trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/docs/architecture/plugin_architecture.md
Trevon 72647bbaf7
 chore: stabilize CI, faiss, metadata, docker, and simeon (#25) 
- CI warm: ccache dep, continue-on-error, 60min timeout, YAMS_DISABLE_FAISS
- Faiss: YAMS_HAS_FAISS guard, Apple Clang disable, OpenMP detection
- Metadata: retry constraint failures, deduplicate batch UPSERT entries
- Docker: wget for Zig, gfortran, YAMS_DISABLE_FAISS=1
- CI config: free-disk-space v1.3.1, libomp-18-dev, experimental trigger
- Weekly benchmarks: faiss disable, ccache, diagnostics capture
- Simeon: bump to 7763a97, EntropyRouter, concept_count
- CorpusStats: extraction readiness fields
- Remove committed artifacts (.wrangler, .artifacts, dispatcher_external)
- Fix .vscode/settings.json hardcoded path

Signed-off-by: trvon <git@trevon.dev>
2026-05-07 21:07:59 -04:00

2.9 KiB
Raw Permalink Blame History

Architecture: Plugin and Content Extraction

This overview reflects the current implementation of the plugin stack and content extraction pipeline. Every item below maps directly to production code.

For the user-facing plugin guide (writing and configuring plugins), see Plugins.

Manifests and Capability Negotiation

Source: src/daemon/resource/plugin_loader.cpp

  • PluginLoader::loadPlugin opens a shared object, probes exported functions such as getProviderName, createOnnxProvider, and createGraphAdapterV1, and records which interfaces the plugin implements.
  • Manifests declared in docs/spec/plugin_metadata.schema.json describe versioning and required capabilities; loader logic validates the manifest against this schema before registering providers.
  • Graph adapter factories are tracked in an in-process registry via registerGraphAdapter / createGraphAdapter so knowledge-graph integrations can be selected at runtime.

Trust and Admission

Source: src/daemon/resource/plugin_host.cpp

  • PluginHost::trust/untrust maintain a canonicalized allow-list persisted alongside the daemon data directory.
  • PluginHost::enumerateTrusted filters discovered libraries against that list, which ServiceManager::autoloadPluginsNow uses to decide which plugins can be materialized.

Native ABI Loader

Source: src/daemon/resource/abi_plugin_loader.cpp

  • Resolves ABI entry points, maps them to C++ adapter classes, and converts plugin callbacks into daemon-facing interfaces (storage, model providers, etc.).
  • Handles symbol version mismatches and throws structured errors that bubble back to the caller.

Content Extraction Pipeline

Content extraction is driven by plugins implementing the content_extractor_v1 interface. The daemon coordinates extraction during ingestion:

  1. Plugin discovery: The daemon scans for extractors via PluginLoader at startup, matching file types to capable plugins.
  2. Extraction dispatch: During add/ingest, the PostIngestQueue dispatches documents to registered extractors based on MIME type (e.g., PDF extractor, tree-sitter symbol extractor).
  3. Symbol extraction: The tree-sitter plugin (plugin-symbols) parses source files and emits function/class/import definitions into the knowledge graph, enabling symbol-aware search.
  4. PDF extraction: PDF extractor plugins extract text content, which is then indexed for full-text and vector search.

Plugin Interfaces

Interface Purpose Entry points
Model provider Embedding generation (ONNX) createOnnxProvider, getProviderName
Graph adapter Knowledge graph backend createGraphAdapterV1
Content extractor Text/symbol extraction content_extractor_v1 ABI
Search provider Plugin-contributed search results search_provider_v1 ABI
Storage plugin Object storage backends (S3/R2) See Storage Plugin v1