Every kNN edge was also written in reverse, so each semantic_neighbor
edge looked reciprocal and topology's reciprocalOnly filtering, SGC
smoothing over reciprocal edges, and graph reciprocity metrics filtered
nothing. Write only the edge from a document to each of its k nearest
neighbours; a reverse edge now exists exactly when the pair is mutual.
Semantic maintenance pruned one-way edges on the assumption that edges
were symmetric; with directed kNN edges that would erase every
non-mutual neighbour, so it now removes only self-loops (and no longer
reports a fabricated community size for the scoped pass).
Edge reads truncated by insertion order: getEdgesBidirectional sorted by
id and getEdgesTo had no order, so a LIMIT kept arbitrary or oldest
edges. Both now return strongest first, and the topology extractor reads
a document's own neighbours separately from its incoming edges so a
hub's many incoming edges cannot evict them.
Existing databases keep their mirrored edges until the semantic
neighbour graph is rebuilt.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py.
- Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile.
- Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests.
- Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test
fixtures to run symbol-specific test cases against schema v39.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
