Use pinned, governed pages for metadata, content, vectors and topology.
Preflight identity/content/model prerequisites before writes and continue
independent records without softening authentication, capacity or I/O errors.
Apply topology deletions, nodes and edges in one real KG batch with bounded
transaction-local reads. Preserve ordinary-read isolation, report exact edge
deferrals independently of bounded diagnostic samples and avoid empty commits.
Retract absent local topology only from exact committed own observations and
conditionally erase the same envelope, including off-resident/stale-winner
regressions. Probe local KG presence with fallible scalar SQL rather than
materializing properties; retain SQL errors as errors, not absent records.
The public C++ KG/batch interfaces gain virtual capabilities. Defaults provide
source-level hard errors only: installed/old-binary ABI compatibility is NOT
established and providers must be rebuilt/reviewed before promotion.
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores
The vector and topology catch-up sweeps restart a full pass as soon as
one ends, so an idle node reads and re-checks every node, edge, vector,
and committed topology record forever. On a 3-node mesh limited to 2
CPUs each this held every node at ~24% of a core with nothing changing.
New cases drive the production backfill through ServiceManager:
- after a completed pass with no writes, further cycles serve no sweep
item and take no graph-store connection;
- a node, edge, and vector written after a pass are published by the
next pass while unchanged records keep their committed operation;
- a node deleted after the sweeps went idle is retracted with its edge.
Store seams: KnowledgeGraphStore::changeGeneration() and
VectorDatabase::changeGeneration() move on writes and batch commits,
not on reads.
Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32,
acquisitions 157 == 97); kg_store and vector_smoke do not compile.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): run memory_sync catch-up sweeps only when a store changed
The vector and topology catch-up sweeps in publishMemorySyncBackfill
restarted a full pass as soon as one ended. An idle node therefore read
and re-checked every node, edge, vector, and committed topology record
forever, and every published record cost each peer a delta exchange and
an apply. On a 3-node mesh limited to 2 CPUs each this held every node
at ~24% of a core with nothing changing; on a real corpus the first
pass never ends.
Each store now exposes a change generation: KnowledgeGraphStore counts
write transactions (single writes through the write pool and write
batch commits), VectorDatabase counts inserts, updates, and deletes. A
sweep records the generation a pass began at and, once the pass
completes, does not start another until the generation differs. Writes
behind a running pass's cursor move the generation past the pass's own
reading, so the following pass still covers them; deletes move it too,
so the retraction phase still runs after a rebuild drops records. With
nothing changed, a cycle costs each sweep one atomic read.
No config knob: the gate is a property of the stores, not a tunable.
Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases),
catch2_kg_store, catch2_vector_submodule.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover document lookup and --scope-cwd listing (#280)
Red tests for the two graph CLI bugs in #280:
- GetResponse graph fields and file name must survive the proto wire.
- DocumentService graph lookup by hash and by name must return the
same documents as traversing the doc:<hash> node.
- The document graph JSON must name the document when only name is set.
- KG path-range scoping must match document nodes by label and
snapshot path nodes by $.path.
- listByType --scope-cwd must keep document and path nodes under the
client cwd (including via a symlinked cwd) outside src/ and include/.
The existing code-roots dispatcher case now places its out-of-scope
file outside the scope root, since the scope is the whole cwd.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280)
yams graph <hash> and --name returned graphEnabled=false with no
related documents while --node-key doc:<hash> listed neighbours:
- The GetResponse proto binding dropped graphEnabled, related and
fileName, so socket clients never saw graph data.
- DocumentService started traversal at blob:<hash>, which has no
outgoing edges, and filtered out semantic_neighbor. Start at the
doc:<hash> node and allow semantic_neighbor.
- GraphQueryService did not hydrate documentHash for doc: nodes, so
reached documents were skipped.
- The presenter read only fileName for "document"; fall back to name.
--list-type <t> --scope-cwd returned nothing:
- The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the
whole cwd, lexical and symlink-resolved.
- The KG range predicate ignored document labels and $.path on
snapshot path nodes. Match both.
Signed-off-by: trvon <me@trevon.dev>
* docs(skills): correct graph document lookup examples
--format dot is only rendered for node traversal, so show it with
--node-key doc:<hash>. Describe --scope-cwd as cwd-scoped.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): add keyset node/edge scans and exact edge lookup
The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no
ORDER BY, so a topology rebuild that deleted and reinserted cluster
nodes between two pages shifted the offset and the sweep skipped live
nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never
reused, so a cursor on the last id returned is stable under concurrent
inserts and deletes and survives restarts.
getNodesAfterId/getEdgesAfterId scan in id order from such a cursor.
getEdgesBetween returns every edge between two nodes for one relation;
getEdgesFrom truncates at its page limit, which made "is this edge
present" checks on high-degree nodes unreliable.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish topology edge endpoints and retract deleted ones
On a Docker mesh that converges on documents, topology deferrals never
drained: peers held member_of and semantic_neighbor edges whose endpoint
node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80).
Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral
named a doc or cluster node that its origin still held but had not
published. The origin's sweep was still inside the "document" type: no
"path", "topology_cluster" or "topology_snapshot" node had been published
by any node, while the doc nodes' edges into them had been.
The backfill swept node types in sorted order and published each node's
edges right after the node, so an edge reached peers long before its
target node of a later type; at the backfill's rate that is tens of
minutes. Its paging also made some of those waits permanent:
- Nodes were paged per type with OFFSET and no ORDER BY. A topology
rebuild deletes every cluster node and inserts replacements, so the
offset skipped live nodes whose member_of edges were published.
- Per-type offsets were never reset, so a node was visited once: edges
added after its sweep (semantic neighbors, new memberships) and
in-place node updates were never published.
- Nothing published deletions. Worse, apply() wrote this node's own
winners back into its graph, resurrecting the cluster nodes and edges
its rebuild had just deleted, so the deletion could not be observed.
- Peers deferred an edge whose endpoint was tombstoned forever.
The topology backfill now runs repeating keyset passes: every node by
id, then every edge by id published together with both endpoint nodes
(endpoints first, so a peer never holds a writer's edge without them),
then this writer's committed topology winners by logical key, tombstoning
those the local graph no longer holds. Cursors are stable ids/keys, so
inserts and deletes between items cannot shift them, and a restart just
begins a new pass; unchanged records publish nothing.
apply() still validates every record but never applies the writer's own
ones: the local graph is authoritative for them. An edge whose endpoint
is known deleted (tombstoned, or an own record the local graph no longer
holds) is dropped instead of deferred, and applies again if the endpoint
is re-published. Edge lookups during apply use the exact endpoint query
instead of a page-limited adjacency scan.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory-sync): publish documents before derived records in backfill
With the topology sweep now publishing every node and edge, a 3-node x
40-doc mesh took longer until every document was listed on every node
(603 s against 426 s before the topology fix). Debug logs showed why:
a publish takes most of the backfill's 100 ms per-cycle time budget, so
each cycle published about one item, and the fair rotation over
documents, vectors and topology handed two of every three cycles to
derived records while peers waited for the documents.
The backfill now serves domains in priority order: documents, then
vectors, then topology, each taking items until it has nothing left this
cycle. Vectors and topology defer on a peer until their document
arrives, so publishing the document first never adds a deferral. In the
same mesh, run concurrently with both other builds, every document is
listed on every node after 140 s (426 s before the topology fix, 603 s
with it alone), and deferrals still drain to 0.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:
- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
the audited machine) that `yams init` / `yams config grammar` built
and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
every start with two warnings (the ABI host's skip plus the plugin
manager's "load failed"), and the refusal flipped the plugin host FSM
through Failed.
Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.
The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.
Signed-off-by: trvon <me@trevon.dev>
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.
Shutdown now tells in-flight work to stop before joining:
- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
on every leased connection under the pool mutex, which keeps each
lease open while it is interrupted. DatabaseManager exposes it for
both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
join wait slice through a new joinWithTimeout hook, because SQLite
drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
before the interrupt, and publishes it as AppContext::cancellationSignal.
DocumentService::list and GrepService::grep check it between query
and hydration steps and between document batches, and return
OperationCancelled; SearchService checks it on entry and forwards it
as SearchExecutionContext::hostCancellationSignal, which the engine's
existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
componentTimeout is 0, and a canceled search stops waiting for the
query embedding / concept tasks. Those tasks now own copies of their
inputs instead of capturing the caller's stack, so abandoning them is
safe.
Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.
Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.
Signed-off-by: trvon <me@trevon.dev>
getSnapshots, getSnapshotInfo and batchGetSnapshotInfo read only each
document's latest snapshot_id key, so a re-stored document disappeared
from every earlier snapshot's file count, and a snapshot whose documents
were all re-stored vanished from the list.
- getSnapshots lists tree_snapshots, document_snapshots and latest keys.
It deliberately skips legacy snapshot_id:<id> keys, which would be a
full-table scan on large corpora until the backfill finishes.
- batchGetSnapshotInfo counts members from document_snapshots, legacy
keys not yet moved, and latest keys, each looked up by id through an
index; label and git commit come from tree_snapshots first.
- getSnapshotInfo delegates to the batch query, removing three
near-duplicate queries.
Signed-off-by: trvon <me@trevon.dev>
Every store added two metadata rows per snapshot to the document,
snapshot_id:<id> and snapshot_time:<id>, and nothing removed them.
Content-addressed documents are shared across paths and passes, so they
piled up: on one 93 GB database they were 43.1M of 44.0M metadata rows,
one file carried 52,367 snapshots, and 'yams list' timed out hydrating
2.69M rows for its 100 documents. Meanwhile findDocumentsBySnapshot read
only the latest snapshot_id, so a re-stored document dropped out of every
earlier snapshot.
- Migration 42 creates document_snapshots(document_id, snapshot_id,
snapshot_time), keyed on (document_id, snapshot_id). It only creates the
table, so startup stays fast.
- The insert transaction records membership; stores keep only the latest
snapshot_id/snapshot_time keys.
- findDocumentsBySnapshot and file history read membership, falling back
to legacy keys through one shared parser (legacySnapshotEntries).
- getMetadataForDocuments skips legacy keys via idx_metadata_doc_key: on
the 93 GB database, 0.21 s instead of 18.5 s for list's 100 documents.
- RepairService moves legacy keys in the background, 1000 snapshots per
batch, 200 ms apart, only below Warning pressure.
Signed-off-by: trvon <me@trevon.dev>
Read and write pools share the slow-holder threshold, but the warning
always said 'slow write-connection hold'. A slow read-only 'yams list'
(tag client_list, 42 s) was reported as a write hold, which pointed the
investigation at a nonexistent write. Say read or write from the pool's
readOnly config.
Signed-off-by: trvon <me@trevon.dev>
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
Migration 40 deleted every kg_edges row whose relation was calls,
implements, includes, ... and every node of a symbol type such as class or
interface. Graph ingest through MCP and IPC accepts any type and relation,
so an upgrade from v0.19 irreversibly removed user-modelled data like a
'class' node or an 'implements' edge.
Select the extractor's nodes by provenance and key shape instead
(<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its
kg_doc_entities rows reference) and delete edges that are tagged with the
extractor or touch those nodes. The new test fails before the change.
kgEdgeDensity counted every kg_edges row, including the ~16
semantic_neighbor edges per document that 8-NN embedding neighbours
write in both directions, plus file-lineage and entity-attachment edges.
Any embedded corpus therefore looked like a rich knowledge graph, so the
SCIENTIFIC-profile gate that should zero the KG leg on flat prose never
fired and graph richness scaled from embedding edges.
Add kgRelationalEdgeCount/Density, which exclude embedding kNN, file
lineage and layout, text segments, and entity-to-document attachments,
and use it for hasRichGraphTopology() and the tuner's richness scaling.
kgEdgeCount keeps its meaning as the reconciled-graph signal. The live
overlay carries the reconciled relational count forward.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
Migration 40 pruned every topology snapshot when the latest pointer was
missing or named a snapshot that did not exist, and aborted on malformed
pointer JSON. Prune only when the pointer is valid JSON naming an
existing snapshot node.
Migration 41 matched topology keys with LIKE, which is case-insensitive
in SQLite and also stopped counting user keys such as Topology.*. Use
GLOB, and rebuild the topology counts on rollback so down leaves
metadata_value_counts consistent. Neither migration has shipped in a
release or nightly.
- Use v39 metadata DB template for legacy symbol metadata test cases in
metadata_schema_catch2_test.cpp and kg_store_catch2_test.cpp to avoid
running against schema v40 where symbol tables are dropped.
- Use yams::test::make_temp_sqlite_path instead of raw getenv in
metadata_schema_catch2_test.cpp and prune the stale portability allowlist entry.
- Add site directory to SKIP_PARTS in check_repository_metadata.py to prevent
local docs builds from triggering retired marker warnings.
Signed-off-by: trvon <me@trevon.dev>
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py.
- Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile.
- Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests.
- Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test
fixtures to run symbol-specific test cases against schema v39.
Signed-off-by: trvon <me@trevon.dev>
Drop legacy Tree-sitter symbol extraction subsystem, associated database
tables, triggers, and AST edge types:
- Migration 40 drops symbol_metadata, document_symbol_extraction_state,
symbol_metadata_fts, associated triggers, and prunes historical bloated
topology snapshots while preserving active snapshot topology,
semantic_neighbor edges, and non-symbol observed_as provenance edges.
Exclude nl_entity:% nodes from kg_nodes / kg_edges pruning to preserve
canonicalized method-type technology entities.
Provides complete downSQL restoring symbol tables, indexes, FTS virtual table,
and sync triggers for clean v39 rollbacks.
- In EntityGraphService, ensure locally resolved documentDbId propagates to
WriteBatch for knowledgeGraphDocumentId to ensure accurate enrichment
token completion, and only populate batch->documentIdToDelete for snapshot
documents with a resolved document database ID.
- In SymbolEnricher and SearchService, restore one-directional symbol-contains-query
matching with non-empty checks to prevent short substrings matching broad queries.
- In tests/unit/metadata/symbol_metadata_schema_test.cpp, migrate test DB up to
version 39 so tests pass against their target version prior to v40 table drops.
- Remove Tree-sitter grammar download and compile workflows from CLI init
- Default plugin-symbols to false.
- Fix macOS daemon client and CLI executable resolution using
_NSGetExecutablePath with candidate fallbacks.
- Add migration validation tests and symbol enrichment unit tests.
Signed-off-by: trvon <me@trevon.dev>
Reserve slots before unlocking for connection creation. Route replacement
and maintenance through the same capacity accounting and wake waiters
when reservations settle.
Reproduce concurrent over-admission with bounded creation barriers.
Reanchor existing portability exceptions without adding exemptions.
Signed-off-by: trvon <me@trevon.dev>
* test(ci): stabilize main matrix (#112)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): preview experimental promotion (#111)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): reconcile draft promotion safely (#113)
* ci(release): reconcile draft promotion
Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.
Signed-off-by: trvon <me@trevon.dev>
* ci: isolate hook Git environments
Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): reject invalid candidate evidence
Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve completion and retrieval contracts
Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence asynchronous enrichment completion
Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.
Signed-off-by: trvon <me@trevon.dev>
* fix: bound semantic graph output reservation
Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence standalone embedding repair completion
Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve enrichment failure reporting
Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test: align explicit hash prefix boundaries
Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.
Signed-off-by: trvon <me@trevon.dev>
* test: align UI hash validation boundary
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): load tag filters once per request, not once per document
pathSearch and searchByHashPrefix filtered candidates through
metadataHasTags(), which for every document called getAllMetadata()
(four retries with backoff on failure), concatenated every metadata
key and value into a string, and logged that string at info level.
A path search over a few thousand documents was a few thousand
metadata round trips and a few thousand info lines.
Both paths now load tags for the whole candidate set with one
batchGetDocumentTags() call (the API GraphComponent already used) and
filter with a pure matcher. The legacy fallback that also accepted a
metadata row with key "tag" is gone: nothing writes that form and the
repository's own tag APIs never honoured it, so search and repository
now agree on what a tag is.
A counting repository subclass pins the shape: one batch call and zero
getAllMetadata calls for both the path and the hash-prefix routes.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
Correction to the claim above that nothing writes the legacy `key = "tag"`
form: the repair downloads pass did (tag values "downloaded", "host:<h>",
"scheme:<s>"), and the search fallback was its only reader. That writer now
uses the same `tag:<name>` keys as every other tag writer, so `--tags
downloaded` keeps matching documents the repair pass tagged. And
batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it
an unbounded candidate set; an oversized list used to fail the statement
and, with this change, would have emptied the result instead of degrading
per document.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hydrate topology routing candidates in batches
The routing session called getDocumentByHash once per hash in three
places on the product routing path: ranked-candidate admission, cluster
expansion, and the graph-neighbor diagnostic trace (memoized, but still
one statement per distinct hash across four stages). Each now issues
one batchGetDocumentsByHash for its whole set, the same API the
topology input extractor already used, and the docLookupMicros timing
covers the batch. Cluster expansion applies the maxDocs cut before
hydrating so no document is fetched only to be discarded.
A session-level test with one seed and eight graph neighbours under a
counting repository fails on the previous code (17 single lookups, no
batch) and passes now (0 single lookups). The full topology suite is
unchanged.
portability_allowlist.txt: one topology test anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): bound hydration and preserve errors
Retain legacy tags and honor per-connection SQLite bind limits.
Keep hash chunks in one read snapshot and release it on failure.
Failed lookups are not missing documents or successful empty searches.
Reject partial route admission and preserve tagged path errors.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): count uncached prepares in Database::CacheStats
Database::prepare() compiles a statement every call; prepareCached()
reuses one. The cache stats reported hits and misses but nothing about
the bypass, so a hot path that re-prepared fixed SQL on every row was
invisible. CacheStats gains uncachedPrepares, incremented under the
existing cache mutex and carried through move construction and
assignment.
This is the measurement primitive for the following hot-path changes:
a test can warm a path, then assert that N more calls add zero uncached
prepares.
portability_allowlist.txt: one database.cpp anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(metadata): cache KG statements and batch the scorer's neighbor fetch
Measured with the new uncachedPrepares counter on a single-connection
pool. Before: 64 warmed neighbors() calls plus 32 document lookups cost
306 uncached statement compilations, and one SimpleKGScorer::score()
over eight query terms cost 620. After: 0 and 1.
Three things were compiling SQL per call on the search path:
- The sqlite KG store used Database::prepare() for fixed SQL in
neighbors, getDocumentIdByHash/ByPath, ensureBlobNode,
ensureDocumentNode, ensurePathNode (both lookups), selectKgNodeIdByKey,
getNodeById, resolveAliasExact/Fuzzy, getEdgesFrom/To,
getDocEntitiesForDocument, and searchNodesByLabel. All now use
prepareCached (85 -> 70 raw prepares in the file; the rest are
dynamic IN-list or maintenance queries).
- ConnectionPool::isConnectionValid ran prepare("SELECT 1") on every
acquire and again on every release, so each pooled call paid two
compilations before doing any work. The probe is now cached.
- Database::tableExists compiled its sqlite_master probe every time;
the fuzzy alias resolver calls it per query token.
SimpleKGScorer builds its 1-hop neighbor union with one
getEdgesFromBatch call instead of neighbors() per query node.
kMaxCacheSize rises from 64 to 128: 90 cached call sites now share a
connection when KG and repository work interleave, and the cache
evicts first-found, not least-recently-used.
Tests pin both numbers: the store test asserts zero uncached prepares
across 96 warmed lookups; the scorer test asserts fewer than eight on
a second run (previously one per query node from neighbors() alone).
portability_allowlist.txt: two anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(metadata): bound batched edge query parameters
Respect connection-specific SQLite limits and deduplicate chunk inputs.
Preserve directional ranking and a single read snapshot across chunks;
roll back the savepoint on query failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(embedding): complete embeddings by derivation token
cbc4959c added derivation tokens (begin/complete with a per-attempt
generation, invalidated by document_content triggers) but nothing
produced them: beginDocumentEmbeddingDerivation had no callers and the
completion op's derivations field was never written, while every ingest
still paid for the triggers and the per-hash NOT EXISTS guard.
processEmbedJob now mints one token per document it will embed, before
any input snapshot is read, and completes by token instead of by hash.
A document whose content changes while it is being embedded keeps
has_embedding = 0 and is re-derived later, instead of publishing vectors
for text the corpus no longer holds. Repair jobs travel the same
channel, so they get the same guarantee. The recipe binds the
preparation policy, embedding space, provider version, and dimension
through the embeddingDerivationRecipe helper that cbc4959c also left
unused.
Repository: batchBeginDocumentEmbeddingDerivations and
batchCompleteDocumentEmbeddingDerivations do a whole job in one
transaction each (the single-token completion loop in WriteCoordinator
opened one transaction per document). WriteCoordinator uses the batch
form and takes its stats lock once.
A small EmbeddingDerivationLedger maps job hashes to their tokens so
completion is issued for exactly the documents that produced vectors.
Behavior note: a re-embed (skipExisting = false) of an already-embedded
document resets its readiness at mint time; if that job then fails, the
document reads as not embedded until the next attempt. That is the
attempt semantics the token table was built for.
Not included: the legacy hash completion path is now unproduced and is
removed in a follow-up so the coalescer change is reviewed on its own.
tests/scripts/portability_allowlist.txt: 13 EmbeddingService.cpp
line anchors reflowed; no new environment reads.
The startup reconcile (ServiceManager, "Reconciling document_embeddings_status
from vector rows") used to mark every document with vector rows as embedded,
which would have undone the stale-token guarantee on the next daemon start: a
document whose content changed under an in-flight embed has vector rows for the
old text. reconcileDocumentEmbeddingStatusByHashes now leaves a document with an
open derivation (document_embedding_derivations.completed = 0) at has_embedding =
0, so it is re-derived rather than published; the reconciled-count query applies
the same predicate. Test: an open derivation stays unembedded across a reconcile
and flips once its token completes.
Signed-off-by: trvon <me@trevon.dev>
* fix(embedding): require current recipe to skip
Re-derive legacy or mismatched recipes rather than trusting readiness
alone. Treat derivation lookup errors as failed gathering.
Cover service-level recipe changes and legacy readiness, and characterize
atomic batch completion rollback and retry.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
* refactor(tuning): prune dormant runtime knobs
Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.
* refactor(vector): retire dormant backends
Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.
Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.
* refactor(plugins): remove legacy SDK tooling
Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.
* build(vector): remove stale FAISS overrides
* refactor(ipc): remove dormant FSM snapshots
* refactor(ingest): unify bounded KG enqueue
* fix(plugins): harden allocation contracts
Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).
* fix(storage): define empty object reads
Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.
* refactor(storage): trim through string views
* test: prune dormant harness helpers
Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.
* chore(cpp): initialize analyzer-facing locals
* build(simeon): advance analyzer cleanup
* refactor(config): drop dead daemon client keys
* refactor(ipc): inject daemon client transport
Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.
* refactor(retrieval): route CLI through daemon
Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.
* refactor(graph): move CLI scope into daemon
Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.
* build: isolate test-only runtime controls
* refactor(metadata): decouple lock telemetry
* fix(services): return on request timeout
* fix(retrieval): enforce workspace scope
* fix(graph): scope nodes before pagination
* fix(grep): drop binary result lines
* fix(daemon): bound checkpoint wait
* fix(repair): use KG health checks
* fix(storage): always verify compressed CRC
* perf(vector): skip unchanged PQ snapshots
* build(zyp): exclude bench from default build
* build: protect Meson files from formatter
* refactor: remove unreachable production APIs
* refactor(daemon): remove dormant runtime state
* fix(vector): harden daemon provider calls
Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.
* refactor(storage): remove unusable adapters
* fix(cli): unify in-process add requests
* fix(tsan): preserve async worker progress
* test(cli): isolate daemon-dependent suites
* fix(repair): expose queue backoff progress
* fix(graph): handle qualified C++ symbols
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector)!: remove dead transition APIs
Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.
BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): harden PQ snapshot lifecycle
Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): expose route coordinates
Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.
Signed-off-by: trvon <me@trevon.dev>
* feat(xplan): add vector decision gates
Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): formalize coordinate routing
Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): serialize SymSpell readers
SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make mutations atomic
Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.
* fix(search)!: surface SymSpell write errors
Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.
BREAKING CHANGE: SymSpellSearch mutations now return Result values.
* feat(topology): formalize candidate expansion
* feat(vector): add coordinate-aware embeddings
* feat(topology): preserve protected relation cover
* feat(search): add graph candidate rescue
* feat(daemon): expose retrieval hydration telemetry
* feat(xplan): add rescue and hydration gates
* refactor(content): dedupe media handlers
Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.
* feat(lifecycle): prune stale corpus data
Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.
* feat(search): add topology candidate rescue
Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.
* feat(search): expose evidence paths
* docs: define mobile host contract
* formal: model bounded candidate rescue
* fix(prune): inspect extensionless signatures
* fix(prune): tolerate stale extensions
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): harden model state ownership
Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.
* refactor(daemon): consolidate ingest pollers
Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.
* refactor(daemon): unify repair execution
Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.
* refactor(daemon): unify runtime tuning
Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.
* refactor(daemon)!: remove dead IPC surfaces
Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.
BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.
* refactor(daemon): hide test hooks
Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.
* fix(daemon): avoid replaying successful adds
A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.
* bench(ingest): enforce lossless pipeline runs
Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.
* bench(search): expose cold seed ingest timings
Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.
* perf(ingest): batch deferred content indexing
Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.
* feat(metadata): expose insert writer metrics
Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.
* fix(daemon): retry transient refused connects
The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.
* bench(ingest): attribute submission path costs
Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.
* perf(ingest): batch durable document storage
Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.
* perf(ingest): coalesce bulk post-processing
Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.
* bench(ingest): record post-ingest batch size
Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.
* perf(ingest): batch queued storage waves
Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.
* perf(ingest): route bulk enrichment batches
Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.
* test(daemon): initialize task fixtures
* perf(ingest): batch post-ingest dispatch
* perf(ingest): widen directory storage waves
Borrow from a full wave when one document would otherwise fall through to single-document storage.
* perf(storage): bound batch writer fanout
Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.
* perf(ingest): align storage batch drains
Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.
* fix(ingest): commit metadata state atomically
Fold first-version, extracted-title, and embedding-completion state
into existing transactions.
Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.
* bench(ingest): measure production submission
Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.
* fix(xplan): preserve native failures
Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.
* fix(daemon): preserve embedding success hashes
* fix(daemon): serialize pooled socket shutdown
* fix(daemon): harden batch publication shutdown
* fix(storage): use compat jthread shim
* fix(daemon): avoid poller template shadowing
* fix(search): keep vector pipeline test enabled
* fix: harden shutdown and fallback paths
Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
