7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
211db15606 |
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup: std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine threshold could exceed 1.0 for a legal pair of identical vectors. The same work was also admitted while the daemon was still initializing, so the 300-second starvation guard forced a pass before readiness. + semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the existing embed policy header, shared by the streamed and corpus branches; non-finite vectors or scores produce no neighbor and legal scores are bounded before ranking and threshold selection + startupMaintenanceEligible() requires ServiceManagerState::Ready, not stopping and idle, and is evaluated before ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no longer manufacture missing readiness + StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter line reporting state, phase and phase_elapsed_ms instead of a debug-level state number; a 53-minute startup previously logged nothing actionable + regression through the public backfill and corpus-rebuild APIs using a 64-D vector whose old float inverse norm rounds its self-cosine above 1 - removed the duplicated inverse-norm and cosine lambdas from EmbeddingService - renamed __test_setModelProvider/__test_setKgStore to testing_* because they used reserved identifiers, across 11 files - re-anchored 13 pre-existing raw-environment allowlist entries to their new line numbers; no new exception was added Startup is still blocked by corpus-sized memory-sync scans; this change stops the abort and names the blocking phase, it does not make startup fast. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
33243d8472 |
fix(memory-sync): stop re-walking the op store on every publish (#319)
* fix(storage): bound filesystem listPage residency listPage() walked the whole op store and retained every matching key before slicing one page. At 1025 records and a 512-record page it held 1025 keys where 513 is the working set, and the retained set grew with the store. Traverse once into a max-heap holding the smallest eligible limit+1 keys. The walk itself stays complete, so a traversal error still returns an error instead of a partial page. Exclusive lexical cursor, ordering, Windows key decoding and the SIZE_MAX limit stay as they were. Add a YAMS_TESTING-only observation seam that counts real walks, entries visited and peak retained keys, so the traversal cost can be measured instead of inferred. The new traversal shifts two reviewed getenv allowlist line numbers. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): keep backfill progress past an oversized vector A record above the resident limit failed publication, the domain cursor never advanced, and every later cycle retried the same row. Valid vectors behind it never reconciled, so one oversized value could pin a domain indefinitely. The daemon log showed this repeating every ~100 s. Classify the row only after a publish fails with InvalidArgument and the same serialized record is independently confirmed to exceed the immutable limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep the local copy, and revisit it when a new source generation or service limit appears. A generic backend InvalidArgument stays retryable, and healthy rows still serialize only once. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): reuse a bounded complete listing across publishes Every store invalidated the cached listing pages, so a publish followed by syncFully walked the whole op store again, once per 512-record page. On the profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of it system time, with listPage at 55% of user samples. Keep a complete key-only prefix snapshot, built only from normal contiguous pages starting at an empty cursor on the concrete filesystem backend. The snapshot is capped by key count and key bytes, served only in lexical order, and dropped as incomplete unless the walk reached the end. Any mutation invalidates it before the backend call; a successful store or remove then applies the exact key to matching snapshots, and a failure drops them because the write may have committed. Initialising the async path disables cache reuse permanently rather than racing a future. Cache maintenance is best effort: it runs after the backend result is known and any allocation failure clears the cache and keeps the committed result. Opaque cursors are untouched, so only the built-in filesystem backend gets snapshot treatment. Active publishes now cost the same number of walks as a warm sweep instead of four, six and nine at 511, 513 and 1025 records. Signed-off-by: trvon <git@trevon.dev> * test(bench): measure memory-sync listing cost across arms Add a standalone harness that drives a real filesystem op store through MemorySyncLoop, one scenario per process so no cache state survives a repeat, and reports wall, user, system and peak RSS per run. It uses production APIs only, so the same source builds against a baseline and a candidate tree and neither arm needs a measurement seam ported into it. The driver seeds one corpus per size, interleaves the two arms inside each repeat, and writes results.jsonl plus a median report. Enumeration counts are not re-derived here; the unit suite owns those. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
f7ca95427d |
fix(daemon): run memory_sync catch-up sweeps only when a store changed (#295)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores The vector and topology catch-up sweeps restart a full pass as soon as one ends, so an idle node reads and re-checks every node, edge, vector, and committed topology record forever. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing. New cases drive the production backfill through ServiceManager: - after a completed pass with no writes, further cycles serve no sweep item and take no graph-store connection; - a node, edge, and vector written after a pass are published by the next pass while unchanged records keep their committed operation; - a node deleted after the sweeps went idle is retracted with its edge. Store seams: KnowledgeGraphStore::changeGeneration() and VectorDatabase::changeGeneration() move on writes and batch commits, not on reads. Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32, acquisitions 157 == 97); kg_store and vector_smoke do not compile. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): run memory_sync catch-up sweeps only when a store changed The vector and topology catch-up sweeps in publishMemorySyncBackfill restarted a full pass as soon as one ended. An idle node therefore read and re-checked every node, edge, vector, and committed topology record forever, and every published record cost each peer a delta exchange and an apply. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing; on a real corpus the first pass never ends. Each store now exposes a change generation: KnowledgeGraphStore counts write transactions (single writes through the write pool and write batch commits), VectorDatabase counts inserts, updates, and deletes. A sweep records the generation a pass began at and, once the pass completes, does not start another until the generation differs. Writes behind a running pass's cursor move the generation past the pass's own reading, so the following pass still covers them; deletes move it too, so the retraction phase still runs after a rebuild drops records. With nothing changed, a cycle costs each sweep one atomic read. No config knob: the gate is a property of the stores, not a tunable. Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases), catch2_kg_store, catch2_vector_submodule. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
1b6671528d |
fix(memory-sync): publish committed vectors and wrap the vector sweep (#266)
* test(memory-sync): cover vectors committed behind the backfill cursor The vector backfill cursor (document_hash, chunk_id) never wraps, and the backfill is the only outbound path for vectors, so a vector added or re-embedded behind the cursor after the first sweep is never published to peers. Add red tests for: - the sweep starting over to publish a new and a re-embedded vector that sort below its last position; - vectors an embedding job commits being published within the next backfill cycle, ahead of a sweep that would need many items to reach them; - the topology sweep still publishing its share while the vector sweep always has work. Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op seam for the embedding commit notification; the fix fills it in. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish committed vectors and wrap the vector sweep The vector backfill cursor (document_hash, chunk_id) was set once and reset only by a daemon restart, and the backfill is the only outbound path for vectors. After the first full sweep, a vector added or re-embedded under a document hash that sorts below the cursor was never published to peers. Topology had the same bug and now sweeps in repeating passes; vectors did not. Eager publish: the embedding service already reports the documents a job committed vectors for (successHashes, after the vector insert and before readiness is published) through the callback that requested the topology rebuild. That callback is renamed setEmbeddingsCommittedCallback and the daemon now also passes the hashes to ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues them (bounded at 4096 documents, deduplicated, dropped past the cap): it runs on the embedding worker, and the memory-sync service's lifetime and outbound budget belong to the sync cycle. The backfill serves the queue as its own domain right after documents, one vector row per item, walking each document's chunks with getVectorsPage. The queue is in memory; a restart loses it, but the sweep then starts over anyway. Catch-up sweep: when the vector sweep reaches the end of the table it resets its cursor and sits out the rest of the cycle, as the topology pass does, so the next pass publishes whatever changed behind it. publishIfChanged keeps repeat passes from writing unchanged records. R7's skip logic is kept: an unpublishable row is skipped and the cursor moves past it; it is logged on the first pass (and when committed) but not again on every later pass. Fairness: with both sweeps repeating, both always have work, so the strict priority order (documents, vectors, topology) would hand the vector sweep every item and starve topology. Documents and committed vectors keep priority; the two sweeps now alternate item by item, and the turn persists across cycles so a one-item budget still alternates. A sweep that is done for the cycle leaves its turns to the other. This was chosen over alternating whole passes because it needs one field of state, keeps both sweeps' latency bounded by their own size rather than the other's, and reuses the existing per-item budget loop unchanged. Signed-off-by: trvon <me@trevon.dev> * test(memory-sync): pin the vector-store environment in the cursor-wrap cases Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44daa47b94 |
fix(memory-sync): skip bad vector records instead of stalling the stage (#265)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
|
Trevon
|
8aa2a33720 |
fix(memory-sync): publish topology edge endpoints and retract deleted topology (#250)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* feat(metadata): add keyset node/edge scans and exact edge lookup The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no ORDER BY, so a topology rebuild that deleted and reinserted cluster nodes between two pages shifted the offset and the sweep skipped live nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never reused, so a cursor on the last id returned is stable under concurrent inserts and deletes and survives restarts. getNodesAfterId/getEdgesAfterId scan in id order from such a cursor. getEdgesBetween returns every edge between two nodes for one relation; getEdgesFrom truncates at its page limit, which made "is this edge present" checks on high-degree nodes unreliable. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish topology edge endpoints and retract deleted ones On a Docker mesh that converges on documents, topology deferrals never drained: peers held member_of and semantic_neighbor edges whose endpoint node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80). Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral named a doc or cluster node that its origin still held but had not published. The origin's sweep was still inside the "document" type: no "path", "topology_cluster" or "topology_snapshot" node had been published by any node, while the doc nodes' edges into them had been. The backfill swept node types in sorted order and published each node's edges right after the node, so an edge reached peers long before its target node of a later type; at the backfill's rate that is tens of minutes. Its paging also made some of those waits permanent: - Nodes were paged per type with OFFSET and no ORDER BY. A topology rebuild deletes every cluster node and inserts replacements, so the offset skipped live nodes whose member_of edges were published. - Per-type offsets were never reset, so a node was visited once: edges added after its sweep (semantic neighbors, new memberships) and in-place node updates were never published. - Nothing published deletions. Worse, apply() wrote this node's own winners back into its graph, resurrecting the cluster nodes and edges its rebuild had just deleted, so the deletion could not be observed. - Peers deferred an edge whose endpoint was tombstoned forever. The topology backfill now runs repeating keyset passes: every node by id, then every edge by id published together with both endpoint nodes (endpoints first, so a peer never holds a writer's edge without them), then this writer's committed topology winners by logical key, tombstoning those the local graph no longer holds. Cursors are stable ids/keys, so inserts and deletes between items cannot shift them, and a restart just begins a new pass; unchanged records publish nothing. apply() still validates every record but never applies the writer's own ones: the local graph is authoritative for them. An edge whose endpoint is known deleted (tombstoned, or an own record the local graph no longer holds) is dropped instead of deferred, and applies again if the endpoint is re-published. Edge lookups during apply use the exact endpoint query instead of a page-limited adjacency scan. Signed-off-by: trvon <me@trevon.dev> * perf(memory-sync): publish documents before derived records in backfill With the topology sweep now publishing every node and edge, a 3-node x 40-doc mesh took longer until every document was listed on every node (603 s against 426 s before the topology fix). Debug logs showed why: a publish takes most of the backfill's 100 ms per-cycle time budget, so each cycle published about one item, and the fair rotation over documents, vectors and topology handed two of every three cycles to derived records while peers waited for the documents. The backfill now serves domains in priority order: documents, then vectors, then topology, each taking items until it has nothing left this cycle. Vectors and topology defer on a peer until their document arrives, so publishing the document first never adds a deferral. In the same mesh, run concurrently with both other builds, every document is listed on every node after 140 s (426 s before the topology fix, 603 s with it alone), and deferrals still drain to 0. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
7be908e5b7 |
fix(p2p): stop the memory-sync apply/publish deadlock and report apply health (#243)
* fix(daemon): never gate memory-sync publish on inbound apply applyMemorySyncWinners() ran the inbound stages content -> metadata -> vector -> topology and returned on the first stage failure, before the node's own outbound backfill. In a mesh where every node holds an inbound record whose prerequisite only a peer can publish (an embedding whose content blob has not replicated, an edge whose target node has not), every node refused to publish, so the prerequisites never arrived: a stable mutual deadlock with healthy transport and nothing converging. The outbound publish now runs after the inbound stages whatever their outcome; only a stop request ends a cycle early. Each inbound stage also runs even when an earlier one failed. Cross-stage dependencies (metadata and vectors need local content bytes, edges need their endpoint nodes) are checked per record against the local stores, so a failed stage shows up downstream as unapplied records rather than as skipped stages. Stages keep their prerequisite order so a prerequisite landing in a cycle is visible to its dependents in that same cycle. A two-node test drives real ServiceManager apply cycles over a shared store, each node holding an edge whose target only the other can publish; it converges in a bounded number of rounds only with this change. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): defer records whose prerequisite has not replicated Vector, metadata, and topology apply treated a missing replicated prerequisite as a hard error and rejected the whole batch: one embedding whose content blob had not arrived, or one edge whose target node had not, blocked every other winner in that stage on every cycle. These are ordering races between independently published records, not failures. Each adapter now defers such a record instead: it is skipped, the rest of the batch applies, and its logical key is reported through deferredKeys(). Nothing marks it applied, so it remains a winner in the replicated index and the next cycle retries it once the prerequisite lands; apply needs no pending queue of its own and loses nothing across restarts. An edge whose endpoint is being deleted in the same batch is deferred too rather than inserted against a stale node id. Malformed records, probe failures, and store errors still fail the stage. Content blobs are independent, so the content stage now attempts every blob and fails afterwards with the first genuine error; a corrupt blob no longer starves the blobs after it. A blob whose bytes are not hydrated in the sync cache yet is deferred like the other stages. Signed-off-by: trvon <me@trevon.dev> * feat(p2p): report memory-sync apply health in status The mesh deadlock was invisible: every node reported failed_cycles=0 and a climbing successful_cycles while nothing converged, because a cycle counts as successful once reconciliation completes, whatever the apply stages did afterwards. Deferred records would be just as invisible without a signal. ServiceManager now keeps an ApplyHealth record of the apply path: - deferred records per stage, and the age of the oldest one still pending, with a warning once a deferral passes ten minutes (its prerequisite likely never arrived). Ages are tracked for at most 4096 keys per stage, keeping the oldest; counts stay exact. - hard failures per stage, plus the last failure's stage, message, and age. - apply_cycles and apply_failed_cycles. failed_cycles keeps meaning reconciliation failures; a cycle whose apply stage hard-failed is counted in apply_failed_cycles instead of passing as a plain success. - publish_skipped_cycles (delete outbox preempted the cycle) and publish_failed_cycles (an outbound backfill domain failed). The fields travel through MemorySyncStatus, MemorySyncResponse, and protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the first line, an `apply` line with per-stage counts, a last-apply-failure line, and an `apply` object in --json) and to the memory-sync section of `yams daemon status`. The p2p status formatter moves behind a header so it can be tested without a daemon. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): keep content deferrals when the same scan hits a failure The content stage scans every blob and fails afterwards with the first genuine error, but the apply cycle discarded the deferred keys that the same scan collected whenever it failed. A corrupt blob therefore hid every blob still waiting for its bytes: the deferred count stayed at its previous value (0 on the first pass) and the stale-deferral warning never fired for a prerequisite that was actually missing. A stage pass now carries both outcomes. The content-blob scan moves into ContentBlobSyncAdapter::applyWinners, which reports the applied count, the deferred keys, and the first failure of one complete scan, and ApplyHealth::recordPass records the failure and replaces the deferred set together. A stage that returns an error still leaves the deferred set as last observed, since an aborted scan says nothing about prerequisites. The tracked-deferral cap already carried every still-deferred tracked key before admitting new ones, but its carry loop had a cap check that read as if it could stop early. Drop it (the tracked set never exceeds the cap) and pin the behaviour with tests that place the oldest key after more than the cap's worth of new keys and that refill a full tracked set. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |