Use pinned, governed pages for metadata, content, vectors and topology.
Preflight identity/content/model prerequisites before writes and continue
independent records without softening authentication, capacity or I/O errors.
Apply topology deletions, nodes and edges in one real KG batch with bounded
transaction-local reads. Preserve ordinary-read isolation, report exact edge
deferrals independently of bounded diagnostic samples and avoid empty commits.
Retract absent local topology only from exact committed own observations and
conditionally erase the same envelope, including off-resident/stale-winner
regressions. Probe local KG presence with fallible scalar SQL rather than
materializing properties; retain SQL errors as errors, not absent records.
The public C++ KG/batch interfaces gain virtual capabilities. Defaults provide
source-level hard errors only: installed/old-binary ABI compatibility is NOT
established and providers must be rebuilt/reviewed before promotion.
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
