10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
8aa2a33720 |
fix(memory-sync): publish topology edge endpoints and retract deleted topology (#250)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* feat(metadata): add keyset node/edge scans and exact edge lookup The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no ORDER BY, so a topology rebuild that deleted and reinserted cluster nodes between two pages shifted the offset and the sweep skipped live nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never reused, so a cursor on the last id returned is stable under concurrent inserts and deletes and survives restarts. getNodesAfterId/getEdgesAfterId scan in id order from such a cursor. getEdgesBetween returns every edge between two nodes for one relation; getEdgesFrom truncates at its page limit, which made "is this edge present" checks on high-degree nodes unreliable. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish topology edge endpoints and retract deleted ones On a Docker mesh that converges on documents, topology deferrals never drained: peers held member_of and semantic_neighbor edges whose endpoint node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80). Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral named a doc or cluster node that its origin still held but had not published. The origin's sweep was still inside the "document" type: no "path", "topology_cluster" or "topology_snapshot" node had been published by any node, while the doc nodes' edges into them had been. The backfill swept node types in sorted order and published each node's edges right after the node, so an edge reached peers long before its target node of a later type; at the backfill's rate that is tens of minutes. Its paging also made some of those waits permanent: - Nodes were paged per type with OFFSET and no ORDER BY. A topology rebuild deletes every cluster node and inserts replacements, so the offset skipped live nodes whose member_of edges were published. - Per-type offsets were never reset, so a node was visited once: edges added after its sweep (semantic neighbors, new memberships) and in-place node updates were never published. - Nothing published deletions. Worse, apply() wrote this node's own winners back into its graph, resurrecting the cluster nodes and edges its rebuild had just deleted, so the deletion could not be observed. - Peers deferred an edge whose endpoint was tombstoned forever. The topology backfill now runs repeating keyset passes: every node by id, then every edge by id published together with both endpoint nodes (endpoints first, so a peer never holds a writer's edge without them), then this writer's committed topology winners by logical key, tombstoning those the local graph no longer holds. Cursors are stable ids/keys, so inserts and deletes between items cannot shift them, and a restart just begins a new pass; unchanged records publish nothing. apply() still validates every record but never applies the writer's own ones: the local graph is authoritative for them. An edge whose endpoint is known deleted (tombstoned, or an own record the local graph no longer holds) is dropped instead of deferred, and applies again if the endpoint is re-published. Edge lookups during apply use the exact endpoint query instead of a page-limited adjacency scan. Signed-off-by: trvon <me@trevon.dev> * perf(memory-sync): publish documents before derived records in backfill With the topology sweep now publishing every node and edge, a 3-node x 40-doc mesh took longer until every document was listed on every node (603 s against 426 s before the topology fix). Debug logs showed why: a publish takes most of the backfill's 100 ms per-cycle time budget, so each cycle published about one item, and the fair rotation over documents, vectors and topology handed two of every three cycles to derived records while peers waited for the documents. The backfill now serves domains in priority order: documents, then vectors, then topology, each taking items until it has nothing left this cycle. Vectors and topology defer on a peer until their document arrives, so publishing the document first never adds a deferral. In the same mesh, run concurrently with both other builds, every document is listed on every node after 140 s (426 s before the topology fix, 603 s with it alone), and deferrals still drain to 0. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e0d15dc680 |
fix(p2p): stop quarantining honest writers for concurrent-session races (#245)
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
7be908e5b7 |
fix(p2p): stop the memory-sync apply/publish deadlock and report apply health (#243)
* fix(daemon): never gate memory-sync publish on inbound apply applyMemorySyncWinners() ran the inbound stages content -> metadata -> vector -> topology and returned on the first stage failure, before the node's own outbound backfill. In a mesh where every node holds an inbound record whose prerequisite only a peer can publish (an embedding whose content blob has not replicated, an edge whose target node has not), every node refused to publish, so the prerequisites never arrived: a stable mutual deadlock with healthy transport and nothing converging. The outbound publish now runs after the inbound stages whatever their outcome; only a stop request ends a cycle early. Each inbound stage also runs even when an earlier one failed. Cross-stage dependencies (metadata and vectors need local content bytes, edges need their endpoint nodes) are checked per record against the local stores, so a failed stage shows up downstream as unapplied records rather than as skipped stages. Stages keep their prerequisite order so a prerequisite landing in a cycle is visible to its dependents in that same cycle. A two-node test drives real ServiceManager apply cycles over a shared store, each node holding an edge whose target only the other can publish; it converges in a bounded number of rounds only with this change. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): defer records whose prerequisite has not replicated Vector, metadata, and topology apply treated a missing replicated prerequisite as a hard error and rejected the whole batch: one embedding whose content blob had not arrived, or one edge whose target node had not, blocked every other winner in that stage on every cycle. These are ordering races between independently published records, not failures. Each adapter now defers such a record instead: it is skipped, the rest of the batch applies, and its logical key is reported through deferredKeys(). Nothing marks it applied, so it remains a winner in the replicated index and the next cycle retries it once the prerequisite lands; apply needs no pending queue of its own and loses nothing across restarts. An edge whose endpoint is being deleted in the same batch is deferred too rather than inserted against a stale node id. Malformed records, probe failures, and store errors still fail the stage. Content blobs are independent, so the content stage now attempts every blob and fails afterwards with the first genuine error; a corrupt blob no longer starves the blobs after it. A blob whose bytes are not hydrated in the sync cache yet is deferred like the other stages. Signed-off-by: trvon <me@trevon.dev> * feat(p2p): report memory-sync apply health in status The mesh deadlock was invisible: every node reported failed_cycles=0 and a climbing successful_cycles while nothing converged, because a cycle counts as successful once reconciliation completes, whatever the apply stages did afterwards. Deferred records would be just as invisible without a signal. ServiceManager now keeps an ApplyHealth record of the apply path: - deferred records per stage, and the age of the oldest one still pending, with a warning once a deferral passes ten minutes (its prerequisite likely never arrived). Ages are tracked for at most 4096 keys per stage, keeping the oldest; counts stay exact. - hard failures per stage, plus the last failure's stage, message, and age. - apply_cycles and apply_failed_cycles. failed_cycles keeps meaning reconciliation failures; a cycle whose apply stage hard-failed is counted in apply_failed_cycles instead of passing as a plain success. - publish_skipped_cycles (delete outbox preempted the cycle) and publish_failed_cycles (an outbound backfill domain failed). The fields travel through MemorySyncStatus, MemorySyncResponse, and protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the first line, an `apply` line with per-stage counts, a last-apply-failure line, and an `apply` object in --json) and to the memory-sync section of `yams daemon status`. The p2p status formatter moves behind a header so it can be tested without a daemon. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): keep content deferrals when the same scan hits a failure The content stage scans every blob and fails afterwards with the first genuine error, but the apply cycle discarded the deferred keys that the same scan collected whenever it failed. A corrupt blob therefore hid every blob still waiting for its bytes: the deferred count stayed at its previous value (0 on the first pass) and the stale-deferral warning never fired for a prerequisite that was actually missing. A stage pass now carries both outcomes. The content-blob scan moves into ContentBlobSyncAdapter::applyWinners, which reports the applied count, the deferred keys, and the first failure of one complete scan, and ApplyHealth::recordPass records the failure and replaces the deferred set together. A stage that returns an error still leaves the deferred set as last observed, since an aborted scan says nothing about prerequisites. The tracked-deferral cap already carried every still-deferred tracked key before admitting new ones, but its carry loop had a cap check that read as if it could stop early. Drop it (the tracked set never exceeds the cap) and pin the behaviour with tests that place the oldest key after more than the cap's worth of new keys and that refill a full tracked set. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
trvon
|
25d171dd62
|
ci(fuzz): run the fuzz lane on the self-hosted Forgejo runner
Fuzzing belongs on our own compute rather than GitHub-hosted runners. Move the smoke job to .forgejo/workflows on the self-hosted label, pin actions to code.forgejo.org, and add a preflight that fails clearly when the runner lacks clang's libFuzzer runtime or the build tools. Note that once .forgejo/workflows exists, Forgejo runs only those workflows and stops executing the .github/workflows copies. |
||
|
trvon
|
34b9f7346e
|
docs(p2p): describe the fuzz lane, its oracles and remaining gaps | ||
|
Trevon
|
9c19452e7d |
test(p2p): add direct control fuzz harnesses (#74)
Add libFuzzer/AFL++ harnesses for the direct-P2P handshake/state/history and delta/bootstrap control parsers. Fuzzers invoke the same production parser seams as the live wire path, so coverage cannot drift from real validation. A protocol-v3 seed exercises exact-version rejection after the v4 upgrade. Bound noninteractive AFL++ campaigns with AFL_FUZZ_SECONDS and register focused fuzz-smoke tests without duplicating broad protocol/delta registrations. Clarify that peer/cycle status is local health telemetry rather than global convergence, and document the remaining decision-grade live-performance gate. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
1c30839d7f |
fix(p2p): harden bootstrap trust (#72)
Require authenticated direct stores and explicit operator enrollment while hardening private keys and trust files across Unix and Windows. Sign local control artifacts, bind them to a protected store generation, and serialize async initialization shutdown. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
0eac7c2fd9 |
fix(p2p): harden transport availability (#71)
Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
dbe7aea900 |
fix(p2p): resume bounded history catch-up (#70)
Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
5a53cdfa90 |
feat(p2p): add authenticated direct replication (#68)
* fix(memory-sync): keep replication responsive Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle. Reconstructed from ff62d3f8, 8e232721, and e60f513d. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated direct transport Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged. Reconstructed from 96f9db88 and c46de77b. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): exchange causal deltas Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies. Reconstructed from 1b8be146 and 41883338. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): integrate direct sync services Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport. Reconstructed from a19aaa44. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): apply replicated document state Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths. Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5. Signed-off-by: trvon <git@trevon.dev> * test(p2p): register direct sync gates Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout. Reconstructed from 360b9115 and 89416ccb. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): require bounded peer enrollment Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity. Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): persist quarantine and deletes Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects. Reconstructed from 74ce01e0 and 19888604. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated cold bootstrap Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume. Reconstructed from 8a9f5acf. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |