Dispatch explicit committed memory reads on admitted workers, checking the
governor before queueing and before I/O. Require the coordinator instead of
using the synchronous strand fallback; keep cached APIs and wire opcodes
unchanged and preserve task-record validation.
Use the paged content adapter, distinguish typed local admission deferral
from hard errors and retain known deferred totals/ages after incomplete
scans. Forward exact topology totals independently of diagnostic samples
and keep attempted cycles distinct from complete scans and successful apply.
Add real daemon IPC tests for authenticated off-resident reads, pressure
health, exact 70-edge status/retry accounting and authenticated cold document
retrieval through the existing MCP get handler, DaemonClient IPC and CAS.
Reanchor existing dispatcher environment-read entries without new exceptions.
Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader
fault/readiness/P2P convergence and patched resource measurements remain open.
Signed-off-by: trvon <git@trevon.dev>
Use pinned, governed pages for metadata, content, vectors and topology.
Preflight identity/content/model prerequisites before writes and continue
independent records without softening authentication, capacity or I/O errors.
Apply topology deletions, nodes and edges in one real KG batch with bounded
transaction-local reads. Preserve ordinary-read isolation, report exact edge
deferrals independently of bounded diagnostic samples and avoid empty commits.
Retract absent local topology only from exact committed own observations and
conditionally erase the same envelope, including off-resident/stale-winner
regressions. Probe local KG presence with fallible scalar SQL rather than
materializing properties; retain SQL errors as errors, not absent records.
The public C++ KG/batch interfaces gain virtual capabilities. Defaults provide
source-level hard errors only: installed/old-binary ABI compatibility is NOT
established and providers must be rebuilt/reviewed before promotion.
Signed-off-by: trvon <git@trevon.dev>
Expose owner-validated committed views, bounded exclusive-prefix pages and
fallible exact historical payload reads without per-value reconciliation.
Keep cached reads nonblocking and separate from explicit committed I/O.
Add governed adapter indexes, revocable escaped-handle lifetime checks and
typed local pressure/contention deferral. Release the loop lock only across
audited native payload fetches and recheck cancellation/quarantine afterward.
Preserve the process-wide recursive callback serializer while releasing the
lifecycle gate before its drain on both restart and stop. Recheck workers and
callback generations after the gap; retain all cancellation and join guards.
Wire the optional local-yield callback through the config factory alongside
the constructor change.
Add native SDK concurrency, pressure, lifetime, off-resident and callback-order
regressions. Arbitrary blocking backend/callback shutdown is not signed off.
Signed-off-by: trvon <git@trevon.dev>
Spill exclusive native-filesystem sweep observations and committed winners
into bounded SQLite workspaces on the corpus filesystem. Preserve complete
operation/vector fork observations and validate payloads before publishing
an immutable generation; keep resident caches separate from full authority.
Add fallible committed/pinned reads, bounded adapter bookkeeping, cooperative
filesystem traversal, secure scratch allocation/cleanup and pinned lifetime
limits. Retain exact durable erase intents and condition own-record erasure
on the full current envelope, including a post-reconciliation check.
Preserve the existing shared-store, cache, value and scan limits. Reanchor
only the existing filesystem environment-read entries and register SQLite
and layout-independent test hooks in the memory-sync library.
Bootstrap import/finalize/restart, broader ingress/scratch faults and final
SDK ABI/security/performance signoff remain separate work.
Signed-off-by: trvon <git@trevon.dev>
A complete sweep fails closed when its retained state cannot hold the
corpus, returning ResourceExhausted ("complete-sweep candidate budget
exhausted"). That is correct: a truncated sweep must never commit a fork
decision. What was missing is a failure state. Every caller retried
immediately, and each attempt re-listed the whole op store, so a corpus
larger than maxTrackedIdentities pinned a core indefinitely.
Observed on a 65,612-object op store with 24,689 index records against
maxTrackedIdentities=8192: one thread in
publish -> reconcileBeforeWrite -> syncFully -> sync
-> FilesystemBackend::listPage("index/")
-> visitFilesystemKeys -> recursive_directory_iterator
92% of it in kernel time, 7h26m CPU over 7h29m wall in a single run, and
17,486 "backfill domain failed: complete-sweep candidate budget
exhausted" lines. The listing snapshot cache could not help: it never
completes, because the sweep aborts before reaching the end.
Remember the unsatisfiable sweep and gate retries behind bounded
exponential backoff (5s..600s) so one attempt per window replaces one
attempt per caller. A single retry is still admitted, because one
exhaustion can be a store that is about to fit (an erase or quarantine
landing between attempts); the latch starts on consecutive failures and
clears when a sweep completes.
Only the sweep's own budget sites set sweepBoundExhausted_. A
resource-governor denial reports ResourceExhausted too, but it is
transient backpressure and must keep retrying freely, otherwise pressure
that has already lifted suppresses convergence for a whole window.
Fail-closed behaviour is unchanged: the wrapper never converts a failure
into success, nothing commits until a sweep completes, and the memoized
error carries the original code and message. Cancellation is never
memoized, so shutdown is not delayed.
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): bound whole-sweep candidate state
Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision.
Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default.
The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(storage): bound filesystem listPage residency
listPage() walked the whole op store and retained every matching key before
slicing one page. At 1025 records and a 512-record page it held 1025 keys
where 513 is the working set, and the retained set grew with the store.
Traverse once into a max-heap holding the smallest eligible limit+1 keys.
The walk itself stays complete, so a traversal error still returns an error
instead of a partial page. Exclusive lexical cursor, ordering, Windows key
decoding and the SIZE_MAX limit stay as they were.
Add a YAMS_TESTING-only observation seam that counts real walks, entries
visited and peak retained keys, so the traversal cost can be measured
instead of inferred. The new traversal shifts two reviewed getenv allowlist
line numbers.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep backfill progress past an oversized vector
A record above the resident limit failed publication, the domain cursor
never advanced, and every later cycle retried the same row. Valid vectors
behind it never reconciled, so one oversized value could pin a domain
indefinitely. The daemon log showed this repeating every ~100 s.
Classify the row only after a publish fails with InvalidArgument and the
same serialized record is independently confirmed to exceed the immutable
limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep
the local copy, and revisit it when a new source generation or service
limit appears. A generic backend InvalidArgument stays retryable, and
healthy rows still serialize only once.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): reuse a bounded complete listing across publishes
Every store invalidated the cached listing pages, so a publish followed by
syncFully walked the whole op store again, once per 512-record page. On the
profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of
it system time, with listPage at 55% of user samples.
Keep a complete key-only prefix snapshot, built only from normal contiguous
pages starting at an empty cursor on the concrete filesystem backend. The
snapshot is capped by key count and key bytes, served only in lexical order,
and dropped as incomplete unless the walk reached the end. Any mutation
invalidates it before the backend call; a successful store or remove then
applies the exact key to matching snapshots, and a failure drops them
because the write may have committed. Initialising the async path disables
cache reuse permanently rather than racing a future.
Cache maintenance is best effort: it runs after the backend result is known
and any allocation failure clears the cache and keeps the committed result.
Opaque cursors are untouched, so only the built-in filesystem backend gets
snapshot treatment.
Active publishes now cost the same number of walks as a warm sweep instead
of four, six and nine at 511, 513 and 1025 records.
Signed-off-by: trvon <git@trevon.dev>
* test(bench): measure memory-sync listing cost across arms
Add a standalone harness that drives a real filesystem op store through
MemorySyncLoop, one scenario per process so no cache state survives a
repeat, and reports wall, user, system and peak RSS per run. It uses
production APIs only, so the same source builds against a baseline and a
candidate tree and neither arm needs a measurement seam ported into it.
The driver seeds one corpus per size, interleaves the two arms inside each
repeat, and writes results.jsonl plus a median report. Enumeration counts
are not re-derived here; the unit suite owns those.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* test(search): rename a road beam local that collides with the Windows far macro
windows.h defines far and near as empty macros, so `const auto far = ...`
in search_vector_stage_road_beam_catch2_test.cpp does not compile with MSVC
(C2513), and the Windows Tests job has failed since the road beam tests
landed. Rename it to deep.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory_sync): skip rescans of an unchanged direct op store
The sync worker rescanned the whole op store every cycle: a full walk of the
sharded objects tree, then a read, hash, decode and signature check of every
index envelope. On an idle 3-node mesh (2 CPUs per node, 5,256 objects) that
cost about 1.1 s of CPU per 5 s cycle on each node, mostly in getdents/openat.
In direct transport the daemon is the op store's only writer. The loop now
counts its own backend writes and reuses the last complete scan until it
writes again or a writer is quarantined. Shared-store and temporary namespaces
keep rescanning every cycle.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory_sync): keep exclusive-backend listings until the loop writes
FilesystemBackend::listPage walks every shard directory of the op store for
any prefix. After the rescan skip, each idle direct-transport cycle still
listed the store twice: replaying ready erases and inspecting the erase
outbox. The loop's write-counting backend now keeps listing pages on an
exclusive backend and drops them on any store, remove or clear.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Reassert backend cancellation under the lifecycle lock. Protect fake
backend condition-variable predicates against lost wakeups.
Fix Windows fixture encoding, shell selection, and rooted-path checks.
Keep the unresolved rename failure diagnostic-only.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): give the CLI fixture the shared corpus scope it needs
94e948c6 made createMemorySyncService fail closed unless
memory_sync.corpus_scope is "shared" and updated the service-manager
test, but not the p2p CLI fixture, whose eight cases have failed at
construction ever since ("replication requires
memory_sync.corpus_scope=shared"). The fixture now sets the scope; the
cases run again and pass.
Signed-off-by: trvon <me@trevon.dev>
* refactor(memory_sync): move MemorySyncLoop bodies into a translation unit
include/yams/memory_sync/memory_sync.h was a 3,527-line header-only
class: 120 inline method definitions, 2,900 lines of implementation,
pulled into every translation unit that includes daemon.h or the p2p
protocol header. The memory-sync subsystem had no source directory at
all.
MemorySyncLoop's bodies now live in src/memory_sync/memory_sync.cpp,
compiled once into a new yams_memory_sync static library that follows
the yams_wal pattern. The header keeps the declarations, the nested
types, the constructor, one-line accessors, and the YAMS_TESTING hooks
(those must stay inline because the library is not built with the
testing define). The daemon library and its declared dependency link
the new library, as do the five memory-sync test binaries.
Zero semantic change: the move is mechanical (a scripted split that
copies each body verbatim, drops default arguments and static from the
definitions, and qualifies nested return types). All seven memory-sync
test binaries and the service-manager suite pass unchanged.
memory_sync.h: 3,527 -> 569 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): move TuneAdvisor method bodies out of the header
TuneAdvisor.h was 3,345 lines, almost all of it the bodies of ~300
static accessors and setters, included by 60 translation units. Each
of them recompiled the whole policy surface on every touch, and any
change to one knob rebuilt most of the daemon.
The bodies now live in src/daemon/components/TuneAdvisor.cpp (which
previously held only the system-memory probe). The header keeps the
declarations, the static inline override slots (unchanged, so the
ODR story for process-wide tuning state is exactly what it was), the
YAMS_TESTING hooks, and the two methods whose bodies are
preprocessor-conditional on YAMS_TESTING; those must stay inline
because the library is not built with the testing define and moving
them would silently switch tests to the production branch.
Mechanical move by the same scripted split used for memory_sync.h:
bodies copied verbatim, static and default arguments dropped from the
definitions, nested return types qualified. The one template
(readConfiguredOverridesSnapshot) stays in the header because three
other translation units instantiate it.
The header is included well outside src/daemon (vector, repair, app
services, MCP, the CLI doctor, the ONNX plugin), and those targets do
not link yams_daemon. They used to get the bodies for free from the
header, so the move needs a home they can all link: the new
yams_tune_advisor static library (src/daemon/tune_advisor/meson.build)
builds TuneAdvisor.cpp alone, and yams_daemon, yams_vector,
yams_repair, yams_app_services, yams_mcp, yams_cli, the ONNX plugin,
and the standalone snapshot test depend on it.
Executables compiled with -DYAMS_TESTING link a second build of the
same file, yams_tune_advisor_testing, ahead of the production archive.
Under that define compatibilityEnvironment() reads the process
environment live instead of the startup snapshot, which every
env-driven tuning test relies on; before the split each such TU
instantiated the inline bodies itself with the define. Meson emits
dependencies in list order and the first archive defining a symbol
wins, so the order is load-bearing: test_deps, the benchmark
dependency lists, and the mobile ABI test all put the testing archive
first, and tests/meson.build says so. The testing archive is built
under the same predicate that enters tests/ (a plain debug configure
included), so `meson setup --buildtype=debug` keeps working. The
standalone snapshot test keeps the production build on purpose.
No behavior change; a full debug compile is clean and the tuning
suites pass unchanged.
TuneAdvisor.h: 3,345 -> 1,170 lines.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Compile the direct-P2P test-only machinery out of production builds
(YAMS_DAEMON_TEST_HOOKS_ENABLED now wraps the reconnect-loop hook storage,
invoker, and call site, matching the already-guarded setter). Replace the
single shared writer key in p2p_manager tests with one distinct key pair per
writer identity so origin-binding regressions (a record claiming origin X but
signed by Y's key) fail verification. Extract the copy-pasted bytes/text/TempDir
helpers into tests/common/p2p_test_helpers.h and reuse them across the
transport/delta/manager suites. Replace a timing-sensitive fixed sleep in the
transport session-expiry test with a bounded poll that accepts both deadline
manifestations, and gate the memory-sync-service concurrent-lifecycle stress
case behind YAMS_RUN_STRESS_TESTS=1 per the existing FTS5 pattern. Deduplicate
the byte-identical commitment-JSON serialization inside MemorySyncLoop
(coldBootstrapRoot / persistReplicationCheckpoint) into one static helper.
Signed-off-by: trvon <git@trevon.dev>
Bound pre-authentication JSON nesting depth to 32 so a pathologically nested
control frame cannot overflow the session-thread stack before any trust
decision. Raise the control-frame cap to 512 KiB to cover the worst-case
authenticated state frame (~300 KiB across 256 writers x 256-byte ids). Gate
MemorySyncLoop::applyDeltas through the resource governor so a peer cannot
flood deltas on the direct replication path. Account delta/bootstrap wire bytes
using the exact received frame size instead of a compact re-serialization.
Signed-off-by: trvon <git@trevon.dev>
Require authenticated direct stores and explicit operator enrollment while hardening private keys and trust files across Unix and Windows. Sign local control artifacts, bind them to a protected store generation, and serialize async initialization shutdown.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Route the s3 backend through StorageBackendFactory::parseURL so scheme
detection, authority credentials, and query params flow through the same
path as the daemon's storage resolver, and reject a backend=s3 config
whose path is not an s3:// URL.
Thin p2p command over the memory-sync service: publish <key> <value|@file>,
read <key>, and status (merged index summary + backend identity). Binary
values are hex-encoded in JSON output; config resolves from [memory_sync]
with --backend/--path/--node-id overrides.
Typed MemorySyncDaemonConfig parsed from the [memory_sync] TOML section
(enabled, node_id, backend, path, sync_interval_ms) plus
createMemorySyncService() which resolves filesystem/s3 backends and
constructs the service. No YAMS_* env knobs.
Daemon-facing service owns the backend + sync loop and reconciles on a
configurable interval via a stoppable worker thread. Typed
MemorySyncConfig (nodeId, syncIntervalMs); start/stop idempotent and
stop noexcept-safe. Backend injected so filesystem and S3/R2 resolution
stay with the caller.
Publish memory records as content-addressed blobs plus a
version-vector-tagged index record per write; sync lists the index,
pulls records, and LWW-merges to a deterministic winner per logical
key. Backend injected (filesystem / S3-R2 interchangeable).
Version vector (element-wise max merge, dominates/concurrent partial
order) plus hybrid timestamp and LWW resolution: causally-later writes
win, concurrent writes resolve deterministically by (ts, origin). JSON
serialization via nlohmann. Header-only, no library deps.
