Dispatch explicit committed memory reads on admitted workers, checking the
governor before queueing and before I/O. Require the coordinator instead of
using the synchronous strand fallback; keep cached APIs and wire opcodes
unchanged and preserve task-record validation.
Use the paged content adapter, distinguish typed local admission deferral
from hard errors and retain known deferred totals/ages after incomplete
scans. Forward exact topology totals independently of diagnostic samples
and keep attempted cycles distinct from complete scans and successful apply.
Add real daemon IPC tests for authenticated off-resident reads, pressure
health, exact 70-edge status/retry accounting and authenticated cold document
retrieval through the existing MCP get handler, DaemonClient IPC and CAS.
Reanchor existing dispatcher environment-read entries without new exceptions.
Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader
fault/readiness/P2P convergence and patched resource measurements remain open.
Signed-off-by: trvon <git@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): add keyset node/edge scans and exact edge lookup
The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no
ORDER BY, so a topology rebuild that deleted and reinserted cluster
nodes between two pages shifted the offset and the sweep skipped live
nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never
reused, so a cursor on the last id returned is stable under concurrent
inserts and deletes and survives restarts.
getNodesAfterId/getEdgesAfterId scan in id order from such a cursor.
getEdgesBetween returns every edge between two nodes for one relation;
getEdgesFrom truncates at its page limit, which made "is this edge
present" checks on high-degree nodes unreliable.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish topology edge endpoints and retract deleted ones
On a Docker mesh that converges on documents, topology deferrals never
drained: peers held member_of and semantic_neighbor edges whose endpoint
node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80).
Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral
named a doc or cluster node that its origin still held but had not
published. The origin's sweep was still inside the "document" type: no
"path", "topology_cluster" or "topology_snapshot" node had been published
by any node, while the doc nodes' edges into them had been.
The backfill swept node types in sorted order and published each node's
edges right after the node, so an edge reached peers long before its
target node of a later type; at the backfill's rate that is tens of
minutes. Its paging also made some of those waits permanent:
- Nodes were paged per type with OFFSET and no ORDER BY. A topology
rebuild deletes every cluster node and inserts replacements, so the
offset skipped live nodes whose member_of edges were published.
- Per-type offsets were never reset, so a node was visited once: edges
added after its sweep (semantic neighbors, new memberships) and
in-place node updates were never published.
- Nothing published deletions. Worse, apply() wrote this node's own
winners back into its graph, resurrecting the cluster nodes and edges
its rebuild had just deleted, so the deletion could not be observed.
- Peers deferred an edge whose endpoint was tombstoned forever.
The topology backfill now runs repeating keyset passes: every node by
id, then every edge by id published together with both endpoint nodes
(endpoints first, so a peer never holds a writer's edge without them),
then this writer's committed topology winners by logical key, tombstoning
those the local graph no longer holds. Cursors are stable ids/keys, so
inserts and deletes between items cannot shift them, and a restart just
begins a new pass; unchanged records publish nothing.
apply() still validates every record but never applies the writer's own
ones: the local graph is authoritative for them. An edge whose endpoint
is known deleted (tombstoned, or an own record the local graph no longer
holds) is dropped instead of deferred, and applies again if the endpoint
is re-published. Edge lookups during apply use the exact endpoint query
instead of a page-limited adjacency scan.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory-sync): publish documents before derived records in backfill
With the topology sweep now publishing every node and edge, a 3-node x
40-doc mesh took longer until every document was listed on every node
(603 s against 426 s before the topology fix). Debug logs showed why:
a publish takes most of the backfill's 100 ms per-cycle time budget, so
each cycle published about one item, and the fair rotation over
documents, vectors and topology handed two of every three cycles to
derived records while peers waited for the documents.
The backfill now serves domains in priority order: documents, then
vectors, then topology, each taking items until it has nothing left this
cycle. Vectors and topology defer on a peer until their document
arrives, so publishing the document first never adds a deferral. In the
same mesh, run concurrently with both other builds, every document is
listed on every node after 140 s (426 s before the topology fix, 603 s
with it alone), and deferrals still drain to 0.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Apply the existing configured timeout scale to the integration executable. Observe periodic backfill attempts while retaining all publication assertions; no forced backfill or production scheduling change. Full TSan memory-sync suite passed.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
