7e744805 (#241) registered daemon_harness_full_isolation with get_option('integration-daemon-timeout'). ca944de5 removed that option, so every Tests lane on experimental fails at meson setup: "Option integration-daemon-timeout does not exist". Use daemon_integration_timeout, as the other daemon_harness tests do. Configured on grim with the default, asan, ubsan, mobile and macOS option sets from tests.yml.
Signed-off-by: trvon <me@trevon.dev>
Wait for caller startup before measuring completion and keep synchronous
console I/O out of the checkpoint wait test. Preserve the 20 ms timeout
and 250 ms assertion, and verify the queued/timeout telemetry.
Keep a guarded rescue executor alive for late posts so the unbounded-wait
negative control fails without leaving an unjoinable caller.
Signed-off-by: trvon <git@trevon.dev>
* test(memory-sync): cover vectors committed behind the backfill cursor
The vector backfill cursor (document_hash, chunk_id) never wraps, and
the backfill is the only outbound path for vectors, so a vector added
or re-embedded behind the cursor after the first sweep is never
published to peers.
Add red tests for:
- the sweep starting over to publish a new and a re-embedded vector
that sort below its last position;
- vectors an embedding job commits being published within the next
backfill cycle, ahead of a sweep that would need many items to reach
them;
- the topology sweep still publishing its share while the vector sweep
always has work.
Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op
seam for the embedding commit notification; the fix fills it in.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish committed vectors and wrap the vector sweep
The vector backfill cursor (document_hash, chunk_id) was set once and
reset only by a daemon restart, and the backfill is the only outbound
path for vectors. After the first full sweep, a vector added or
re-embedded under a document hash that sorts below the cursor was never
published to peers. Topology had the same bug and now sweeps in
repeating passes; vectors did not.
Eager publish: the embedding service already reports the documents a
job committed vectors for (successHashes, after the vector insert and
before readiness is published) through the callback that requested the
topology rebuild. That callback is renamed setEmbeddingsCommittedCallback
and the daemon now also passes the hashes to
ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues
them (bounded at 4096 documents, deduplicated, dropped past the cap):
it runs on the embedding worker, and the memory-sync service's lifetime
and outbound budget belong to the sync cycle. The backfill serves the
queue as its own domain right after documents, one vector row per item,
walking each document's chunks with getVectorsPage. The queue is in
memory; a restart loses it, but the sweep then starts over anyway.
Catch-up sweep: when the vector sweep reaches the end of the table it
resets its cursor and sits out the rest of the cycle, as the topology
pass does, so the next pass publishes whatever changed behind it.
publishIfChanged keeps repeat passes from writing unchanged records.
R7's skip logic is kept: an unpublishable row is skipped and the
cursor moves past it; it is logged on the first pass (and when
committed) but not again on every later pass.
Fairness: with both sweeps repeating, both always have work, so the
strict priority order (documents, vectors, topology) would hand the
vector sweep every item and starve topology. Documents and committed
vectors keep priority; the two sweeps now alternate item by item, and
the turn persists across cycles so a one-item budget still alternates.
A sweep that is done for the cycle leaves its turns to the other. This
was chosen over alternating whole passes because it needs one field of
state, keeps both sweeps' latency bounded by their own size rather than
the other's, and reuses the existing per-item budget loop unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): pin the vector-store environment in the cursor-wrap cases
Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): cover vector records that cannot apply or publish
One bad vector record should cost only itself. Add red tests for the
vector sync adapter (corrupt payload, dimension mismatches, non-hash
document identity, key/identity mismatch sorted between two good
records), for the daemon apply cycle reporting such a record in apply
health while the good embedding lands, and for the outbound backfill
publishing good vectors past a row with no model identity.
Add a YAMS_TESTING hook to inject a vector database into the
ServiceManager so the daemon vector stage and backfill can be driven
in-process.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): skip bad vector records instead of stalling the stage
Inbound: VectorSyncAdapter::apply() returned an error on the first
winner it could not apply (corrupt payload, dimension or identity
mismatch, non-hash document id, a model the store cannot hold for its
chunk, a failed store write or content probe). It validated the whole
batch before any write, so one such winner kept every vector winner
from applying on every cycle until a newer winner replaced it.
Each winner is now applied, deferred, or skipped on its own, as the
content stage does. A skipped winner is reported through failure()
(first failure, prefixed with its logical key), which the daemon maps
to ApplyStagePass::failure, so apply health counts it as a vector stage
failure while good records land. Skipped winners are never marked
applied and are retried on the next pass. Winners that disagree on a
chunk's model are all skipped, so the outcome does not depend on
arrival order. Reconciliation and index rebuild failures still end the
pass.
Outbound: the backfill threw on any vector publish error without
advancing its (document_hash, chunk_id) cursor, so a row that can
never publish (for example one with no model id) stopped the vector
sweep on that row for good. Such rows are now skipped with a warning,
as unrecoverable documents are; transient failures still propagate
and are retried next cycle.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): pin the vector-store environment in the vector cases
CI runs the unit lanes with YAMS_DISABLE_VECTORS=1, YAMS_SQLITE_VEC_SKIP_INIT=1 and YAMS_VDB_IN_MEMORY=1. VectorDatabase then skips creating its tables, so every vector insert in these cases failed under the CI environment while passing in a clean shell. Clear those variables for each case that attaches a vector store, as the init command tests do.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
A CompressedStorageEngine wrapping an engine without a base path used a
fixed /tmp/yams-compressed-wrapper as its own base, and the StorageEngine
base creates objects/ and temp/ there. With the daemon now running as a
systemd user unit for each user (UMask=0077), whichever user's daemon
started first owned that directory, and every other user's daemon with
compression enabled (the yams init default) failed at startup with
"Failed to create storage directory: Permission denied" and never became
ready. Found by the package validation: two lingering users, reboot, the
second daemon stuck initializing.
Suffix the scratch directory with the real UID on POSIX.
Signed-off-by: trvon <me@trevon.dev>
With the daemon shipped as a systemd user unit, the CLI must not fight
systemd for it:
- Auto-start (DaemonClient::startDaemon): when the unit is installed but
not running and would run exactly the daemon the client expects (same
socket, data dir and config as the default resolution, default
binary), start it with `systemctl --user start` instead of forking a
copy that a later login would race for the socket. Flag or
environment overrides (--socket, --data-dir, YAMS_DAEMON_SOCKET,
YAMS_DATA_DIR, YAMS_CONFIG, YAMS_DAEMON_BIN) keep the fork path.
- `yams daemon start|stop|restart` use `systemctl --user` for an
installed/active unit (start/restart without custom path options,
stop/restart without --socket), falling back to IPC if systemctl
fails.
- `yams daemon install --user` with no options enables the packaged unit
(`systemctl --user enable --now`) instead of writing a copy into
~/.config/systemd/user that would shadow it; with options it still
writes that override and says so. `uninstall --user` reloads the
manager after removing an override and explains how to mask the
globally enabled packaged unit for one account.
The decision is a pure function (userUnitMatchesPaths) with Catch2
coverage; systemctl is only run with whitelisted verbs and only when a
user manager is reachable ($XDG_RUNTIME_DIR set, Linux).
Signed-off-by: trvon <me@trevon.dev>
The packaged service becomes a systemd user unit that runs each user's
own daemon on the CLI's default per-user socket, with that user's
config and data. There is no system daemon left to discover, and the
daemon always runs as the same user as the CLI, so both workarounds for
the system-service design go:
- Revert "feat(cli): find the packaged system daemon socket without
configuration" (client-side /run/yams/yams-daemon.sock discovery, the
systemd refusal in `daemon stop|restart`, the spawn refusal and the
status mismatch suppression, plus its tests).
- Revert "fix(cli): send file content to a daemon that runs as another
user" (peer-UID probe and content upload in `yams add`).
Per-user socket resolution is back to: explicit flag/env/config, then
$XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback).
Signed-off-by: trvon <me@trevon.dev>
The resolution tests used ELF names (libonnxruntime.so.1) everywhere,
but the policy only treats .dylib files as runtime libraries on macOS,
so the macOS test lane would have failed. Fixtures now use the host's
names (libonnxruntime.1.dylib / libonnxruntime.1.23.0.dylib on macOS)
and build paths with fs::path; the test is not registered on Windows,
where resolution is the DLL search for onnxruntime.dll.
grim (Linux, Debug): 7/7 cases, 34 assertions pass.
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin cached compiled CoreML models directly in the model
directory. With the plugin now able to run on either the system or the
bundled ONNX Runtime, a cache compiled by one version was reused by the
other: on macOS, after a run on Homebrew's 1.30 the bundled 1.23 failed
every embedding batch with 'Non-zero status code returned while running
..._CoreML_... node'. The default cache is now
<model dir>/coreml-ort-<runtime version>; YAMS_COREML_CACHE_DIR still
overrides it.
Verified on macOS arm64: a run on the system runtime creates
coreml-ort-1.30.0, a run with Homebrew's copy hidden uses the bundled
runtime, creates coreml-ort-1.23.0 and embeds new documents without
CoreML errors.
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
`yams add <path>` passes the path to the daemon, which reads the file
itself. The packaged system daemon runs as its own account with
ProtectHome= and PrivateTmp=, so adding ~/notes.txt failed with
"Permission denied" and a /tmp file was accepted but never stored (the
daemon's /tmp is private).
When SO_PEERCRED/getpeereid shows the daemon's UID differs from ours,
the CLI now reads single files itself and sends their bytes (up to
15 MiB, under the 16 MiB IPC frame) named by their absolute path.
Directory adds stay daemon-side walks (include/exclude patterns and
ignore files), so for such a daemon they now fail up front with an
explanation instead of queueing a walk that finds nothing. Same-user
daemons keep the path-based flow.
Signed-off-by: trvon <me@trevon.dev>
The deb, rpm and Arch packages start yams-daemon.service on
/run/yams/yams-daemon.sock, but clients only looked at the per-user
default (XDG_RUNTIME_DIR, /var/run as root, /tmp/yams-daemon-<uid>.sock).
With no YAMS_DAEMON_SOCKET set, root and normal users silently fell back
to an in-process daemon and a private store in ~/.local/share/yams, so
the installed service was never used.
Client-side resolution now goes: an explicit socket (flag, env, config),
the per-user daemon when its socket exists or a personal data directory
was named (flag, env, core.data_dir from yams init), then the system
socket when it exists and is connectable (root or the yams group), and
finally the per-user default. Daemons keep binding what
resolve_runtime_paths() names; `yams daemon install --user` writes the
per-user socket, never the discovered one.
Because systemd owns the system unit, `yams daemon stop|restart` refuse
it with a systemctl hint and the client never tries to spawn a daemon on
that socket. `daemon status` no longer flags the system corpus as a data
directory mismatch. socket_utils also gains a peer-UID probe used by the
next change.
The two socket default tests now use the per-user resolver so they do
not depend on whether the build host runs the packaged service.
Signed-off-by: trvon <me@trevon.dev>
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.
Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
the old default), conan-generators-dir, cli11-cmake-path,
onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
integration-smoke-timeout, integration-daemon-timeout and stress-iters:
replaced by the same constants at their call sites (smoke keeps its
test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
folded into enable-integration-tests. Debug builds still register every
integration suite; nothing set the sub-toggles.
Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).
Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).
Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.
Signed-off-by: trvon <me@trevon.dev>
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
The deferred-drain check treated a missing value as zero, so a node
whose p2p status call failed or returned non-JSON satisfied it. Drain
now needs a numeric 0 from every node, and so does the quarantined
writer check once any node reports the field; an unreadable status is
reported as such instead of passing.
Signed-off-by: trvon <me@trevon.dev>
converge proves convergence with yams get once yams list shows every
hash. When those reads kept failing until the timeout, the listing gap
was empty, so converge.json and REPORT.md claimed every document was
readable next to converged=false. The final yams get failures are now
kept (unreadable_via_get) and used for the readable counts whenever the
read-path check ran.
Signed-off-by: trvon <me@trevon.dev>
seed floored --docs-per-node to whole rounds of topics: 10 documents
with 8 topics seeded 8, 0 still seeded a full round, and --topics 0
raised ZeroDivisionError. Both counts are now validated, and the last
round is truncated so every node gets exactly the requested number.
Signed-off-by: trvon <me@trevon.dev>
up --force only skipped the existing-state guard. Compose then recreated
the services on the old named volumes, so nodes came up with the stale
/node/ready marker, keys and config of the previous mesh, and services
of a larger previous mesh kept running as orphans. A forced up now runs
down (volumes, orphans, image) for the previous project first and starts
under a fresh project name.
Signed-off-by: trvon <me@trevon.dev>
A mesh of honest writers must never durably quarantine one of them, but
converge only checked document reachability and deferral drain, so a run
that quarantined a peer could still pass. converge now records
quarantined_writers per node and exits 4 when any node reports one.
Every up/run also gets its own compose project (yamsmesh-<random>, or
--project), so containers, network, volumes and image tag of concurrent
meshes on one host no longer collide.
Signed-off-by: trvon <me@trevon.dev>
Production meshes (transport=direct, full mesh, operator-pinned mutual TLS)
stalled at a fixed record count with failed_cycles=0 because a failed inbound
apply stage skipped the node's own publish. Reproducing that needs several
real daemons with enrolled pins and a corpus of embeddings and topology
edges authored on different nodes, so add tests/integration/p2p_mesh:
- Dockerfile and node entrypoint that run prebuilt yams/yams-daemon from a
host build dir or a released .deb, one isolated volume per node.
- mesh.py (stdlib only) with up/seed/converge/status/chaos/run/down. It
generates writer keys and manifests, enrolls every peer pin and connects
a full mesh the way an operator does, and records apply-health counters
when the build reports them.
Local/manual tooling only; not wired into any CI.
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): add keyset node/edge scans and exact edge lookup
The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no
ORDER BY, so a topology rebuild that deleted and reinserted cluster
nodes between two pages shifted the offset and the sweep skipped live
nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never
reused, so a cursor on the last id returned is stable under concurrent
inserts and deletes and survives restarts.
getNodesAfterId/getEdgesAfterId scan in id order from such a cursor.
getEdgesBetween returns every edge between two nodes for one relation;
getEdgesFrom truncates at its page limit, which made "is this edge
present" checks on high-degree nodes unreliable.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish topology edge endpoints and retract deleted ones
On a Docker mesh that converges on documents, topology deferrals never
drained: peers held member_of and semantic_neighbor edges whose endpoint
node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80).
Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral
named a doc or cluster node that its origin still held but had not
published. The origin's sweep was still inside the "document" type: no
"path", "topology_cluster" or "topology_snapshot" node had been published
by any node, while the doc nodes' edges into them had been.
The backfill swept node types in sorted order and published each node's
edges right after the node, so an edge reached peers long before its
target node of a later type; at the backfill's rate that is tens of
minutes. Its paging also made some of those waits permanent:
- Nodes were paged per type with OFFSET and no ORDER BY. A topology
rebuild deletes every cluster node and inserts replacements, so the
offset skipped live nodes whose member_of edges were published.
- Per-type offsets were never reset, so a node was visited once: edges
added after its sweep (semantic neighbors, new memberships) and
in-place node updates were never published.
- Nothing published deletions. Worse, apply() wrote this node's own
winners back into its graph, resurrecting the cluster nodes and edges
its rebuild had just deleted, so the deletion could not be observed.
- Peers deferred an edge whose endpoint was tombstoned forever.
The topology backfill now runs repeating keyset passes: every node by
id, then every edge by id published together with both endpoint nodes
(endpoints first, so a peer never holds a writer's edge without them),
then this writer's committed topology winners by logical key, tombstoning
those the local graph no longer holds. Cursors are stable ids/keys, so
inserts and deletes between items cannot shift them, and a restart just
begins a new pass; unchanged records publish nothing.
apply() still validates every record but never applies the writer's own
ones: the local graph is authoritative for them. An edge whose endpoint
is known deleted (tombstoned, or an own record the local graph no longer
holds) is dropped instead of deferred, and applies again if the endpoint
is re-published. Edge lookups during apply use the exact endpoint query
instead of a page-limited adjacency scan.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory-sync): publish documents before derived records in backfill
With the topology sweep now publishing every node and edge, a 3-node x
40-doc mesh took longer until every document was listed on every node
(603 s against 426 s before the topology fix). Debug logs showed why:
a publish takes most of the backfill's 100 ms per-cycle time budget, so
each cycle published about one item, and the fair rotation over
documents, vectors and topology handed two of every three cycles to
derived records while peers waited for the documents.
The backfill now serves domains in priority order: documents, then
vectors, then topology, each taking items until it has nothing left this
cycle. Vectors and topology defer on a peer until their document
arrives, so publishing the document first never adds a deferral. In the
same mesh, run concurrently with both other builds, every document is
listed on every node after 140 s (426 s before the topology fix, 603 s
with it alone), and deferrals still drain to 0.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
tests/integration/search/meson.build reassigned catch2_main_dep to Catch2's
own catch2-with-main. Meson variables are global, so every test executable
defined after that subdir (catch2_search_service among them) linked Catch2's
main instead of tests/support/catch2_main.cpp. The Windows hang stack showed
main coming from Catch2's catch_main.cpp, which is why the CRT report setup
in the shared main never ran there.
Use the shared catch2_main_dep for the hybrid search integration binaries
too.
Signed-off-by: trvon <me@trevon.dev>
supportedExtensions() returned its static set by value, so capabilities()
constructed the extension vector from begin() of one temporary and end()
of another. That range is undefined behavior. libstdc++ and libc++ release
iterators happened to walk the first copy to completion, but the MSVC debug
STL rejects it in _Verify_range with a modal "Microsoft Visual C++ Runtime
Library" assertion dialog. HandlerRegistry::selectBest calls capabilities()
for every candidate, so every lines: search and every applyExtractScope call
hung forever on Windows Debug builds.
Return the shared set by const reference and take begin()/end() from that
one instance. New text_basic_handler unit tests pin the shared instance,
the capability list, registry selection and line-range extraction; the old
code also fails them under -D_GLIBCXX_DEBUG with "function requires a valid
iterator range".
Signed-off-by: trvon <me@trevon.dev>
Three catch2_search_service cases hang on windows-latest with no output at
all, so the diagnose workflow could only name them. A silent hang on a
Debug MSVC build is often a modal CRT assert or abort dialog that nobody
can dismiss.
Before killing a hung case the workflow now prints its stdout/stderr tail
(run with -s -d yes), its top-level windows, any WER helper, and every
thread's stack plus !analyze -v -hang from cdb.exe. A test_filter input
limits the run to a Catch2 test spec for faster iterations.
The shared Catch2 main now routes CRT warnings, errors and assertions to
stderr and disables the Windows error-reporting dialogs, so a future
Debug assertion fails the case with its message instead of hanging.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
* fix(init): align fresh-install embedding dimensions and vector sentinel
A fresh `yams init` emitted dimension-conflict warnings and created a
vector store whose dimension did not match the configured embedding model:
Embedding policy: embedding dimension conflict:
embeddings.embedding_dim='1024' overrides compatibility key
vector_database.embedding_dim='384' (and vector_index.dimension='384')
Root causes:
- getLatestConfigDefaults() hardcoded vector_database.embedding_dim and
vector_index.dimension to 384 while [embeddings] used 1024 for the
default Simeon backend, so the generated config was self-inconsistent.
- init resolved the vector dimension only by scanning the optional ONNX
model list, leaving it at 0 for the default "simeon-default", and never
wrote vectors_sentinel.json.
- updateV2Config searched for [vector_database] keys (enabled/model/
model_path) that do not exist in the v3 schema and never updated the
dimension keys.
Fixes:
- Default vector_database.embedding_dim and vector_index.dimension to 1024.
- Add resolveModelDimension() and stamp vectors_sentinel.json from the
vector DB's effective dimension only when this init created the store,
so a forced re-init cannot overwrite a real stored dimension.
- Rewrite updateV2Config to set enable/embedding_dim for [vector_database],
enable/preferred_model/backend/embedding_dim for [embeddings], and
dimension for [vector_index] consistently; disable the enable flags when
semantic search is declined.
- Non-interactive init now defaults to semantic search enabled, matching
the interactive default.
- Keep compatibility dimension keys aligned during v1/v2 migration so an
existing 384-d install never receives a 1024-d sibling default.
* fix(init): prompt for the tuning profile once
`execute()` asked for the tuning profile before the idempotency check and
`handleAlreadyInitialized()` asked again for an already-initialized instance,
so `yams init` showed the tuning prompt twice. Collect the profile once on the
fresh path (after the idempotency check); the already-initialized path keeps
its single prompt.
Add interactive regression tests with a scripted stdin helper:
- already-initialized init prompts for tuning exactly once and persists the
chosen profile;
- declining semantic search disables both `[vector_database]` and
`[embeddings]`, writes no dimension sentinel, and still prompts for tuning
once (also covers the fresh interactive path).
---------
Co-authored-by: trvon <git@trevon.dev>
Full isolation still let get_runtime_dir() consume the ambient XDG_RUNTIME_DIR, so yams-daemon.status.json landed outside the temp root. Relocate XDG_RUNTIME_DIR (POSIX) and APPDATA/LOCALAPPDATA/USERPROFILE (Windows) under the run root, and scrub YAMS_EMBED_BACKEND, YAMS_PREFERRED_MODEL, YAMS_RERANKER_MODEL and YAMS_EMBED_DIM, which ConfigResolver ranks above TOML, so the generated config is deterministic. Apply the same to ingestion_e2e_bench and extend the isolation test with a decoy runtime dir and model overrides.
Signed-off-by: trvon <me@trevon.dev>
repair_ability_bench built its harness without any isolation, so the daemon
resolved config from ambient YAMS_CONFIG, ~/.config/yams and HOME and could
index real files. multi_client_ingestion_bench isolated state but not config,
HOME or the XDG cache/data dirs, and ingestion_e2e_bench left HOME untouched.
Use DaemonHarnessOptions::isolateEnvironment in repair_ability_bench (with an
explicit simeon embeddings config so the embed fault kind stays repairable)
and both multi_client_ingestion_bench option builders; an explicit
YAMS_BENCH_CONFIG still wins. In ingestion_e2e_bench set HOME and
XDG_DATA_HOME to the fixture root and unset YAMS_STORAGE and the daemon
socket aliases. Shift the affected portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
DaemonHarness only relocated state when isolateState/isolateConfig were set,
and even then left HOME, XDG_DATA_HOME, XDG_CACHE_HOME and ambient
YAMS_DAEMON_SOCKET_PATH/YAMS_STORAGE/YAMS_DATA_DIR visible to the daemon, so a
run could still resolve a developer's config, data and cache.
Add DaemonHarnessOptions::isolateEnvironment (and DaemonHarness::isolatedOptions())
which implies isolateState and isolateConfig, points HOME and every XDG base
directory at the run's temp root, and scrubs the ambient config/data/socket
variables for the daemon's lifetime, restoring them through ScopedEnvVar RAII.
The new Catch2 case exports a decoy config.toml (distinctive data_dir marker) and
decoy XDG/HOME/YAMS_* values, then asserts the daemon's config path, data dir and
every ambient path resolver stay under the temp root, the decoy is never read or
populated, and the ambient environment is restored after stop.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the pre-September upload code, which passed
--force, while the workflow still pins wrangler 4.69.0. That version's
'r2 object put' has no --force option (it overwrites without prompting)
and exits with 'Unknown argument: force', so the v0.20.3 release failed
its first R2 upload and no apt, yum or Arch repository was published.
Immutable objects are still checked with 'get' before any write.
Signed-off-by: trvon <me@trevon.dev>
WranglerObjectStore passes str(Path("/wrangler")) to the command, which
is \\wrangler on Windows, so the Windows lane failed the literal
"/wrangler" comparison.
Signed-off-by: trvon <me@trevon.dev>
CI runs the unit lanes with YAMS_SQLITE_VEC_SKIP_INIT=1, so the test's
VectorDatabase came up without vector tables and the post-VACUUM insert
failed on every platform. The case already clears the other vector
toggles; clear this one too.
Signed-off-by: trvon <me@trevon.dev>
When the daemon holds background work back for a busy host, a user
looking at a stale topology or an unfinished repair scan needs to see
why. Status had no host signal at all.
Carry the governor's host sample through the metrics snapshot and
StatusResponse: source (psi, loadavg), whether the host counts as busy,
the PSI cpu/io/memory "some avg10" values or the per-CPU load average,
and which deferrable work is currently held back. The proto transport
piggybacks the fields on request_counts like the other string status
fields; the native serializer appends them as optional tail fields.
`yams daemon status` (brief and detailed) shows a Host Load row, e.g.
"Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and
`yams status --json` gains a host_pressure object. Document the new
[tuning.resource] keys next to the other typed tuning sections.
Signed-off-by: trvon <me@trevon.dev>
requestTopologyRebuild() re-posted itself immediately whenever ingest,
post-ingest, embedding or KG work was still in flight. It released its
in-progress flag before re-posting, so every request that arrived while
the daemon was busy (one per embedding batch) could win the flag and
start its own re-post loop. Those loops never slept: they occupied the
worker pool until the ingest drained, and a slower drain on a loaded
host meant more of them for longer.
Per-thread sampling during a 1899-file ingest on a 2x oversubscribed
48-core host showed ~36 worker threads at 100% for the whole drain. For
the same ingest (120s window, background work only):
daemon CPU seconds before after
idle host 41, 41 23, 20, 19
loaded host 96, 95 28, 30, 30 (deferral off)
Arm a single timer instead: a busy daemon re-checks after 500ms, a
rebuild deferred for host pressure after 5s, and only one retry is
armed at a time. post_ingest_drain still requests a rebuild directly
when the pipeline empties, so topology freshness does not wait on the
timer in the common case (idle-host topology completion is unchanged at
~7s after ingest).
Add a ServiceManager test for the deferral path: with the host marked
busy the rebuild is held and the governor reports it deferred; once the
host calms down nothing is held back and, when a vector database is
available, the armed retry builds the artifacts without a new request.
Signed-off-by: trvon <me@trevon.dev>
Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
The governor only measured the daemon itself (RSS, own CPU), so it could
not tell that the machine was busy. On a loaded host the daemon still
started background work at full width, and that work ran far less
efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU
seconds on an idle 48-core host and 96 CPU seconds with the host 2x
oversubscribed (stress-ng), and took 4x longer to finish.
Sample a cheap host-wide signal on the existing governor tick, at most
once per second: Linux PSI "some avg10" for cpu, io and memory, falling
back to the one-minute load average per online CPU; on macOS the load
average plus the kernel VM pressure level. A busy host is a flag with a
release band (signals must drop to 75% of the threshold to clear) so
work does not flap around a threshold.
Host load stays a separate dimension from the governor's pressure
level. Folding it into the level would reuse the Warning response,
which blocks model loads and trims foreground concurrency; a busy build
on the same machine should not stop a search from loading its model.
Background scheduling consumes the flag in a follow-up change.
Thresholds are typed config under [tuning.resource]: host_pressure,
host_cpu_pressure_pct (40), host_io_pressure_pct (30),
host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in
the runtime tuning status with their provenance.
Signed-off-by: trvon <me@trevon.dev>
With -Denable-mobile-bindings=true, libyams_mobile no longer compiled:
- diff/cat called IDocumentService::retrieveDocument, which was replaced
by retrieve(); they now fetch stored bytes through retrieve() with
acceptCompressed left false, so content comes back uncompressed, and
map service errors through map_error_code instead of NOT_FOUND.
- repair awaited DaemonClient::call() as a future and read response
fields that no longer exist; it now goes through daemon_call() and
folds RepairResponse::operationResults into the C result.
- YAMS_PROFILE_ZONE is not a defined macro; use YAMS_ZONE_SCOPED_N.
- doctor built its version with std::string(char); emit JSON instead.
- yams_mobile_string_view_destroy, _cat_result_destroy and
_restore_result_destroy were declared but never defined.
The repair/diff/cat/restore/model/doctor declarations sat outside the
extern "C" block and inside #ifdef __cplusplus, so C and FFI hosts could
not see them and C++ builds exported mangled names. Move them into the
C block.
Daemon-mode contexts handed their own worker_pool executor to the
process-wide connection pool keyed by socket path. The pool outlived
yams_mobile_context_destroy, so a second context reused sockets bound to
a freed thread pool (ASAN heap-use-after-free in AsioConnection) and the
daemon was left with connections that never drained, which then tripped
a use-after-free in daemon shutdown. Contexts now use single-use
connections and disconnect the client before joining the worker pool.
Tests: add an embedded cat/diff test (exact bytes, line diff, identical
docs, not-found mapping) and a daemon test that creates and destroys two
contexts and checks the server returns to its baseline connection count.
This proves the Linux x86_64 host build and host-side behavior only. It
does not show that an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
v0.20 stopped producing code-symbol nodes and calls/references/
inherits/implements edges, and migration 40 deleted the existing ones.
GraphContextService::impact() and affectedTests() had already been
reduced to returning a "code-symbol graph was removed" warning, so
`yams graph --impact`, `--affected-tests` and `--test-pattern` (and the
MCP graph actions `impact` and `affected_tests`) could only ever print
that warning.
Remove the dead surface:
- CLI: the three flags, their execute/render helpers, and the --impact
mention in the --global help text.
- MCP: the two actions, their schema entries and the changed_files /
test_pattern request fields. Unknown graph actions now fail with
InvalidArgument instead of silently running a default query.
- Service: impact()/affectedTests() and their request/response structs.
The GraphImpact/GraphAffectedTests IPC messages stay for wire
compatibility. The dispatcher now answers them directly with the same
removal warning, without touching the metadata repository.
Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph
node labels and aliases and walk edges of any relation, so they still
work on document, path, version and entity graphs.
Tests: the service-level impact/affected-tests case moves to the
dispatcher graph suite; the sandbox smoke test now expects --impact to
be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and
blast-radius questions at `yams grep`.
Signed-off-by: trvon <me@trevon.dev>
Mode::None is the initial value of mode_ and means "no subcommand
parsed". execute() only reached it through a catch-all default, which
also hid any future Mode enumerator that someone forgot to dispatch.
Give None its own case (same InvalidArgument error as before) and drop
the default so -Wswitch flags an unhandled mode at compile time. The
enumerator stays: removing it would make an unparsed command silently
default to Init.
Add a CLI test that `yams session` with no subcommand is rejected; the
None path stays a defensive guard because require_subcommand() stops
that case at parse time.
Signed-off-by: trvon <me@trevon.dev>
The ConnectionFsm state classes produce transitions through Next, and
to_public() maps every ImplState back to the public enum. No state class
ever returned ImplState::ReadingHeader, so that mapping was unreachable:
on_readable() bypassed the state classes and called transition() with a
hard-coded switch instead.
ReadingHeader itself is live (the request handler drives on_readable()
and checks for the state), so wire it rather than remove it: add an
on_readable() hook to StateBase, have ConnectedState return
ReadingHeader, and let ConnectionFsm::on_readable() dispatch through the
current state like every other event.
ConnectedState::on_header_parsed() returned ReadingPayload or
WritingHeader, neither of which the legality matrix allows from
Connected, so every such transition was rejected. Drop the override; the
observable behavior (header events are ignored until on_readable()) is
unchanged.
Pin the transition table with a Catch2 case covering on_readable in each
reachable state, header-only and payload frames, streaming keep-alive,
close, error, and accept, and give the previously assertion-free
"Body parsing" section real expectations.
Signed-off-by: trvon <me@trevon.dev>
ClientConfig::acceptCompressed was written by RetrievalService's config
mapping and compared in the CLI client-pool equality operator, but no
DaemonClient or transport code ever read it, so it could not change
behavior. Compression is negotiated per request through
GetRequest::acceptCompressed, which RetrievalOptions::acceptCompressed
still feeds via makeGetRequest.
Drop the field, the mapping, the equality term (configs that differed
only in this flag now share a pooled client, which is what their
behavior already was), and the mapping-test assertion. The per-request
flag and its tests are unchanged.
Signed-off-by: trvon <me@trevon.dev>
YamsDaemon::stop() stopped the SocketServer and destroyed it while the
IOCoordinator threads were still running, and only then stopped and
destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for
connections to drain, and the handle_connection coroutine frames keep a
raw pointer to the server in their cleanup guard. A connection that
outlived the drain window either resumed on an I/O thread after the
server was freed, or had its frame destroyed together with the
io_context, and both paths decremented the connection counters of the
freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The
ServiceManager-init failure path stopped the server the same way and
left the I/O threads running.
Tear the IPC layer down in one place, teardownSocketServer(): stop the
server (which still releases its acceptors and accept strand while the
io_context runs), stop and join the I/O threads so no handler can
resume, destroy the IOCoordinator so the io_context destroys the pending
handler frames while the server is valid, and only then destroy the
server. All daemon stop and startup-failure paths use it.
The unit test stops the io_context under the connection handlers so
they are deterministically pending when the teardown runs; the daemon
integration test parks the I/O threads past the drain window.
Signed-off-by: trvon <me@trevon.dev>
#216 and #220 each shifted the two reviewed getenv reads in
RequestDispatcher.cpp and each updated the allowlist on its own; merged
together the entries point at the old lines and the portability check fails.
Signed-off-by: trvon <me@trevon.dev>
