* test(vector): pin vec0 ranking to cosine order on non-unit rows
vec0 ranks by L2 distance, which matches cosine order only for unit
vectors. Insert rows whose L2 order disagrees with cosine order and
assert vec0 returns cosine order, agrees with the exact cosine scan,
skips zero rows and rejects a zero query. Pins a vector-enabled env
so the cases run under the CI test env.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): rank vec0 by cosine on unit-normalized copies
vec0 ranks by L2 distance, which matches cosine order only when both
sides are unit length. Stored rows reach the vectors table at whatever
magnitude the provider, config, import or sync produced, and the vec0
aux table copied them as-is, so vec0 selected and ordered rows
differently from the Simeon PQ and exact cosine paths.
The vec0 table is a derived index rebuilt from vectors.embedding, so
write the unit direction into it and leave the source blob, which
export/sync and exact scoring read, untouched. Normalize the query the
same way. Zero rows are left out of vec0 and never match; a zero or
non-finite query is rejected as the exact scan rejects it. Returned
rows are ordered by their cosine score with the PQ tie-break.
No schema change. The aux table is rebuilt on first use in each
process (ready dims are in-memory), so existing non-unit rows are
normalized on the next rebuild without a repair step.
Signed-off-by: trvon <me@trevon.dev>
* test(topology): pin the k-means stored centroid to the unit direction
Spherical k-means assigns against unit centroids but stored the shorter
arithmetic mean. Assert the stored centroid is unit length along the
member mean direction, with and without boundary spill. Add a SOAR case
showing that with unit centroids the spill follows angular distance,
and that a short mean centroid reorders the candidates.
Reformat the touched k-means test file to the lint formatter.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): store the unit k-means centroid
The k-means engine clusters by cosine against normalized centroids,
then stored the unnormalized member mean as centroidEmbedding. Routing
reads the centroid by cosine and does not care, but SOAR boundary spill
measures Euclidean residuals against it. A loose cluster's short mean
adds a radial term to those residuals, so spill candidates were ranked
in a different space from the one the clusters were built and routed
in.
Store the unit mean direction for k-means, which is the converged
Lloyd centroid. Connected-component and Louvain keep the arithmetic
mean, as their tests and contract require. Correct the stale field
comment.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(wal): pin v1 entry encoding to a Clang-written fixture
store_block_entry_clang_v1.bin is one StoreBlock entry serialized by the
current wal_entry.cpp built with Clang 21, which zero-fills the 4 bytes of
trailing Header padding. The new cases decode and verify it, re-encode the
same logical entry and require identical bytes, and round-trip fresh
entries through serialize/deserialize/verifyChecksum.
GCC 16 leaves that padding uninitialized in stableHeaderBytes(), so these
cases fail there.
Signed-off-by: trvon <me@trevon.dev>
* fix(wal): encode entry header field by field, never via padding
stableHeaderBytes() built the checksummed header with `Header stable{}`
plus member assignments and then memcpy'd sizeof(Header) bytes. That is
aggregate initialization, so the 4 bytes of trailing padding (offsets
44..47) are not guaranteed to be zero. Clang 21 memsets the object first;
GCC 16 at -O0 stores only the members, so stack garbage reached both the
serialized entry and the CRC, and verifyChecksum() failed on GCC builds.
Encode and decode the header field by field into a zero-filled 48-byte
array at the existing offsets, with static_asserts pinning every offset
and size. The checksum is CRC32 over that canonical encoding plus the
payload, which is exactly what Clang builds have always written, so their
WAL files verify unchanged. deserialize() also accepts an entry whose CRC
covers nonzero padding as stored on disk (written by earlier GCC builds)
and normalizes its in-memory checksum. WALFile and WALManager recovery
read headers through the new WALEntry::decodeHeader().
TransactionData (uint64 + uint32) has 4 bytes of trailing padding and was
copied whole into the payload, relying on the same unguaranteed zeroing
(GCC 16 and Clang 21 happen to zero it today); it is now encoded field by
field too. The remaining payload structs are padding-free, which
appendObjectBytes/readObject now enforce with
has_unique_object_representations_v.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
7e744805 (#241) registered daemon_harness_full_isolation with get_option('integration-daemon-timeout'). ca944de5 removed that option, so every Tests lane on experimental fails at meson setup: "Option integration-daemon-timeout does not exist". Use daemon_integration_timeout, as the other daemon_harness tests do. Configured on grim with the default, asan, ubsan, mobile and macOS option sets from tests.yml.
Signed-off-by: trvon <me@trevon.dev>
Wait for caller startup before measuring completion and keep synchronous
console I/O out of the checkpoint wait test. Preserve the 20 ms timeout
and 250 ms assertion, and verify the queued/timeout telemetry.
Keep a guarded rescue executor alive for late posts so the unbounded-wait
negative control fails without leaving an unjoinable caller.
Signed-off-by: trvon <git@trevon.dev>
* docs: MSI ships the daemon; p2p sync shipped in 0.20.0
install.md said the MSI lacks yams-daemon.exe and yams-mcp-server.exe.
Both ship from the next published release (packaging/windows/yams.wxs).
0.20.4 was not published, so name 0.20.3 as the last MSI without them.
roadmap.md listed p2p sync as unshipped. Whole-corpus replication
shipped in 0.20.0. List what is still open.
Signed-off-by: trvon <me@trevon.dev>
* docs: give the ONNX Runtime search order
install.md said only that a system copy is preferred over the bundled
one. State the order from plugins/ort_runtime/ort_runtime_resolution.h:
runtime_library pin, system copy >= 1.23, bundled lib/yams/onnxruntime/,
with YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR overriding all three.
Quote the real log line.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): cover vectors committed behind the backfill cursor
The vector backfill cursor (document_hash, chunk_id) never wraps, and
the backfill is the only outbound path for vectors, so a vector added
or re-embedded behind the cursor after the first sweep is never
published to peers.
Add red tests for:
- the sweep starting over to publish a new and a re-embedded vector
that sort below its last position;
- vectors an embedding job commits being published within the next
backfill cycle, ahead of a sweep that would need many items to reach
them;
- the topology sweep still publishing its share while the vector sweep
always has work.
Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op
seam for the embedding commit notification; the fix fills it in.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish committed vectors and wrap the vector sweep
The vector backfill cursor (document_hash, chunk_id) was set once and
reset only by a daemon restart, and the backfill is the only outbound
path for vectors. After the first full sweep, a vector added or
re-embedded under a document hash that sorts below the cursor was never
published to peers. Topology had the same bug and now sweeps in
repeating passes; vectors did not.
Eager publish: the embedding service already reports the documents a
job committed vectors for (successHashes, after the vector insert and
before readiness is published) through the callback that requested the
topology rebuild. That callback is renamed setEmbeddingsCommittedCallback
and the daemon now also passes the hashes to
ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues
them (bounded at 4096 documents, deduplicated, dropped past the cap):
it runs on the embedding worker, and the memory-sync service's lifetime
and outbound budget belong to the sync cycle. The backfill serves the
queue as its own domain right after documents, one vector row per item,
walking each document's chunks with getVectorsPage. The queue is in
memory; a restart loses it, but the sweep then starts over anyway.
Catch-up sweep: when the vector sweep reaches the end of the table it
resets its cursor and sits out the rest of the cycle, as the topology
pass does, so the next pass publishes whatever changed behind it.
publishIfChanged keeps repeat passes from writing unchanged records.
R7's skip logic is kept: an unpublishable row is skipped and the
cursor moves past it; it is logged on the first pass (and when
committed) but not again on every later pass.
Fairness: with both sweeps repeating, both always have work, so the
strict priority order (documents, vectors, topology) would hand the
vector sweep every item and starve topology. Documents and committed
vectors keep priority; the two sweeps now alternate item by item, and
the turn persists across cycles so a one-item budget still alternates.
A sweep that is done for the cycle leaves its turns to the other. This
was chosen over alternating whole passes because it needs one field of
state, keeps both sweeps' latency bounded by their own size rather than
the other's, and reuses the existing per-item budget loop unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): pin the vector-store environment in the cursor-wrap cases
Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): cover vector records that cannot apply or publish
One bad vector record should cost only itself. Add red tests for the
vector sync adapter (corrupt payload, dimension mismatches, non-hash
document identity, key/identity mismatch sorted between two good
records), for the daemon apply cycle reporting such a record in apply
health while the good embedding lands, and for the outbound backfill
publishing good vectors past a row with no model identity.
Add a YAMS_TESTING hook to inject a vector database into the
ServiceManager so the daemon vector stage and backfill can be driven
in-process.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): skip bad vector records instead of stalling the stage
Inbound: VectorSyncAdapter::apply() returned an error on the first
winner it could not apply (corrupt payload, dimension or identity
mismatch, non-hash document id, a model the store cannot hold for its
chunk, a failed store write or content probe). It validated the whole
batch before any write, so one such winner kept every vector winner
from applying on every cycle until a newer winner replaced it.
Each winner is now applied, deferred, or skipped on its own, as the
content stage does. A skipped winner is reported through failure()
(first failure, prefixed with its logical key), which the daemon maps
to ApplyStagePass::failure, so apply health counts it as a vector stage
failure while good records land. Skipped winners are never marked
applied and are retried on the next pass. Winners that disagree on a
chunk's model are all skipped, so the outcome does not depend on
arrival order. Reconciliation and index rebuild failures still end the
pass.
Outbound: the backfill threw on any vector publish error without
advancing its (document_hash, chunk_id) cursor, so a row that can
never publish (for example one with no model id) stopped the vector
sweep on that row for good. Such rows are now skipped with a warning,
as unrecoverable documents are; transient failures still propagate
and are retried next cycle.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): pin the vector-store environment in the vector cases
CI runs the unit lanes with YAMS_DISABLE_VECTORS=1, YAMS_SQLITE_VEC_SKIP_INIT=1 and YAMS_VDB_IN_MEMORY=1. VectorDatabase then skips creating its tables, so every vector insert in these cases failed under the CI environment while passing in a clean shell. Clear those variables for each case that attaches a vector store, as the init command tests do.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
The system-service entries were superseded by the per-user unit in the same stack, one ci fix was listed twice, and the promotion merge is not a change.
Signed-off-by: trvon <me@trevon.dev>
Switching the main test user's embedding backend to onnxruntime (no
model installed) and back left state behind that the later phases then
tripped over. Probe the runtime with a third user whose own config
enables plugin autoload with the auto backend (config_version = 3 so
the daemon does not migrate it to the simeon default), and keep the main
user on yams init's configuration throughout.
Signed-off-by: trvon <me@trevon.dev>
A CompressedStorageEngine wrapping an engine without a base path used a
fixed /tmp/yams-compressed-wrapper as its own base, and the StorageEngine
base creates objects/ and temp/ there. With the daemon now running as a
systemd user unit for each user (UMask=0077), whichever user's daemon
started first owned that directory, and every other user's daemon with
compression enabled (the yams init default) failed at startup with
"Failed to create storage directory: Permission denied" and never became
ready. Found by the package validation: two lingering users, reboot, the
second daemon stuck initializing.
Suffix the scratch directory with the real UID on POSIX.
Signed-off-by: trvon <me@trevon.dev>
First runs of the user-unit validation on all four lanes found harness
gaps, fixed here:
- Arch's `su -` does not run pam_systemd, so the login shell had no
XDG_RUNTIME_DIR and the CLI looked for /tmp/yams-daemon-<uid>.sock;
the checks now provide the runtime dir a real login session gets and
report whether su set it.
- Ubuntu 24.04's systemctl --user needs the user D-Bus bus: the
Debian/Ubuntu lane image gains dbus-user-session (standard on real
installs) and the checks pass the bus address when present.
- CLI commands fall back to an in-process daemon instead of spawning
one, so the lifecycle check uses `yams daemon start` (which starts the
unit) rather than expecting an ordinary command to start it.
- `yams init`'s default embedder is the built-in simeon backend, which
skips the onnx model provider by design. The onnx phase now checks
that init's config autoloads plugins, then switches [embeddings]
backend to onnxruntime to prove the bundled runtime loads (from the
user journal), and restores init's config so the later phases run the
user's real configuration.
docs/install.md no longer claims ordinary CLI commands start the unit:
`yams daemon start` and on-demand starters do.
Signed-off-by: trvon <me@trevon.dev>
The package validation now exercises the per-user service:
- service: the user unit and user preset are installed, the unit is
enabled for all users (systemctl --global), no system unit remains,
and the unit does not hide the user's home or /tmp.
- user: a normal user with lingering gets a user manager; the unit is
enabled and active without a manual start and `systemd-analyze --user
verify` is clean; the socket is in $XDG_RUNTIME_DIR and owned by the
user; through a login shell with no YAMS_* variables the CLI reports
status, adds from $HOME and /tmp, searches, stores in
~/.local/share/yams and logs to ~/.local/state/yams, and the only
daemon is the unit's MainPID; `yams daemon stop`, CLI auto-start and
`yams daemon restart` go through systemd; `yams daemon install --user`
enables the packaged unit without writing an override; a second user
cannot reach the first one's daemon.
- onnx: after `yams init` the restarted user daemon logs "Using bundled
ONNX Runtime" and `yams plugin health` lists the onnx plugin.
- persist: with lingering the user's daemon is back after a reboot with
the data intact.
- upgrade from 0.20.3: the old system unit is stopped, removed and
leaves no enablement links, /var/lib/yams is left alone, the user unit
is enabled for all users and the user's daemon and CLI work.
- remove: running user daemons are stopped, global enablement is gone,
no daemon process is left, user corpora survive (deb: also purge).
- minimal: bare base images still install with declared dependencies
only; the account checks are gone with the yams account.
The Fedora lane image replaces the systemd-user PAM stack, which cannot
run in a container (unix_chkpwd, pam_selinux, pam_loginuid), with a
minimal one that still runs pam_systemd, so user@.service starts.
Signed-off-by: trvon <me@trevon.dev>
The package scripts only addressed user managers that had a D-Bus user
bus socket (/run/user/<uid>/bus). Debian and Ubuntu without
dbus-user-session have none, so upgrades did not reload or restart running
user daemons and removal left them running on deleted binaries. Detect a
manager by its private socket (/run/user/<uid>/systemd/private), which
systemctl --user uses directly, and pass only XDG_RUNTIME_DIR.
Signed-off-by: trvon <me@trevon.dev>
The generated config.toml (yams init) set daemon.socket_path =
"/tmp/yams-daemon.sock" and pid_file = "/tmp/yams-daemon.pid". Both are
the same for every user on a host, so with the packaged systemd user unit
two users' daemons fought for one socket, and an initialized user's
daemon left $XDG_RUNTIME_DIR for a world-visible /tmp path. Write them
empty so the CLI and the daemon fall back to the per-user defaults
($XDG_RUNTIME_DIR/yams-daemon.sock, /tmp/yams-daemon-<uid>.sock without a
runtime dir). Configs written before keep their explicit values.
Signed-off-by: trvon <me@trevon.dev>
With the daemon shipped as a systemd user unit, the CLI must not fight
systemd for it:
- Auto-start (DaemonClient::startDaemon): when the unit is installed but
not running and would run exactly the daemon the client expects (same
socket, data dir and config as the default resolution, default
binary), start it with `systemctl --user start` instead of forking a
copy that a later login would race for the socket. Flag or
environment overrides (--socket, --data-dir, YAMS_DAEMON_SOCKET,
YAMS_DATA_DIR, YAMS_CONFIG, YAMS_DAEMON_BIN) keep the fork path.
- `yams daemon start|stop|restart` use `systemctl --user` for an
installed/active unit (start/restart without custom path options,
stop/restart without --socket), falling back to IPC if systemctl
fails.
- `yams daemon install --user` with no options enables the packaged unit
(`systemctl --user enable --now`) instead of writing a copy into
~/.config/systemd/user that would shadow it; with options it still
writes that override and says so. `uninstall --user` reloads the
manager after removing an override and explains how to mask the
globally enabled packaged unit for one account.
The decision is a pure function (userUnitMatchesPaths) with Catch2
coverage; systemctl is only run with whitelisted verbs and only when a
user manager is reachable ($XDG_RUNTIME_DIR set, Linux).
Signed-off-by: trvon <me@trevon.dev>
The service is now user-bound: each user runs their own daemon with
their own config ($XDG_CONFIG_HOME/yams/config.toml), corpus
($XDG_DATA_HOME/yams) and socket ($XDG_RUNTIME_DIR/yams-daemon.sock),
which is exactly what the CLI already resolves. The system service, its
yams account/group, the group socket and the /etc/yams/config.toml
configuration go away.
- Unit: /usr/lib/systemd/user/yams-daemon.service runs
`yams-daemon --foreground` with default path resolution,
Restart=on-failure, WantedBy=default.target, and light hardening
(NoNewPrivileges, LockPersonality, RestrictRealtime, RestrictSUIDSGID,
native syscalls only). No ProtectHome/PrivateTmp: the daemon reads the
files its user adds.
- Enablement: /usr/lib/systemd/user-preset/80-yams.preset plus
`systemctl --global preset` on first install (and on upgrade from a
release with the system unit), so the unit starts with every user's
next session; administrators override with a user preset or
`systemctl --global disable`, and later upgrades keep that choice.
Package scripts never start per-user instances: they reload running
user managers, try-restart running user daemons on upgrade and stop
them on removal (`systemctl --user` via runuser per /run/user/<uid>),
then `systemctl --global disable` on removal.
- Migration from 0.20.x and the system-service design: deb preinst, rpm
%pre and Arch pre_upgrade stop and disable the old system unit while
its file is still installed (deb also purges deb-systemd-helper
state). /var/lib/yams, the old system corpus that the CLI never
reached, is left alone and documented; user corpora are never
touched.
- The sysusers entry, account creation and the deb's passwd and
init-system-helpers dependencies are dropped; the deb purge no longer
deletes /var/lib/yams.
- Install paths move to lib/systemd/user and lib/systemd/user-preset in
meson, the Arch stage/PKGBUILD and the install-surface policy.
- docs/install.md: the user service, enabling it now
(`systemctl --user enable --now`), lingering, plugins via the user's
config, opting out, upgrade and removal behaviour, and the 0.20.3
leftover.
Signed-off-by: trvon <me@trevon.dev>
The packaged service becomes a systemd user unit that runs each user's
own daemon on the CLI's default per-user socket, with that user's
config and data. There is no system daemon left to discover, and the
daemon always runs as the same user as the CLI, so both workarounds for
the system-service design go:
- Revert "feat(cli): find the packaged system daemon socket without
configuration" (client-side /run/yams/yams-daemon.sock discovery, the
systemd refusal in `daemon stop|restart`, the spawn refusal and the
status mismatch suppression, plus its tests).
- Revert "fix(cli): send file content to a daemon that runs as another
user" (peer-UID probe and content upload in `yams add`).
Per-user socket resolution is back to: explicit flag/env/config, then
$XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback).
Signed-off-by: trvon <me@trevon.dev>
The packaged service had no configuration file it would read: with
HOME=/var/lib/yams it looked for /var/lib/yams/.config/yams/config.toml,
which nothing creates. Plugins load only with [daemon]
auto_load_plugins = true, so the onnx and glint plugins and the bundled
ONNX Runtime shipped in the packages were never used by the service
("No plugins loaded"), and an administrator had no obvious place to turn
them on.
The unit now sets YAMS_CONFIG=/etc/yams/config.toml (the package ships
no file; ProtectSystem=strict leaves /etc readable). With
auto_load_plugins = true there, the sandboxed daemon loads all three
plugins and logs "Using bundled ONNX Runtime 1.23.0 from
/usr/lib/yams/onnxruntime/libonnxruntime.so.1 (no system copy found)",
ready=true. docs/install.md describes the file and how to check which
runtime was loaded; the validation's onnx phase writes it, restarts the
unit and checks the bundled runtime and plugin health.
Signed-off-by: trvon <me@trevon.dev>
`meson install` of a release build with --with-onnx failed:
ERROR: Tried to install symlink to missing file
.../usr/lib/yams/onnxruntime/libonnxruntime.so.1
Meson installs symlinks in declaration order and refuses one whose target
is not installed yet. list-onnxruntime-libs.py printed them in name order,
so libonnxruntime.so -> .so.1 came before .so.1 -> .so.1.23.0. Emit each
chain from the real file outwards (by link depth). Found by building the
packages the release way (setup.sh Release --with-onnx, meson install).
Signed-off-by: trvon <me@trevon.dev>
The lane images boot systemd and carry the tools the checks use, so
systemd-sysusers always existed and the deb postinst's broken `adduser`
fallback was never exercised.
Each lane now also installs the package in the raw distro image
(debian:trixie-slim, ubuntu:24.04, fedora:42, the Arch base) with no
systemd running and nothing preinstalled, so only declared dependencies
are present. It checks that the install finishes (deb: dpkg status
"install ok installed", not half-configured), that the yams account and
group exist, and that `yams add` + `yams search` work without the
service.
A new onnx phase checks, on the systemd lane, that the package ships the
private runtime in <libdir>/yams/onnxruntime and the onnx plugin, that
the system daemon logs "Using bundled ONNX Runtime", and that
`yams plugin health` lists the onnx plugin.
Signed-off-by: trvon <me@trevon.dev>
The deb postinst created the account with systemd-sysusers and fell back
to `adduser`, which the package never depended on. In a minimal Debian or
Ubuntu container (no systemd, no adduser) installation stopped in
postinst with "adduser: not found" and left the package half-configured.
The postinst still prefers the shipped sysusers.d entry, then falls back
to groupadd/useradd from `passwd`, now a declared dependency (Priority:
required, so it costs nothing). If neither works it prints what to do,
skips enabling the service and exits 0: the CLI does not need the account,
so a missing account no longer breaks the install. The rpm %post gets the
same groupadd/useradd fallback when systemd-sysusers cannot run; Arch
already relies on pacman's sysusers hook plus an explicit
systemd-sysusers call.
Signed-off-by: trvon <me@trevon.dev>
The resolution tests used ELF names (libonnxruntime.so.1) everywhere,
but the policy only treats .dylib files as runtime libraries on macOS,
so the macOS test lane would have failed. Fixtures now use the host's
names (libonnxruntime.1.dylib / libonnxruntime.1.23.0.dylib on macOS)
and build paths with fs::path; the test is not registered on Windows,
where resolution is the DLL search for onnxruntime.dll.
grim (Linux, Debug): 7/7 cases, 34 assertions pass.
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin cached compiled CoreML models directly in the model
directory. With the plugin now able to run on either the system or the
bundled ONNX Runtime, a cache compiled by one version was reused by the
other: on macOS, after a run on Homebrew's 1.30 the bundled 1.23 failed
every embedding batch with 'Non-zero status code returned while running
..._CoreML_... node'. The default cache is now
<model dir>/coreml-ort-<runtime version>; YAMS_COREML_CACHE_DIR still
overrides it.
Verified on macOS arm64: a run on the system runtime creates
coreml-ort-1.30.0, a run with Homebrew's copy hidden uses the bundled
runtime, creates coreml-ort-1.23.0 and embeds new documents without
CoreML errors.
Signed-off-by: trvon <me@trevon.dev>
On macOS meson.build puts the Homebrew onnxruntime prefix first when
normalizing ONNX Runtime linkage, so the private bundle copied
/opt/homebrew/opt/onnxruntime/lib even in a Conan build. That dylib links
other Homebrew libraries (abseil, protobuf, re2) by absolute path, so the
bundled copy would not load on a Mac without them. Bundle only from the
dependency's own libdir (pkg-config/CMake), skipping Homebrew prefixes;
the Homebrew formula uses Homebrew's onnxruntime directly.
Verified on macOS arm64: the bundle source changes from
/opt/homebrew/opt/onnxruntime/lib to the Conan package libdir.
Signed-off-by: trvon <me@trevon.dev>
rpm's dependency generator advertised every ELF library in the package
as a system soname, including the bundled ONNX Runtime
(libonnxruntime.so.1()(64bit)), so the yams RPM could satisfy other
packages' ONNX Runtime dependencies, and it emitted requires on the
private libyams_onnx_resource/libzpdf it ships itself. Filter provides
from /usr/lib/yams and drop requires on those private sonames.
Signed-off-by: trvon <me@trevon.dev>
Packages now carry the private fallback ONNX Runtime in
/usr/lib/yams/onnxruntime/ (libonnxruntime.so -> .so.1 -> .so.1.x.y and
libonnxruntime_providers_shared.so). Without an allowlist entry the
layout check rejects them as unreviewed paths. The pattern admits only
those library names under that directory.
Signed-off-by: trvon <me@trevon.dev>
No package carried ONNX Runtime: tools/yams-cli tried to install_subdir()
the runtime's libdir, but read it with get_variable('libdir') from the
dependency meson.build had already normalized into a declare_dependency,
which has no variables, so it installed nothing. Packaged ONNX and Glint
plugins therefore only worked where the user had installed a compatible
runtime themselves.
meson.build now records the directories it resolved ONNX Runtime from,
and the runtime install bundles the shared library from the first that
holds one into <libdir>/yams/onnxruntime/ (install tag runtime). A small
configure-time helper (scripts/list-onnxruntime-libs.py) lists the real
files and the soname symlinks, so the chain libonnxruntime.so ->
.so.1 -> .so.1.23.0 is recreated instead of tripled, and only the core
library and providers_shared are taken (no static archives, CMake or
pkg-config files). The plugins use this copy only when no compatible
system copy exists. Windows is unchanged: onnxruntime.dll already ships
next to the executables in the MSI and is found by the loader search.
- scripts/split-debug-symbols.sh leaves the bundled runtime
byte-identical: it is an upstream release build, already stripped, and
re-stripping would void a macOS code signature.
- install_surface_policy: <libdir>/yams/onnxruntime/* is allowed and
exempt from first-party export/hardening checks; the Glint size budget
grows to 850,000 bytes (711,352 measured with real ONNX Runtime code).
- Homebrew formulas depend on Homebrew's onnxruntime, which the plugins
prefer, so they drop the archive's private copy (rm_rf
lib/yams/onnxruntime).
- Arch: optdepends now reads 'onnxruntime: use the system ONNX Runtime
instead of the bundled copy' (PKGBUILD and the CI .PKGINFO).
Measured on grim (Linux x86_64 release, --with-onnx, stripped staged
payload): 87,883,247 -> 110,236,706 bytes (+22.4 MB: the 22,207,288-byte
libonnxruntime.so.1.23.0, 14,632-byte providers_shared, and +111 KB of
real GLiNER code in yams_glint.so).
Signed-off-by: trvon <me@trevon.dev>
On Linux the Glint plugin always built in mock mode: meson found the ONNX
Runtime headers but its find_library('onnxruntime') looked in a libdir
taken from the already-normalized dependency, found nothing, and dropped
YAMS_USE_ONNX_RUNTIME, so GLiNER never ran (the v0.20.3 deb's glint has
no ONNX Runtime dependency either). On macOS it linked Homebrew's copy
directly.
Glint now uses the shared ONNX Runtime loader like the ONNX plugin: it
compiles against the headers only (ORT_API_MANUAL_INIT through
ort_cxx_api_wrapper.h), and before creating the session it resolves a
runtime with the same system-first, bundled-fallback policy. If none is
available the session reports "ONNX Runtime unavailable: ..." and entity
extraction is disabled instead of silently running the mock. Plugin
health gains an onnx_runtime object (available, source, version, path or
error), and [plugins.glint] runtime_library pins a copy.
Verified on grim (Linux, release, staged tree in a clean debian:testing
container): "Glint plugin: ONNX Runtime enabled (loaded at runtime)" at
configure, "[Glint] Loaded ONNX model with 6 inputs, 5 outputs" at run
time, and the daemon's title+NL stage extracts entities ("nl=5" for a
document naming people, companies, a place and a date).
Signed-off-by: trvon <me@trevon.dev>
