The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
On the Arch package build the executables link the system libmagic, whose
dependencies (libzstd, libz) are also linked statically from Conan. ld then
exported the private copies so libmagic would bind to them: 600+ dynamic
symbols per binary (ZSTD_*, deflate*, inflate*), interposing one library
copy into another, and scripts/ci/check_install_surface.py failed the
Arch stage (export budget 64, forbidden third-party exports). The Debian
build, without that system dependency, exported 43.
Link optimized Linux builds with --exclude-libs,ALL so symbols from static
archives never enter .dynsym. YAMS executables export nothing plugins rely
on (plugins resolve against their own dependencies and the C ABI), so this
only removes the leak.
Signed-off-by: trvon <me@trevon.dev>
The install-surface cleanup moved libyams_onnx_resource from /usr/lib to
/usr/lib/yams, matching scripts/ci/install_surface_policy.json. The
package validation allowlist only accepted libyams_* directly in the
library directory and would have failed every lane; accept both, and
nested paths under /usr/share/doc/yams like the policy does.
Signed-off-by: trvon <me@trevon.dev>
The systemd fix added lib/sysusers.d/yams.conf without an install tag, so
'meson install --tags runtime' dropped it, and the install-surface policy
did not allow the path, so the release gate would have failed on it.
Signed-off-by: trvon <me@trevon.dev>
rpm expands macros inside comments. The generated spec's comment named
%systemd_postun_with_restart, which systemd-rpm-macros defines with
required arguments, so rpmbuild on Fedora 42 (rpm 4.20 with the macros
installed) stopped with "The %systemd_postun_with_restart macro
requires some arguments". Debian and Ubuntu builders do not define the
macro, which is why release CI never saw it. Write the names as %% in
the comments, including the %postun mention in the %post scriptlet.
Checked with rpmbuild -bb of the generated spec in fedora:42 with
systemd-rpm-macros: the previous spec fails, this one builds and its
%post carries the systemd-sysusers call.
Signed-off-by: trvon <me@trevon.dev>
x86_64 runs on a loaded host showed three harness problems:
- The Debian/Ubuntu and Fedora lane images lacked pam_systemd, so
user@.service could not start and `yams daemon install --user` looked
broken. Install libpam-systemd / systemd-pam. Where a host still
cannot start a user manager in a container, the check reports INFO
instead of failing, since that says nothing about yams.
- `systemctl --user -M user@` needs machined; query the user manager as
that user instead. The "CLI uses its unit" check now also requires that
the only daemon running for the user is the unit's MainPID, so a
CLI-spawned second daemon no longer passes.
- A container reboot can sit in 'starting' past 60 s under load; wait up
to three minutes and print pending jobs if it still does not boot.
Signed-off-by: trvon <me@trevon.dev>
Running the gate against v0.20.3 and branch packages on Debian, Ubuntu,
Fedora and Arch showed harness gaps, fixed here:
- The driver ran only under bash 4+ (mapfile, empty arrays under set -u);
it now works with macOS's bash 3.2 so it runs on Docker Desktop too.
- Paths dropped by the image's dpkg path-exclude (docs) or rpm nodocs
no longer fail the permission check, and rpm's /usr/lib/.build-id links
and the package doc/license directories are on the allowlist.
- The upgrade check pins the system socket, so it measures whether the
corpus survived rather than whether an older CLI can discover it.
- Debian keeps enablement links until purge, so for deb the dangling-link
check runs after purge.
- New per-user service check: `yams daemon install --user` beside the
system unit starts a working user unit, and that user's CLI uses it.
The unit comment and install guide now describe the socket as group
read/write with no access for others (the daemon sets owner rwx, group
rw), rather than a literal 0660.
Signed-off-by: trvon <me@trevon.dev>
`yams daemon install --user` writes WorkingDirectory=<data dir> into the
unit. On an account that never ran yams, ~/.local/share/yams does not
exist yet, so systemd failed every start with 200/CHDIR and restarted
the unit forever while the command printed "[OK] ... enabled and
started". Create the directory before writing the unit and fail clearly
if that is impossible.
Found by the new per-user service check in package validation.
Signed-off-by: trvon <me@trevon.dev>
The old package-validate.sh only checked that the unit came up and that
`yams daemon status` answered with YAMS_DAEMON_SOCKET set by hand, so it
passed while nobody's CLI ever reached the system daemon, the socket was
world-writable and `yams add ~/file` failed.
package-validate.sh stays the single entry point (same --only/--deb/
--rpm/--arch flags the workflow uses) and now drives
package-validate-checks.sh inside Debian, Ubuntu, Fedora and Arch
containers booted with systemd as PID 1 (private cgroup namespace; the
host's cgroups are not shared). Per lane it checks: install with no
manual steps; the installed file list against a runtime allowlist with
no headers, static libs, pkg-config or CMake files; no world-writable,
setuid or non-root files; resolvable DT_NEEDED and $ORIGIN-or-/usr/lib
RPATH/RUNPATH; unit enabled and active as the yams account; socket group
yams and closed to others; `systemd-analyze verify` and a
`systemd-analyze security` exposure budget; root, a yams-group member
and a non-member using the CLI with no YAMS_* variables (add from $HOME
and /tmp, search, shared corpus, no private fallback store, clear
directory error, non-member denied, `daemon stop` deferring to systemd);
state after a container reboot; removal and deb purge. With
--upgrade-from-<fmt> it installs an older package, stores a document,
upgrades and checks the restart and the document. Every check reports
PASS/FAIL, a summary table follows, and any FAIL fails the run.
release.yml's existing validation steps (the gate before artifacts are
uploaded for publishing) pass the latest published release's packages as
the upgrade source. The unreferenced packaging/systemd/Dockerfile is
removed; the lane images take a BASE_IMAGE argument and add binutils
and util-linux for the checks.
Signed-off-by: trvon <me@trevon.dev>
The packaged yams-daemon.service ran under DynamicUser= and its
ExecStartPost chmod'ed /run/yams/yams-daemon.sock to 0666, so every
local account could read, add and delete documents in the system
corpus. It also used the deprecated PermissionsStartOnly= and a light
sandbox (ProtectSystem=full only).
The unit now runs as a static `yams` system account created from
/usr/lib/sysusers.d/yams.conf (deb postinst, rpm %post and the Arch
install script run systemd-sysusers before starting it). The daemon's
own socket mode (owner rwx, group rw) with Group=yams and UMask=0007
gives a group-only socket, and /run/yams, /var/lib/yams and
/var/log/yams are 0750, so access is granted by adding a user to the
`yams` group. The chmod and PermissionsStartOnly are gone.
Sandboxing: ProtectSystem=strict, ProtectHome, PrivateTmp,
PrivateDevices, empty CapabilityBoundingSet, kernel/clock/hostname/
cgroup protection, ProtectProc=invisible, RestrictAddressFamilies
(unix, inet, inet6, netlink), RestrictNamespaces, LockPersonality,
RemoveIPC and a @system-service syscall filter without @privileged and
@resources. MemoryDenyWriteExecute stays off for ONNX Runtime.
Package scripts: Arch pre_remove now disables as well as stops (it left
a dangling multi-user.target.wants link), and deb purge removes
/var/lib/yams and /var/log/yams, including the /var/{lib,log}/private
copies earlier DynamicUser= releases left. Upgrades keep the corpus:
systemd moves the DynamicUser= state back to /var/lib/yams on first
start under User=yams.
docs/install.md describes the group, the client socket order, and what
the sandboxed service can and cannot read.
Signed-off-by: trvon <me@trevon.dev>
`yams add <path>` passes the path to the daemon, which reads the file
itself. The packaged system daemon runs as its own account with
ProtectHome= and PrivateTmp=, so adding ~/notes.txt failed with
"Permission denied" and a /tmp file was accepted but never stored (the
daemon's /tmp is private).
When SO_PEERCRED/getpeereid shows the daemon's UID differs from ours,
the CLI now reads single files itself and sends their bytes (up to
15 MiB, under the 16 MiB IPC frame) named by their absolute path.
Directory adds stay daemon-side walks (include/exclude patterns and
ignore files), so for such a daemon they now fail up front with an
explanation instead of queueing a walk that finds nothing. Same-user
daemons keep the path-based flow.
Signed-off-by: trvon <me@trevon.dev>
The deb, rpm and Arch packages start yams-daemon.service on
/run/yams/yams-daemon.sock, but clients only looked at the per-user
default (XDG_RUNTIME_DIR, /var/run as root, /tmp/yams-daemon-<uid>.sock).
With no YAMS_DAEMON_SOCKET set, root and normal users silently fell back
to an in-process daemon and a private store in ~/.local/share/yams, so
the installed service was never used.
Client-side resolution now goes: an explicit socket (flag, env, config),
the per-user daemon when its socket exists or a personal data directory
was named (flag, env, core.data_dir from yams init), then the system
socket when it exists and is connectable (root or the yams group), and
finally the per-user default. Daemons keep binding what
resolve_runtime_paths() names; `yams daemon install --user` writes the
per-user socket, never the discovered one.
Because systemd owns the system unit, `yams daemon stop|restart` refuse
it with a systemctl hint and the client never tries to spawn a daemon on
that socket. `daemon status` no longer flags the system corpus as a data
directory mismatch. socket_utils also gains a peer-UID probe used by the
next change.
The two socket default tests now use the per-user resolver so they do
not depend on whether the build host runs the packaged service.
Signed-off-by: trvon <me@trevon.dev>
Meson rewrites the install name of installed dylibs to their absolute
install path and points consumers at it, so the staged macOS payload
referenced /opt/homebrew/lib/yams/libyams_onnx_resource.dylib: yams-cli
from an unzipped release archive (or a staging dir) died in dyld
("Library not loaded ... no such file"), and Homebrew depended on its
prefix symlinks to resolve it.
A darwin-only Meson install script (scripts/macos-rpath-install-names.sh,
tag runtime) restores @rpath/libyams_onnx_resource.dylib as the library
id and in yams-cli/yams-daemon and the plugins, which already carry
@executable_path/../lib/yams and @loader_path/.. RPATHs.
Verified on macOS arm64 (ONNX + Glint release build): otool -L shows
@rpath/libyams_onnx_resource.dylib; a `zip -ry` archive of the stage
unzipped to an unrelated directory runs `yams --version`, codesign -v
passes, and the Glint plugin dlopens from there.
Signed-off-by: trvon <me@trevon.dev>
On a macOS Release build staged and split with split-debug-symbols.sh,
check_install_surface reported every binary as shipping a symbol table.
The only stab left after strip -S -x is the 'OPT radr://5614542' entry
ld64 always writes; it carries no debug information. Unstripped binaries
still trip the check (135,012 stabs in an unsplit yams-cli).
Verified on macOS arm64 (ONNX + Glint build): RED with 8 false
'stripped' violations before, GREEN after.
Signed-off-by: trvon <me@trevon.dev>
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.
Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
the old default), conan-generators-dir, cli11-cmake-path,
onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
integration-smoke-timeout, integration-daemon-timeout and stress-iters:
replaced by the same constants at their call sites (smoke keeps its
test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
folded into enable-integration-tests. Debug builds still register every
integration suite; nothing set the sub-toggles.
Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).
Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).
Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.
Signed-off-by: trvon <me@trevon.dev>
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
The audit found -fstack-protector-strong, RELRO/BIND_NOW, NX and
_FORTIFY_SOURCE=2 but no stack clash protection or control-flow
protection.
- -fstack-clash-protection on GCC/Clang where supported.
- Linux x86_64: -fcf-protection=full; Linux AArch64:
-mbranch-protection=standard (get_supported_arguments drops either
where the compiler rejects it).
- _FORTIFY_SOURCE=3 (with -U first so a toolchain default does not
produce redefinition warnings) for optimized builds; glibc < 2.34
treats it as level 2.
FORTIFY was not inert: the shipped v0.20.3 yams-cli imports
__memcpy_chk, and a grim (clang 21, glibc 2.43) build imports six
__*_chk functions. The count stays small by construction because most
libc calls in the binaries come from the Conan static archives (spdlog,
fmt, SQLite, libcurl, OpenSSL), which are compiled without FORTIFY;
level 3 did not change the count on this tree.
The x86 IBT/SHSTK property note is the AND of every linked object, so
the binaries still carry none: the Conan archives are not built with
-fcf-protection. check_install_surface keeps that as a warning; making
it a requirement needs CET-built dependencies.
Measured on grim (release -O3, stripped staged payload):
file before after delta
bin/yams-cli 33,261,496 33,712,056 +1.4%
bin/yams-daemon 28,033,432 28,389,784 +1.3%
bin/yams-mcp-server 16,573,864 16,741,800 +1.0%
libyams_object_storage_s3.so 7,837,680 7,854,064 +0.2%
yams_zyp.so 80,112 80,112 0.0%
Signed-off-by: trvon <me@trevon.dev>
- macOS zips were built with `zip -r`, which follows symlinks: bin/yams
(-> yams-cli) shipped as a second 34 MB copy and each versioned dylib
link became another full file. `zip -ry` stores the links.
- RPMs used rpm's gzip-class default payload (37.5 MB vs 28.6 MB for the
zstd .deb of the same tree); the spec now sets w19.zstdio (rpm >= 4.14).
The spec's License also said Apache-2.0.
- The Arch PKGBUILD and the CI .PKGINFO generator declared Apache-2.0
(the project is GPL-3.0-or-later) and depended on liburing, libarchive,
taglib, sqlite, openssl, curl and boost-libs, none of which the
installed binaries load: readelf shows they need only libstdc++,
libgcc_s, libc and libm, so depends is gcc-libs and glibc.
- packaging/appimage was never built by any workflow; it and the
*.AppImage globs in release.yml and the release manifest go away.
- packaging/homebrew/yams.rb was a stale CMake-era formula; release.yml
renders the tap formula from yams.rb.template, which stays.
Signed-off-by: trvon <me@trevon.dev>
Each executable links the whole static core, so most of it is unreachable
from any single binary, and every PIE carried ~0.9 MiB of RELA relative
relocations.
Optimized, non-coverage builds now compile with -ffunction-sections
-fdata-sections and link with --gc-sections and --build-id (the latter
keeps the debug split's .build-id lookup independent of the toolchain
default); lld gets --icf=safe when it is the linker. Linux links add
-z pack-relative-relocs (DT_RELR): it needs glibc >= 2.36 at runtime,
below the GLIBC_2.38 floor shipped binaries already require (v0.20.3
objdump -T), and the link probe drops it where the toolchain cannot emit
it. macOS uses -dead_strip. Debug, -O0 and coverage builds are unchanged.
Measured on grim (Linux x86_64, release -O3, stripped staged payload):
file before after delta
bin/yams-cli 36,733,304 33,261,496 -9.5%
bin/yams-daemon 30,685,792 28,033,432 -8.6%
bin/yams-mcp-server 19,865,472 16,573,864 -16.6%
libyams_object_storage_s3.so 8,239,016 7,837,680 -4.9%
libyams_onnx_resource.so 481,488 477,472 -0.8%
staged runtime total 92.0 MiB 82.7 MiB -10.1%
yams-daemon .rela.dyn: 0xd9158 (889,176 B) -> 0xb250 RELA + 0x2380 RELR.
The staged binaries init, add and search a document with a private HOME.
Signed-off-by: trvon <me@trevon.dev>
`./setup.sh Profiling` used Debug as its base build type, so Conan built
every dependency (spdlog, fmt, SQLite, Boost, protobuf, ...) unoptimized
while YAMS itself was -O2. Zone timings and allocation profiles from that
tree misattribute time to dependency code that is fast in shipped
binaries.
Profiling now uses the Release Conan profile and the Release Meson
configuration (-O3 -g, NDEBUG, PIE) plus Tracy, and keeps frame pointers
(-fno-omit-frame-pointer, -mno-omit-leaf-frame-pointer when supported)
for Tracy/perf/Instruments call stacks. TSan stays off for Profiling, as
before; --tsan still opts in.
Configure options, build/profiling on grim:
before: conan build_type=Debug; buildtype=debugoptimized (-O2 -g),
b_ndebug=true, enable-profiling=true, enable-tsan=false,
no frame-pointer flags
after: conan build_type=Release; buildtype=release, debug=true (-O3 -g),
b_ndebug=true, b_pie=true, enable-profiling=true,
enable-tsan=false, b_sanitize=none, -fno-omit-frame-pointer
Signed-off-by: trvon <me@trevon.dev>
Release used buildtype=debugoptimized (-O2 -g, strip=false) and relied on
scripts/prune-runtime-install.sh to strip the staged payload, discarding
the debug info. Crash reports from shipped binaries could not be
symbolicated and the build type label did not match what shipped.
- setup.sh/setup.ps1 Release and the Meson project default use
buildtype=release. setup.sh adds -Ddebug=true for Release so -O3 code
keeps DWARF for the split, and passes -Db_pie=true explicitly (also a
project default) so reconfigured build dirs pick it up. Windows keeps
release without -g so no PDBs land in the MSI stage. Debug/Fuzzing are
unchanged (-O0 -g, assertions on).
- scripts/split-debug-symbols.sh replaces prune-runtime-install.sh: ELF
files get objcopy --only-keep-debug (compressed) into <debug-dir>/
<path>.debug plus .build-id/xx/yyyy.debug links, strip
--strip-unneeded and --add-gnu-debuglink; Mach-O files get dsymutil
dSYMs and strip -S -x. Without a debug dir it only strips.
- The release job splits into $BUILD_DIR/debug-symbols and uploads it as
a debug-symbols-<suffix> workflow artifact (30 days) instead of
shipping or dropping it. build-deb.sh, the Dockerfile, PKGBUILD and
build-arch-pkg.sh call the split script in strip-only mode.
Measured on grim (Linux x86_64, clang 21, stripped staged payload):
file debugoptimized release -O3 delta
bin/yams-cli 35,528,968 36,733,304 +3.4%
bin/yams-daemon 29,731,312 30,685,792 +3.2%
bin/yams-mcp-server 19,504,928 19,865,472 +1.8%
libyams_onnx_resource.so 477,280 481,488 +0.9%
libyams_object_storage_s3.so 8,226,608 8,239,016 +0.2%
yams_zyp.so 79,936 80,040 +0.1%
Debug symbols: 309 MiB (compressed .debug files); gdb resolves
`info line main` on the stripped daemon through the .build-id tree.
check_install_surface: no debug sections or .symtab remain; the
remaining violations are the export leaks fixed later in this series.
Signed-off-by: trvon <me@trevon.dev>
`meson install` of a Release tree staged 2.4 GiB: all 28 internal static
libraries (1.3 GiB), the whole include/ tree (459 files), 22 pkg-config
files for libraries that were never meant to be linked from outside, and
the ONNX resource library in the system libdir. Packages hid this by
deleting include/, *.a and pkgconfig/ after the fact.
- Internal static libraries are `install: false` and their pkg.generate()
calls are gone, as is the top-level yams.pc (it required yams-api, which
is not installable). Benchmarks are no longer installed either.
- Headers are not installed: no installable C++ library exists to link
them against. The one SDK surface, the mobile C ABI, installs
mobile_bindings.h and its .pc under the 'devel' install tag.
- Runtime installs carry explicit install_tag: 'runtime' where Meson
cannot infer one (data, systemd unit/preset, the yams alias symlink,
Windows DLL copies, the libzpdf install script), so
`meson install --tags runtime` stages the same set as a plain install.
- libyams_onnx_resource moves from <libdir> to the private <libdir>/yams
on Linux/macOS, without a soname version since nothing outside this
build links it. yams-cli/yams-daemon RUNPATH becomes $ORIGIN/../lib/yams
(@executable_path on macOS) and the ONNX/Glint plugins use $ORIGIN/..
so they resolve it and the bundled ONNX Runtime, which also moves to
<libdir>/yams (the ONNX plugin's runtime loader already searches its
parent directory). Windows keeps its existing DLL layout.
- Homebrew templates and the macOS install smoke follow the new layout.
- scripts/prune-runtime-install.sh only strips now.
Measured on grim (Linux x86_64 Release, prefix /usr/local):
staged paths 522 -> 11, staged bytes 2.4 GiB -> 1.0 GiB (unstripped)
check_install_surface path violations: 6 -> 0
`--tags devel` stages nothing; ldd resolves libyams_onnx_resource from
bin/../lib/yams and the staged `yams --version` runs.
Signed-off-by: trvon <me@trevon.dev>
Packages have only looked small because scripts/prune-runtime-install.sh
deletes headers and static archives and strips after `meson install`. A
plain install of a Release tree stages 2.4 GiB: 28 internal static
libraries (1.3 GiB), 459 headers and 22 .pc files, unstripped binaries
(yams-cli 497 MiB), an S3 plugin exporting 11,154 symbols including
OpenSSL, libcurl, zlib, spdlog and nlohmann, and an ONNX resource library
in the system libdir.
scripts/ci/check_install_surface.py stages `meson install --destdir` for
a build dir (or reads a staged tree) and asserts it against
scripts/ci/install_surface_policy.json: an allowed runtime path set and a
forbidden devel set, required paths, per-binary size budgets, exported
symbol budgets and forbidden third-party export patterns, $ORIGIN-only
RUNPATH, ELF hardening (PIE, BIND_NOW, GNU_RELRO, NX; CET and FORTIFY
reported as warnings) and stripped binaries. ELF inspection uses
readelf/nm; Mach-O uses otool/nm. Standard library only.
The release packaging job runs it on the exact staged payload after the
existing runtime asset checks. On current experimental it is RED
(Linux x86_64 Release, grim):
raw meson install: 43 violations (paths, debug info, exports, sizes)
after prune script: 29 violations (exports, libdir placement)
Size budgets start at the measured stripped baseline plus ~10% and are
tightened by the follow-up build changes.
Signed-off-by: trvon <me@trevon.dev>
The deferred-drain check treated a missing value as zero, so a node
whose p2p status call failed or returned non-JSON satisfied it. Drain
now needs a numeric 0 from every node, and so does the quarantined
writer check once any node reports the field; an unreadable status is
reported as such instead of passing.
Signed-off-by: trvon <me@trevon.dev>
converge proves convergence with yams get once yams list shows every
hash. When those reads kept failing until the timeout, the listing gap
was empty, so converge.json and REPORT.md claimed every document was
readable next to converged=false. The final yams get failures are now
kept (unreadable_via_get) and used for the readable counts whenever the
read-path check ran.
Signed-off-by: trvon <me@trevon.dev>
seed floored --docs-per-node to whole rounds of topics: 10 documents
with 8 topics seeded 8, 0 still seeded a full round, and --topics 0
raised ZeroDivisionError. Both counts are now validated, and the last
round is truncated so every node gets exactly the requested number.
Signed-off-by: trvon <me@trevon.dev>
up --force only skipped the existing-state guard. Compose then recreated
the services on the old named volumes, so nodes came up with the stale
/node/ready marker, keys and config of the previous mesh, and services
of a larger previous mesh kept running as orphans. A forced up now runs
down (volumes, orphans, image) for the previous project first and starts
under a fresh project name.
Signed-off-by: trvon <me@trevon.dev>
A mesh of honest writers must never durably quarantine one of them, but
converge only checked document reachability and deferral drain, so a run
that quarantined a peer could still pass. converge now records
quarantined_writers per node and exits 4 when any node reports one.
Every up/run also gets its own compose project (yamsmesh-<random>, or
--project), so containers, network, volumes and image tag of concurrent
meshes on one host no longer collide.
Signed-off-by: trvon <me@trevon.dev>
Production meshes (transport=direct, full mesh, operator-pinned mutual TLS)
stalled at a fixed record count with failed_cycles=0 because a failed inbound
apply stage skipped the node's own publish. Reproducing that needs several
real daemons with enrolled pins and a corpus of embeddings and topology
edges authored on different nodes, so add tests/integration/p2p_mesh:
- Dockerfile and node entrypoint that run prebuilt yams/yams-daemon from a
host build dir or a released .deb, one isolated volume per node.
- mesh.py (stdlib only) with up/seed/converge/status/chaos/run/down. It
generates writer keys and manifests, enrolls every peer pin and connects
a full mesh the way an operator does, and records apply-health counters
when the build reports them.
Local/manual tooling only; not wired into any CI.
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): add keyset node/edge scans and exact edge lookup
The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no
ORDER BY, so a topology rebuild that deleted and reinserted cluster
nodes between two pages shifted the offset and the sweep skipped live
nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never
reused, so a cursor on the last id returned is stable under concurrent
inserts and deletes and survives restarts.
getNodesAfterId/getEdgesAfterId scan in id order from such a cursor.
getEdgesBetween returns every edge between two nodes for one relation;
getEdgesFrom truncates at its page limit, which made "is this edge
present" checks on high-degree nodes unreliable.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish topology edge endpoints and retract deleted ones
On a Docker mesh that converges on documents, topology deferrals never
drained: peers held member_of and semantic_neighbor edges whose endpoint
node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80).
Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral
named a doc or cluster node that its origin still held but had not
published. The origin's sweep was still inside the "document" type: no
"path", "topology_cluster" or "topology_snapshot" node had been published
by any node, while the doc nodes' edges into them had been.
The backfill swept node types in sorted order and published each node's
edges right after the node, so an edge reached peers long before its
target node of a later type; at the backfill's rate that is tens of
minutes. Its paging also made some of those waits permanent:
- Nodes were paged per type with OFFSET and no ORDER BY. A topology
rebuild deletes every cluster node and inserts replacements, so the
offset skipped live nodes whose member_of edges were published.
- Per-type offsets were never reset, so a node was visited once: edges
added after its sweep (semantic neighbors, new memberships) and
in-place node updates were never published.
- Nothing published deletions. Worse, apply() wrote this node's own
winners back into its graph, resurrecting the cluster nodes and edges
its rebuild had just deleted, so the deletion could not be observed.
- Peers deferred an edge whose endpoint was tombstoned forever.
The topology backfill now runs repeating keyset passes: every node by
id, then every edge by id published together with both endpoint nodes
(endpoints first, so a peer never holds a writer's edge without them),
then this writer's committed topology winners by logical key, tombstoning
those the local graph no longer holds. Cursors are stable ids/keys, so
inserts and deletes between items cannot shift them, and a restart just
begins a new pass; unchanged records publish nothing.
apply() still validates every record but never applies the writer's own
ones: the local graph is authoritative for them. An edge whose endpoint
is known deleted (tombstoned, or an own record the local graph no longer
holds) is dropped instead of deferred, and applies again if the endpoint
is re-published. Edge lookups during apply use the exact endpoint query
instead of a page-limited adjacency scan.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory-sync): publish documents before derived records in backfill
With the topology sweep now publishing every node and edge, a 3-node x
40-doc mesh took longer until every document was listed on every node
(603 s against 426 s before the topology fix). Debug logs showed why:
a publish takes most of the backfill's 100 ms per-cycle time budget, so
each cycle published about one item, and the fair rotation over
documents, vectors and topology handed two of every three cycles to
derived records while peers waited for the documents.
The backfill now serves domains in priority order: documents, then
vectors, then topology, each taking items until it has nothing left this
cycle. Vectors and topology defer on a peer until their document
arrives, so publishing the document first never adds a deferral. In the
same mesh, run concurrently with both other builds, every document is
listed on every node after 140 s (426 s before the topology fix, 603 s
with it alone), and deferrals still drain to 0.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
tests/integration/search/meson.build reassigned catch2_main_dep to Catch2's
own catch2-with-main. Meson variables are global, so every test executable
defined after that subdir (catch2_search_service among them) linked Catch2's
main instead of tests/support/catch2_main.cpp. The Windows hang stack showed
main coming from Catch2's catch_main.cpp, which is why the CRT report setup
in the shared main never ran there.
Use the shared catch2_main_dep for the hybrid search integration binaries
too.
Signed-off-by: trvon <me@trevon.dev>
supportedExtensions() returned its static set by value, so capabilities()
constructed the extension vector from begin() of one temporary and end()
of another. That range is undefined behavior. libstdc++ and libc++ release
iterators happened to walk the first copy to completion, but the MSVC debug
STL rejects it in _Verify_range with a modal "Microsoft Visual C++ Runtime
Library" assertion dialog. HandlerRegistry::selectBest calls capabilities()
for every candidate, so every lines: search and every applyExtractScope call
hung forever on Windows Debug builds.
Return the shared set by const reference and take begin()/end() from that
one instance. New text_basic_handler unit tests pin the shared instance,
the capability list, registry selection and line-range extraction; the old
code also fails them under -D_GLIBCXX_DEBUG with "function requires a valid
iterator range".
Signed-off-by: trvon <me@trevon.dev>
Three catch2_search_service cases hang on windows-latest with no output at
all, so the diagnose workflow could only name them. A silent hang on a
Debug MSVC build is often a modal CRT assert or abort dialog that nobody
can dismiss.
Before killing a hung case the workflow now prints its stdout/stderr tail
(run with -s -d yes), its top-level windows, any WER helper, and every
thread's stack plus !analyze -v -hang from cdb.exe. A test_filter input
limits the run to a Catch2 test spec for faster iterations.
The shared Catch2 main now routes CRT warnings, errors and assertions to
stderr and disables the Windows error-reporting dialogs, so a future
Debug assertion fails the case with its message instead of hanging.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
The build needs third_party/symspell (tests.yml initializes it); without it
meson stopped at 'Nonexistent build file third_party/symspell/meson.build'
before any test ran.
Signed-off-by: trvon <me@trevon.dev>
* fix(init): align fresh-install embedding dimensions and vector sentinel
A fresh `yams init` emitted dimension-conflict warnings and created a
vector store whose dimension did not match the configured embedding model:
Embedding policy: embedding dimension conflict:
embeddings.embedding_dim='1024' overrides compatibility key
vector_database.embedding_dim='384' (and vector_index.dimension='384')
Root causes:
- getLatestConfigDefaults() hardcoded vector_database.embedding_dim and
vector_index.dimension to 384 while [embeddings] used 1024 for the
default Simeon backend, so the generated config was self-inconsistent.
- init resolved the vector dimension only by scanning the optional ONNX
model list, leaving it at 0 for the default "simeon-default", and never
wrote vectors_sentinel.json.
- updateV2Config searched for [vector_database] keys (enabled/model/
model_path) that do not exist in the v3 schema and never updated the
dimension keys.
Fixes:
- Default vector_database.embedding_dim and vector_index.dimension to 1024.
- Add resolveModelDimension() and stamp vectors_sentinel.json from the
vector DB's effective dimension only when this init created the store,
so a forced re-init cannot overwrite a real stored dimension.
- Rewrite updateV2Config to set enable/embedding_dim for [vector_database],
enable/preferred_model/backend/embedding_dim for [embeddings], and
dimension for [vector_index] consistently; disable the enable flags when
semantic search is declined.
- Non-interactive init now defaults to semantic search enabled, matching
the interactive default.
- Keep compatibility dimension keys aligned during v1/v2 migration so an
existing 384-d install never receives a 1024-d sibling default.
* fix(init): prompt for the tuning profile once
`execute()` asked for the tuning profile before the idempotency check and
`handleAlreadyInitialized()` asked again for an already-initialized instance,
so `yams init` showed the tuning prompt twice. Collect the profile once on the
fresh path (after the idempotency check); the already-initialized path keeps
its single prompt.
Add interactive regression tests with a scripted stdin helper:
- already-initialized init prompts for tuning exactly once and persists the
chosen profile;
- declining semantic search disables both `[vector_database]` and
`[embeddings]`, writes no dimension sentinel, and still prompts for tuning
once (also covers the fresh interactive path).
---------
Co-authored-by: trvon <git@trevon.dev>
Full isolation still let get_runtime_dir() consume the ambient XDG_RUNTIME_DIR, so yams-daemon.status.json landed outside the temp root. Relocate XDG_RUNTIME_DIR (POSIX) and APPDATA/LOCALAPPDATA/USERPROFILE (Windows) under the run root, and scrub YAMS_EMBED_BACKEND, YAMS_PREFERRED_MODEL, YAMS_RERANKER_MODEL and YAMS_EMBED_DIM, which ConfigResolver ranks above TOML, so the generated config is deterministic. Apply the same to ingestion_e2e_bench and extend the isolation test with a decoy runtime dir and model overrides.
Signed-off-by: trvon <me@trevon.dev>
repair_ability_bench built its harness without any isolation, so the daemon
resolved config from ambient YAMS_CONFIG, ~/.config/yams and HOME and could
index real files. multi_client_ingestion_bench isolated state but not config,
HOME or the XDG cache/data dirs, and ingestion_e2e_bench left HOME untouched.
Use DaemonHarnessOptions::isolateEnvironment in repair_ability_bench (with an
explicit simeon embeddings config so the embed fault kind stays repairable)
and both multi_client_ingestion_bench option builders; an explicit
YAMS_BENCH_CONFIG still wins. In ingestion_e2e_bench set HOME and
XDG_DATA_HOME to the fixture root and unset YAMS_STORAGE and the daemon
socket aliases. Shift the affected portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
DaemonHarness only relocated state when isolateState/isolateConfig were set,
and even then left HOME, XDG_DATA_HOME, XDG_CACHE_HOME and ambient
YAMS_DAEMON_SOCKET_PATH/YAMS_STORAGE/YAMS_DATA_DIR visible to the daemon, so a
run could still resolve a developer's config, data and cache.
Add DaemonHarnessOptions::isolateEnvironment (and DaemonHarness::isolatedOptions())
which implies isolateState and isolateConfig, points HOME and every XDG base
directory at the run's temp root, and scrubs the ambient config/data/socket
variables for the daemon's lifetime, restoring them through ScopedEnvVar RAII.
The new Catch2 case exports a decoy config.toml (distinctive data_dir marker) and
decoy XDG/HOME/YAMS_* values, then asserts the daemon's config path, data dir and
every ambient path resolver stay under the temp root, the decoy is never read or
populated, and the ambient environment is restored after stop.
Signed-off-by: trvon <me@trevon.dev>
* fix(windows): ship yams-daemon and yams-mcp-server in the MSI
The Windows release already builds both and meson install stages them next
to yams.exe, but yams.wxs only packaged yams.exe, so daemon-backed commands
and yams serve failed after an MSI install. Package both, and make
build-msi.ps1 fail when a required executable is missing from the stage or a
staged executable is not packaged.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): fall back to the stable R2 and signing secrets for experimental
The experimental channel needs EXPERIMENTAL_* Cloudflare and GPG secrets that
were never created, so even a passing nightly could not publish and
repo.yamsmemory.ai/experimental/* has never existed. Fall back to the stable
token, account and key when the channel-specific secrets are unset; setting
them later still takes precedence.
Signed-off-by: trvon <me@trevon.dev>
* ci: add a manual Windows workflow that finds hanging test cases
search_service times out at 360 s on Windows in every Tests run, which keeps
experimental from going green and blocks every nightly. The meson log only
shows the timeout. This workflow builds once and runs each Catch2 case on its
own with a per-case timeout, then reports durations and the cases that hang.
It runs only on workflow_dispatch or a push to a diagnose/** branch.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
