* test(search,daemon,storage): assert behaviour instead of frozen literals
Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.
Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.
Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.
The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.
Signed-off-by: trvon <git@trevon.dev>
* ci(tests): summarise Meson failures without flooding the log
A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.
Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.
Signed-off-by: trvon <git@trevon.dev>
* build(deb): validate candidate identity, mapping and upgrade ordering
The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.
Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): keep init vector tests hermetic and the sentinel tied to disk
cli_init_command and cli_commands failed on every platform since #247.
The Tests workflow runs unit lanes with YAMS_VDB_IN_MEMORY=1 and vectors
disabled, so the vector store init creates is in memory: no vectors.db
was written (init_command_catch2_test.cpp:350) and a forced re-init,
seeing no file, overwrote the 384 sentinel with 1024 (:455).
The two cases assert on the on-disk store, so they now pin that
environment with ScopedEnvVar instead of inheriting the CI one.
Init also stamped a dimension sentinel for the in-memory store even
though nothing was written to disk. Stamp it only when this init created
vectors.db, and cover the in-memory case.
Signed-off-by: trvon <me@trevon.dev>
* fix(test): ignore worker-thread stages in memory sync serialization test
daemon_memory_sync_apply_serialization failed on Windows with
contentStages 3 == 2 and "memory_sync stage observer threw at
apply.after_content". The memory_sync worker reconciles once as soon as
it starts and runs the apply callback on its thread. On a slow host that
first cycle lands after the harness reports ready, inside the test's
observer window, so a third apply hit the observer and set an already
satisfied promise.
Reproduced on Linux under CPU load (1 of 192 runs, same signature).
The observers now count only stages reported from the test's own caller
threads, and the attempts check allows an extra worker attempt. 0 of 384
runs fail under the same load.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): do not mmap vectors.db on Windows
repair_maintenance failed on Windows only: the live vectors.db VACUUM
reported "4 MB -> 4 MB" and the file never shrank
(repair_maintenance_catch2_test.cpp:301). The standalone VACUUM case,
with no other connection open, passes there.
The vector backend connection sets mmap_size=256MB. On Windows,
SetEndOfFile from another handle fails with ERROR_USER_MAPPED_FILE while
a view of the file is mapped, and SQLite's winTruncate ignores that
error, so the truncate at the end of the VACUUM connection's checkpoint
is silently skipped. Linux and macOS can truncate a mapped file.
Skip the mmap pragma on Windows so the idle VACUUM in RepairService can
reclaim disk.
Signed-off-by: trvon <me@trevon.dev>
* ci: name the red Tests streak when a nightly is refused
The nightly gate refused with only "no successful Tests run for SHA",
so a run of red Tests on experimental stayed silent. On refusal, also
report the Tests conclusion for that SHA, the newest failing Tests run on
experimental with its SHA and failing jobs, and the last green run, as
annotations and in the job summary. The gate itself is unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): claim the clean-shutdown stamp exclusively on Windows
db_recovery failed on Windows with "clean shutdown stamp has only one
consumer" (db_recovery_test.cpp:108): both concurrent consumers
rejected a valid stamp.
The claim renamed the stamp with MoveFileExW. That opens the source
sharing read, write and delete and renames through the handle, so two
callers can both open the stamp and both succeed, the second moving the
first's claim file away; their opens also fail each other's reads with
ERROR_SHARING_VIOLATION. Driving the real stamp code from two threads on
Windows x64, 205 of 2000 rounds ended with both consumers rejecting it
("stamp missing or malformed"); never with both trusting it.
On Windows, open the stamp with no sharing, retrying briefly while
another opener holds it, and rename through that handle. A concurrent
consumer then finds no stamp. Same driver: 2000 of 2000 rounds with
exactly one trusted consumer and no leftover claim files. The dropped
MOVEFILE_WRITE_THROUGH only affects cross-volume copy moves; the claim is
a same-directory rename. POSIX keeps rename(2).
The test now captures both consumers' reasons on failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): size the onnx resource budget for aarch64 page alignment
libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget.
Signed-off-by: trvon <me@trevon.dev>
* fix(plugins): declare the S3 legacy factory with the plugin export macro
93f24394 marked the S3 plugin's yams_plugin_create/destroy_object_storage definitions YAMS_PLUGIN_API so they stay exported under hidden visibility, but include/yams/plugins/object_storage.h still declared them without it. MSVC rejects a definition whose linkage differs from the first declaration (C2375), so the v0.20.4 Windows release job failed to compile s3_plugin.cpp. GCC and Clang accept the mismatch, which is why the Linux and macOS lanes passed.
The header now includes yams/plugins/abi.h and declares both functions YAMS_PLUGIN_API, matching how abi.h declares the rest of the plugin C ABI. Checked with MSVC 14.51 (x64 target): a translation unit with the s3_plugin.cpp definitions fails with C2375 against the old header and compiles against the new one.
Signed-off-by: trvon <me@trevon.dev>
* fix(build): drop install_tag from the mobile install_headers call
324be33a passed install_tag: 'devel' to install_headers(), which accepts no such keyword. Meson 1.12 fails the configure with "install_headers got unknown keyword arguments install_tag" whenever -Denable-mobile-bindings=true, which broke the Tests linux-x86_64-mobile lane on experimental. Meson tags installed headers 'devel' by itself, so the header still ships only with the devel set (checked with meson introspect --install-plan and meson install --tags devel|runtime --dry-run on grim).
Signed-off-by: trvon <me@trevon.dev>
* fix(packaging): install the Windows onnx resource DLL next to the executables
yams-cli.exe, yams-daemon.exe and yams-mcp-server.exe import
yams_onnx_resource-0.dll at load time, but Meson installed it to lib\ and
yams.wxs never packaged it. The shipped v0.20.3 MSI therefore installs a
yams.exe that cannot start: extracting it (msiexec /a) and running
`yams.exe --version` exits 0xC00004BC with no output, and dumpbin
/dependents lists yams_onnx_resource-0.dll. The release job never runs the
staged CLI on Windows, so CI stayed green.
Drop the explicit Windows install_dir so Meson installs the DLL to bindir
(the import library stays in libdir) and add it to the MSI. With the change
`meson install` stages bin\yams_onnx_resource-0.dll and the staged
`yams.exe --version` prints the version.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): smoke-test the staged Windows CLI before building the MSI
The v0.20.3 MSI shipped a yams.exe that exits 0xC00004BC because a load-time DLL was not packaged, and nothing in the release job ran the staged binary. Run the staged yams.exe --version with a 60 s timeout after meson install and fail the job on a non-zero exit. Checked on a Windows host against the old stage layout (exits 0xC00004BC, step fails) and the fixed one (prints 0.20.5, step passes).
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
The old package-validate.sh only checked that the unit came up and that
`yams daemon status` answered with YAMS_DAEMON_SOCKET set by hand, so it
passed while nobody's CLI ever reached the system daemon, the socket was
world-writable and `yams add ~/file` failed.
package-validate.sh stays the single entry point (same --only/--deb/
--rpm/--arch flags the workflow uses) and now drives
package-validate-checks.sh inside Debian, Ubuntu, Fedora and Arch
containers booted with systemd as PID 1 (private cgroup namespace; the
host's cgroups are not shared). Per lane it checks: install with no
manual steps; the installed file list against a runtime allowlist with
no headers, static libs, pkg-config or CMake files; no world-writable,
setuid or non-root files; resolvable DT_NEEDED and $ORIGIN-or-/usr/lib
RPATH/RUNPATH; unit enabled and active as the yams account; socket group
yams and closed to others; `systemd-analyze verify` and a
`systemd-analyze security` exposure budget; root, a yams-group member
and a non-member using the CLI with no YAMS_* variables (add from $HOME
and /tmp, search, shared corpus, no private fallback store, clear
directory error, non-member denied, `daemon stop` deferring to systemd);
state after a container reboot; removal and deb purge. With
--upgrade-from-<fmt> it installs an older package, stores a document,
upgrades and checks the restart and the document. Every check reports
PASS/FAIL, a summary table follows, and any FAIL fails the run.
release.yml's existing validation steps (the gate before artifacts are
uploaded for publishing) pass the latest published release's packages as
the upgrade source. The unreferenced packaging/systemd/Dockerfile is
removed; the lane images take a BASE_IMAGE argument and add binutils
and util-linux for the checks.
Signed-off-by: trvon <me@trevon.dev>
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.
Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
the old default), conan-generators-dir, cli11-cmake-path,
onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
integration-smoke-timeout, integration-daemon-timeout and stress-iters:
replaced by the same constants at their call sites (smoke keeps its
test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
folded into enable-integration-tests. Debug builds still register every
integration suite; nothing set the sub-toggles.
Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).
Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).
Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.
Signed-off-by: trvon <me@trevon.dev>
- macOS zips were built with `zip -r`, which follows symlinks: bin/yams
(-> yams-cli) shipped as a second 34 MB copy and each versioned dylib
link became another full file. `zip -ry` stores the links.
- RPMs used rpm's gzip-class default payload (37.5 MB vs 28.6 MB for the
zstd .deb of the same tree); the spec now sets w19.zstdio (rpm >= 4.14).
The spec's License also said Apache-2.0.
- The Arch PKGBUILD and the CI .PKGINFO generator declared Apache-2.0
(the project is GPL-3.0-or-later) and depended on liburing, libarchive,
taglib, sqlite, openssl, curl and boost-libs, none of which the
installed binaries load: readelf shows they need only libstdc++,
libgcc_s, libc and libm, so depends is gcc-libs and glibc.
- packaging/appimage was never built by any workflow; it and the
*.AppImage globs in release.yml and the release manifest go away.
- packaging/homebrew/yams.rb was a stale CMake-era formula; release.yml
renders the tap formula from yams.rb.template, which stays.
Signed-off-by: trvon <me@trevon.dev>
Release used buildtype=debugoptimized (-O2 -g, strip=false) and relied on
scripts/prune-runtime-install.sh to strip the staged payload, discarding
the debug info. Crash reports from shipped binaries could not be
symbolicated and the build type label did not match what shipped.
- setup.sh/setup.ps1 Release and the Meson project default use
buildtype=release. setup.sh adds -Ddebug=true for Release so -O3 code
keeps DWARF for the split, and passes -Db_pie=true explicitly (also a
project default) so reconfigured build dirs pick it up. Windows keeps
release without -g so no PDBs land in the MSI stage. Debug/Fuzzing are
unchanged (-O0 -g, assertions on).
- scripts/split-debug-symbols.sh replaces prune-runtime-install.sh: ELF
files get objcopy --only-keep-debug (compressed) into <debug-dir>/
<path>.debug plus .build-id/xx/yyyy.debug links, strip
--strip-unneeded and --add-gnu-debuglink; Mach-O files get dsymutil
dSYMs and strip -S -x. Without a debug dir it only strips.
- The release job splits into $BUILD_DIR/debug-symbols and uploads it as
a debug-symbols-<suffix> workflow artifact (30 days) instead of
shipping or dropping it. build-deb.sh, the Dockerfile, PKGBUILD and
build-arch-pkg.sh call the split script in strip-only mode.
Measured on grim (Linux x86_64, clang 21, stripped staged payload):
file debugoptimized release -O3 delta
bin/yams-cli 35,528,968 36,733,304 +3.4%
bin/yams-daemon 29,731,312 30,685,792 +3.2%
bin/yams-mcp-server 19,504,928 19,865,472 +1.8%
libyams_onnx_resource.so 477,280 481,488 +0.9%
libyams_object_storage_s3.so 8,226,608 8,239,016 +0.2%
yams_zyp.so 79,936 80,040 +0.1%
Debug symbols: 309 MiB (compressed .debug files); gdb resolves
`info line main` on the stripped daemon through the .build-id tree.
check_install_surface: no debug sections or .symtab remain; the
remaining violations are the export leaks fixed later in this series.
Signed-off-by: trvon <me@trevon.dev>
Packages have only looked small because scripts/prune-runtime-install.sh
deletes headers and static archives and strips after `meson install`. A
plain install of a Release tree stages 2.4 GiB: 28 internal static
libraries (1.3 GiB), 459 headers and 22 .pc files, unstripped binaries
(yams-cli 497 MiB), an S3 plugin exporting 11,154 symbols including
OpenSSL, libcurl, zlib, spdlog and nlohmann, and an ONNX resource library
in the system libdir.
scripts/ci/check_install_surface.py stages `meson install --destdir` for
a build dir (or reads a staged tree) and asserts it against
scripts/ci/install_surface_policy.json: an allowed runtime path set and a
forbidden devel set, required paths, per-binary size budgets, exported
symbol budgets and forbidden third-party export patterns, $ORIGIN-only
RUNPATH, ELF hardening (PIE, BIND_NOW, GNU_RELRO, NX; CET and FORTIFY
reported as warnings) and stripped binaries. ELF inspection uses
readelf/nm; Mach-O uses otool/nm. Standard library only.
The release packaging job runs it on the exact staged payload after the
existing runtime asset checks. On current experimental it is RED
(Linux x86_64 Release, grim):
raw meson install: 43 violations (paths, debug info, exports, sizes)
after prune script: 29 violations (exports, libdir placement)
Size budgets start at the measured stripped baseline plus ~10% and are
tightened by the follow-up build changes.
Signed-off-by: trvon <me@trevon.dev>
Three catch2_search_service cases hang on windows-latest with no output at
all, so the diagnose workflow could only name them. A silent hang on a
Debug MSVC build is often a modal CRT assert or abort dialog that nobody
can dismiss.
Before killing a hung case the workflow now prints its stdout/stderr tail
(run with -s -d yes), its top-level windows, any WER helper, and every
thread's stack plus !analyze -v -hang from cdb.exe. A test_filter input
limits the run to a Catch2 test spec for faster iterations.
The shared Catch2 main now routes CRT warnings, errors and assertions to
stderr and disables the Windows error-reporting dialogs, so a future
Debug assertion fails the case with its message instead of hanging.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
The build needs third_party/symspell (tests.yml initializes it); without it
meson stopped at 'Nonexistent build file third_party/symspell/meson.build'
before any test ran.
Signed-off-by: trvon <me@trevon.dev>
* fix(windows): ship yams-daemon and yams-mcp-server in the MSI
The Windows release already builds both and meson install stages them next
to yams.exe, but yams.wxs only packaged yams.exe, so daemon-backed commands
and yams serve failed after an MSI install. Package both, and make
build-msi.ps1 fail when a required executable is missing from the stage or a
staged executable is not packaged.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): fall back to the stable R2 and signing secrets for experimental
The experimental channel needs EXPERIMENTAL_* Cloudflare and GPG secrets that
were never created, so even a passing nightly could not publish and
repo.yamsmemory.ai/experimental/* has never existed. Fall back to the stable
token, account and key when the channel-specific secrets are unset; setting
them later still takes precedence.
Signed-off-by: trvon <me@trevon.dev>
* ci: add a manual Windows workflow that finds hanging test cases
search_service times out at 360 s on Windows in every Tests run, which keeps
experimental from going green and blocks every nightly. The meson log only
shows the timeout. This workflow builds once and runs each Catch2 case on its
own with a per-case timeout, then reports durations and the cases that hang.
It runs only on workflow_dispatch or a push to a diagnose/** branch.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the wrangler upload path, which passes
--content-type, and removed the S3 "ContentType" field this step grepped
for. Act Sanity has failed on experimental since then.
Signed-off-by: trvon <me@trevon.dev>
The snippet wrote /etc/pacman.d/yams.conf but nothing includes that file,
so pacman never saw the repository. Append the [yams] section to
/etc/pacman.conf instead.
Signed-off-by: trvon <me@trevon.dev>
${{ cond && '' || 'experimental/' }} always yields 'experimental/' because
'' is falsy in Actions expressions, so the generated USAGE.txt for stable
releases pointed YUM and Arch users at experimental paths that return 404.
Invert the expression. The YUM instructions also claimed an unsigned
repository although repomd.xml is signed whenever a channel key is set;
enable repo_gpgcheck with the channel key in that case.
Signed-off-by: trvon <me@trevon.dev>
libyams_mobile stopped compiling without anyone noticing because no CI
lane built it: every lane forced YAMS_ENABLE_MOBILE_BINDINGS=false, and
the "Check mobile ABI surface" step was commented out and pointed at
scripts/ci/check_mobile_abi.sh, which was never committed.
Add a linux-x86_64-mobile matrix entry that configures with
enable-mobile-bindings and enable-mobile-tests, compiles only the
yams_mobile and catch2_mobile_abi targets, runs the ABI check, and runs
the mobile meson suite. Other lanes keep mobile bindings off.
Add scripts/ci/check_mobile_abi.sh. It compiles the public header as C11
and C++20 and takes the address of every YAMS_MOBILE_API function, so a
declaration hidden from C (the extern "C" bug fixed in the previous
commit) fails. It then checks that the library exports exactly the
declared yams_mobile_* functions with unmangled names.
This lane proves the Linux x86_64 host build only. It does not show that
an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
This reverts commit 296e7320.
296e7320 moved R2 publication from wrangler (authenticated by
CLOUDFLARE_API_TOKEN) to boto3 over R2's S3 API, which needs
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (and EXPERIMENTAL_* pairs). Those
secrets were never configured, so the first stable release built after
the change (v0.20.2, run 36084509789) failed at 'Publish channel package
repositories to R2' with 'Missing channel-specific Cloudflare R2 S3
authority', and the APT/RPM/Arch repositories were not updated.
Restore the wrangler path so releases publish with the existing token.
archlinux/archlinux only publishes linux/amd64, so the native aarch64 Arch
lane failed with 'exec format error' before provisioning. Both Arch
Dockerfiles now take ARCH_BASE_IMAGE; the aarch64 matrix entry pins
ghcr.io/menci/archlinuxarm by digest and x86_64 keeps the official image.
package-validate.sh forwards ARCH_BASE_IMAGE to the systemd substrate.
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.
The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.
Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
Default builds no longer pull onnxruntime: conan enable_onnx and the conan
profiles default to False, meson enable-onnx defaults to disabled and skips
onnxruntime discovery entirely, and setup.sh/setup.ps1 only enable ONNX
(and the ONNX-backed Glint plugin) with --with-onnx / -WithOnnx.
YAMS_DISABLE_ONNX=true still forces it off.
This removes a build blocker on hosts where conan tries to build
onnxruntime from source (e.g. ROCm/MIGraphX on Linux). Release, Docker, and
Arch packaging pass the flag explicitly, so shipped artifacts are unchanged
until ONNX is dropped.
Fuzzing belongs on our own compute rather than GitHub-hosted runners.
Move the smoke job to .forgejo/workflows on the self-hosted label, pin
actions to code.forgejo.org, and add a preflight that fails clearly when
the runner lacks clang's libFuzzer runtime or the build tools.
Note that once .forgejo/workflows exists, Forgejo runs only those
workflows and stops executing the .github/workflows copies.
Build the libFuzzer targets with clang through setup.sh Fuzzing and run
each for 60 seconds over the generated seeds on pull requests touching
the P2P, codec or fuzzing sources, nightly, and on demand. Reproducers
are uploaded when a harness fails.
A stable workflow_dispatch took the release tag from GITHUB_REF_NAME
without checking it, so dispatching from main published a release
tagged main. Require refs/tags/v* or refs/tags/yams-v*.
Fetch main and experimental with explicit refspecs in the candidate
preflight so a tag named main cannot shadow the branch, and drop the
workflow_run trust path from draft reconciliation now that the workflow
is dispatch-only.
- Decouple draft-promotion.yml from automated Tests workflow_run triggers,
making promotion draft creation manual via workflow_dispatch to prevent
GitHub from auto-chaining active experimental feature PRs into a stack
targeting main.
- Enforce and document the mandatory Promotion Merge Policy across
reconcile_promotion_pr.py, validate_release_candidate.py, and draft-promotion.yml
step summary: promotions must use a merge commit (--no-ff) or fast-forward,
never squash-merge into main, preserving conventional commit history and
unbroken git ancestry.
- Unhide refactor commits in release-please-config.json so architectural
refactorings are tracked under Changed in CHANGELOG.md.
- Update unit tests in test_reconcile_promotion_pr.py and
test_validate_release_candidate.py for the new policy invariants.
Signed-off-by: trvon <me@trevon.dev>
* ci(tests): free unused toolchains on the macOS runner
The macOS ARM64 lane hit "zstd: error 70 ... No space left on device"
followed by a linker failure. Free multi-gigabyte toolchains this build
never uses before anything is built: Android SDK, dotnet, PowerShell and
Mono.
Deliberately keep Xcode/clang, Homebrew, cmake, ccache, zig, boost and the
Rust toolchain: meson.build builds the libSQL subproject only when cargo is
found and silently falls back to plain sqlite3 otherwise, so deleting
~/.cargo would switch this lane database backend without failing anything.
This does not address the nightly release gate, which currently fails on an
unrelated flaky batch-count assertion in daemon_background_processing.
Signed-off-by: trvon <me@trevon.dev>
* ci(tests): clarify SQLite warning-only fallback in macOS free-disk step
Update comment in .github/workflows/tests.yml to accurately describe that
when Cargo is missing, meson.build:555-572 emits warnings before falling
back to standard SQLite3 without failing the build.
Addresses Copilot review comment on PR #174.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Prune CI artifacts and stream metadata scans. Preserve relocated CLI
loader dependencies, reject incomparable path roots, and make deadline
and corruption fixtures test their stated contracts.
Keep unresolved Windows publication and hook failures diagnostic-only.
Signed-off-by: trvon <me@trevon.dev>
* test(ci): stabilize main matrix (#112)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): preview experimental promotion (#111)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): reconcile draft promotion safely (#113)
* ci(release): reconcile draft promotion
Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.
Signed-off-by: trvon <me@trevon.dev>
* ci: isolate hook Git environments
Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): reject invalid candidate evidence
Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve completion and retrieval contracts
Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence asynchronous enrichment completion
Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.
Signed-off-by: trvon <me@trevon.dev>
* fix: bound semantic graph output reservation
Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence standalone embedding repair completion
Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve enrichment failure reporting
Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test: align explicit hash prefix boundaries
Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.
Signed-off-by: trvon <me@trevon.dev>
* test: align UI hash validation boundary
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* chore(repo): land the retrieval contract and drop root cruft
Lands work that had been sitting uncommitted in the working tree:
- docs/architecture/retrieval_contracts.md plus its executable form,
tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py,
now registered in meson as retrieval_contract_unit (suite: policy).
.gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore.
- AGENTS.md: the Development Memory section, and the product default
recorded as hybrid_assist + shadow with topology-off as the control,
which is what search_topology_stage already does.
docs/benchmarks/index.md gains the matching search_shadow_cost plan.
- .gitleaks.toml, and a Secret scan workflow that actually runs it on
push and pull requests (nothing in .github invoked gitleaks before).
Deletes three root files with zero references anywhere in the tree:
bench_profile.json (ad-hoc profiling leftover), CppProperties.json
(Visual Studio config; the repo standardises on Meson), and spdlog.pc
(a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would
mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan).
docs/plans/ (two p2p designs already shipped in 5a53cdfa and d012d95b)
was ingested into YAMS with task=desloppification source=plan, hydrated
via yams cat and byte-compared before removal, per the new AGENTS.md
rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a
plans directory.
Also reflows two stale line anchors in raw_environment_allowlist.txt
for tools/yams-cli/main.cpp (shifted by 11c39157) so the policy suite
passes again; no new environment writes.
Not included: the external/yams-repo submodule pointer in the working
tree was a stale local clone two commits behind, not an intended bump.
Signed-off-by: trvon <me@trevon.dev>
* test(policy): make the configuration-authority ratchet bidirectional
check_configuration_authority.py only reported keys and TOML readers
present in code but missing from the allowlists, so the reviewed
YAMS_* surface could grow but never shrink: production_environment_keys.txt
even said removals "may remain grandfathered". An allowlist that only
accretes stops describing the code.
The policy now also fails on allowlist entries with no remaining
occurrence in src/, include/, or tools/, naming the lines to delete.
Key removals in later commits therefore have to update the allowlist in
the same change, and the count in the success banner becomes a real
measure of the environment surface (432 today).
stale_entries() exposes the two stale sets for tooling; three unit
tests cover stale key, stale reader, and the returned names.
Signed-off-by: trvon <me@trevon.dev>
* tools(policy): report where every configuration key is referenced
Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in
production_environment_keys.txt and each dotted TOML key literally
resolved in ConfigResolver.cpp, count whole-token references per tree
bucket (src, test, bench, docs, plugin; build output and the allowlists
themselves excluded) and assign a verdict: KEEP when anything outside
product code exercises or documents the key, CANDIDATE when only
product code mentions it, NOT_IN_SRC when the allowlist is stale.
The report is evidence, not a decision. From here on a key deletion
must paste its CANDIDATE rows into the change description; the first
audit pass hand-listed keys that turned out to have bench and test
consumers, which is exactly the mistake a mechanical inventory avoids.
Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys
(103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit.
The report counts only files git tracks (git ls-files) when the root is a
work tree, so an ignored local bench harness cannot turn a CANDIDATE into a
KEEP on one machine and not another; outside a work tree it falls back to
scanning the directory, which is what the synthetic-tree test exercises.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Push-triggered builds (local pre-push gate and GitHub Actions Tests)
ran meson compile without a job cap, saturating every host/runner core.
Default to -j4; keep the memory-sensitive linux-arm64 lane at 2.
Override the local lane with YAMS_MACOS_NINJA_JOBS.
* refactor(tuning): prune dormant runtime knobs
Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.
* refactor(vector): retire dormant backends
Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.
Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.
* refactor(plugins): remove legacy SDK tooling
Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.
* build(vector): remove stale FAISS overrides
* refactor(ipc): remove dormant FSM snapshots
* refactor(ingest): unify bounded KG enqueue
* fix(plugins): harden allocation contracts
Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).
* fix(storage): define empty object reads
Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.
* refactor(storage): trim through string views
* test: prune dormant harness helpers
Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.
* chore(cpp): initialize analyzer-facing locals
* build(simeon): advance analyzer cleanup
* refactor(config): drop dead daemon client keys
* refactor(ipc): inject daemon client transport
Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.
* refactor(retrieval): route CLI through daemon
Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.
* refactor(graph): move CLI scope into daemon
Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.
* build: isolate test-only runtime controls
* refactor(metadata): decouple lock telemetry
* fix(services): return on request timeout
* fix(retrieval): enforce workspace scope
* fix(graph): scope nodes before pagination
* fix(grep): drop binary result lines
* fix(daemon): bound checkpoint wait
* fix(repair): use KG health checks
* fix(storage): always verify compressed CRC
* perf(vector): skip unchanged PQ snapshots
* build(zyp): exclude bench from default build
* build: protect Meson files from formatter
* refactor: remove unreachable production APIs
* refactor(daemon): remove dormant runtime state
* fix(vector): harden daemon provider calls
Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.
* refactor(storage): remove unusable adapters
* fix(cli): unify in-process add requests
* fix(tsan): preserve async worker progress
* test(cli): isolate daemon-dependent suites
* fix(repair): expose queue backoff progress
