Startup ordering defects were only observable by waiting minutes on a live
daemon, and the one that aborted it took a live restart loop to find. Give the
startup contracts a deterministic harness that runs in milliseconds.
+ startup_fuzz_support.h drives the real ServiceManagerFsm and the real
DeferralGate with virtual time: legal phase advances, injected failure,
shutdown and terminal reset, idle/busy, pressure/calm, starvation deadlines
and cancellation
+ oracles fail closed on shutdown resurrection without reset, failure-to-ready,
eligibility disagreeing with fsm.isReady(), admission outside the virtual
deadline, and ineligible work touching the admission gate; fuzzed IEEE-754
components must never yield non-finite or out-of-range scores
+ catch2_startup_replay runs named regressions plus 1024 seeded traces per run
and reports the root seed and action offset for replay
+ fuzz_startup builds the same harness for libFuzzer/AFL, with a seed generator
that refuses to overwrite different existing files
- the fuzzing lane no longer defaults to TSan: harnesses link
-fsanitize=fuzzer,address,undefined and the two runtimes cannot share a
process; combining them now fails with an explanation instead of producing a
binary that cannot start
- documented the lane invocation, target naming and the coverage limits,
including that governor call ordering inside the poller is not covered
Scope: contract-valid single-owner schedules. Executor interleavings, late
callbacks, concurrent FSM instances and filesystem/model faults are not modeled.
Signed-off-by: trvon <git@trevon.dev>
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.
Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
the old default), conan-generators-dir, cli11-cmake-path,
onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
integration-smoke-timeout, integration-daemon-timeout and stress-iters:
replaced by the same constants at their call sites (smoke keeps its
test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
folded into enable-integration-tests. Debug builds still register every
integration suite; nothing set the sub-toggles.
Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).
Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).
Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.
Signed-off-by: trvon <me@trevon.dev>
`./setup.sh Profiling` used Debug as its base build type, so Conan built
every dependency (spdlog, fmt, SQLite, Boost, protobuf, ...) unoptimized
while YAMS itself was -O2. Zone timings and allocation profiles from that
tree misattribute time to dependency code that is fast in shipped
binaries.
Profiling now uses the Release Conan profile and the Release Meson
configuration (-O3 -g, NDEBUG, PIE) plus Tracy, and keeps frame pointers
(-fno-omit-frame-pointer, -mno-omit-leaf-frame-pointer when supported)
for Tracy/perf/Instruments call stacks. TSan stays off for Profiling, as
before; --tsan still opts in.
Configure options, build/profiling on grim:
before: conan build_type=Debug; buildtype=debugoptimized (-O2 -g),
b_ndebug=true, enable-profiling=true, enable-tsan=false,
no frame-pointer flags
after: conan build_type=Release; buildtype=release, debug=true (-O3 -g),
b_ndebug=true, b_pie=true, enable-profiling=true,
enable-tsan=false, b_sanitize=none, -fno-omit-frame-pointer
Signed-off-by: trvon <me@trevon.dev>
Release used buildtype=debugoptimized (-O2 -g, strip=false) and relied on
scripts/prune-runtime-install.sh to strip the staged payload, discarding
the debug info. Crash reports from shipped binaries could not be
symbolicated and the build type label did not match what shipped.
- setup.sh/setup.ps1 Release and the Meson project default use
buildtype=release. setup.sh adds -Ddebug=true for Release so -O3 code
keeps DWARF for the split, and passes -Db_pie=true explicitly (also a
project default) so reconfigured build dirs pick it up. Windows keeps
release without -g so no PDBs land in the MSI stage. Debug/Fuzzing are
unchanged (-O0 -g, assertions on).
- scripts/split-debug-symbols.sh replaces prune-runtime-install.sh: ELF
files get objcopy --only-keep-debug (compressed) into <debug-dir>/
<path>.debug plus .build-id/xx/yyyy.debug links, strip
--strip-unneeded and --add-gnu-debuglink; Mach-O files get dsymutil
dSYMs and strip -S -x. Without a debug dir it only strips.
- The release job splits into $BUILD_DIR/debug-symbols and uploads it as
a debug-symbols-<suffix> workflow artifact (30 days) instead of
shipping or dropping it. build-deb.sh, the Dockerfile, PKGBUILD and
build-arch-pkg.sh call the split script in strip-only mode.
Measured on grim (Linux x86_64, clang 21, stripped staged payload):
file debugoptimized release -O3 delta
bin/yams-cli 35,528,968 36,733,304 +3.4%
bin/yams-daemon 29,731,312 30,685,792 +3.2%
bin/yams-mcp-server 19,504,928 19,865,472 +1.8%
libyams_onnx_resource.so 477,280 481,488 +0.9%
libyams_object_storage_s3.so 8,226,608 8,239,016 +0.2%
yams_zyp.so 79,936 80,040 +0.1%
Debug symbols: 309 MiB (compressed .debug files); gdb resolves
`info line main` on the stripped daemon through the .build-id tree.
check_install_surface: no debug sections or .symtab remain; the
remaining violations are the export leaks fixed later in this series.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
build-arch-pkg.sh and the Arch PKGBUILD export YAMS_CONAN_ARCH from the
pacman/uname label (aarch64), which Conan rejects, failing the Arch
aarch64 release lane at conan install.
With -Dbuild-fuzzers=true every object carries ASan/UBSan and
SanitizerCoverage, whose symbols only the fuzzer executable provides, but
Meson's default b_lundef=true links shared libraries such as
libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to
link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for
Fuzzing builds and warn when a libFuzzer configuration keeps it on.
./setup.sh Fuzzing runs Conan with the Debug build type, which writes its
toolchain under build/fuzzing/build-debug, but setup only looked under
build-fuzzing and failed with 'Conan meson toolchain file not found'.
This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug
as the Profiling mode already does.
Default builds no longer pull onnxruntime: conan enable_onnx and the conan
profiles default to False, meson enable-onnx defaults to disabled and skips
onnxruntime discovery entirely, and setup.sh/setup.ps1 only enable ONNX
(and the ONNX-backed Glint plugin) with --with-onnx / -WithOnnx.
YAMS_DISABLE_ONNX=true still forces it off.
This removes a build blocker on hosts where conan tries to build
onnxruntime from source (e.g. ROCm/MIGraphX on Linux). Release, Docker, and
Arch packaging pass the flag explicitly, so shipped artifacts are unchanged
until ONNX is dropped.
./setup.sh Profiling configured Meson as buildtype=debug (-O0) and turned
TSAN on by default, so Tracy zone timings described unoptimized,
instrumented code. Use debugoptimized and leave TSAN off unless --tsan is
passed.
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py.
- Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile.
- Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests.
- Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test
fixtures to run symbol-specific test cases against schema v39.
Signed-off-by: trvon <me@trevon.dev>
Remove legacy Tree-sitter symbol extractor plugin (plugins/symbol_extractor_treesitter),
the unused plugin-symbols build option, and associated tests and benchmark targets:
- Drop plugins/symbol_extractor_treesitter/ (grammar_loader, symbol_extractor, plugin_api).
- Remove plugin-symbols Meson option and packaging references.
- Remove dedicated symbol plugin tests and benchmark targets (symbol_extraction_bench,
symbol_extractor_all_langs_catch2_test, symbol_extractor_edge_cases_catch2_test,
symbol_extractor_solidity_catch2_test, symbol_extractor_query_test,
symbol_extractor_validation_test).
- Clean up dead symbol extraction plugin helper and test from hnsw_entity_search_bench.
- Prune obsolete portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter
Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close
* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler
* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly
ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.
* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict
setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.
* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build
* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs
* fix(build): restore hooks and shrink srht
Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.
Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.
Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.
* refactor(storage): drop dead code, add coverage
* refactor(cli): polish grep/search graph UX
* fix(daemon): stabilize shutdown and CI paths
- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work
* test(daemon): add db vacuum coverage
* test(daemon): add WAL startup recovery
- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown
* refactor(grep): consolidate indexed content scan
* fix(grep): remove pre-scan candidate truncation
* refactor(runtime): split startup, harden storage
* feat(bench): add manifest retrieval corpus
* fix(bench): run fusion experiment on simeon
* fix(cli): sharpen search graph hints
* fix(build): refresh version metadata
* fix(ci): stabilize sync search and srht packaging
* fix(ci): unblock windows and shrink artifacts
* fix(ci): stabilize linux runtime checks
* fix(build): shrink srht manifest
