* perf(daemon): coalesce metadata writes
* refactor(metadata): split repository ops
* fix(metadata): harden repository invariants
* feat: test coverage and continued refactor work in metadata code
* fix(graph): handle path-like explore queries
- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH
* fix(daemon): quiesce shutdown background work
- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain
* fix(daemon): cancel repair work on shutdown
Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.
* perf(metadata): reduce sqlite lock contention
* perf(metadata): flatten batch content retries
* fix(daemon): align shutdown wait budgets
A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.
* perf(metadata): bound feedback write waits
* perf(metadata): prep batch content before tx
* fix(daemon): honor no-embeddings tasks
Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.
* perf(daemon): batch fallback metadata writes
* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements
* perf(metadata): flatten insert retries
* fix(metadata): clear CI portability blockers
Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.
* perf(metadata): batch writes and steady bench
* fix(daemon): skip inactive post-ingest pollers
* perf(metadata): tighten tag and path caches
* perf(metadata): cache small batch upserts
* fix(daemon): harden embedded host teardown
* feat(docs): updating docs policies and language
* perf(metadata): tune doc update retries
* perf(metadata): bound download retries
* perf(metadata): bound download metadata waits
* bench(metadata): scope post-ingest content path
* fix(daemon): export embedded test hooks
* fix(daemon): harden test helper linkage
* fix(build): add missing standard includes
* fix(core): add assertions and harden daemon lifecycle
* fix(daemon): tighten shutdown lifecycle invariants
* fix(test): add PressureLimitedPoller capability-sleep shutdown test
* refactor(vector): harden backend contracts
Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.
Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.
* fix(vector): address audit findings plus hardening
Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.
Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.
Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.
* fix(test): add PressureLimitedPoller wake-timer shutdown tests
* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec
Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.
Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.
* fix(daemon): harden PressureLimitedPoller inFlightCounter null check
* fix(daemon): harden PostIngestQueue wait_until deadline checks
* fix(daemon): clean up recovery sentinels regardless of salvage count
Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.
Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.
* feat(vec0): restore persisted ANN index on daemon restart
After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index. When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.
Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.
* fix: enable DCHECK by default and assert metadata counter consistency
* fix(services): assert document deletion ordering content-before-metadata
* fix(test): add fork-based assertion validation tests
* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE
* fix(ci): enable -Denable-dcheck=true in all test CI lanes
* fix(test): address assertion review findings
* refactor(daemon): isolate poller config assertion checks
* refactor(services): reduce deletion and poller complexity
* style(ci): normalize test workflow spacing
* refactor(services): extract list query helpers
* refactor(services): extract list hydration helpers
* refactor(daemon): extract metrics snapshot helpers
* refactor(daemon): extract metrics counter snapshot
* refactor(daemon): extract metrics resource snapshot
* feat(graph): add daemon-backed explore
Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.
Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.
* fix(services): prevent prune delete metadata orphans
* fix(cli): honor data-dir in graph explore
- bypass daemon explore when the CLI pins a data dir so\n temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n context
* fix(daemon): make lifecycle stop completion safe
* fix(daemon): gate db-open startup against shutdown
* fix(metadata): serialize FTS indexed counter updates
* test(daemon): simulate vector lock contention cross-process
* test(daemon): isolate post-ingest identity batch
* style(daemon): normalize test include order
* refactor(graph): split cli graph helpers
Keep graph command paths smaller and easier to evolve.
Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.
* fix(config): harden expand_tilde and fix Windows XDG state path doubling
- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment
* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax
* fix(static-analysis): strip trailing text from nosemgrep comments
* fix(doctor): add logging to daemon_check empty catch blocks
* fix(db_repair): add logging to empty catch blocks
* fix(chunking): add precondition checks for rabin polynomial and windowSize
* fix(safety): add null precondition checks for Fold proved null-deref sites
* fix(daemon): widen init-failure dispatch, add null/dbz preconditions
InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.
* ci(ubsan): use restricted build targets to avoid runner timeout
* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV
* fix(config): cast parseTomlU32 to size_t for batch fields
* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
* fix(storage): add MutexPool and hash-length precondition assertions
* fix(storage): add reference counter lifecycle assertions
- YAMS_PRECONDITION on increment size constraints
- YAMS_DCHECK on decrement zero-row detection
- YAMS_ASSERT on transaction AUTOINCREMENT ID
- YAMS_PRECONDITION on transaction active guard
- YAMS_POSTCONDITION on commit/rollback state
* fix(storage): add atomic-write and store postcondition assertions
* fix(ingestion): add batch concurrency and accounting assertions
* fix(storage): add remove counter and retrieve-read DCHECKs
* fix(daemon): add WorkCoordinator double-start precondition
* bench(storage): add store/retrieve throughput microbenchmark
* fix(daemon): replace WorkCoordinator double-start throw with precondition
* test(daemon): add fork-based double-start precondition test
* test(ingestion): add batch concurrency edge-case tests
* fix(storage): remove overly-aggressive decrement DCHECK, fix transaction test
* test(daemon): fork-based precondition test — known Catch2 SIGABRT artifact
Catch2 v3.12.0 signal handler catches child SIGABRT via abort(),
marking test as FAILED even though CHECK passes correctly.
Affects all fork-based assertion tests. Test logic is correct.
* refactor(test): remove duplicate fork test, defer to assert_catch2
* fix(review): downgrade transactionId to DCHECK, use unique benchmark hashes
* fix(review): document PRECONDITION+throw pattern, downgrade exists() to DCHECK
* test(ingestion): add maxConcurrent early-return branch coverage
* perf(storage): eliminate zero-init in retrieveRaw, fix benchmark build
retrieveRaw bottleneck: 73% of samples in vector<std::byte>(N) zero-init
that gets overwritten by file.read(). Use make_unique_for_overwrite (C++20)
to allocate uninitialized, then assign into vector.
Before/after (TSan, 1MB Retrieve): 41.7ms -> 9.0ms (4.6x faster)
Also fix Google Benchmark build: add benchmark_main_dep (provides main())
to all 7 benchmark targets. Fix storage benchmark to use valid hex hashes.
0 assertion violations across all benchmark runs.
* perf(storage): skip redundant temp-dir check, use hex table for names
Two micro-optimizations in getTempPath() hot path:
1. Drop ensureDirectories() call — temp dir already created at ctor,
stat() on every store was wasted work.
2. Replace 16 fmt::format("{:x}") calls with pre-computed hex char
table + push_back. Avoids format machinery per store.
0 opengrep findings.
* assert(storage): DCHECK temp dir, buffer, size invariants in hot paths
Coverage for the three changed functions:
getTempPath:
- DCHECK temp dir still exists (catch external cleanup in dev/CI)
- DCHECK temp name length matches TEMP_NAME_LENGTH
atomicWrite:
- DCHECK temp file exists after successful ofstream write
retrieveRaw:
- DCHECK buffer non-null for positive file sizes
- DCHECK assigned vector size matches fileSize
All benchmarks pass with zero assertion violations (TSan + Release).
* style(storage): fmt wrap DCHECK lines
* fix(review): downgrade PRECONDITION→DCHECK for catchable throws, guard zero-size UB
getObjectPath: YAMS_PRECONDITION always aborts — throw below is dead code.
Use YAMS_DCHECK instead (aborts in dev/CI, compiles out in release so
throw fires for backward-compatible catchable std::invalid_argument).
retrieveRaw: fileSize==0 → make_unique_for_overwrite(0) may return null,
assign(null,null) is UB. Early-return empty RawObject for zero-size files.
meson.build: guard benchmark blocks with benchmark_main_dep.found() too.
* fix(review): drop PRECONDITION dead throw, fix test comments
reference_counter: downgrade YAMS_PRECONDITION→DCHECK for transaction
active_ guard (aborts dev/CI, throw fires in release). Add matching
DCHECKs to decrement() and pruneReference() for consistency.
Fix inaccurate test comments claiming fork-test coverage for
WorkCoordinator double-start invariant.
* fix(review): DCHECK store postcondition, cast fileSize, rm dup main
store: downgrade YAMS_POSTCONDITION→DCHECK for exists() check.
Always-on POSTCONDITION adds stat() syscall to hot store() path;
debug-only DCHECK is the correct tier.
retrieveRaw: explicit static_cast<size_t>(fileSize) in assign
to avoid signed/unsigned narrowing warnings.
meson.build: remove benchmark_main_dep from metadata_query_bench_deps.
metadata_path_query_bench.cpp and tree_list_filter_bench.cpp both
define BENCHMARK_MAIN() — linking benchmark_main would be duplicate.
WorkCoordinator.h: update @throws→@pre in doxygen comment to match
YAMS_PRECONDITION(!started_) behavior.
* fix(bench): move pre-store to SetUp, PauseTiming unreliable
PauseTiming before first benchmark iteration caused single-iteration
runs in some Google Benchmark versions. Move pre-store into Fixture::SetUp
where it runs outside the timing loop naturally.
* fix(bench): check store/retrieve results, fix comment copy-vs-move
SetUp: throw if pre-store fails instead of silently continuing.
Retrieve: SkipWithError if retrieveRaw returns error.
Comment: assign() copies, not moves — fix misleading wording.
* fix(bench): rm benchmark_main_dep from targets with own main
ingestion_e2e_bench, retrieval_quality_bench, fusion_strategy_experiment
all define their own main() — linking benchmark_main_dep would cause
duplicate symbol at link time.
* fix(test): rename work_coordinator section, no longer tests double-start
Section title "Rejects start when already running" implied
double-start coverage but only checked worker count after first
start. Renamed to "Starts with correct worker count".
* test(refcount): add closed-state throw coverage for increment/decrement
After commit, increment/decrement must fail:
- Dev/CI (DCHECK): YAMS_DCHECK aborts, covered by fork tests
- Release (NDEBUG): DCHECK compiles out, throw fires
Conditional REQUIRE_THROWS_AS for release path.
* fix(bench): split meson guard for own-main vs benchmark_main targets
Targets with BENCHMARK_MAIN (metadata_path_query, tree_list_filter,
retrieval_quality, fusion_experiment, ingestion_e2e) only need
benchmark_dep. Storage/retrieval benchmarks need benchmark_main_dep.
Split into two guarded blocks to avoid skipping own-main targets
when benchmark_main is unavailable.
* fix(refcount): downgrade increment size PRECONDITION to DCHECK+error return
compressedSize>0 and uncompressedSize>=compressedSize come from user
content (block sizes). PRECONDITION would abort the process on corrupted
or malicious blocks. Return ErrorCode::InvalidArgument instead so
callers can handle gracefully.
* fix(daemon): defer startup maintenance
Make metadata repository readiness the startup boundary while moving search build, recovery cleanup, salvage, and VACUUM into coordinated background or pre-publication paths.\n\nKeep status honest for operators by separating database readiness from maintenance phase and surfacing recovery artifacts.
* fix(core): add assertions and harden daemon lifecycle
* fix(daemon): tighten shutdown lifecycle invariants
* fix(test): add PressureLimitedPoller capability-sleep shutdown test
* refactor(vector): harden backend contracts
Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.
Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.
* fix(vector): address audit findings plus hardening
Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.
Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.
Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.
* fix(test): add PressureLimitedPoller wake-timer shutdown tests
* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec
Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.
Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.
* fix(daemon): harden PressureLimitedPoller inFlightCounter null check
* fix(daemon): harden PostIngestQueue wait_until deadline checks
* fix(daemon): clean up recovery sentinels regardless of salvage count
Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.
Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.
* feat(vec0): restore persisted ANN index on daemon restart
After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index. When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.
Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.
* fix: enable DCHECK by default and assert metadata counter consistency
* fix(services): assert document deletion ordering content-before-metadata
* fix(test): add fork-based assertion validation tests
* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE
* fix(ci): enable -Denable-dcheck=true in all test CI lanes
* fix(test): address assertion review findings
* refactor(daemon): isolate poller config assertion checks
* refactor(services): reduce deletion and poller complexity
* style(ci): normalize test workflow spacing
* refactor(services): extract list query helpers
* refactor(services): extract list hydration helpers
* refactor(daemon): extract metrics snapshot helpers
* refactor(daemon): extract metrics counter snapshot
* refactor(daemon): extract metrics resource snapshot
* feat(graph): add daemon-backed explore
Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.
Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.
* fix(services): prevent prune delete metadata orphans
* fix(cli): honor data-dir in graph explore
- bypass daemon explore when the CLI pins a data dir so\n temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n context
* fix(daemon): make lifecycle stop completion safe
* fix(daemon): gate db-open startup against shutdown
* fix(metadata): serialize FTS indexed counter updates
* test(daemon): simulate vector lock contention cross-process
* test(daemon): isolate post-ingest identity batch
* style(daemon): normalize test include order
* refactor(graph): split cli graph helpers
Keep graph command paths smaller and easier to evolve.
Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.
* fix(config): harden expand_tilde and fix Windows XDG state path doubling
- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment
* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax
* fix(static-analysis): strip trailing text from nosemgrep comments
* fix(doctor): add logging to daemon_check empty catch blocks
* fix(db_repair): add logging to empty catch blocks
* fix(chunking): add precondition checks for rabin polynomial and windowSize
* fix(safety): add null precondition checks for Fold proved null-deref sites
* fix(daemon): widen init-failure dispatch, add null/dbz preconditions
InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.
* ci(ubsan): use restricted build targets to avoid runner timeout
* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV
* fix(config): cast parseTomlU32 to size_t for batch fields
* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter
Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close
* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler
* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly
ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.
* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict
setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.
* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build
* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs
* fix(build): restore hooks and shrink srht
Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.
Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.
Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.
* refactor(storage): drop dead code, add coverage
* refactor(cli): polish grep/search graph UX
* fix(daemon): stabilize shutdown and CI paths
- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work
* test(daemon): add db vacuum coverage
* test(daemon): add WAL startup recovery
- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown
* refactor(grep): consolidate indexed content scan
* fix(grep): remove pre-scan candidate truncation
* refactor(runtime): split startup, harden storage
* feat(bench): add manifest retrieval corpus
* fix(bench): run fusion experiment on simeon
* fix(cli): sharpen search graph hints
* fix(build): refresh version metadata
* fix(ci): stabilize sync search and srht packaging
* fix(ci): unblock windows and shrink artifacts
* fix(ci): stabilize linux runtime checks
* fix(build): shrink srht manifest
