Commit graph

74 commits

Author SHA1 Message Date
trvon
d152b5697d
chore(tooling): mark dependency-managed includes 2026-08-19 08:43:17 -06:00
trvon
a83897eeb9
test(ci): normalize Windows expectations 2026-08-19 05:25:52 -06:00
trvon
5a24effbae
fix(daemon): harden async shutdown 2026-08-17 21:43:12 -06:00
trvon
be24d07c27
fix(ci): stabilize sanitizer validation
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
2026-08-08 11:42:33 -04:00
trvon
30618eb113
refactor(config): consolidate runtime tuning 2026-08-05 20:53:18 -04:00
trvon
8b501a344e
refactor(config): consolidate embedding policy 2026-08-05 20:53:18 -04:00
trvon
b319f0de6f
perf(daemon): trust clean metadata restarts 2026-08-03 17:52:58 -04:00
trvon
d03ffd07b3
refactor(plugins): centralize lifecycle policy 2026-08-03 17:52:58 -04:00
Trevon
0dd3985db4
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs

Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.

* refactor(vector): retire dormant backends

Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.

Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.

* refactor(plugins): remove legacy SDK tooling

Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.

* build(vector): remove stale FAISS overrides

* refactor(ipc): remove dormant FSM snapshots

* refactor(ingest): unify bounded KG enqueue

* fix(plugins): harden allocation contracts

Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).

* fix(storage): define empty object reads

Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.

* refactor(storage): trim through string views

* test: prune dormant harness helpers

Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.

* chore(cpp): initialize analyzer-facing locals

* build(simeon): advance analyzer cleanup

* refactor(config): drop dead daemon client keys

* refactor(ipc): inject daemon client transport

Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.

* refactor(retrieval): route CLI through daemon

Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.

* refactor(graph): move CLI scope into daemon

Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.

* build: isolate test-only runtime controls

* refactor(metadata): decouple lock telemetry

* fix(services): return on request timeout

* fix(retrieval): enforce workspace scope

* fix(graph): scope nodes before pagination

* fix(grep): drop binary result lines

* fix(daemon): bound checkpoint wait

* fix(repair): use KG health checks

* fix(storage): always verify compressed CRC

* perf(vector): skip unchanged PQ snapshots

* build(zyp): exclude bench from default build

* build: protect Meson files from formatter

* refactor: remove unreachable production APIs

* refactor(daemon): remove dormant runtime state

* fix(vector): harden daemon provider calls

Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.

* refactor(storage): remove unusable adapters

* fix(cli): unify in-process add requests

* fix(tsan): preserve async worker progress

* test(cli): isolate daemon-dependent suites

* fix(repair): expose queue backoff progress
2026-07-31 14:16:45 -04:00
trvon
e290deb944
test(daemon): isolate sanitizer lifecycle fixtures 2026-07-29 22:30:32 -04:00
trvon
91fb1a32d9
fix(rm): make deletion safe and observable 2026-07-29 10:46:44 -04:00
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
Trevon
090a5ca69c
fix(testing): stabilize TSAN coverage, harden daemon shutdown, and speed up KG/storage paths (#53)
* perf(search, metadata): add KG write buffer, score cache, adaptive budgets

* perf(indexing): wire KGWriteBuffer into indexing service

   Converts per-document addEdgesUnique calls into batched flush via
   KGWriteBuffer.  500-doc ingest: 500 SQL round-trips → 1 batch.

   Verified: 14.2× edge insert gap (14.8K→195K edges/s) in kg_edge bench.
   Tests: 10/10 Catch2 pass (105 assertions).

* perf(storage): add fsync durability, parallel batch, read integrity

* test(storage): add stress, crash-sim, cold-cache, batch benchmarks

* fix: CI optimizations and work coordinator shutdown fix

* fix(testing): stabilize TSAN coverage gate

* fix: harden KG flush, score cache, and benches

   - keep KG buffers intact when batch commit fails
   - count flushed edges per batch instead of lifetime totals
   - guard the Simeon score cache and clear it on rebuild
   - fall back to fsync when F_FULLFSYNC fails on macOS
   - remove interactive sudo from cold-cache benchmarks
2026-06-26 17:57:02 -04:00
trvon
a21dc6ae02
fix: test fixes for ci
Signed-off-by: trvon <me@trevon.dev>
2026-06-21 19:57:02 -04:00
Trevon
5bda4d77a2
perf+assert(storage): eliminate zero-init, temp-path overhead, add DCHECK invariants (#42)
* fix(core): add assertions and harden daemon lifecycle

* fix(daemon): tighten shutdown lifecycle invariants

* fix(test): add PressureLimitedPoller capability-sleep shutdown test

* refactor(vector): harden backend contracts

Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.

Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.

* fix(vector): address audit findings plus hardening

Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.

Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.

Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.

* fix(test): add PressureLimitedPoller wake-timer shutdown tests

* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec

Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.

Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.

* fix(daemon): harden PressureLimitedPoller inFlightCounter null check

* fix(daemon): harden PostIngestQueue wait_until deadline checks

* fix(daemon): clean up recovery sentinels regardless of salvage count

Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.

Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.

* feat(vec0): restore persisted ANN index on daemon restart

After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index.  When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.

Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.

* fix: enable DCHECK by default and assert metadata counter consistency

* fix(services): assert document deletion ordering content-before-metadata

* fix(test): add fork-based assertion validation tests

* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE

* fix(ci): enable -Denable-dcheck=true in all test CI lanes

* fix(test): address assertion review findings

* refactor(daemon): isolate poller config assertion checks

* refactor(services): reduce deletion and poller complexity

* style(ci): normalize test workflow spacing

* refactor(services): extract list query helpers

* refactor(services): extract list hydration helpers

* refactor(daemon): extract metrics snapshot helpers

* refactor(daemon): extract metrics counter snapshot

* refactor(daemon): extract metrics resource snapshot

* feat(graph): add daemon-backed explore

Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.

Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.

* fix(services): prevent prune delete metadata orphans

* fix(cli): honor data-dir in graph explore

- bypass daemon explore when the CLI pins a data dir so\n  temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n  reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n  context

* fix(daemon): make lifecycle stop completion safe

* fix(daemon): gate db-open startup against shutdown

* fix(metadata): serialize FTS indexed counter updates

* test(daemon): simulate vector lock contention cross-process

* test(daemon): isolate post-ingest identity batch

* style(daemon): normalize test include order

* refactor(graph): split cli graph helpers

Keep graph command paths smaller and easier to evolve.

Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.

* fix(config): harden expand_tilde and fix Windows XDG state path doubling

- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
  on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
  to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment

* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax

* fix(static-analysis): strip trailing text from nosemgrep comments

* fix(doctor): add logging to daemon_check empty catch blocks

* fix(db_repair): add logging to empty catch blocks

* fix(chunking): add precondition checks for rabin polynomial and windowSize

* fix(safety): add null precondition checks for Fold proved null-deref sites

* fix(daemon): widen init-failure dispatch, add null/dbz preconditions

InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.

* ci(ubsan): use restricted build targets to avoid runner timeout

* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV

* fix(config): cast parseTomlU32 to size_t for batch fields

* fix(ci): use static helper for Windows timer leak, restrict ubsan tests

* fix(storage): add MutexPool and hash-length precondition assertions

* fix(storage): add reference counter lifecycle assertions

- YAMS_PRECONDITION on increment size constraints
- YAMS_DCHECK on decrement zero-row detection
- YAMS_ASSERT on transaction AUTOINCREMENT ID
- YAMS_PRECONDITION on transaction active guard
- YAMS_POSTCONDITION on commit/rollback state

* fix(storage): add atomic-write and store postcondition assertions

* fix(ingestion): add batch concurrency and accounting assertions

* fix(storage): add remove counter and retrieve-read DCHECKs

* fix(daemon): add WorkCoordinator double-start precondition

* bench(storage): add store/retrieve throughput microbenchmark

* fix(daemon): replace WorkCoordinator double-start throw with precondition

* test(daemon): add fork-based double-start precondition test

* test(ingestion): add batch concurrency edge-case tests

* fix(storage): remove overly-aggressive decrement DCHECK, fix transaction test

* test(daemon): fork-based precondition test — known Catch2 SIGABRT artifact

Catch2 v3.12.0 signal handler catches child SIGABRT via abort(),
marking test as FAILED even though CHECK passes correctly.
Affects all fork-based assertion tests. Test logic is correct.

* refactor(test): remove duplicate fork test, defer to assert_catch2

* fix(review): downgrade transactionId to DCHECK, use unique benchmark hashes

* fix(review): document PRECONDITION+throw pattern, downgrade exists() to DCHECK

* test(ingestion): add maxConcurrent early-return branch coverage

* perf(storage): eliminate zero-init in retrieveRaw, fix benchmark build

retrieveRaw bottleneck: 73% of samples in vector<std::byte>(N) zero-init
that gets overwritten by file.read(). Use make_unique_for_overwrite (C++20)
to allocate uninitialized, then assign into vector.

Before/after (TSan, 1MB Retrieve): 41.7ms -> 9.0ms (4.6x faster)

Also fix Google Benchmark build: add benchmark_main_dep (provides main())
to all 7 benchmark targets. Fix storage benchmark to use valid hex hashes.

0 assertion violations across all benchmark runs.

* perf(storage): skip redundant temp-dir check, use hex table for names

Two micro-optimizations in getTempPath() hot path:
1. Drop ensureDirectories() call — temp dir already created at ctor,
   stat() on every store was wasted work.
2. Replace 16 fmt::format("{:x}") calls with pre-computed hex char
   table + push_back. Avoids format machinery per store.

0 opengrep findings.

* assert(storage): DCHECK temp dir, buffer, size invariants in hot paths

Coverage for the three changed functions:

getTempPath:
  - DCHECK temp dir still exists (catch external cleanup in dev/CI)
  - DCHECK temp name length matches TEMP_NAME_LENGTH

atomicWrite:
  - DCHECK temp file exists after successful ofstream write

retrieveRaw:
  - DCHECK buffer non-null for positive file sizes
  - DCHECK assigned vector size matches fileSize

All benchmarks pass with zero assertion violations (TSan + Release).

* style(storage): fmt wrap DCHECK lines

* fix(review): downgrade PRECONDITION→DCHECK for catchable throws, guard zero-size UB

getObjectPath: YAMS_PRECONDITION always aborts — throw below is dead code.
Use YAMS_DCHECK instead (aborts in dev/CI, compiles out in release so
throw fires for backward-compatible catchable std::invalid_argument).

retrieveRaw: fileSize==0 → make_unique_for_overwrite(0) may return null,
assign(null,null) is UB. Early-return empty RawObject for zero-size files.

meson.build: guard benchmark blocks with benchmark_main_dep.found() too.

* fix(review): drop PRECONDITION dead throw, fix test comments

reference_counter: downgrade YAMS_PRECONDITION→DCHECK for transaction
active_ guard (aborts dev/CI, throw fires in release). Add matching
DCHECKs to decrement() and pruneReference() for consistency.

Fix inaccurate test comments claiming fork-test coverage for
WorkCoordinator double-start invariant.

* fix(review): DCHECK store postcondition, cast fileSize, rm dup main

store: downgrade YAMS_POSTCONDITION→DCHECK for exists() check.
Always-on POSTCONDITION adds stat() syscall to hot store() path;
debug-only DCHECK is the correct tier.

retrieveRaw: explicit static_cast<size_t>(fileSize) in assign
to avoid signed/unsigned narrowing warnings.

meson.build: remove benchmark_main_dep from metadata_query_bench_deps.
metadata_path_query_bench.cpp and tree_list_filter_bench.cpp both
define BENCHMARK_MAIN() — linking benchmark_main would be duplicate.

WorkCoordinator.h: update @throws→@pre in doxygen comment to match
YAMS_PRECONDITION(!started_) behavior.

* fix(bench): move pre-store to SetUp, PauseTiming unreliable

PauseTiming before first benchmark iteration caused single-iteration
runs in some Google Benchmark versions. Move pre-store into Fixture::SetUp
where it runs outside the timing loop naturally.

* fix(bench): check store/retrieve results, fix comment copy-vs-move

SetUp: throw if pre-store fails instead of silently continuing.
Retrieve: SkipWithError if retrieveRaw returns error.
Comment: assign() copies, not moves — fix misleading wording.

* fix(bench): rm benchmark_main_dep from targets with own main

ingestion_e2e_bench, retrieval_quality_bench, fusion_strategy_experiment
all define their own main() — linking benchmark_main_dep would cause
duplicate symbol at link time.

* fix(test): rename work_coordinator section, no longer tests double-start

Section title "Rejects start when already running" implied
double-start coverage but only checked worker count after first
start. Renamed to "Starts with correct worker count".

* test(refcount): add closed-state throw coverage for increment/decrement

After commit, increment/decrement must fail:
- Dev/CI (DCHECK): YAMS_DCHECK aborts, covered by fork tests
- Release (NDEBUG): DCHECK compiles out, throw fires

Conditional REQUIRE_THROWS_AS for release path.

* fix(bench): split meson guard for own-main vs benchmark_main targets

Targets with BENCHMARK_MAIN (metadata_path_query, tree_list_filter,
retrieval_quality, fusion_experiment, ingestion_e2e) only need
benchmark_dep. Storage/retrieval benchmarks need benchmark_main_dep.

Split into two guarded blocks to avoid skipping own-main targets
when benchmark_main is unavailable.

* fix(refcount): downgrade increment size PRECONDITION to DCHECK+error return

compressedSize>0 and uncompressedSize>=compressedSize come from user
content (block sizes). PRECONDITION would abort the process on corrupted
or malicious blocks. Return ErrorCode::InvalidArgument instead so
callers can handle gracefully.

* fix(daemon): defer startup maintenance

Make metadata repository readiness the startup boundary while moving search build, recovery cleanup, salvage, and VACUUM into coordinated background or pre-publication paths.\n\nKeep status honest for operators by separating database readiness from maintenance phase and surfacing recovery artifacts.
2026-06-04 22:44:12 -04:00
Trevon
2d01ba314c
Assertion hardening, metadata correctness, & daemon lifecycle fixes (#41)
* fix(core): add assertions and harden daemon lifecycle

* fix(daemon): tighten shutdown lifecycle invariants

* fix(test): add PressureLimitedPoller capability-sleep shutdown test

* refactor(vector): harden backend contracts

Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.

Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.

* fix(vector): address audit findings plus hardening

Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.

Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.

Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.

* fix(test): add PressureLimitedPoller wake-timer shutdown tests

* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec

Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.

Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.

* fix(daemon): harden PressureLimitedPoller inFlightCounter null check

* fix(daemon): harden PostIngestQueue wait_until deadline checks

* fix(daemon): clean up recovery sentinels regardless of salvage count

Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.

Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.

* feat(vec0): restore persisted ANN index on daemon restart

After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index.  When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.

Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.

* fix: enable DCHECK by default and assert metadata counter consistency

* fix(services): assert document deletion ordering content-before-metadata

* fix(test): add fork-based assertion validation tests

* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE

* fix(ci): enable -Denable-dcheck=true in all test CI lanes

* fix(test): address assertion review findings

* refactor(daemon): isolate poller config assertion checks

* refactor(services): reduce deletion and poller complexity

* style(ci): normalize test workflow spacing

* refactor(services): extract list query helpers

* refactor(services): extract list hydration helpers

* refactor(daemon): extract metrics snapshot helpers

* refactor(daemon): extract metrics counter snapshot

* refactor(daemon): extract metrics resource snapshot

* feat(graph): add daemon-backed explore

Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.

Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.

* fix(services): prevent prune delete metadata orphans

* fix(cli): honor data-dir in graph explore

- bypass daemon explore when the CLI pins a data dir so\n  temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n  reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n  context

* fix(daemon): make lifecycle stop completion safe

* fix(daemon): gate db-open startup against shutdown

* fix(metadata): serialize FTS indexed counter updates

* test(daemon): simulate vector lock contention cross-process

* test(daemon): isolate post-ingest identity batch

* style(daemon): normalize test include order

* refactor(graph): split cli graph helpers

Keep graph command paths smaller and easier to evolve.

Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.

* fix(config): harden expand_tilde and fix Windows XDG state path doubling

- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
  on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
  to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment

* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax

* fix(static-analysis): strip trailing text from nosemgrep comments

* fix(doctor): add logging to daemon_check empty catch blocks

* fix(db_repair): add logging to empty catch blocks

* fix(chunking): add precondition checks for rabin polynomial and windowSize

* fix(safety): add null precondition checks for Fold proved null-deref sites

* fix(daemon): widen init-failure dispatch, add null/dbz preconditions

InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.

* ci(ubsan): use restricted build targets to avoid runner timeout

* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV

* fix(config): cast parseTomlU32 to size_t for batch fields

* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
2026-06-04 12:53:42 -04:00
Trevon
b603738f44
Stabilize sanitizer lanes and daemon concurrency (#40)
* fix(repair): keep RPC progress alive

* fix(daemon): propagate DB open failure to FSM and add shutdown cancellation

* fix(db): classify FTS5 inverted-index errors as repairable, not transient locks

* fix(daemon): clean up stale corrupt files after salvage and suppress optional readiness keys

* test(db_recovery): update FTS5 classification test for new transient-lock exclusion

* test(gc): add garbage collector unit tests for collection, dry-run, scheduling

* test(repair): expand repair manager tests with hash-verified recovery and source ordering

* fix(gc): dry-run mode no longer increments blocksDeleted/bytesReclaimed

* fix(mock,mcp): add atomic bool for lock consistency, move field assignments

* style: fix unnecessary-copy, dead-store, and const-refable findings from Infer

* fix(daemon): increase default shutdown timeout from 15s to 120s

* fix(daemon): demote non-critical warnings to info (FTS5, VACUUM, WorkCoordinator timeout)

* fix(db): re-enable WAL auto-checkpoint to prevent unbounded WAL growth

* test(pool): add WAL auto-checkpoint and concurrency coverage for connection pool

* test(pool): add exhaustion, stats, lifecycle, and read-only connection pool tests

* test(pool): add tagging, WAL journal, reuse, and busy timeout tests

* test(metrics): relax session snippet assertion to tolerate stub behavior

* fix(test): stabilize sanitizer triage failures

* fix(build): wire version and zyp deps

* fix(daemon): close TSAN race clusters

* fix(test): guard optional zyp dep

* fix(test): address PR review findings
2026-06-01 12:14:43 -04:00
Trevon
bfe7cd94ca
test: expand storage-readiness coverage — 48.1%→59.8% broad, 74.3% narrow (#38)
* fix(daemon): avoid socket cancel on stop

Signal active sockets with shutdown only so pending I/O wakes without entering Boost.Asio's reactor cancel path during teardown.

* fix(metadata): open read pools read-only

* ci: fix coverage report generation

* fix(metadata): tag unscoped pool holders

* test(daemon): avoid ipc teardown takeover

* ci: skip plugin coverage objects

* fix(daemon): repair completed FTS5 ghosts

* build(vector): guard unusable FAISS headers

* ci: use clang gcov for coverage

* test(daemon): stabilize arm CI flakes

* Harden search and compression numeric config

* fix(daemon): reconcile orphan vector embeddings

* fix: address vector cleanup review comments

* chore(main): release 0.16.1

* Harden search ranking divisors

* Harden snippet distance and command registration

* test(storage): add readiness coverage gate

Document the distributed sync readiness surface and add direct adapter, refcount, and GC failure-path coverage.

* test(storage): cover strict resolver fallback

* fix(search): tighten retrieval ranking

Honor component weights in default RRF, over-fetch vector chunks before document aggregation, and persist topology centroids for sparse-guided routing.

* fix(search): route weak queries through topology

* fix(search): improve rerank and weak queries

Wire provider-backed cross reranking, add direct Simeon candidates for weak lexical pools, and report stage-level retrieval metrics.

* fix(storage): honor explicit plugin names

Add readiness coverage for factory, URL backend, and compressed storage failure paths.

* fix(storage): avoid remote size listing

* fix(api): rollback failed content adds

* test(storage): cover engine readiness

* test(storage): cover compressed readiness

* fix(core): harden delete and recovery paths

Route prune and local delete fallback through DocumentService so content, metadata, vectors, and KG cleanup stay consistent.

Treat transient SQLite quick_check lock failures as ResourceBusy, refresh IPC activity on streaming writes, and preserve reranker config in tuner seeding.

* test(storage): cover URL backend readiness

* fix(api): harden content rollback

* feat(storage): add list seam, URL listing, and write-edge coverage

* feat(api): reconcile ambiguous remote manifest writes

* feat(storage): add deep compressed scan validation

* test(storage): add plugin loader error path coverage

* fix(gc): prune reference rows after collection

* test(api,storage): add health, compact, gc, and verify edge coverage

* docs: update storage-readiness coverage to 71.8%

* test(metadata): add connection pool shutdown-while-leased coverage

* test(metadata): add tree builder snapshot root stability coverage

* test(vector): add before-init and after-close operation rejection coverage

* test(metadata): add database post-close, open-failure, and cache-clear coverage

* test(integrity): add P2P wrong-hash rejection + batch matrix resolution

* docs: audit and resolve all matrix rows — 18 fully covered

* test(metadata): add failed-migration and rollback coverage

* test(storage): add getBlockSize-invalid-key, empty-verify, PUT/DELETE retry coverage

* feat(storage): add file-open and rename failure hooks + ContentStore retrieve edge coverage

* docs: add broad-surface coverage with per-subsystem breakdown

* test(integrity): add chunk validator manifest validation coverage

* test(vector): add batch insert, delete, and update operation coverage

* docs: update integrity subsystem coverage to 46.4%

* docs: consolidate coverage report — clean matrix, remove boilerplate

* docs: update coverage with complete-suite numbers — 59.8% broad, 74.3% narrow

* test(storage): add URLBackend HEAD exists with 2xx coverage

* bump submodules

* fix: address PR review — curl_slist leak, ec ordering, CAS rollback, production hooks

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-25 22:58:15 -04:00
trvon
77cab7f6e9
fix(ci): stabilize PR checks
Signed-off-by: trvon <me@trevon.dev>
2026-05-21 16:18:08 -04:00
trvon
afe5074abb
fix(ci): stabilize test matrix
- Guard post-ingest callbacks before teardown.
- Skip unavailable vector/provider paths in CI.
- Fix Windows assumptions and trim ASAN benchmarks.
2026-05-18 16:36:21 -04:00
trvon
2d5c6625eb
fix: download logic improvements and ci test fixes. Stagging download fixes seperate from search features 2026-05-04 13:29:17 -04:00
trvon
0204072397
fix(asan): Addressing asan test regressions and suite failures from recent changes 2026-05-01 09:18:39 -04:00
trvon
f825bc3cf5
feat: ci/test regressions updates, reinstated onnxruntime (simeon will still be default), expanding topology instrumentation in search, newsletter update with throughput comparison 2026-04-26 17:03:37 -04:00
trvon
49b40d5471
fix(ci): ci build fixes for windows, linux, macos tests lanes 2026-04-25 12:55:02 -04:00
trvon
8de19b84f0
feat: adding instrumentation and config options for build and repair system. Bumping simeon and improving topology system with experimental findings 2026-04-24 17:38:24 -04:00
trvon
7de4d95a43
feat: repair logic improvements, expanding config options to disable rebuilding/repair for memory constrained systems 2026-04-24 10:53:15 -04:00
trvon
df0c78cf29
feat: test fixes, metric expansion, build improvements for gliner 2026-04-23 08:48:33 -04:00
trvon
0975e3e740
fix: fixing neon build and memory pressure issues 2026-04-22 15:41:05 -04:00
trvon
d8cb0764d8
feat: address test drift, removing hnsw remants for simeon backend implementation, continued refactor of vector backend 2026-04-22 15:03:36 -04:00
trvon
4f92b4ed39
fix: correctness fix for abalation test, fixes for daemon response (regression is present underload/repair) 2026-04-20 13:20:24 -04:00
trvon
a01bd767ce
feat: bumping simeon, improving graph building, clustering, configuration tuning of said system and updating benchmark files. Additional fixes for daemon readiness and status return underload. 2026-04-20 09:31:45 -04:00
trvon
5eab5b9d48
feat: refining readiness strings with constexpr. Final release testing for v0.13.0 2026-04-17 10:57:30 -04:00
trvon
11ef10141f
fix: addressing UX issue with daemon status readiness 2026-04-17 10:32:05 -04:00
trvon
4a3dce881f
feat: test imrpovements, infer fixes in search changes from previous commits 2026-04-14 18:36:09 -04:00
trvon
a930776655
feat: cleaning up test suite 2026-04-14 10:25:54 -04:00
trvon
e261420351
fix: suspected deadlock fixes in repair, bugs addressed in request logic. Continued triage from infer output and test failures 2026-04-11 18:45:22 -04:00
trvon
ce36736e63
feat: search / retrieval optimizations for performance and accuracy 2026-04-09 21:34:43 -04:00
trvon
e8a6296797
fix: re-ranker embedding generation fix, test failures drift fix, search improvements for graph query expansion and gliner improvements for building graph 2026-04-06 18:38:54 -04:00
trvon
d25cc39e12
fix: vector and embedding count fixes related to repair functionality. Improvements for search and tuning scope of candidates 2026-04-05 08:30:28 -04:00
trvon
9a1ed082d4
feat: optimizations for data returns and harden daemon status and normalize MCP/version responses 2026-04-04 17:33:59 -04:00
trvon
106aa0b1c8
feat: additional dev leakage cleanup for query and tick logic 2026-04-02 17:27:51 -04:00
trvon
86ec015eea
feat: massive commit. Download logic updates with expanding controls (stop, start, resume etc.) from daemon control. This is used in my archive openclaw agent. This commit also includes work to harden onnxruntime abi access. this should allow loading newer abi's when builds are pinned to 1.23.0. 2026-03-21 17:12:47 -04:00
trvon
f3de749787
feat: test improvements, doc updates for download expansion 2026-03-21 10:30:00 -04:00
trvon
81cc38a3cf
fix: expanding utf-8 sanitization and expand search tracing and test coverage 2026-03-21 10:12:42 -04:00
trvon
943393af6d
fix: fixing search regression when fuzzy search triggered and testing improvements 2026-03-21 09:00:54 -04:00
trvon
554bf47366
fix: test coverage updates and address protobuf bug in search 2026-03-20 22:15:36 -04:00
trvon
9e918b723e
feat: test coverage improvements, list hardening for wildcard usage, search and benchmark optimizations for retrievals 2026-03-20 20:42:57 -04:00
trvon
b0c8d96e86
fix: optimizations for hnsw rebuilding logic, test expansions and improvements 2026-03-20 17:43:27 -04:00
trvon
727d5b38a1
fix: additional cli and search optimizations to reduce latency, test coverage improvements 2026-03-20 13:14:31 -04:00
trvon
c781640dee
fix: improving daemon repsonsiveness and instrumented the daemon client for improved benchmark tracing 2026-03-20 11:02:52 -04:00