Commit graph

38 commits

Author SHA1 Message Date
Trevon
2d01ba314c
Assertion hardening, metadata correctness, & daemon lifecycle fixes (#41)
* fix(core): add assertions and harden daemon lifecycle

* fix(daemon): tighten shutdown lifecycle invariants

* fix(test): add PressureLimitedPoller capability-sleep shutdown test

* refactor(vector): harden backend contracts

Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.

Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.

* fix(vector): address audit findings plus hardening

Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.

Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.

Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.

* fix(test): add PressureLimitedPoller wake-timer shutdown tests

* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec

Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.

Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.

* fix(daemon): harden PressureLimitedPoller inFlightCounter null check

* fix(daemon): harden PostIngestQueue wait_until deadline checks

* fix(daemon): clean up recovery sentinels regardless of salvage count

Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.

Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.

* feat(vec0): restore persisted ANN index on daemon restart

After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index.  When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.

Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.

* fix: enable DCHECK by default and assert metadata counter consistency

* fix(services): assert document deletion ordering content-before-metadata

* fix(test): add fork-based assertion validation tests

* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE

* fix(ci): enable -Denable-dcheck=true in all test CI lanes

* fix(test): address assertion review findings

* refactor(daemon): isolate poller config assertion checks

* refactor(services): reduce deletion and poller complexity

* style(ci): normalize test workflow spacing

* refactor(services): extract list query helpers

* refactor(services): extract list hydration helpers

* refactor(daemon): extract metrics snapshot helpers

* refactor(daemon): extract metrics counter snapshot

* refactor(daemon): extract metrics resource snapshot

* feat(graph): add daemon-backed explore

Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.

Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.

* fix(services): prevent prune delete metadata orphans

* fix(cli): honor data-dir in graph explore

- bypass daemon explore when the CLI pins a data dir so\n  temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n  reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n  context

* fix(daemon): make lifecycle stop completion safe

* fix(daemon): gate db-open startup against shutdown

* fix(metadata): serialize FTS indexed counter updates

* test(daemon): simulate vector lock contention cross-process

* test(daemon): isolate post-ingest identity batch

* style(daemon): normalize test include order

* refactor(graph): split cli graph helpers

Keep graph command paths smaller and easier to evolve.

Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.

* fix(config): harden expand_tilde and fix Windows XDG state path doubling

- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
  on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
  to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment

* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax

* fix(static-analysis): strip trailing text from nosemgrep comments

* fix(doctor): add logging to daemon_check empty catch blocks

* fix(db_repair): add logging to empty catch blocks

* fix(chunking): add precondition checks for rabin polynomial and windowSize

* fix(safety): add null precondition checks for Fold proved null-deref sites

* fix(daemon): widen init-failure dispatch, add null/dbz preconditions

InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.

* ci(ubsan): use restricted build targets to avoid runner timeout

* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV

* fix(config): cast parseTomlU32 to size_t for batch fields

* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
2026-06-04 12:53:42 -04:00
trvon
8772cc8e54
fix(docs,ci): addressing breaking changes in zig v0.16.x in Dockerfile, docs update and code cleanup 2026-04-25 11:27:18 -04:00
trvon
d7ab4844a1
fix: small fixes for benchmarks and documentation update 2026-04-17 22:32:04 -04:00
trvon
f4fffd73a0
feat: instrumentation improvements for benchmarks, shell completition improvements, packaging updates for completions and docs updates 2026-04-13 21:34:18 -04:00
trvon
3c2f61b863
feat: expansion of topology system into repair and ingestion pipelines 2026-04-13 17:19:13 -04:00
trvon
ad8d7ac059
feat: imrpovements for repair and graph clustering logic and search usage 2026-04-04 21:16:41 -04:00
trvon
49a106aeef
fix: cleaning up ENV leakage from debugging 2026-04-02 16:58:33 -04:00
trvon
970f253380
fix: cleaning up macos build detection logic 2026-04-02 15:43:33 -04:00
trvon
2e20416e96
feat: adding fuzz test for new download logic, removing r2 endpoint from tests, decoupling embedding / onnxruntime logic for work from previous commit 2026-03-21 18:08:23 -04:00
trvon
f3de749787
feat: test improvements, doc updates for download expansion 2026-03-21 10:30:00 -04:00
trvon
f3399c001e
feat: adding perf optimizations for search and grep based on previous commit. Improving model logic for future embeddinggemma and improved colbert support. Re-ranking fixes for search to improve retrieval accuracy. 2026-03-11 14:15:33 -04:00
trvon
d92f66d2f6
fix: fixing daemon regression for startup logic resulting in ipc eof, expanding release please to update conan file version, expanding benchmark docs 2026-03-07 21:38:35 -05:00
trvon
8af4d3524e
feat: starting remote storage work, config v3, and config migration 2026-03-07 10:34:48 -05:00
trvon
44733e9114
feat: list and documentation updates 2026-03-06 19:22:40 -05:00
trvon
099bbef70d
feat: large update. Documentation update, config and plugin hardning and improvements related to loading trust. Exposed trust state in the doctor command with hints on improvements. Bug fixes and test improvements to reduce flakiness and TSAN issues 2026-02-26 11:21:29 -05:00
trvon
2e87c3c947
feat: search/grep retrieval based improvements, test expansions and bug fixes 2026-02-24 19:17:55 -05:00
trvon
8256917e5e
feat: Hardening moodel logic for embedings. Adding trace-linked feedback event persistence checkpoint 2026-02-12 23:42:21 -05:00
trvon
08ce3bee22 docs and cli improvements 2026-01-22 03:11:57 +00:00
trevon
b1acc52040
watch additions and new tests 2025-12-29 03:14:16 -05:00
trevon
2e687eef79
docs improvements, tuning manager now scales postingestqueue, stats exposed in yams stats 2025-12-20 13:36:59 -05:00
trevon
c4e8099f07
improvements for daemon service management 2025-12-02 12:03:40 -05:00
trevon
ebe81823db
improvements for daemon lifecycle, tree repair, and mcp regression 2025-11-26 14:22:35 -05:00
trvon
c0fde23eb5
small bug fixes for retrieval 2025-11-06 17:00:34 -05:00
trvon
b065e6a2c1
v0.7.5 2025-10-11 00:34:31 -04:00
trvon
7b9da100ff
v0.7.3 staging 2025-10-07 11:18:35 -04:00
Trevon
956c43b80b
v0.6.12 experiemental 2025-09-11 11:28:42 -04:00
Trevon
78f6585b9c
v0.6.5 2025-09-04 16:46:05 -04:00
Trevon
e5c0357218
v0.6.4 2025-09-04 15:51:26 -04:00
Trevon
d0342a4ed6
v0.4.0 staging 2025-08-20 12:17:48 -04:00
Trevon
d27c481d6b
v0.2.0 2025-08-13 23:11:02 -04:00
Trevon
96342464ca
v0.1.5 2025-08-13 19:25:31 -04:00
Trevon
10e49d95db
patching github actions and cli docs 2025-08-12 22:26:13 -04:00
Trevon
bb1041656d
v0.1.0 2025-08-12 20:52:21 -04:00
Trevon
a384715bee
g++ compability updates and mcp updates for feature parity 2025-08-11 21:21:05 -04:00
Trevon
847fefb97a
CHANGELOG updated. v0.0.4 and test improvements 2025-08-11 18:40:05 -04:00
Trevon
a74a276f6b
changelog updates v0.0.3 2025-08-11 11:11:19 -04:00
Trevon
41617a74a5
improvements tracked in @CHANGELOG.MD v0.0.2 2025-08-11 01:17:41 -04:00
Trevon
917aecf514
yams 2025-08-10 23:41:49 -04:00