Commit graph

75 commits

Author SHA1 Message Date
Trevon
2d01ba314c
Assertion hardening, metadata correctness, & daemon lifecycle fixes (#41)
* fix(core): add assertions and harden daemon lifecycle

* fix(daemon): tighten shutdown lifecycle invariants

* fix(test): add PressureLimitedPoller capability-sleep shutdown test

* refactor(vector): harden backend contracts

Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.

Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.

* fix(vector): address audit findings plus hardening

Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.

Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.

Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.

* fix(test): add PressureLimitedPoller wake-timer shutdown tests

* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec

Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.

Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.

* fix(daemon): harden PressureLimitedPoller inFlightCounter null check

* fix(daemon): harden PostIngestQueue wait_until deadline checks

* fix(daemon): clean up recovery sentinels regardless of salvage count

Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.

Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.

* feat(vec0): restore persisted ANN index on daemon restart

After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index.  When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.

Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.

* fix: enable DCHECK by default and assert metadata counter consistency

* fix(services): assert document deletion ordering content-before-metadata

* fix(test): add fork-based assertion validation tests

* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE

* fix(ci): enable -Denable-dcheck=true in all test CI lanes

* fix(test): address assertion review findings

* refactor(daemon): isolate poller config assertion checks

* refactor(services): reduce deletion and poller complexity

* style(ci): normalize test workflow spacing

* refactor(services): extract list query helpers

* refactor(services): extract list hydration helpers

* refactor(daemon): extract metrics snapshot helpers

* refactor(daemon): extract metrics counter snapshot

* refactor(daemon): extract metrics resource snapshot

* feat(graph): add daemon-backed explore

Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.

Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.

* fix(services): prevent prune delete metadata orphans

* fix(cli): honor data-dir in graph explore

- bypass daemon explore when the CLI pins a data dir so\n  temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n  reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n  context

* fix(daemon): make lifecycle stop completion safe

* fix(daemon): gate db-open startup against shutdown

* fix(metadata): serialize FTS indexed counter updates

* test(daemon): simulate vector lock contention cross-process

* test(daemon): isolate post-ingest identity batch

* style(daemon): normalize test include order

* refactor(graph): split cli graph helpers

Keep graph command paths smaller and easier to evolve.

Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.

* fix(config): harden expand_tilde and fix Windows XDG state path doubling

- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
  on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
  to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment

* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax

* fix(static-analysis): strip trailing text from nosemgrep comments

* fix(doctor): add logging to daemon_check empty catch blocks

* fix(db_repair): add logging to empty catch blocks

* fix(chunking): add precondition checks for rabin polynomial and windowSize

* fix(safety): add null precondition checks for Fold proved null-deref sites

* fix(daemon): widen init-failure dispatch, add null/dbz preconditions

InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.

* ci(ubsan): use restricted build targets to avoid runner timeout

* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV

* fix(config): cast parseTomlU32 to size_t for batch fields

* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
2026-06-04 12:53:42 -04:00
Trevon
bfe7cd94ca
test: expand storage-readiness coverage — 48.1%→59.8% broad, 74.3% narrow (#38)
* fix(daemon): avoid socket cancel on stop

Signal active sockets with shutdown only so pending I/O wakes without entering Boost.Asio's reactor cancel path during teardown.

* fix(metadata): open read pools read-only

* ci: fix coverage report generation

* fix(metadata): tag unscoped pool holders

* test(daemon): avoid ipc teardown takeover

* ci: skip plugin coverage objects

* fix(daemon): repair completed FTS5 ghosts

* build(vector): guard unusable FAISS headers

* ci: use clang gcov for coverage

* test(daemon): stabilize arm CI flakes

* Harden search and compression numeric config

* fix(daemon): reconcile orphan vector embeddings

* fix: address vector cleanup review comments

* chore(main): release 0.16.1

* Harden search ranking divisors

* Harden snippet distance and command registration

* test(storage): add readiness coverage gate

Document the distributed sync readiness surface and add direct adapter, refcount, and GC failure-path coverage.

* test(storage): cover strict resolver fallback

* fix(search): tighten retrieval ranking

Honor component weights in default RRF, over-fetch vector chunks before document aggregation, and persist topology centroids for sparse-guided routing.

* fix(search): route weak queries through topology

* fix(search): improve rerank and weak queries

Wire provider-backed cross reranking, add direct Simeon candidates for weak lexical pools, and report stage-level retrieval metrics.

* fix(storage): honor explicit plugin names

Add readiness coverage for factory, URL backend, and compressed storage failure paths.

* fix(storage): avoid remote size listing

* fix(api): rollback failed content adds

* test(storage): cover engine readiness

* test(storage): cover compressed readiness

* fix(core): harden delete and recovery paths

Route prune and local delete fallback through DocumentService so content, metadata, vectors, and KG cleanup stay consistent.

Treat transient SQLite quick_check lock failures as ResourceBusy, refresh IPC activity on streaming writes, and preserve reranker config in tuner seeding.

* test(storage): cover URL backend readiness

* fix(api): harden content rollback

* feat(storage): add list seam, URL listing, and write-edge coverage

* feat(api): reconcile ambiguous remote manifest writes

* feat(storage): add deep compressed scan validation

* test(storage): add plugin loader error path coverage

* fix(gc): prune reference rows after collection

* test(api,storage): add health, compact, gc, and verify edge coverage

* docs: update storage-readiness coverage to 71.8%

* test(metadata): add connection pool shutdown-while-leased coverage

* test(metadata): add tree builder snapshot root stability coverage

* test(vector): add before-init and after-close operation rejection coverage

* test(metadata): add database post-close, open-failure, and cache-clear coverage

* test(integrity): add P2P wrong-hash rejection + batch matrix resolution

* docs: audit and resolve all matrix rows — 18 fully covered

* test(metadata): add failed-migration and rollback coverage

* test(storage): add getBlockSize-invalid-key, empty-verify, PUT/DELETE retry coverage

* feat(storage): add file-open and rename failure hooks + ContentStore retrieve edge coverage

* docs: add broad-surface coverage with per-subsystem breakdown

* test(integrity): add chunk validator manifest validation coverage

* test(vector): add batch insert, delete, and update operation coverage

* docs: update integrity subsystem coverage to 46.4%

* docs: consolidate coverage report — clean matrix, remove boilerplate

* docs: update coverage with complete-suite numbers — 59.8% broad, 74.3% narrow

* test(storage): add URLBackend HEAD exists with 2xx coverage

* bump submodules

* fix: address PR review — curl_slist leak, ec ordering, CAS rollback, production hooks

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-25 22:58:15 -04:00
trvon
ff7696a0ff
ci: addressing warm issues with ci. staging bug fixes and search research work 2026-04-29 17:22:20 -04:00
trvon
f825bc3cf5
feat: ci/test regressions updates, reinstated onnxruntime (simeon will still be default), expanding topology instrumentation in search, newsletter update with throughput comparison 2026-04-26 17:03:37 -04:00
trvon
42ee44cef4
fix(ci): ci test and bug fixes. Improvements for testing topology construction 2026-04-26 13:08:55 -04:00
trvon
8de19b84f0
feat: adding instrumentation and config options for build and repair system. Bumping simeon and improving topology system with experimental findings 2026-04-24 17:38:24 -04:00
trvon
7de4d95a43
feat: repair logic improvements, expanding config options to disable rebuilding/repair for memory constrained systems 2026-04-24 10:53:15 -04:00
trvon
e88578a9fc
feat: adding support for phss vector search, test fixes, daemon status regression fix, ci fixes for docker builds with new third-party library 2026-04-23 11:02:18 -04:00
trvon
d8cb0764d8
feat: address test drift, removing hnsw remants for simeon backend implementation, continued refactor of vector backend 2026-04-22 15:03:36 -04:00
trvon
be0feac605
feat: refactor vector engine to reduce monoliths in codebase 2026-04-22 08:36:55 -04:00
trvon
a3f1d529c9
fix: addressing unoptimal copies of vectors in repair path 2026-04-22 02:09:58 -04:00
trvon
4617bf4d8f
feat: further removing hnsw references and centralizing on simeon pq implementation 2026-04-22 01:10:57 -04:00
trvon
d669c009d0
feat: removing experiments and unifying compression in embeddings storage. 2026-04-21 18:13:37 -04:00
trvon
1b6f137132
fix: prune fixes for build artifacts, readiness improvements and hnsw api unification 2026-04-21 17:03:58 -04:00
trvon
bdcff5b636
fix: HNSW and topology rebuilds are now batched in repair. Cleaning up from previous migration work 2026-04-21 13:58:10 -04:00
trvon
6c2c31f2bc
fix: cli clean up of old comments and duplicate code. search improvements as we migrate yams to simeon. 2026-04-21 10:20:01 -04:00
trvon
8c313fc023
feat: expanding topology system and search for better testing and simeon integration 2026-04-19 09:25:28 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos 2026-04-12 18:16:10 -04:00
trvon
ef3d925797
feat: continued corectness improvements, ANN / vectordb improvements 2026-04-12 12:53:29 -04:00
trvon
57b7e20657
fix: concurrency and race condition fixes 2026-04-12 11:37:06 -04:00
trvon
61d92c9f5a
fix: address daemon starvation/deadlock paths and aligning ownership of run loop lifecycle 2026-04-12 10:04:50 -04:00
trvon
814672f4dd
feat: removing unnecessary abstraction and adding some optimizations for embeddings generation 2026-04-07 18:04:34 -04:00
trvon
825d9aa666
feat: improving repair logic and search engine metrics for tuning logic 2026-04-06 11:23:02 -04:00
trvon
c32db6e34d
fix: windows build fixes and test instrumentation to resolve ANN precision issues 2026-03-29 08:18:58 -04:00
trvon
c182221b48
feat: updating staging 2026-03-27 10:45:36 -04:00
trvon
6874b2220e
feat: TurboQuant search and ANN compression improvements. CLi client regression fixes 2026-03-25 23:59:42 -04:00
trvon
8b393049ba
feat: TurboQuant now supports end-to-end packed-vector storage, reranking, and direct compressed ANN traversal with persisted per-coordinate calibration and validated multi-dimension performance. 2026-03-25 23:00:16 -04:00
trvon
169230645a
feat: TurboQuant integration for re-ranking, starting with benchmarks, then moving to integration 2026-03-25 17:40:35 -04:00
trvon
7609d6355b
fix: continued turboquant work. added additional api's and improved testing / perf tests 2026-03-25 17:19:57 -04:00
trvon
2922835242
fix: cli regression fixes and turboquant improvements for storage retrieval logic 2026-03-25 16:42:26 -04:00
trvon
b7d1b46cc7
turboquant: align all docs and math, add IP scoring prototype task
Benchmark extension:
- turboquant_bench.cpp: measures decoded MAE vs QJL-corrected MAE
- benchmark confirms full QJL is WORSE than decoded-only on random vectors
2026-03-25 16:42:12 -04:00
trvon
1bc6b7d6d1
feat: TurboQuant imrpovements 2026-03-25 11:27:18 -04:00
trvon
147f5ca052
feat: moving changelogs to archive, implementing TurboQuant then will integrate in vector engine 2026-03-25 08:15:26 -04:00
trvon
2381c90c86
feat: cleaning up codebase and improving treesitter plugin 2026-03-23 16:57:12 -04:00
trvon
15e049daee
feat: test suite improvements. Migrating remaining tests to catch2. Improving hnsw for usage in benchmarks 2026-03-22 07:56:54 -04:00
trvon
f4b3576e8e
fix: bug fixes from test regressions and test infrastructure improvements 2026-03-21 19:52:39 -04:00
trvon
b0c8d96e86
fix: optimizations for hnsw rebuilding logic, test expansions and improvements 2026-03-20 17:43:27 -04:00
trvon
727d5b38a1
fix: additional cli and search optimizations to reduce latency, test coverage improvements 2026-03-20 13:14:31 -04:00
trvon
513d43098c
feat: hardening client reponses and improving search engine precision through optimization loop 2026-03-12 11:07:18 -04:00
trvon
cc28b8a343
feat: instrumented embedding for deeper insight into hotspots. Fixed CoreML scaling that resulted in high ram usage due to VRAM calculation. Expanded testing for embedding and draining of work 2026-02-11 12:07:08 -05:00
trvon
448f5e47b4
fix: fixing repairmanager logic, windows build issues, linting, resource scaling 2026-02-07 22:07:56 -05:00
trvon
36c587e776
bug fixes and test improvements 2026-01-24 11:49:48 -05:00
trvon
da12ead280
additional bug fixes 2026-01-24 00:32:05 -05:00
trvon
4442c3f485
additional windows compat fixes 2026-01-21 13:27:55 -05:00
trvon
dabb828e7a
stagging changes for release of v0.7.11 . Working on improving retrieval 2026-01-19 18:59:06 -05:00
trvon
b7aa881de2
test fixes, reranking model loading improvements 2026-01-18 22:27:30 -05:00
trvon
67a3c521e0
search engine tests, hnsw bug fix and test fixes 2026-01-17 22:13:08 -05:00
trvon
c7743269b4
bug fixes for model loading logic, dynamic vec table dimensions, cli refactor and search bug fixes 2026-01-08 23:10:58 -05:00
trvon
027fdc001e
clangd fixes, moved symbol extractor to vectordb, investigating slow vector search performance 2026-01-07 10:38:05 -05:00
trvon
c7c48e8934
removed hnsw vector manager. search optimizations 2026-01-05 23:39:18 -05:00