Commit graph

30 commits

Author SHA1 Message Date
Trevon
7076e4b283
feat(daemon): shutdown hardening + test helper linkage fixes (#44)
* perf(daemon): coalesce metadata writes

* refactor(metadata): split repository ops

* fix(metadata): harden repository invariants

* feat: test coverage and continued refactor work in metadata code

* fix(graph): handle path-like explore queries

- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH

* fix(daemon): quiesce shutdown background work

- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain

* fix(daemon): cancel repair work on shutdown

Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.

* perf(metadata): reduce sqlite lock contention

* perf(metadata): flatten batch content retries

* fix(daemon): align shutdown wait budgets

A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.

* perf(metadata): bound feedback write waits

* perf(metadata): prep batch content before tx

* fix(daemon): honor no-embeddings tasks

Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.

* perf(daemon): batch fallback metadata writes

* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements

* perf(metadata): flatten insert retries

* fix(metadata): clear CI portability blockers

Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.

* perf(metadata): batch writes and steady bench

* fix(daemon): skip inactive post-ingest pollers

* perf(metadata): tighten tag and path caches

* perf(metadata): cache small batch upserts

* fix(daemon): harden embedded host teardown

* feat(docs): updating docs policies and language

* perf(metadata): tune doc update retries

* perf(metadata): bound download retries

* perf(metadata): bound download metadata waits

* bench(metadata): scope post-ingest content path

* fix(daemon): export embedded test hooks

* fix(daemon): harden test helper linkage

* fix(build): add missing standard includes
2026-06-08 16:29:44 -04:00
Trevon
2d01ba314c
Assertion hardening, metadata correctness, & daemon lifecycle fixes (#41)
* fix(core): add assertions and harden daemon lifecycle

* fix(daemon): tighten shutdown lifecycle invariants

* fix(test): add PressureLimitedPoller capability-sleep shutdown test

* refactor(vector): harden backend contracts

Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.

Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.

* fix(vector): address audit findings plus hardening

Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.

Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.

Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.

* fix(test): add PressureLimitedPoller wake-timer shutdown tests

* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec

Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.

Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.

* fix(daemon): harden PressureLimitedPoller inFlightCounter null check

* fix(daemon): harden PostIngestQueue wait_until deadline checks

* fix(daemon): clean up recovery sentinels regardless of salvage count

Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.

Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.

* feat(vec0): restore persisted ANN index on daemon restart

After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index.  When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.

Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.

* fix: enable DCHECK by default and assert metadata counter consistency

* fix(services): assert document deletion ordering content-before-metadata

* fix(test): add fork-based assertion validation tests

* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE

* fix(ci): enable -Denable-dcheck=true in all test CI lanes

* fix(test): address assertion review findings

* refactor(daemon): isolate poller config assertion checks

* refactor(services): reduce deletion and poller complexity

* style(ci): normalize test workflow spacing

* refactor(services): extract list query helpers

* refactor(services): extract list hydration helpers

* refactor(daemon): extract metrics snapshot helpers

* refactor(daemon): extract metrics counter snapshot

* refactor(daemon): extract metrics resource snapshot

* feat(graph): add daemon-backed explore

Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.

Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.

* fix(services): prevent prune delete metadata orphans

* fix(cli): honor data-dir in graph explore

- bypass daemon explore when the CLI pins a data dir so\n  temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n  reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n  context

* fix(daemon): make lifecycle stop completion safe

* fix(daemon): gate db-open startup against shutdown

* fix(metadata): serialize FTS indexed counter updates

* test(daemon): simulate vector lock contention cross-process

* test(daemon): isolate post-ingest identity batch

* style(daemon): normalize test include order

* refactor(graph): split cli graph helpers

Keep graph command paths smaller and easier to evolve.

Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.

* fix(config): harden expand_tilde and fix Windows XDG state path doubling

- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
  on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
  to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment

* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax

* fix(static-analysis): strip trailing text from nosemgrep comments

* fix(doctor): add logging to daemon_check empty catch blocks

* fix(db_repair): add logging to empty catch blocks

* fix(chunking): add precondition checks for rabin polynomial and windowSize

* fix(safety): add null precondition checks for Fold proved null-deref sites

* fix(daemon): widen init-failure dispatch, add null/dbz preconditions

InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.

* ci(ubsan): use restricted build targets to avoid runner timeout

* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV

* fix(config): cast parseTomlU32 to size_t for batch fields

* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
2026-06-04 12:53:42 -04:00
trvon
0b61c0818a
docs: clean up docs and ci fixes for tests 2026-04-25 00:08:18 -04:00
trvon
4f92b4ed39
fix: correctness fix for abalation test, fixes for daemon response (regression is present underload/repair) 2026-04-20 13:20:24 -04:00
trvon
a930776655
feat: cleaning up test suite 2026-04-14 10:25:54 -04:00
trvon
68af81d66b
Merge branch 'experimental' 2026-04-09 09:31:25 -04:00
trvon
01c72ec828
fix: test and ci fixes and search improvements. Adding a router for tuning logic 2026-04-09 09:31:00 -04:00
trvon
d1225ab465
fix: test fixes related to metadata optimizations and some search tuning checkpoints before refactor experiment 2026-04-08 21:07:01 -04:00
trvon
82daf7dcf2
feat: additional cleanup from taskflow migration 2026-03-17 05:22:12 -04:00
trvon
4245090359
fix: test improvements and CI harning for slower tests 2026-02-27 20:54:34 -05:00
trvon
55b2dc517f
fix: additional merge fixes and search improvements 2026-02-19 10:24:23 -05:00
trvon
207532f46a
merge: integrate main into experimental (preserve PBI-081 simplification)
Merge main's 93 commits into experimental, preserving architectural updates
decisions:

- Keep ResourceGovernor as centralized pool registry (experimental's PBI-081 Phase 2)
- Translate all PoolManager API calls to ResourceGovernor equivalents
- Remove PoolManager.{h,cpp} from build (absorbed into ResourceGovernor)
- Preserve main's enhancements: RepairService, IOCoordinator, GradientLimiter,
  embed_preparer, GPU info, plugin trust, 60+ new test files
- Take main's version for 28 of 31 conflicts (more comprehensive rewrites)
- Keep experimental's ResourceGovernor.h (expanded pool API)
- Manual merge for .vscode/settings.json (use workspace-relative paths)
2026-02-19 10:01:01 -05:00
trvon
dea5315ae3
feat: streamlining tuning and pool logic in daemon 2026-02-18 16:56:25 -05:00
trvon
33821d93f1
feat: test hardening, added benchmark for multiclient and improve stability to support up to 16 clients before performance degrades, improved parallization and request handling in daemon ingestion pipeline 2026-02-16 15:46:16 -05:00
trvon
94a49472f1
feat: fixing bug with multiple processes access gpu, read/write pools now added in metadata repository, and updated plugins for opencode and vscode 2026-02-14 18:16:57 -05:00
trvon
a44c2fb4af
feat: implementing embedding chunking strategy and two staged ingestion that processes then distributes document data 2026-02-12 00:46:04 -05:00
trvon
1d255d59ce
feat: address bugs with scaling, batching in daemon processing 2026-02-09 17:07:00 -05:00
trvon
ded16c11d4
improvements of parallel ingestion 2026-02-05 11:51:24 -05:00
trvon
2946b231dd
fix: license header file updates, mcp improvements, windows build fixes 2026-01-30 17:23:33 -05:00
trvon
b3fbff0457
fixing model logic to reduce pool size when using cpu 2026-01-20 13:05:42 -05:00
trvon
66361a1efd
test fixes 2026-01-08 23:05:57 -05:00
trvon
b078329bc8
search regressions fixes. removed bktree for symspell 2026-01-07 23:43:12 -05:00
trvon
0557c9b33b
test improvements and linter run 2025-12-31 12:51:28 -05:00
trevon
b1acc52040
watch additions and new tests 2025-12-29 03:14:16 -05:00
trvon
6dda21d40b
test and compatibility fixes. grep UI improvements 2025-12-22 17:20:15 -05:00
trvon
c16939f482
linter 2025-11-28 20:30:41 -05:00
trevon
1de9e31a58
cleaning up tests and exposing windows packaging and builds in ci 2025-11-27 10:54:09 -05:00
trvon
5b23afa99d
adding dedicate graph component for graph operations, updating socket stale logic and fixing CI issues in release. 2025-11-16 13:31:04 -05:00
trvon
b06e9335b9
staging v0.7.8. running benchmarks on updated search engine and improving code for testing/benchmarking 2025-11-13 10:27:10 -05:00
trvon
bbf23bb190
test migrations to catch2, service manager + socket server refactor for better startup performance. search bug fixes 2025-10-29 23:08:09 -04:00