Commit graph

71 commits

Author SHA1 Message Date
Trevon
08322369cb
Search/grep UX, runtime hardening, and CI stabilization (#39)
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter

Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close

* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler

* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly

ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.

* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict

setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.

* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build

* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs

* fix(build): restore hooks and shrink srht

Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.

Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.

Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.

* refactor(storage): drop dead code, add coverage

* refactor(cli): polish grep/search graph UX

* fix(daemon): stabilize shutdown and CI paths

- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work

* test(daemon): add db vacuum coverage

* test(daemon): add WAL startup recovery

- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown

* refactor(grep): consolidate indexed content scan

* fix(grep): remove pre-scan candidate truncation

* refactor(runtime): split startup, harden storage

* feat(bench): add manifest retrieval corpus

* fix(bench): run fusion experiment on simeon

* fix(cli): sharpen search graph hints

* fix(build): refresh version metadata

* fix(ci): stabilize sync search and srht packaging

* fix(ci): unblock windows and shrink artifacts

* fix(ci): stabilize linux runtime checks

* fix(build): shrink srht manifest
2026-05-27 12:15:17 -04:00
Trevon
a62a901e2e
Stabilize daemon recovery and vector reconciliation (#36)
* fix(daemon): avoid socket cancel on stop

Signal active sockets with shutdown only so pending I/O wakes without entering Boost.Asio's reactor cancel path during teardown.

* fix(metadata): open read pools read-only

* ci: fix coverage report generation

* fix(metadata): tag unscoped pool holders

* test(daemon): avoid ipc teardown takeover

* ci: skip plugin coverage objects

* fix(daemon): repair completed FTS5 ghosts

* build(vector): guard unusable FAISS headers

* ci: use clang gcov for coverage

* test(daemon): stabilize arm CI flakes

* Harden search and compression numeric config

* fix(daemon): reconcile orphan vector embeddings

* fix: address vector cleanup review comments
2026-05-23 21:21:55 -04:00
trvon
77cab7f6e9
fix(ci): stabilize PR checks
Signed-off-by: trvon <me@trevon.dev>
2026-05-21 16:18:08 -04:00
trvon
1315843e89
fix(ci): stabilize tests and coverage 2026-05-21 09:26:43 -04:00
trvon
4df44e0c2f
fix(tests): use portable getpid shim 2026-05-20 15:54:13 -04:00
trvon
a930776655
feat: cleaning up test suite 2026-04-14 10:25:54 -04:00
trvon
0c74dcb34e
feat: completitions fixes. Starting topology routing work for semantic clusters 2026-04-13 12:51:59 -04:00
trvon
4c9ce1e1e1
feat: completion fixes and reduce bottleneck of ingestion on add from cli 2026-04-12 23:45:28 -04:00
trvon
49a106aeef
fix: cleaning up ENV leakage from debugging 2026-04-02 16:58:33 -04:00
trvon
2381c90c86
feat: cleaning up codebase and improving treesitter plugin 2026-03-23 16:57:12 -04:00
trvon
9c201afae7
feat: search improvements, test correctness fixes and bug fixes 2026-03-23 13:59:54 -04:00
trvon
d353e53f26
feat: search optimizations (expanding re-ranking work to recovery items that are not strong candidates based on how they are retrieved). Test improvements and TSAN/ASAN fixes. 2026-03-23 09:15:05 -04:00
trvon
2fd9164bda
fix: tsan fixes, test instrumentation improvements and improved sql handler logic 2026-03-22 20:41:48 -04:00
trvon
9c496e674e
fix: correctness fixes for stale sqlite handles (apart of previous work), improvements in test suite and continue work for search optimizations 2026-03-22 20:06:29 -04:00
trvon
9091a9fa5d
fix: ci fixes, coverage improvements and test improvements and bugs fixes 2026-03-22 17:27:34 -04:00
trvon
15e049daee
feat: test suite improvements. Migrating remaining tests to catch2. Improving hnsw for usage in benchmarks 2026-03-22 07:56:54 -04:00
trvon
f4b3576e8e
fix: bug fixes from test regressions and test infrastructure improvements 2026-03-21 19:52:39 -04:00
trvon
554bf47366
fix: test coverage updates and address protobuf bug in search 2026-03-20 22:15:36 -04:00
trvon
9e918b723e
feat: test coverage improvements, list hardening for wildcard usage, search and benchmark optimizations for retrievals 2026-03-20 20:42:57 -04:00
trvon
11304c6d8e
fix: test fixes 2026-03-18 10:12:33 -04:00
trvon
8cb29866a4
feat: resolving bug introduced in socketserver 2026-03-17 18:22:58 -04:00
trvon
bf11646952
feat: additional test and ASAN/TSAN fixes 2026-02-26 16:54:08 -05:00
trvon
b72eb955be
feat: lint run and sandbox detection improvements for non sandbox envs 2026-02-25 13:41:09 -05:00
trvon
38629de1a1
fix: improvement of graph node source and retrieval logic and sandbox fixes in add command 2026-02-25 12:23:25 -05:00
trvon
3b1a3007ab
feat: sandbox testing expansion (sandbox relates to how the daemon communicates in sandbox). Graph optimizations for retrieval and UX improvements. Will continue improving graph system 2026-02-25 11:37:29 -05:00
trvon
c49f0ba759
feat: optimizations based on profile runs in hashing and work coordinator. test updates and correctness fixes 2026-02-24 21:06:43 -05:00
trvon
2e87c3c947
feat: search/grep retrieval based improvements, test expansions and bug fixes 2026-02-24 19:17:55 -05:00
trvon
c3c4d1d684
fix: ASAN fixes in daemon lifecycle. grep testing improvements 2026-02-24 15:41:48 -05:00
trvon
2d412a59cd
feat: search improvements, changelog fixes and test expansions for refactored code
Signed-off-by: trvon <git@trevon.dev>
2026-02-24 15:01:58 -05:00
0xkite
56cb6a66c5
Recreate last 7 commits (signed) 2026-02-24 14:32:58 -05:00
trvon
d86ac665cd
fix: updating conanfile for onnxruntime to respect system version, updated tests for windows compat, update vectors to enforce wal check point so files does not explode in size 2026-02-21 12:20:50 -05:00
trvon
91e212389c
feat: fixing TSAN issues in MCP. implementing cloudflare code mode https://blog.cloudflare.com/code-mode-mcp/ 2026-02-20 20:39:12 -05:00
trvon
207532f46a
merge: integrate main into experimental (preserve PBI-081 simplification)
Merge main's 93 commits into experimental, preserving architectural updates
decisions:

- Keep ResourceGovernor as centralized pool registry (experimental's PBI-081 Phase 2)
- Translate all PoolManager API calls to ResourceGovernor equivalents
- Remove PoolManager.{h,cpp} from build (absorbed into ResourceGovernor)
- Preserve main's enhancements: RepairService, IOCoordinator, GradientLimiter,
  embed_preparer, GPU info, plugin trust, 60+ new test files
- Take main's version for 28 of 31 conflicts (more comprehensive rewrites)
- Keep experimental's ResourceGovernor.h (expanded pool API)
- Manual merge for .vscode/settings.json (use workspace-relative paths)
2026-02-19 10:01:01 -05:00
trvon
dea5315ae3
feat: streamlining tuning and pool logic in daemon 2026-02-18 16:56:25 -05:00
trvon
33821d93f1
feat: test hardening, added benchmark for multiclient and improve stability to support up to 16 clients before performance degrades, improved parallization and request handling in daemon ingestion pipeline 2026-02-16 15:46:16 -05:00
trvon
8f38ee76be
feat: updating metrics for search and retrieval for future learning system. Additional tuning for search performance 2026-02-13 21:09:26 -05:00
trvon
8256917e5e
feat: Hardening moodel logic for embedings. Adding trace-linked feedback event persistence checkpoint 2026-02-12 23:42:21 -05:00
trvon
b071112836
feat: test coverage improvements. Scaling bug addresses 2026-02-04 20:20:18 -05:00
trvon
210a6f005f
test improvements, bug fix for tuning clamps 2026-01-25 19:08:21 -05:00
trvon
da12ead280
additional bug fixes 2026-01-24 00:32:05 -05:00
trvon
5af411d45c
test improvements 2026-01-23 23:00:56 -05:00
trvon
80ea3ca869
testing improvements 2026-01-14 16:27:42 -05:00
trvon
d7271b16a7
tests: Fix cross-platform integration test failures
- Add runLoop thread to GrepServiceExpectationsIT for request processing
- Increase daemon startup timeout to 15s for CI environments
- Add settling time after daemon Ready state
- Add BEIR dataset infrastructure for SciFact benchmark
2026-01-04 15:25:08 -05:00
trvon
92ccd287b5
daemon communication imrpovements and bug fixes related to tests 2026-01-02 11:55:58 -05:00
trevon
708d9076b8
continued test migration, updating symbol extraction plugin to address plug, search improvements 2025-12-12 15:23:44 -05:00
trvon
d03f94834b
linter run from windows work 2025-12-09 21:11:09 -05:00
trevon
8c58e9d961
testing improvements 2025-12-08 20:52:24 -05:00
trevon
d24a961e2e
checkpoint system, removed hybrid search engine after its been deprecated, build updates 2025-12-02 15:21:31 -05:00
trevon
1de9e31a58
cleaning up tests and exposing windows packaging and builds in ci 2025-11-27 10:54:09 -05:00
trvon
2913adcd20
pre-release v0.7.8 2025-11-14 11:58:19 -05:00