Commit graph

15 commits

Author SHA1 Message Date
Trevon
090a5ca69c
fix(testing): stabilize TSAN coverage, harden daemon shutdown, and speed up KG/storage paths (#53)
* perf(search, metadata): add KG write buffer, score cache, adaptive budgets

* perf(indexing): wire KGWriteBuffer into indexing service

   Converts per-document addEdgesUnique calls into batched flush via
   KGWriteBuffer.  500-doc ingest: 500 SQL round-trips → 1 batch.

   Verified: 14.2× edge insert gap (14.8K→195K edges/s) in kg_edge bench.
   Tests: 10/10 Catch2 pass (105 assertions).

* perf(storage): add fsync durability, parallel batch, read integrity

* test(storage): add stress, crash-sim, cold-cache, batch benchmarks

* fix: CI optimizations and work coordinator shutdown fix

* fix(testing): stabilize TSAN coverage gate

* fix: harden KG flush, score cache, and benches

   - keep KG buffers intact when batch commit fails
   - count flushed edges per batch instead of lifetime totals
   - guard the Simeon score cache and clear it on rebuild
   - fall back to fsync when F_FULLFSYNC fails on macOS
   - remove interactive sudo from cold-cache benchmarks
2026-06-26 17:57:02 -04:00
trvon
5b8d3e452b
docs: prompt updates 2026-06-10 10:58:36 -04:00
Trevon
7076e4b283
feat(daemon): shutdown hardening + test helper linkage fixes (#44)
* perf(daemon): coalesce metadata writes

* refactor(metadata): split repository ops

* fix(metadata): harden repository invariants

* feat: test coverage and continued refactor work in metadata code

* fix(graph): handle path-like explore queries

- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH

* fix(daemon): quiesce shutdown background work

- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain

* fix(daemon): cancel repair work on shutdown

Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.

* perf(metadata): reduce sqlite lock contention

* perf(metadata): flatten batch content retries

* fix(daemon): align shutdown wait budgets

A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.

* perf(metadata): bound feedback write waits

* perf(metadata): prep batch content before tx

* fix(daemon): honor no-embeddings tasks

Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.

* perf(daemon): batch fallback metadata writes

* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements

* perf(metadata): flatten insert retries

* fix(metadata): clear CI portability blockers

Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.

* perf(metadata): batch writes and steady bench

* fix(daemon): skip inactive post-ingest pollers

* perf(metadata): tighten tag and path caches

* perf(metadata): cache small batch upserts

* fix(daemon): harden embedded host teardown

* feat(docs): updating docs policies and language

* perf(metadata): tune doc update retries

* perf(metadata): bound download retries

* perf(metadata): bound download metadata waits

* bench(metadata): scope post-ingest content path

* fix(daemon): export embedded test hooks

* fix(daemon): harden test helper linkage

* fix(build): add missing standard includes
2026-06-08 16:29:44 -04:00
trvon
5ff4b720f3
fix(cli): improve graph-guided grep 2026-06-04 22:46:10 -04:00
trvon
84015b39b7
build(ci,tests): cleaning build system and updates related to ci test failures and compat 2026-04-25 14:48:44 -04:00
trvon
cc49704791
feat: bench harness data_dir reuse + richer telemetry, doc updates
Topology ablation bench: read YAMS_BENCH_DAEMON_DATA_DIR so multiple
daemon spawns can share a stable data_dir. Lets the topology tuner's
persisted MAB state carry over between bench reps; without this, every
fresh DaemonHarness spawn cold-starts UCB1 and picks the alphabetically-
first arm. Pairs with the TopologyTuner persistence (already on HEAD).

Bench also gains per-surface vector-leg counters (vec_unique_doc_counts,
topology_weak_routing_applied, vector_tier_skip_reason, embedding_status,
vector_db_dim/query_embedding_dim) and a modeOf reducer for categorical
metrics — Phase F1-era richer attribution carried forward.

Docs: AGENTS.md / daemon_architecture / retrieval_precision_optimization /
PROMPT-eng-codex updates from the parallel sessions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-23 20:26:26 -04:00
trvon
e2c8ba043b
fix: removing gtest and continued cleanup work of test suite 2026-04-14 12:33:17 -04:00
trvon
d5967deedf
fix: make Gemma CoreML fallback automatic 2026-04-02 13:56:42 -04:00
trvon
efbf102eae
feat: refactoring search graph logic and adding sematic graph edges for better relationships 2026-03-13 23:25:03 -04:00
trvon
f3399c001e
feat: adding perf optimizations for search and grep based on previous commit. Improving model logic for future embeddinggemma and improved colbert support. Re-ranking fixes for search to improve retrieval accuracy. 2026-03-11 14:15:33 -04:00
0xkite
56cb6a66c5
Recreate last 7 commits (signed) 2026-02-24 14:32:58 -05:00
trvon
8256917e5e
feat: Hardening moodel logic for embedings. Adding trace-linked feedback event persistence checkpoint 2026-02-12 23:42:21 -05:00
trvon
b071112836
feat: test coverage improvements. Scaling bug addresses 2026-02-04 20:20:18 -05:00
trvon
4707e18119
committing agent file 2026-01-30 17:56:24 -05:00
trvon
1e50b0c531
Improve daemon shutdown reliability and timeout handling
SocketServer:
- Increase connection handler wait timeout from 100ms to 2s
- Increase accept loop wait timeout from 500ms to 3s
- Prevents race conditions where handlers are cancelled mid-operation

WorkCoordinator:
- Replace thread detach with extended join attempt
- Prevents use-after-free from detached threads accessing freed resources

Daemon Command:
- Increase shutdown wait timeout from 3s to 6s
- Increase default waitForDaemonStop timeout from 5s to 10s
- Allows time for graceful SocketServer shutdown

These changes address SIGABRT and SIGTERM issues during daemon shutdown
by ensuring async operations complete before resources are freed.
2026-01-29 11:55:07 -05:00