Commit graph

11 commits

Author SHA1 Message Date
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
Trevon
7df64b1161
perf(search): bound topology routing and harden runtime paths (#59)
* fix: address review build issues

* perf(search): bound topology routing

Cache route membership, shortlist centroids with static ANN, and route SPQ through explicit allowed-document budgets.

* test(xplan): measure routed vector work

Pin benchmark binary identity and export route, cache, and phase costs for repeated equal-budget comparisons.

* test(search): use portable jthread wrapper

* fix(daemon): serialize detached cancellation

* fix(cli): honor pinned daemon sockets

* fix(xplan): support Windows contract imports

* fix(list): apply metadata filters before paging

* docs(prompts): require memory hydration

Treat search results as candidates and hydrate selected artifacts before use. Quote argument hints for strict YAML loaders.

* test(cli): keep add on pinned daemon

* test(metadata): isolate performance timing

* fix(graph): tolerate stale snippet ranges

Source files may change before persisted symbol metadata is refreshed. Omit stale snippets instead of aborting graph exploration.

* test(daemon): await request readiness

Reuse the retrying list probe before asserting post-shutdown error quality under sanitizer load.

* ci(release): use hosted cmake

Resolve Conan CMake tool requirements from PATH so cold ARM64 caches do not depend on cmake.org. Default BUILD_DIR for failure cleanup.
2026-07-19 14:53:10 -04:00
Trevon
771deebc84
fix: harden CI and metadata stability (#48)
* feat: dead code removal, profling foudnation improvements, addressing code smells in reference db implementation

* fix: test fixes and optimmizations for benchmarks

* fix: harden audit findings

   Reject invalid tokenizer ids, fix move-order hazards, bound manifest
   and audio IO, and guard numeric/process edge cases found by Fold.

* fix: route SymSpell writes through coordinator

* test: fix batch content initializers

* fix(ci): restore cross-platform builds

* fix: address Copilot review findings
2026-06-11 16:42:54 -04:00
Trevon
9cb6421593
feat: daemon stability fixes, vector search optimizations, and docs refresh (#46)
* fix(daemon): resolve sign-conversion warnings

* fix(daemon): add IO thread startup barrier

* refactor: consolidate toLowerCopy clones

* fix(daemon): add atexit handler for static host map cleanup

* fix(build): add missing include, fix daemon_helpers syntax, use atomic barrier

* feat(docs): docs cleanup, and metadata refactor updates

* fix(daemon): resolve Windows-only PostIngestQueue shutdown SIGSEGV (#45)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat: submodule bump, multiclient optimizations and benchmark optimizations

* feat(perf): reduced contention with update to placement of flushPathCacheBuffer()

* fix(metadata): invalidate path cache on document mutation

* fix(metadata): hydrate cached counters before first write

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-10 10:30:38 -04:00
trvon
d67d5d63ac
Remediate Fold SAST findings
Centralize duplicated helpers and harden numeric, timing, IPC, SQLite retry, config, plugin, metrics, and async content-store paths identified during Fold SAST triage.\n\nAdd targeted tests for shared utilities and preserve per-task YAMS checkpoints.
2026-05-27 22:41:44 -04:00
trvon
0b61c0818a
docs: clean up docs and ci fixes for tests 2026-04-25 00:08:18 -04:00
trvon
d8cb0764d8
feat: address test drift, removing hnsw remants for simeon backend implementation, continued refactor of vector backend 2026-04-22 15:03:36 -04:00
trvon
4617bf4d8f
feat: further removing hnsw references and centralizing on simeon pq implementation 2026-04-22 01:10:57 -04:00
trvon
d669c009d0
feat: removing experiments and unifying compression in embeddings storage. 2026-04-21 18:13:37 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos 2026-04-12 18:16:10 -04:00
trvon
ef3d925797
feat: continued corectness improvements, ANN / vectordb improvements 2026-04-12 12:53:29 -04:00