Commit graph

8 commits

Author SHA1 Message Date
Trevon
9883a4382b
refactor(vector): decouple embedding generator policy (#83)
* refactor(vector): add injectable executor for the embedding daemon bridge

await_with_timeout spawned coroutines unconditionally on the daemon
global IO context, coupling the vector-layer embedding generator to
daemon globals. EmbeddingConfig gains an optional
boost::asio::any_io_executor executorOverride; the bridge uses it when
present and keeps the global executor as the default — identical
behavior for all existing callers (embedding_service, model providers,
CLI all construct without an override).

Lead 1 plan step (b) of the embedding_generator vector->daemon
decomposition (Brick audit follow-up).

Validation: full ninja build green; meson test yams:service_manager,
vector_submodule, yams:embedding_lifecycle, yams:embedding_recovery —
all green.

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector): split DaemonBackend into own TU

Keep daemon client/IPC headers out of the vector-layer generator TU;
EmbeddingGenerator builds the backend via createDaemonBackend factory
on the existing IEmbeddingBackend seam. Behavior-preserving.

Brick audit lead 1: 5 file-level cycles hubbed on
embedding_generator.cpp.

Signed-off-by: trvon <me@trevon.dev>

* revert(vector): keep bridge executor out of public EmbeddingConfig

Pre-push gate caught it: the config layer includes the vector header
transitively (ConfigResolver.h -> document_chunker.h ->
embedding_generator.h) and has no boost asio include path, so
<boost/asio/any_io_executor.hpp> in the public header breaks the
config build.

Executor selection stays in embedding_daemon_backend.cpp (daemon-facing
TU by design); per-instance injection needs the config->vector include
direction fixed first — recorded as follow-up.

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector): trust resolved backend policy

Remove EmbeddingGenerator's ambient ConfigResolver read. Callers now own
backend-policy resolution; EmbeddingService marks its already-resolved
runtime policy unconditionally. Keep TuneAdvisor only for the process-wide
concurrency gate, where instance-local policy would not coordinate admission.

Tests pin typed backend authority against conflicting environment state.

Signed-off-by: trvon <me@trevon.dev>

* chore(vector): remove stale config include

The generator no longer reads config helpers after backend policy moved to
callers, so drop the misleading DaemonBackend comment and unused include.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-02 10:12:46 -04:00
trvon
8b501a344e
refactor(config): consolidate embedding policy 2026-08-05 20:53:18 -04:00
trvon
5066fc0893
test: isolate process environment 2026-08-04 21:31:39 -04:00
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
trvon
f825bc3cf5
feat: ci/test regressions updates, reinstated onnxruntime (simeon will still be default), expanding topology instrumentation in search, newsletter update with throughput comparison 2026-04-26 17:03:37 -04:00
trvon
42ee44cef4
fix(ci): ci test and bug fixes. Improvements for testing topology construction 2026-04-26 13:08:55 -04:00
trvon
bdcff5b636
fix: HNSW and topology rebuilds are now batched in repair. Cleaning up from previous migration work 2026-04-21 13:58:10 -04:00
trvon
8c313fc023
feat: expanding topology system and search for better testing and simeon integration 2026-04-19 09:25:28 -04:00