23 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
211db15606 |
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup: std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine threshold could exceed 1.0 for a legal pair of identical vectors. The same work was also admitted while the daemon was still initializing, so the 300-second starvation guard forced a pass before readiness. + semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the existing embed policy header, shared by the streamed and corpus branches; non-finite vectors or scores produce no neighbor and legal scores are bounded before ranking and threshold selection + startupMaintenanceEligible() requires ServiceManagerState::Ready, not stopping and idle, and is evaluated before ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no longer manufacture missing readiness + StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter line reporting state, phase and phase_elapsed_ms instead of a debug-level state number; a 53-minute startup previously logged nothing actionable + regression through the public backfill and corpus-rebuild APIs using a 64-D vector whose old float inverse norm rounds its self-cosine above 1 - removed the duplicated inverse-norm and cosine lambdas from EmbeddingService - renamed __test_setModelProvider/__test_setKgStore to testing_* because they used reserved identifiers, across 11 files - re-anchored 13 pre-existing raw-environment allowlist entries to their new line numbers; no new exception was added Startup is still blocked by corpus-sized memory-sync scans; this change stops the abort and names the blocking phase, it does not make startup fast. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
beef87e66c |
bench(retrieval): seed one-way kNN semantic edges on request (#308)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
Benchmark seeding wrote only mutual neighbor pairs. The connected topology engine keeps every reciprocal edge at or above min_edge_score inside its component, so with mutual-only seeding no raw neighbor crosses a cluster boundary and the engine builds no roads. The Eval 2 smoke showed roads_absent on every connected-engine query. The daemon stores one-way kNN edges (GraphComponent treats them as ordinary non-mutual neighbors). YAMS_BENCH_SEED_SEMANTIC_DIRECTED=1 seeds them too, as single directed edges. Default off keeps existing plans unchanged. The pair loop moves into buildSemanticNeighborSeedEdges in the bench support library, sorted by (source, target), with a unit test for both modes. A topology test pins the mutual-only case: no roads on the connected engine until a one-way edge exists. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
81a051fc2e |
feat(search): walk along roads (#307)
Let RouteOnly and Augment expand every routed hit along its cluster's walk order. The snapshot cache loader resolves each cluster's `orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`, `TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with `orderKind` None or a permutation that fails `validateMemberPermutation` resolves to nothing. The store's `orderUnavailableReason` is carried on the session result and emitted as `topology_order_unavailable_reason` only when set. When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex` after the road beam: the selected routes first, then the road-expanded clusters, keeping only clusters with an order and counting the rest in `roadWalkClustersUnordered`. The index borrows its member views from the snapshot it keeps alive; the certificate is never touched. The vector stage walks after the certified and road groups. Every routed hit so far (certified route hits and road-beam hits) anchors +-window positions along each order it belongs to, bounded to that order (`walkTopologyRoadOrders`). Anchoring on route hits too means the walk rescues the ordinal neighbours of the semantically closest members even with `road_hops` = 0; the plan's "around road hits" reads as every hit a road-era routed query produced. The walked set, deduplicated against the hits, is one extra routed query whose quota is its own size (ranking ordinal neighbours by similarity and truncating would reintroduce the filter the walk bypasses); its rows and distance evaluations land in the routed diagnostics like road candidates. Shadow and Narrow never walk. Config: `search.topology.road_walk_window` (default 0, off) on `SearchEngineConfig::topologyRoadWalkWindow` (copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by SearchEngineManager. `makeTopologyRoutingOptions` requests the window only for Augment and a gate-open RouteOnly, like the beam. The window is excluded from `topologyRoutingPolicyFingerprint` (comment and test). Defaults leave every stage byte-identical: the PR 2 goldens (692), the RouteOnly cells (417) and the road beam cells (904) are unchanged. Debug keys appear only when the window > 0: `topology_road_walk_window_applied` (0 when nothing was walkable), `topology_road_walk_anchors`, `topology_road_walk_candidates_added`, `topology_road_walk_clusters_unordered`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes `search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and `topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3 can set the order kind per arm and build roads at all). Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp (new catch2_search_vector_stage_road_walk): two reciprocal chains A (six members) and B (four) with a road, built with `TopologyBuildConfig::roadOrder` Time from eventTime so the walk order differs from the similarity ranking; loader resolution including the invalid-permutation drop and the cache carry; fingerprint equality for any window and the option gating per policy; session-level certificate, beam and work invariance with the index listing routes then road clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an unordered cluster counted and skipped, no index without any order, the order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0 equal to the road beam cell with no walk key, window 1 adding the immediate order neighbours of all four hits, window 2 the whole clusters, rows visited growing while router work is unchanged, certified keys unchanged, unordered B contributing nothing, no order anywhere equal to the beam cell plus counters, hops 0 walking the selected route alone; Augment adding the walked documents as rescues; Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads, the bench reads its keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
111988c6e4 |
feat(topology): typed event time and session for topology inputs (#305)
* feat(topology): typed event time and session for topology inputs Add `eventTime` (epoch seconds, UTC) and `sessionId` to `TopologyDocumentInput`. The extractor fills them from two documented metadata keys: `event_time` (integer epoch seconds, or ISO-8601 `YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive datetimes are UTC) takes precedence over the document createdTime; an unparsable value is skipped with a debug log, never an error, and a zero createdTime leaves the field unset. `session_id` is copied verbatim and empty when absent. The keys are only consulted when the extraction loads metadata; otherwise createdTime is the only source. `parseTopologyEventTime` is public for tests and future callers; the calendar math is Hinnant's days_from_civil with range-checked fields. Extraction stats count the event time source and invalid values. Nothing downstream reads the fields yet. Artifacts, codecs and both routing fingerprints are unchanged, which the new test pins. Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp): parser accept/reject tables, metadata precedence over createdTime, invalid ignored, session_id, includeMetadata=false fallback, and byte-identical snapshot payload plus equal construction and road fingerprints with and without the typed fields. Signed-off-by: trvon <me@trevon.dev> * bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar Every exported corpus document now carries `metadata.event_time` (ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM` haystack date, first-seen date per session) and `metadata.session_id`. Document ids, titles, texts, queries and qrels are unchanged in the default mode. `--round-docs` adds one document per session round (`<sid>__r<n>`): a user turn opens a round and the replies that follow join it; a leading assistant turn is its own round. Round docs share the session's `event_time`/`session_id` and add `round_index`; qrels stay at the session level. `--full-haystack` also writes `haystacks.jsonl`, one line per question with `haystack_session_ids` and the normalized `haystack_dates`, so per-question membership survives the global corpus deduplication (the 2026-10-10 preflight lost it). The converter is split into `convert` / `write_outputs`; `--self-test` runs both on a three-session synthetic fixture without a download and is the regression check for the default output. Signed-off-by: trvon <me@trevon.dev> * bench(retrieval): carry BEIR corpus metadata onto ingested documents `retrieval_quality_bench` ingests the BEIR corpus as a directory, which cannot carry per-file metadata, so `event_time`, `session_id` and `round_index` from corpus.jsonl never reached the topology extractor. `BEIRDocument::metadata` now holds the corpus line's `metadata` object (strings as is, numbers and booleans stringified; both the generic and the SciFact loader read it), and `applyBenchmarkDocumentMetadata` copies it onto the stored documents matched by exact file path through the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can repeat it). The bench runs it after the ingest wait, for cold and reused corpora alike, and logs how many documents were updated or not found. Tests: loader parses and coerces metadata; apply helper writes the keys, skips documents without metadata, counts missing ones and is idempotent against a real MetadataRepository. The portability allowlist entries for the two touched bench files are re-pointed at their shifted lines; no raw environment read was added. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f91cd85e13 |
feat(search): road beam in RouteOnly/Augment (#304)
Let the vector stage follow cluster roads from the selected route clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of both stored orientations of every road, per source sorted by weight desc then target cluster id asc; it is empty when the batch carries no roads, whether never built or dropped with `roadUnavailableReason`. The session runs a beam after the certificate is materialized: for `roadHops` hops it scores every unvisited road neighbour of the frontier by road weight times the router's query-centroid similarity ((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster id asc) and dedupes across hops. Each centroid evaluation is cached per cluster and counted once in `routeRepresentativeDistanceEvaluations` and `routeExactRepresentativeDistanceEvaluations`, so routed work stays comparable with the no-road arm. The result lands only in `TopologyRoutingSessionResult::roadExpandedClusterIds` and `roadExpandedDocumentHashGroups` (one member group per cluster, excluding certificate members). `certificate.allowedDocumentHashes`, admission and `topologyRoutingPolicyFingerprint` are untouched; the fingerprint comment and a test pin that the road knobs stay out of it. `VectorStageRequest` carries the road groups with their own quotas (`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly and Augment query them after the certified groups; the certified part of the routed result is identical with roads on or off. Shadow and Narrow never see road groups, and `makeTopologyRoutingOptions` requests hops only for Augment and a gate-open RouteOnly, so a gated-off RouteOnly running as Shadow takes no roads with it. Config: `search.topology.road_hops` (default 0, off) and `search.topology.road_beam_width` (default 4, read when hops > 0) on `SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`, copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy` and applied by SearchEngineManager. Defaults leave every stage byte-identical: the PR 2 goldens (692) and the RouteOnly cells (417) are unchanged. Debug keys appear only when hops > 0: `topology_road_hops_applied`, `topology_road_clusters_added`, `topology_road_candidates_added` (road documents the stage appended), `topology_road_work_centroid_evaluations`, `topology_road_skip_reason` (roads_absent, no_selected_routes, route_low_confidence) and, under stage trace, `topology_road_expanded_cluster_ids`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH write the two TOML keys. Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp (new catch2_search_vector_stage_road_beam): a four-cluster fixture with hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A; adjacency construction and cache carry; fingerprint equality for hops 0 vs 2 and any beam width; session-level certificate invariance with hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0 equals the route-only cell with no road keys, hops 1 gives A u B, hops 2 with beam width 1 gives A u B u C via the heavier road, work counters grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and Narrow take no road groups. ConfigResolver covers both TOML keys. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads are added, the bench reads its two keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
db53357462 |
feat(search): RouteOnly policy behind an evaluation gate (#300)
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
c26294a938 |
feat(topology): opt-in medoid representative and BQ rotation (#277)
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
trvon
|
e7bf9a05be
|
fix(bench): drop removed symbolRank option and tolerate duplicate-content corpora
nfcorpus contains byte-identical documents that collapse to one content hash; skip repeats when building topology inputs instead of aborting. |
||
|
trvon
|
e3f7e42492
|
feat(search): optional graph-rerank community signal from topology
Graph rerank recomputes connected components over mutual semantic_neighbor edges among the candidates on every query, reading each candidate's edges from the KG. The resident topology snapshot already holds corpus-wide clusters built from those edges. Add search.topology.graph_community_source (reciprocal_edges | topology_snapshot). topology_snapshot scores candidates that share a snapshot cluster with the same normalization and falls back to the edge walk when no snapshot is loaded; graph_community_source in debug stats reports which ran. The default stays reciprocal_edges until a benchmark gate shows the snapshot source is at least as good. The retrieval bench can set it for ablations. |
||
|
trvon
|
c766a17bd1
|
Merge branch 'refactor/remove-symbol-abi' into release/v0.20-integration
# Conflicts: # tests/scripts/portability_allowlist.txt |
||
|
trvon
|
ac070d8d2a
|
refactor(search): remove symbol-based result ranking
SymbolEnricher boosted results by looking up code-symbol nodes that no longer exist since v0.20, so it ran KG queries for the top results and never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the symbol_rank_* search stats, and the symbolRank option on the service, retrieval, MCP and CLI request types. The IPC field stays in the wire format but is ignored, and --symbol-rank/--no-symbol-rank remain accepted as hidden no-ops so existing scripts keep working. |
||
|
trvon
|
8912d70e5c
|
perf(bench): expose BQ routing work and new topology levers to benchmarks
Report topology_route_bq_used / topology_route_bq_candidates in search debug stats so benchmarks can tell BQ-shortlisted routing apart from ANN and exhaustive routing. Let retrieval_quality_bench write bq_candidate_limit, bq_prefix_dim, sgc_hops and simeon_rerank_outer_maxsim into its per-run config from YAMS_BENCH_* harness settings, read through getenv_optional. |
||
|
trvon
|
07b9f6b37b
|
fix(topology): bind calibration to routing policy identity
Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
5066fc0893
|
test: isolate process environment | ||
|
Trevon
|
0dd3985db4 |
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width. * refactor(vector): retire dormant backends Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage. Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation. * refactor(plugins): remove legacy SDK tooling Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture. * build(vector): remove stale FAISS overrides * refactor(ipc): remove dormant FSM snapshots * refactor(ingest): unify bounded KG enqueue * fix(plugins): harden allocation contracts Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0). * fix(storage): define empty object reads Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract. * refactor(storage): trim through string views * test: prune dormant harness helpers Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain. * chore(cpp): initialize analyzer-facing locals * build(simeon): advance analyzer cleanup * refactor(config): drop dead daemon client keys * refactor(ipc): inject daemon client transport Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths. * refactor(retrieval): route CLI through daemon Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection. * refactor(graph): move CLI scope into daemon Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct. * build: isolate test-only runtime controls * refactor(metadata): decouple lock telemetry * fix(services): return on request timeout * fix(retrieval): enforce workspace scope * fix(graph): scope nodes before pagination * fix(grep): drop binary result lines * fix(daemon): bound checkpoint wait * fix(repair): use KG health checks * fix(storage): always verify compressed CRC * perf(vector): skip unchanged PQ snapshots * build(zyp): exclude bench from default build * build: protect Meson files from formatter * refactor: remove unreachable production APIs * refactor(daemon): remove dormant runtime state * fix(vector): harden daemon provider calls Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production. * refactor(storage): remove unusable adapters * fix(cli): unify in-process add requests * fix(tsan): preserve async worker progress * test(cli): isolate daemon-dependent suites * fix(repair): expose queue backoff progress |
||
|
trvon
|
9618370d1f
|
feat(search): trace bounded topology rescue
Keep selector behavior default-off while recording complete eligible tails. Direct Simeon fragment scoring remains an experimental comparison path. |
||
|
Trevon
|
3f37e8f564 |
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols Signed-off-by: trvon <me@trevon.dev> * refactor(vector)!: remove dead transition APIs Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use. BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers. Signed-off-by: trvon <me@trevon.dev> * fix(vector): harden PQ snapshot lifecycle Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics. Signed-off-by: trvon <me@trevon.dev> * feat(topology): expose route coordinates Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero. Signed-off-by: trvon <me@trevon.dev> * feat(xplan): add vector decision gates Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries. Signed-off-by: trvon <me@trevon.dev> * feat(topology): formalize coordinate routing Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment. Signed-off-by: trvon <me@trevon.dev> * fix(search): serialize SymSpell readers SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN. Signed-off-by: trvon <me@trevon.dev> * fix(vector): make mutations atomic Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows. * fix(search)!: surface SymSpell write errors Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms. BREAKING CHANGE: SymSpellSearch mutations now return Result values. * feat(topology): formalize candidate expansion * feat(vector): add coordinate-aware embeddings * feat(topology): preserve protected relation cover * feat(search): add graph candidate rescue * feat(daemon): expose retrieval hydration telemetry * feat(xplan): add rescue and hydration gates * refactor(content): dedupe media handlers Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic. * feat(lifecycle): prune stale corpus data Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open. * feat(search): add topology candidate rescue Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit. * feat(search): expose evidence paths * docs: define mobile host contract * formal: model bounded candidate rescue * fix(prune): inspect extensionless signatures * fix(prune): tolerate stale extensions --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
7df64b1161 |
perf(search): bound topology routing and harden runtime paths (#59)
* fix: address review build issues * perf(search): bound topology routing Cache route membership, shortlist centroids with static ANN, and route SPQ through explicit allowed-document budgets. * test(xplan): measure routed vector work Pin benchmark binary identity and export route, cache, and phase costs for repeated equal-budget comparisons. * test(search): use portable jthread wrapper * fix(daemon): serialize detached cancellation * fix(cli): honor pinned daemon sockets * fix(xplan): support Windows contract imports * fix(list): apply metadata filters before paging * docs(prompts): require memory hydration Treat search results as candidates and hydrate selected artifacts before use. Quote argument hints for strict YAML loaders. * test(cli): keep add on pinned daemon * test(metadata): isolate performance timing * fix(graph): tolerate stale snippet ranges Source files may change before persisted symbol metadata is refreshed. Omit stale snippets instead of aborting graph exploration. * test(daemon): await request readiness Reuse the retrying list probe before asserting post-shutdown error quality under sanitizer load. * ci(release): use hosted cmake Resolve Conan CMake tool requirements from PATH so cold ARM64 caches do not depend on cmake.org. Default BUILD_DIR for failure cleanup. |
||
|
Trevon
|
ae15bf317a |
perf(ingest): harden daemon batching and shutdown (#58)
* fix(daemon): harden model state ownership Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads. * refactor(daemon): consolidate ingest pollers Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration. * refactor(daemon): unify repair execution Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths. * refactor(daemon): unify runtime tuning Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift. * refactor(daemon)!: remove dead IPC surfaces Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact. BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame. * refactor(daemon): hide test hooks Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols. * fix(daemon): avoid replaying successful adds A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again. * bench(ingest): enforce lossless pipeline runs Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift. * bench(search): expose cold seed ingest timings Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements. * perf(ingest): batch deferred content indexing Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window. * feat(metadata): expose insert writer metrics Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics. * fix(daemon): retry transient refused connects The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window. * bench(ingest): attribute submission path costs Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs. * perf(ingest): batch durable document storage Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success. * perf(ingest): coalesce bulk post-processing Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota. * bench(ingest): record post-ingest batch size Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system. * perf(ingest): batch queued storage waves Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock. * perf(ingest): route bulk enrichment batches Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing. * test(daemon): initialize task fixtures * perf(ingest): batch post-ingest dispatch * perf(ingest): widen directory storage waves Borrow from a full wave when one document would otherwise fall through to single-document storage. * perf(storage): bound batch writer fanout Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object. * perf(ingest): align storage batch drains Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work. * fix(ingest): commit metadata state atomically Fold first-version, extracted-title, and embedding-completion state into existing transactions. Read committed extraction state so duplicate-content tasks cannot double-count corpus counters. * bench(ingest): measure production submission Drive pipelined files through DocumentIngestionService. Export dispatch, fingerprint, and enqueue costs through xplan. * fix(xplan): preserve native failures Report child signals and bounded stderr before readiness checks so assertions are not masked as missing traces. * fix(daemon): preserve embedding success hashes * fix(daemon): serialize pooled socket shutdown * fix(daemon): harden batch publication shutdown * fix(storage): use compat jthread shim * fix(daemon): avoid poller template shadowing * fix(search): keep vector pipeline test enabled * fix: harden shutdown and fallback paths Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes. |
||
|
Trevon
|
a3d065dd8d |
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish
Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.
* fix(daemon): stop shutdown log storm
Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.
* refactor(cli): delegate serve to MCP binary
Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.
* refactor(cli): move daemon path helpers
Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.
* refactor(daemon): split embedded client
* perf(search): add lean-guided benchmark gates
Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.
Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.
* refactor(config): move daemon policies
* refactor(daemon): extract plugin runtime
* refactor(daemon): untangle service coordinators
Untangle CLI/MCP from daemon service coordinators and the daemon static lib.
- MetadataRepository uses a daemon-neutral tree-diff callback. The
GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
static lib. Drop ResourceGovernor coupling inside the writer loop; batch
sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
its own enqueuePostIngest; app services no longer call ServiceManager
directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
app services link cleanly without daemon.
Validated:
meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
yams-cli --version, yams-mcp-server init, yams serve --help smoke tests
Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.
* fix(cli): keep stdin add local fallback
No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.
Validated:
meson compile -C build/debug -j4 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
meson compile -C build/release -j4 yams-cli yams-mcp-server
meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output
* feat(bench): add xplan multi-arm experiment harness
Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.
* feat(formal): expand topology/search Lean contracts
Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.
* feat(search,topology): construction purity + graph expansion path
Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.
* feat(topology): add KMeans engine and SGC identity contracts
Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.
* feat(daemon): topology readiness gates and selectable reranker
Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.
* chore(search): drop meta_path tuner knobs
Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].
* feat(bench): add topology_purity_validate plan
Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.
* chore: format add_command and config include order
Mechanical clang-format / include reorder only; no behavior change.
* feat(bench): decision-grade xplan loop for search engine
Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.
* feat(graph): denser product semantic_neighbor topK
Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.
* feat(search): extract topology assist and cross-rerank seams
Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.
* feat(bench): product-path search engine ablation plans
Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.
* fix(ci): repair pre-push fallbacks
* feat(bench): lock-free mid-run builds + multi-corpus gates
xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.
* fix(daemon): keep SearchEngine alive via shared_ptr cache
getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.
Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.
* feat(search): route vectors through cached topology
Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.
Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.
* feat(bench): consolidate search decision plans
Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.
Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.
* feat(search): refine topology vector routing
* fix(search): propagate daemon readiness
* feat(bench): measure generalized memory health
* fix(cli): honor in-process fallbacks
* ci: harden smolvm debug builds
* perf(search): route topology through filtered ANN
* feat(bench): measure read-write pressure
* perf(metadata): index symbol lookups on read pool
* fix(daemon): isolate startup reads and maintenance
* feat(bench): replay isolated generalized corpora
* feat(search): add evidence routing pipeline
Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.
* refactor(search): unify topology evidence path
Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.
* refactor(bench): prune superseded search lanes
Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.
* docs(search): retire stale experiment notes
Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.
* refactor(search): prune dead search surfaces
Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.
* fix(cli): clear build warnings
* fix(xplan): reject stale run identity
* refactor(search)!: remove legacy query stack
Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.
BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.
* refactor(search): type value stage traces
Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.
* refactor(search): isolate benchmark env overlays
Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.
* refactor(search): unify topology routing options
Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.
* fix(search): unify tuner config ownership
Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.
* fix(search): snapshot tuner state per query
Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.
* fix(search): enforce topology routing contract
Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.
* fix(daemon): let SQLite reclaim its file
Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.
* fix(metadata): tolerate missing symbol FTS
Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.
* build(fuzz): drop removed query parser target
The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.
* fix(bench): include writes in drain oracle
A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.
* fix(daemon): vacuum padded WAL databases safely
|
||
|
Trevon
|
f90c2819bd |
perf: improve ingestion optimization and profiling infrastructure (#52)
* perf: improving profile and assertions in ingestion and retrieval paths * perf: fix tests and profiling infrastructure * perf: improving profiling implementation and optimizations in ingestion path * perf: ingestion optimizations and profiling improvements * perf(ingest): cache dispatch policy resolution * perf: improving measurements of subsystems * perf(storage): batch ref-count commits * perf(ingest): route metadata writes through dedicated writers * perf: additional optimizations and benchmark docs update * docs(bench): add profiling bundle helpers * docs: make pipeline diagram user-facing |
||
|
Trevon
|
4228a0c5e9 |
chore: housekeeping — rm dead code, consolidate MCP tools, add tests, clean docs (#50)
* implemented stubbed IGraphContextService methods * feat: wire graph communication updates in ipc, mcp and cli * fix: fix call graph recall and add profiling coverage to graph calls sites * fix: fixes for work where we added agent navigation to graph tooling * fix(corpus): clean FTS and KG orphans on document delete * fix(repair): clean orphans, honor cancellation, surface stats * fix(rm): order content removal before metadata/KG delete. keep manifest while content is still referenced * fix(daemon): seed built-in content extractor for readiness * chore(cli): remove dead stubs and unwired helpers * feat(search): named-collection scoping + provable relevance gate * fix(refactor): rm dead stubs, inline simd, add proto/metrics tests * chore: rm dead interfaces/enums, consolidate guards, add tests * refactor: rm dr vaporware, collapse download handlers, clean list flags * docs: rm stale testing investigation logs * fix: address Copilot review findings from PR #50 - relationship_ops: return existing id when INSERT OR IGNORE skips - document_crud_ops: propagate FTS5 cleanup error (was silent skip) - builtin_text_content_extractor: unsigned char cast for tolower, add cctype - mcp_server download_jobs: validate job_id for status/cancel actions - list_command tests: unsigned char cast for isspace, add cctype include |
||
|
Trevon
|
771deebc84 |
fix: harden CI and metadata stability (#48)
* feat: dead code removal, profling foudnation improvements, addressing code smells in reference db implementation * fix: test fixes and optimmizations for benchmarks * fix: harden audit findings Reject invalid tokenizer ids, fix move-order hazards, bound manifest and audio IO, and guard numeric/process edge cases found by Fold. * fix: route SymSpell writes through coordinator * test: fix batch content initializers * fix(ci): restore cross-platform builds * fix: address Copilot review findings |
Renamed from src/search/benchmarks/retrieval_quality_bench.cpp (Browse further)