186 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
d398556ee5 |
fix(tests): replace frozen expectations and harden release gates (#323)
* test(search,daemon,storage): assert behaviour instead of frozen literals Several suites encoded exact internal values that legitimate tuning and ranking changes are allowed to move: SCIENTIFIC-era profile weights and rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation numbers. They failed on every platform after those changes landed, so the real behaviour regressions they should catch were hidden behind noise. Replace the frozen literals with contract assertions: weights are finite, non-negative and normalize, state selection still holds, capability-gated sources stay zeroed, and hybrid results keep matching canonical hash/path sets. Deadlines are not widened and the daemon paths they cover are unchanged. Also add independent corpus and adapter oracles, deterministic collector cancellation and late-result gates, and synchronize ServiceManager shutdown observation, so the tests fail for the behaviour they name. The portability allowlist keeps its two reviewed raw test-env reads; only their line numbers moved. Signed-off-by: trvon <git@trevon.dev> * ci(tests): summarise Meson failures without flooding the log A failing test run printed the whole Meson suite output, so the actual failing test names and their assertion messages were buried and the uploaded logs were dominated by passing-test chatter. Add a bounded reporter that extracts the failing test names plus their surrounding diagnostics, keeps an overall cap on emitted lines, and is unit-tested against representative Meson output. Wire it into the Tests workflow so failures stay readable. Signed-off-by: trvon <git@trevon.dev> * build(deb): validate candidate identity, mapping and upgrade ordering The Debian packaging scripts took filenames and versions on trust, so a misnamed candidate or a wrong package ordering could pass validation while the real install path was never exercised. Fail closed on candidate identity, version and architecture, verify the package-to-source mapping, seed the legacy-only layout, wait on the actual services, and assert install/upgrade/remove ordering for both Debian and RPM in an offline container. Cases that cannot be proven on a host are reported as skipped rather than silently passing. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
3272367ea2 |
fix(search): restore measured scientific ranking (#318)
* fix(search): make the final lexical tie-break a strict weak ordering The final comparator broke near-ties toward the result with less vector support, and its pairwise |a - b| <= epsilon test was not transitive, which is undefined behavior for the std::sort/partial_sort callers in applySearchTopKLimit. Compare tie buckets floor(score / epsilon) instead, then lexical anchor, keyword and vector evidence (more first), then doc id. Behavior with the tie-break off is unchanged. The comparator moves to final_result_order_internal.h so it can be tested directly. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit ca9b82d6ce9db31b6c9608a7d95b63203649e5a7) Signed-off-by: trvon <git@trevon.dev> * fix(search): keep cross-reranked scores above the unreranked tail applyCrossRerank rewrote the window's scores to a 0..1 min-max blend while the tail kept raw fused scores. Later whole-list sorts (concept boost, top-k partial_sort) compared the two scales, so reranked docs could drop below tail docs or out of the top-k. Map the blended score back onto the window's original range, with the floor raised just above the best tail score when needed. The window's reranked order is unchanged and docTraces.finalScore records the mapped score. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit d66c6d304ca4d32b943e164b0d7720fc7f865db1) Signed-off-by: trvon <git@trevon.dev> * fix(search): judge weak tier-1 queries on the raw bm25 scale The weak-query test compared the top Text/SimeonText component score with weakQueryMinTopTextScore, but lexical batches are min-max normalized, so the top hit is always 1.0. With enough hits every query was strong: the weak-query vector fanout boost and weak-query topology expansion never ran. scoreLexicalBatch now also records absoluteLexicalScore, the hit's score on the existing absolute scale (-bm25 / bm25NormDivisor, same multipliers), and classifyTier1TextSignal uses it. Thresholds keep their values; with the default divisor 25, 0.30 is bm25 <= -7.5 and 0.45 is bm25 <= -11.25. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit bc771c5aab1721170fb9e743013ca0efbf048d61) Signed-off-by: trvon <git@trevon.dev> * fix(search): restore measured scientific fusion Give SCIENTIFIC the measured relative Simeon weight of 0.40 after normalizing the existing text/vector pair. The resulting share is 0.285714, matching the benchmark traces. Other profiles remain unchanged. Keep its one-slot semantic rescue preset instead of raising it to three at the corpus layer. Preserve the existing large-prose recall widening. Cover the preset weights, config propagation, community blend, rich-graph ratios and corpus rescue behavior. Signed-off-by: trvon <git@trevon.dev> * fix(search): order non-finite ranking evidence Treat NaN score/evidence keys as negative infinity and invalid tie epsilon as zero. This keeps comparator equivalence transitive without changing finite-input ordering. Add exhaustive triples covering NaN, infinities and invalid epsilon. Signed-off-by: trvon <git@trevon.dev> * docs(search): clarify lexical scoring limits Distinguish the shared single-hit score conversion from FTS5 BM25 calibration. Note that lexical tie buckets can still reorder numerically separated rerank-window and tail scores. No behavior change. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
d2252877ad |
fix(search): skip per-hash score lists under MAX chunk aggregation (#293)
aggregateChunkVectorScores collected every chunk score per document even when the strategy was MAX, which only needs the best record. Skip the list for MAX; the other strategies are unchanged. Test: search_vector_pipeline. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
8881e0b3e9 |
perf(memory_sync): skip idle rescans and listings of a direct op store (#311)
* test(search): rename a road beam local that collides with the Windows far macro windows.h defines far and near as empty macros, so `const auto far = ...` in search_vector_stage_road_beam_catch2_test.cpp does not compile with MSVC (C2513), and the Windows Tests job has failed since the road beam tests landed. Rename it to deep. Signed-off-by: trvon <me@trevon.dev> * perf(memory_sync): skip rescans of an unchanged direct op store The sync worker rescanned the whole op store every cycle: a full walk of the sharded objects tree, then a read, hash, decode and signature check of every index envelope. On an idle 3-node mesh (2 CPUs per node, 5,256 objects) that cost about 1.1 s of CPU per 5 s cycle on each node, mostly in getdents/openat. In direct transport the daemon is the op store's only writer. The loop now counts its own backend writes and reuses the last complete scan until it writes again or a writer is quarantined. Shared-store and temporary namespaces keep rescanning every cycle. Signed-off-by: trvon <me@trevon.dev> * perf(memory_sync): keep exclusive-backend listings until the loop writes FilesystemBackend::listPage walks every shard directory of the op store for any prefix. After the rescan skip, each idle direct-transport cycle still listed the store twice: replaying ready erases and inspecting the erase outbox. The loop's write-counting backend now keeps listing pages on an exclusive backend and drops them on any store, remove or clear. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
beef87e66c |
bench(retrieval): seed one-way kNN semantic edges on request (#308)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
Benchmark seeding wrote only mutual neighbor pairs. The connected topology engine keeps every reciprocal edge at or above min_edge_score inside its component, so with mutual-only seeding no raw neighbor crosses a cluster boundary and the engine builds no roads. The Eval 2 smoke showed roads_absent on every connected-engine query. The daemon stores one-way kNN edges (GraphComponent treats them as ordinary non-mutual neighbors). YAMS_BENCH_SEED_SEMANTIC_DIRECTED=1 seeds them too, as single directed edges. Default off keeps existing plans unchanged. The pair loop moves into buildSemanticNeighborSeedEdges in the bench support library, sorted by (source, target), with a unit test for both modes. A topology test pins the mutual-only case: no roads on the connected engine until a one-way edge exists. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
81a051fc2e |
feat(search): walk along roads (#307)
Let RouteOnly and Augment expand every routed hit along its cluster's walk order. The snapshot cache loader resolves each cluster's `orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`, `TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with `orderKind` None or a permutation that fails `validateMemberPermutation` resolves to nothing. The store's `orderUnavailableReason` is carried on the session result and emitted as `topology_order_unavailable_reason` only when set. When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex` after the road beam: the selected routes first, then the road-expanded clusters, keeping only clusters with an order and counting the rest in `roadWalkClustersUnordered`. The index borrows its member views from the snapshot it keeps alive; the certificate is never touched. The vector stage walks after the certified and road groups. Every routed hit so far (certified route hits and road-beam hits) anchors +-window positions along each order it belongs to, bounded to that order (`walkTopologyRoadOrders`). Anchoring on route hits too means the walk rescues the ordinal neighbours of the semantically closest members even with `road_hops` = 0; the plan's "around road hits" reads as every hit a road-era routed query produced. The walked set, deduplicated against the hits, is one extra routed query whose quota is its own size (ranking ordinal neighbours by similarity and truncating would reintroduce the filter the walk bypasses); its rows and distance evaluations land in the routed diagnostics like road candidates. Shadow and Narrow never walk. Config: `search.topology.road_walk_window` (default 0, off) on `SearchEngineConfig::topologyRoadWalkWindow` (copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by SearchEngineManager. `makeTopologyRoutingOptions` requests the window only for Augment and a gate-open RouteOnly, like the beam. The window is excluded from `topologyRoutingPolicyFingerprint` (comment and test). Defaults leave every stage byte-identical: the PR 2 goldens (692), the RouteOnly cells (417) and the road beam cells (904) are unchanged. Debug keys appear only when the window > 0: `topology_road_walk_window_applied` (0 when nothing was walkable), `topology_road_walk_anchors`, `topology_road_walk_candidates_added`, `topology_road_walk_clusters_unordered`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes `search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and `topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3 can set the order kind per arm and build roads at all). Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp (new catch2_search_vector_stage_road_walk): two reciprocal chains A (six members) and B (four) with a road, built with `TopologyBuildConfig::roadOrder` Time from eventTime so the walk order differs from the similarity ranking; loader resolution including the invalid-permutation drop and the cache carry; fingerprint equality for any window and the option gating per policy; session-level certificate, beam and work invariance with the index listing routes then road clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an unordered cluster counted and skipped, no index without any order, the order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0 equal to the road beam cell with no walk key, window 1 adding the immediate order neighbours of all four hits, window 2 the whole clusters, rows visited growing while router work is unchanged, certified keys unchanged, unordered B contributing nothing, no order anywhere equal to the beam cell plus counters, hops 0 walking the selected route alone; Augment adding the walked documents as rescues; Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads, the bench reads its keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
111988c6e4 |
feat(topology): typed event time and session for topology inputs (#305)
* feat(topology): typed event time and session for topology inputs Add `eventTime` (epoch seconds, UTC) and `sessionId` to `TopologyDocumentInput`. The extractor fills them from two documented metadata keys: `event_time` (integer epoch seconds, or ISO-8601 `YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive datetimes are UTC) takes precedence over the document createdTime; an unparsable value is skipped with a debug log, never an error, and a zero createdTime leaves the field unset. `session_id` is copied verbatim and empty when absent. The keys are only consulted when the extraction loads metadata; otherwise createdTime is the only source. `parseTopologyEventTime` is public for tests and future callers; the calendar math is Hinnant's days_from_civil with range-checked fields. Extraction stats count the event time source and invalid values. Nothing downstream reads the fields yet. Artifacts, codecs and both routing fingerprints are unchanged, which the new test pins. Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp): parser accept/reject tables, metadata precedence over createdTime, invalid ignored, session_id, includeMetadata=false fallback, and byte-identical snapshot payload plus equal construction and road fingerprints with and without the typed fields. Signed-off-by: trvon <me@trevon.dev> * bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar Every exported corpus document now carries `metadata.event_time` (ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM` haystack date, first-seen date per session) and `metadata.session_id`. Document ids, titles, texts, queries and qrels are unchanged in the default mode. `--round-docs` adds one document per session round (`<sid>__r<n>`): a user turn opens a round and the replies that follow join it; a leading assistant turn is its own round. Round docs share the session's `event_time`/`session_id` and add `round_index`; qrels stay at the session level. `--full-haystack` also writes `haystacks.jsonl`, one line per question with `haystack_session_ids` and the normalized `haystack_dates`, so per-question membership survives the global corpus deduplication (the 2026-10-10 preflight lost it). The converter is split into `convert` / `write_outputs`; `--self-test` runs both on a three-session synthetic fixture without a download and is the regression check for the default output. Signed-off-by: trvon <me@trevon.dev> * bench(retrieval): carry BEIR corpus metadata onto ingested documents `retrieval_quality_bench` ingests the BEIR corpus as a directory, which cannot carry per-file metadata, so `event_time`, `session_id` and `round_index` from corpus.jsonl never reached the topology extractor. `BEIRDocument::metadata` now holds the corpus line's `metadata` object (strings as is, numbers and booleans stringified; both the generic and the SciFact loader read it), and `applyBenchmarkDocumentMetadata` copies it onto the stored documents matched by exact file path through the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can repeat it). The bench runs it after the ingest wait, for cold and reused corpora alike, and logs how many documents were updated or not found. Tests: loader parses and coerces metadata; apply helper writes the keys, skips documents without metadata, counts missing ones and is idempotent against a real MetadataRepository. The portability allowlist entries for the two touched bench files are re-pointed at their shifted lines; no raw environment read was added. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f91cd85e13 |
feat(search): road beam in RouteOnly/Augment (#304)
Let the vector stage follow cluster roads from the selected route clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of both stored orientations of every road, per source sorted by weight desc then target cluster id asc; it is empty when the batch carries no roads, whether never built or dropped with `roadUnavailableReason`. The session runs a beam after the certificate is materialized: for `roadHops` hops it scores every unvisited road neighbour of the frontier by road weight times the router's query-centroid similarity ((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster id asc) and dedupes across hops. Each centroid evaluation is cached per cluster and counted once in `routeRepresentativeDistanceEvaluations` and `routeExactRepresentativeDistanceEvaluations`, so routed work stays comparable with the no-road arm. The result lands only in `TopologyRoutingSessionResult::roadExpandedClusterIds` and `roadExpandedDocumentHashGroups` (one member group per cluster, excluding certificate members). `certificate.allowedDocumentHashes`, admission and `topologyRoutingPolicyFingerprint` are untouched; the fingerprint comment and a test pin that the road knobs stay out of it. `VectorStageRequest` carries the road groups with their own quotas (`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly and Augment query them after the certified groups; the certified part of the routed result is identical with roads on or off. Shadow and Narrow never see road groups, and `makeTopologyRoutingOptions` requests hops only for Augment and a gate-open RouteOnly, so a gated-off RouteOnly running as Shadow takes no roads with it. Config: `search.topology.road_hops` (default 0, off) and `search.topology.road_beam_width` (default 4, read when hops > 0) on `SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`, copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy` and applied by SearchEngineManager. Defaults leave every stage byte-identical: the PR 2 goldens (692) and the RouteOnly cells (417) are unchanged. Debug keys appear only when hops > 0: `topology_road_hops_applied`, `topology_road_clusters_added`, `topology_road_candidates_added` (road documents the stage appended), `topology_road_work_centroid_evaluations`, `topology_road_skip_reason` (roads_absent, no_selected_routes, route_low_confidence) and, under stage trace, `topology_road_expanded_cluster_ids`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH write the two TOML keys. Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp (new catch2_search_vector_stage_road_beam): a four-cluster fixture with hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A; adjacency construction and cache carry; fingerprint equality for hops 0 vs 2 and any beam width; session-level certificate invariance with hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0 equals the route-only cell with no road keys, hops 1 gives A u B, hops 2 with beam width 1 gives A u B u C via the heavier road, work counters grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and Narrow take no road groups. ConfigResolver covers both TOML keys. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads are added, the bench reads its two keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
db53357462 |
feat(search): RouteOnly policy behind an evaluation gate (#300)
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
2ba8cb4a86 |
fix(search): vector-stage lifetime on component timeout (#299)
* test(search): vector stage completes into its own state on component timeout
Executor-backed SearchEngine with a vector hydration step that blocks until the
test releases it. With a small componentTimeout the search returns without the
vector component; the test then releases the worker and runs a second search
from the same call site, so the second search's locals reuse the first frame's
stack region. A stage that still writes into the first frame races with the
second search (TSan data race) until the stage state is heap-owned by the
posted work. A second case pins that a component meeting its deadline is still
collected with its diagnostics.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): vector-stage lifetime on component timeout
The posted vector work still reached into the search frame after a
componentTimeout: its injected query wrapper captured `this`, workingConfig
and four frame-local relaxed-retry atomics, and the frame adopted the stage
outcome unconditionally after collectIf, even when the component had timed out
and the worker was still running (a data race on the outcome, and a stale write
into whatever next occupies that stack region once the frame returns).
Give the wrapper a heap home: VectorStageState {SearchEngineConfig config;
relaxedRetryEnabled; RelaxedVectorRetryStats relaxedRetry; VectorStageOutcome
outcome} is captured by shared_ptr in the posted lambda and the query wrapper is
built from it by makeQueryVectorWithRelaxedRetry, which holds the repository
and vector database by value, so the work depends on neither the frame nor the
engine. The sequential path uses the same factory bound to frame locals; the
relaxed-retry atomics become a plain struct only the frame thread touches.
collectAndRecord now returns the ComponentStatus. The frame adopts the stage
state only when the component finished (Success or Failed); after TimedOut or
Canceled the late completion lands in state only the worker references and is
dropped with it, so the diagnostics merge and route-work application are
no-ops. Behavior with no timeout is unchanged.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
3606914222 |
test(search): characterize the vector stage (#297)
Pin the current behaviour of the vector candidate stage in
search_engine.cpp before it is extracted into search_vector_stage.{h,cpp}.
A new catch2_search_vector_stage binary runs the matrix policy
{Shadow, Augment, Narrow commit, Narrow fallback} x backend
{SimeonPqAdc, ExactScan, Vec0L2} with stage trace off and on, plus one
executor-backed fanout case, and compares each cell against explicit
goldens: fused hash/rank/score and vector contribution, raw vector
ComponentResults (trace_component_hits_json), the stage's debug keys
without timings, certificate.work / admission.work via the
topology_route_work_* and admission keys, and the merged vector_search_*
diagnostics. Quirks are pinned as-is: relaxed_vector_primary_hit_count is
last-writer, Vec0 keeps global results unchanged under Shadow/Augment,
Narrow commits only on Simeon PQ because ExactScan and Vec0 never observe
an ANN candidate budget (work_observation_unavailable), and Vec0 leaks a
partial ann_candidate_budget_actual into the unobserved work record. A
hidden [.dump] case regenerates the goldens.
Move the shared topology fixtures (TopologySearchFixture,
buildTwoClusterTopologyBatch, topologyRoutingTestConfig,
runTopologySearch, configureCertifiedTopologyRoute, TopologyVectorEnvPin)
into tests/unit/search/topology_search_test_support.h so both tests use
the same corpus. runTopologySearch takes an optional executor and
configureCertifiedTopologyRoute an options struct; existing call sites are
unchanged. The portability allowlist entry for the raw YAMS_TEST_TMPDIR
read moves with the helper. No product behaviour changes.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
4e97b31fef |
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
|
||
|
Trevon
|
c26294a938 |
feat(topology): opt-in medoid representative and BQ rotation (#277)
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
1d1e8def4d |
fix(search): count augment route members as added candidates (#276)
* test(search): pin augment routed-candidate accounting Under the augment vector policy a document reachable only through a topology route reaches the fused results through the candidate-rescue vector merge, but the routing session and the topology_weak_query_added_candidates stat report zero additions, because the product stage always collects route membership and the session skips the add loop in that mode. Pin the routed-only fixture (x2 in the route, outside lexical Tier-1 and the global vector top-1) for augment, and characterize shadow (identical to topology-disabled) and uncertified narrow (global, no additions). Signed-off-by: trvon <me@trevon.dev> * fix(search): count augment route members as added candidates #57 unified topology onto one evidence path: the product assist stage always collects route membership, and augment, shadow and narrow all consume the certificate's allowed set (augment vector-scores it through the candidate-rescue merge, shadow measures it, narrow filters to it). The session skips the old add loop in that mode, so under augment addedCandidates, duplicateCandidates and topology_weak_query_added_candidates stayed 0 even when routed documents were rescued into the fused results. Under augment only, count materialized allowed-set members that are new to the existing candidate pool as added candidates and the rest as duplicates. The allowed set, its maxDocs bound, the routing policy fingerprint, applied and skip reasons are unchanged; narrow and shadow report no additions as before. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
trvon
|
8aae9245ef
|
fix(search): stop the simeon-lexical build when shutdown begins
During a search storm on a large corpus, daemon shutdown blocked in Phase 6.7 (SearchEngineManager::clearEngine) until the 120 s watchdog forced an exit. Dropping the engine destroys SimeonLexicalBackend, whose destructor joins the BM25 build thread, and the build only looked at its stop token once every 1024 documents, each needing a metadata fetch. Stop was also requested only from that destructor, so the build had no head start while the earlier shutdown phases ran. SimeonLexicalBackend::requestStop() now flags the build without joining it and makes later buildAsync() calls no-ops. The build checks the flag (alongside the jthread token) before every document fetch in the corpus and fragment passes and before fragment geometry. SearchEngine and SearchEngineManager expose requestShutdown(), and ServiceManager calls it in Phase 4, so the build thread has normally exited by the time clearEngine() destroys the engine. simeon's own finalize, concept-mining and PMI-learning steps take no stop token, so a build already inside one of them still runs it to completion; making those interruptible needs a change in the simeon submodule. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
6aacda7913
|
fix(daemon): cancel in-flight request work when shutdown begins
A clean `yams daemon restart` never published the clean-shutdown integrity stamp, so every start paid for a full PRAGMA quick_check (13-27 min on a 93 GB database). Request handlers offloaded to WorkCoordinator threads (list, grep, get, search) kept running through Phase 5: nothing interrupted a long SQLite statement or a long result loop, both 5 s and 30 s joins expired, the workers were abandoned, and the stamp was disabled. Shutdown now tells in-flight work to stop before joining: - ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt on every leased connection under the pool mutex, which keeps each lease open while it is interrupted. DatabaseManager exposes it for both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each join wait slice through a new joinWithTimeout hook, because SQLite drops an interrupt that lands between two statements. - ServiceManager owns a shutdown cancellation flag, set at Phase 4 before the interrupt, and publishes it as AppContext::cancellationSignal. DocumentService::list and GrepService::grep check it between query and hydration steps and between document batches, and return OperationCancelled; SearchService checks it on entry and forwards it as SearchExecutionContext::hostCancellationSignal, which the engine's existing cancellation checks and the fanout collector now honor. - The concept-extraction wait no longer blocks without bound when componentTimeout is 0, and a canceled search stops waiting for the query embedding / concept tasks. Those tasks now own copies of their inputs instead of capturing the caller's stack, so abandoning them is safe. Interruption stays scoped to shutdown: the flag is only set in Phase 4 and cleared on initialize, and interrupting an idle lease does not fail its next statement. Also erase a connection from the leased set when its rollback throws; it was left behind as a dangling pointer, which the interrupt walk would have dereferenced. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
4ff7279c90
|
fix(search): report the scoring mode of the wired cross reranker
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon resolves the Simeon scoring mode from reranker policy (ConfigResolver), so the stat said 0 for every outer-MaxSim run and the scifact ablation could not tell which scorer ran. setCrossReranker() now takes the mode label the wiring resolved; the engine reports it as cross_rerank_scoring_mode and derives the outer-MaxSim stat from it. |
||
|
trvon
|
e3f7e42492
|
feat(search): optional graph-rerank community signal from topology
Graph rerank recomputes connected components over mutual semantic_neighbor edges among the candidates on every query, reading each candidate's edges from the KG. The resident topology snapshot already holds corpus-wide clusters built from those edges. Add search.topology.graph_community_source (reciprocal_edges | topology_snapshot). topology_snapshot scores candidates that share a snapshot cluster with the same normalization and falls back to the edge walk when no snapshot is loaded; graph_community_source in debug stats reports which ran. The default stays reciprocal_edges until a benchmark gate shows the snapshot source is at least as good. The retrieval bench can set it for ablations. |
||
|
trvon
|
e31eda4b55
|
fix(search): measure graph richness from relational edges only
kgEdgeDensity counted every kg_edges row, including the ~16 semantic_neighbor edges per document that 8-NN embedding neighbours write in both directions, plus file-lineage and entity-attachment edges. Any embedded corpus therefore looked like a rich knowledge graph, so the SCIENTIFIC-profile gate that should zero the KG leg on flat prose never fired and graph richness scaled from embedding edges. Add kgRelationalEdgeCount/Density, which exclude embedding kNN, file lineage and layout, text segments, and entity-to-document attachments, and use it for hasRichGraphTopology() and the tuner's richness scaling. kgEdgeCount keeps its meaning as the reconciled-graph signal. The live overlay carries the reconciled relational count forward. |
||
|
trvon
|
c766a17bd1
|
Merge branch 'refactor/remove-symbol-abi' into release/v0.20-integration
# Conflicts: # tests/scripts/portability_allowlist.txt |
||
|
trvon
|
ac070d8d2a
|
refactor(search): remove symbol-based result ranking
SymbolEnricher boosted results by looking up code-symbol nodes that no longer exist since v0.20, so it ran KG queries for the top results and never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the symbol_rank_* search stats, and the symbolRank option on the service, retrieval, MCP and CLI request types. The IPC field stays in the wire format but is ignored, and --symbol-rank/--no-symbol-rank remain accepted as hidden no-ops so existing scripts keep working. |
||
|
trvon
|
7eb64fa8a2
|
test(search): include the BQ index header in the snapshot cache test
The BQ prefix assertions dereference centroidBqIndex, which the routing headers only forward-declare. |
||
|
trvon
|
b9ec471c4c
|
feat(topology): wire SGC smoothing and BQ routing through typed config
sgcHops, sgcNormalize, bqCandidateLimit and bqPrefixDimension existed only as struct fields, so no production path could enable them. Resolve topology.sgc_hops / topology.sgc_normalize into the TopologyManager build config (hops capped at 4 to bound over-smoothing) and search.topology.bq_candidate_limit / bq_prefix_dim into the routing options, request, and fingerprint. The snapshot cache rebuilds only the route index when the BQ prefix changes. All default to off. Drop the snapshot interner, which was allocated but never used. |
||
|
trvon
|
1550dad3be
|
perf(search): flyweight string interning in TopologyRoutingSnapshotCache
- Add StringInterner in include/yams/search/string_interner.h for reusable, thread-compatible flyweight string deduplication with heterogeneous string_view lookups. - Update TopologyRoutingSnapshot in include/yams/search/topology_routing_session.h to store zero-copy string_view keys for clustersById and membershipsByDocumentHash, pointing directly into immutable TopologyArtifactBatch strings. - Add shared_ptr<StringInterner> to TopologyRoutingSnapshot to intern repeated lookup and dynamically generated strings. - Eliminate ~75,000 duplicate heap-allocated std::string objects per cached snapshot, reducing cache footprint by ~70%. - Add Catch2 tests in tests/unit/search/string_interner_catch2_test.cpp covering interning, deduplication, lifetime guarantees, and zero-copy pointer preservation across snapshot upgrades. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
9c7f780d4c
|
refactor(symbols): remove symbol extraction subsystem and add migration 40
Drop legacy Tree-sitter symbol extraction subsystem, associated database tables, triggers, and AST edge types: - Migration 40 drops symbol_metadata, document_symbol_extraction_state, symbol_metadata_fts, associated triggers, and prunes historical bloated topology snapshots while preserving active snapshot topology, semantic_neighbor edges, and non-symbol observed_as provenance edges. Exclude nl_entity:% nodes from kg_nodes / kg_edges pruning to preserve canonicalized method-type technology entities. Provides complete downSQL restoring symbol tables, indexes, FTS virtual table, and sync triggers for clean v39 rollbacks. - In EntityGraphService, ensure locally resolved documentDbId propagates to WriteBatch for knowledgeGraphDocumentId to ensure accurate enrichment token completion, and only populate batch->documentIdToDelete for snapshot documents with a resolved document database ID. - In SymbolEnricher and SearchService, restore one-directional symbol-contains-query matching with non-empty checks to prevent short substrings matching broad queries. - In tests/unit/metadata/symbol_metadata_schema_test.cpp, migrate test DB up to version 39 so tests pass against their target version prior to v40 table drops. - Remove Tree-sitter grammar download and compile workflows from CLI init - Default plugin-symbols to false. - Fix macOS daemon client and CLI executable resolution using _NSGetExecutablePath with candidate fallbacks. - Add migration validation tests and symbol enrichment unit tests. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e4b32e98bd |
fix: fence enrichment completion and reconcile promotion prerequisites (#157)
Some checks failed
Secret scan / gitleaks (push) Failing after 8s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* test(ci): stabilize main matrix (#112) Signed-off-by: trvon <me@trevon.dev> * ci(release): preview experimental promotion (#111) Signed-off-by: trvon <me@trevon.dev> * ci(release): reconcile draft promotion safely (#113) * ci(release): reconcile draft promotion Create promotion visibility independently of release readiness. Leave existing PRs untouched to preserve concurrent maintainer edits. Validate against current main without masking candidate rejection. Signed-off-by: trvon <me@trevon.dev> * ci: isolate hook Git environments Prevent temporary Git fixtures from modifying the calling worktree. Clear repository-local hook variables before the unchanged build gates, and isolate validator and fixture Git subprocesses from caller overrides. Exercise linked-worktree config, ref and index preservation offline. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * fix(ci): reject invalid candidate evidence Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values. Signed-off-by: trvon <me@trevon.dev> * fix: preserve completion and retrieval contracts Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures. Signed-off-by: trvon <me@trevon.dev> * fix: fence asynchronous enrichment completion Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage. Signed-off-by: trvon <me@trevon.dev> * fix: bound semantic graph output reservation Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward. Signed-off-by: trvon <me@trevon.dev> * fix: fence standalone embedding repair completion Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass. Signed-off-by: trvon <me@trevon.dev> * fix: preserve enrichment failure reporting Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged. Signed-off-by: trvon <me@trevon.dev> * test: align explicit hash prefix boundaries Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun. Signed-off-by: trvon <me@trevon.dev> * test: align UI hash validation boundary Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
60855b7b78 |
[11/14] Separate status presentation and centralize shared query helpers (#151)
Some checks failed
Secret scan / gitleaks (push) Failing after 12s
Mirror source to Forgejo / mirror (push) Has been skipped
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* refactor(daemon): one table for repair operation wire codes
The code published under repair_current_operation_code was defined in
three places: a name->code chain in RepairService.cpp, an enum->name
switch beside it, and a code->label switch in the CLI. The two daemon
tables agreed with each other but the CLI copy had already drifted in
wording, and nothing tied the three together.
metric_keys.h now carries the single code<->name table next to the
metric key it describes, with constexpr lookups in both directions.
RepairService's name->code chain delegates to it; the enum->name
switch stays because it is keyed by the private enum. The CLI copy
goes in the next commit with the status renderer.
Signed-off-by: trvon <me@trevon.dev>
* refactor(cli): extract the daemon status renderer
showStatus() in daemon_command.cpp was 1,762 lines: two RPCs, a
bootstrap-file fallback, and two ~400/1,100-line render bodies that
wrote straight to std::cout from inside the command class, reachable
only through a live daemon. The render bodies read nothing from the
command except the expected data directory, the explicit-socket flag,
and a memory-sync fetch they triggered mid-render.
The bodies now live in src/cli/commands/daemon_status_render.cpp as
free functions over StatusResponse plus a small DaemonStatusRenderContext
(configured data dir, explicit-socket flag, optional MemorySyncResponse)
writing to an ostream. The readiness/label helpers that only they used
(humanizeToken, classifyReadiness, isEphemeralDataDir,
daemonDataDirFromStatus, paintStatus, neutralText) move with them.
DaemonCommand keeps the RPCs and the bootstrap fallback; the memory-sync
fetch becomes fetchMemorySyncStatus() and runs just before rendering
instead of in the middle of it, so the section prints in the same place
with the same bounded 3 s timeout.
The move is verbatim: after the mechanical substitutions
(std::cout -> os, expectedDataDir() -> ctx.configuredDataDir, the
memory-sync call, the explicit-socket flag, and the repair label
function) the two bodies match the originals character for character
modulo whitespace. The CLI's private repair-operation switch is gone;
the label now comes from the shared table in metric_keys.h with
underscores rendered as spaces, which reproduces the previous wording
("stuck docs", "block refs", "path tree").
tests/unit/cli/daemon_status_render_catch2_test.cpp drives both
renderers and the memory-sync section from synthetic responses:
overview rows, blockers with init progress, last error, data-dir
mismatch (and its suppression under --socket), temporary data dir,
the active repair operation label, every detailed section, and the
started/not-started memory-sync gate.
daemon_command.cpp: 4,082 -> 2,341 lines; daemon_status_render.cpp:
1,642 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(app): one home for hash predicates, path aliasing, and metadata qualifiers
Three small helpers had grown parallel copies with quietly different
rules:
- "does this look like a hash" existed four times: search_service
(8-64 hex), document_service (6-63), cat_command (6-64), plus a
never-defined utils::looksLikeHash declaration in services.hpp. The
8 vs 6 threshold is deliberate (D3): a bare query token must reach
8 hex characters before it stops being text, while an explicit hash
argument may be 6. yams/common/hash_predicates.h now names the two
predicates apart (looksLikeHashQueryToken, looksLikePartialHashArgument)
over one isHexDigits; document_service's 6..63 bound was deliberate
for the name resolver (a full 64-hex name must not resolve as a prefix
and delete by hash), so its two resolver sites now guard size() != 64
explicitly, as the partial-hash resolution site already did.
- macOS /private aliasing had three incompatible policies: search
appended the alias of /var only, grep canonicalized /var and /tmp
to the /private spelling, and the delete resolver displayed
/private/var as /var. yams/common/fs_utils.h now carries one alias
table ({/var,/tmp} <-> /private/...) with three lexical policies over
it (otherSpelling, canonicalSpelling, displaySpelling) and the
platform-gated wrappers callers use. Declared behavior change, macOS
only: search patterns under /tmp now also match their /private/tmp
spelling, and the resolver displays /private/tmp/... as /tmp/....
- Inline key=value metadata filters were parsed twice: SearchService's
extractStructuredMetadataQuery (exactly one '=', no quoting/grouping
characters, deduplicated by key) and SearchEngine's
parseMetadataFiltersFromQuery (any '=' token, duplicates kept). The
stricter, tested parser moves into yams/search/query_qualifiers.hpp
and both call it. Declared behavior change for the hybrid metadata
component: tokens like name="x" or a=b=c stay in the query instead
of becoming filters, and repeated keys collapse to the last value.
Tests: hash_predicates_catch2_test.cpp pins both thresholds and the
64-character bound; path_alias_catch2_test.cpp pins the alias table,
the three policies, and the platform gating; query_qualifiers_catch2_test.cpp
gains the structured-metadata cases (split, dedupe, malformed tokens,
filters-only). search_service, document_service, grep_service,
service_utilities, and resolve_submodule pass unchanged. Portability
allowlist anchors reflowed for the four files whose lines moved.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): flush status before optional collection
Render useful primary status before waiting for memory-sync details.
Keep both brief and detailed paths behind the tested ordering seam.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
864f677269 |
[09/14] Bound rerank text and reuse search parsing (#149)
* perf(search): preview-based rerank text and compiled path globs Cross-rerank built each window entry's text with a full getContent() read, then kept only the first rerankSnippetMaxChars characters. The window now hydrates with one batchGetContentPreview() call bounded to that many characters; buildCrossRerankText takes the preview instead of the repository. batchGetContentPreview is virtual so a test can count it: three window results cost one preview batch and zero full reads. Path filtering compiled a std::regex from the glob on every call, and the call sits inside per-document loops in pathSearch, the hash-prefix search, and metadata filtering. GlobMatcher (src/app/services) holds the compiled form; wildcardMatch() reuses it through a small per-thread cache. The glob semantics are unchanged and now pinned by a test: '*' stays within one path segment, '**' crosses, '?' is one character, regex metacharacters are literal. yams::common::wildcard_match was considered and rejected because its '*' crosses separators. portability_allowlist.txt: two search_service.cpp anchors reflowed. Signed-off-by: trvon <me@trevon.dev> * perf(search): parse edge anchor properties once, lower the path query once Graph expansion parsed every visited edge's JSON properties with nlohmann::json::parse to read at most two strings, region and scope, in three places (relation weighting, seed anchoring, and neighbour anchoring). Properties strings repeat heavily across edges and most carry no anchor at all. edgeAnchorHints() now returns immediately when the text cannot contain either key and otherwise parses each distinct string once per thread through a small bounded cache. A process-wide parse counter lets the test pin it: one hundred lookups of the same anchored string cost one parse, and un-anchored or malformed strings cost none. computePathMatchScore lowercased and trimmed the query for every candidate document; pathSearch now does that once and passes the lowered query down. The existing path-search tests cover the scoring. portability_allowlist.txt: two search_service.cpp anchors reflowed. edgeAnchorHints returns its two short strings by value: the per-thread memo clears itself at 1024 entries, so a reference into it could dangle across calls even though today's three callers consume the result immediately. Signed-off-by: trvon <me@trevon.dev> * fix(search): preserve hydration failure semantics Stop reranking before scorer calls or score changes when previews fail. Commit anchor hints atomically and recognize escaped JSON keys. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
2178552fb9 |
[08/14] Bound retained corpus and enrichment resources (#148)
* perf(search): hold Simeon concept mining to the corpus byte budget The BM25 pass streams one chunked document at a time and stops when processed text would exceed max_corpus_bytes. The concept-mining pass that follows re-read every document's raw text into one vector and kept it all resident for mine_concepts(), with no budget at all: on a large corpus the build's peak memory was the whole corpus, regardless of the cap the first pass had just enforced. This is the likeliest single contributor to the daemon's multi-gigabyte footprint. The concept pass now applies the same budget to raw bytes as it loads. When the corpus exceeds it, mining is skipped with a warning and the BM25 index is kept; the partially loaded text is released. A corpus within budget is mined exactly as before. Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass under 64 KiB. Over budget, the build completes with zero concepts (previously 81 were mined from ~480 KiB of resident text); with a 4 MiB budget the same corpus mines concepts. Signed-off-by: trvon <me@trevon.dev> * perf(ingest): bound the pending KG job overflow When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob can carry the document's content bytes, so a stalled or slow KG stage turned into unbounded ingest memory on top of the ~36k entries the symbol, entity and KG channels already hold. The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max (typed key on TuneAdvisor, default 16384, no environment twin per AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for this ingest, the drop is counted through the existing kgDropped bus counter (shared with shutdown cancellations), and a throttled warning names the cap. Nothing re-derives the per-document entity graph for a dropped job today; the structural repair pass rebuilds document nodes and orphans but not that stage, so an operator who sees the warning should raise the cap or re-ingest. Bounded memory with a visible drop beats unbounded memory with an invisible stall. Test: with the channel detached and a live coordinator (so the drain coroutine neither cancels nor consumes), three enqueues against a cap of two leave two pending and one dropped; before the cap all three were pending. Signed-off-by: trvon <me@trevon.dev> * fix(ingest): preserve durable KG admission state Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
3811a2b562 |
[07/14] Batch tag and topology hydration (#147)
* perf(search): load tag filters once per request, not once per document pathSearch and searchByHashPrefix filtered candidates through metadataHasTags(), which for every document called getAllMetadata() (four retries with backoff on failure), concatenated every metadata key and value into a string, and logged that string at info level. A path search over a few thousand documents was a few thousand metadata round trips and a few thousand info lines. Both paths now load tags for the whole candidate set with one batchGetDocumentTags() call (the API GraphComponent already used) and filter with a pure matcher. The legacy fallback that also accepted a metadata row with key "tag" is gone: nothing writes that form and the repository's own tag APIs never honoured it, so search and repository now agree on what a tag is. A counting repository subclass pins the shape: one batch call and zero getAllMetadata calls for both the path and the hash-prefix routes. portability_allowlist.txt: two search_service.cpp anchors reflowed. Correction to the claim above that nothing writes the legacy `key = "tag"` form: the repair downloads pass did (tag values "downloaded", "host:<h>", "scheme:<s>"), and the search fallback was its only reader. That writer now uses the same `tag:<name>` keys as every other tag writer, so `--tags downloaded` keeps matching documents the repair pass tagged. And batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it an unbounded candidate set; an oversized list used to fail the statement and, with this change, would have emptied the result instead of degrading per document. Signed-off-by: trvon <me@trevon.dev> * perf(search): hydrate topology routing candidates in batches The routing session called getDocumentByHash once per hash in three places on the product routing path: ranked-candidate admission, cluster expansion, and the graph-neighbor diagnostic trace (memoized, but still one statement per distinct hash across four stages). Each now issues one batchGetDocumentsByHash for its whole set, the same API the topology input extractor already used, and the docLookupMicros timing covers the batch. Cluster expansion applies the maxDocs cut before hydrating so no document is fetched only to be discarded. A session-level test with one seed and eight graph neighbours under a counting repository fails on the previous code (17 single lookups, no batch) and passes now (0 single lookups). The full topology suite is unchanged. portability_allowlist.txt: one topology test anchor reflowed. Signed-off-by: trvon <me@trevon.dev> * fix(search): bound hydration and preserve errors Retain legacy tags and honor per-connection SQLite bind limits. Keep hash chunks in one read snapshot and release it on failure. Failed lookups are not missing documents or successful empty searches. Reject partial route admission and preserve tagged path errors. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4e3f5ec6db |
[06/14] Measure and reduce repeated SQLite prepares (#146)
* feat(metadata): count uncached prepares in Database::CacheStats
Database::prepare() compiles a statement every call; prepareCached()
reuses one. The cache stats reported hits and misses but nothing about
the bypass, so a hot path that re-prepared fixed SQL on every row was
invisible. CacheStats gains uncachedPrepares, incremented under the
existing cache mutex and carried through move construction and
assignment.
This is the measurement primitive for the following hot-path changes:
a test can warm a path, then assert that N more calls add zero uncached
prepares.
portability_allowlist.txt: one database.cpp anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(metadata): cache KG statements and batch the scorer's neighbor fetch
Measured with the new uncachedPrepares counter on a single-connection
pool. Before: 64 warmed neighbors() calls plus 32 document lookups cost
306 uncached statement compilations, and one SimpleKGScorer::score()
over eight query terms cost 620. After: 0 and 1.
Three things were compiling SQL per call on the search path:
- The sqlite KG store used Database::prepare() for fixed SQL in
neighbors, getDocumentIdByHash/ByPath, ensureBlobNode,
ensureDocumentNode, ensurePathNode (both lookups), selectKgNodeIdByKey,
getNodeById, resolveAliasExact/Fuzzy, getEdgesFrom/To,
getDocEntitiesForDocument, and searchNodesByLabel. All now use
prepareCached (85 -> 70 raw prepares in the file; the rest are
dynamic IN-list or maintenance queries).
- ConnectionPool::isConnectionValid ran prepare("SELECT 1") on every
acquire and again on every release, so each pooled call paid two
compilations before doing any work. The probe is now cached.
- Database::tableExists compiled its sqlite_master probe every time;
the fuzzy alias resolver calls it per query token.
SimpleKGScorer builds its 1-hop neighbor union with one
getEdgesFromBatch call instead of neighbors() per query node.
kMaxCacheSize rises from 64 to 128: 90 cached call sites now share a
connection when KG and repository work interleave, and the cache
evicts first-found, not least-recently-used.
Tests pin both numbers: the store test asserts zero uncached prepares
across 96 warmed lookups; the scorer test asserts fewer than eight on
a second run (previously one per query node from neighbors() alone).
portability_allowlist.txt: two anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(metadata): bound batched edge query parameters
Respect connection-specific SQLite limits and deduplicate chunk inputs.
Preserve directional ranking and a single read snapshot across chunks;
roll back the savepoint on query failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
73d408a4a6 |
[05/14] Consolidate configuration authority (#145)
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as test-only benchmark levers, but tests/scripts/report_config_key_usage.py shows 45 of them are set by no test, no bench arm, no script, and no doc: they were levers nobody pulled, each shadowing a SearchEngineConfig field that benches and tests already set directly. The entire YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene note that graph-vector fusion is a parked lever. Removed (CANDIDATE rows of the report, verbatim): YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT YAMS_SEARCH_GRAPH_ENABLE_PATHS YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER YAMS_SEARCH_GRAPH_HOP_DECAY YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW YAMS_SEARCH_GRAPH_MAX_HOPS YAMS_SEARCH_GRAPH_MAX_NEIGHBORS YAMS_SEARCH_GRAPH_MAX_PATHS YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE YAMS_SEARCH_GRAPH_TEXT_WEIGHT YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING YAMS_SEARCH_GRAPH_VECTOR_WEIGHT YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY YAMS_SEARCH_RERANK_TOP_K YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK YAMS_SEARCH_TIERED_MIN_CANDIDATES YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE (different clamps, later block won when both were set); only the current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by two identical blocks and is now read once. A test builds the environment over a recording lookup and asserts none of the retired names is ever consulted, and that the surviving names are consulted exactly once. production_environment_keys.txt shrinks by 45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale. The search_latency_graph_rerank task in .vscode/tasks.json set two of the retired overlays; those assignments are dropped so the task does not claim a configuration the code no longer honours. Signed-off-by: trvon <me@trevon.dev> * refactor(config): drop unread embeddings.* keys and the no-op toggles yams config embeddings enable/disable/tune wrote embeddings.auto_generate, embeddings.generation_delay_ms, and embeddings.batch_size, and the migrator seeded them (plus embeddings.auto_on_add) into every new config.toml. No daemon code reads any of them: the post-ingest queue always dispatches embed jobs when an embed queue exists, and the only batch size the resolvers read is embeddings.runtime.batch_size. The toggle was a switch wired to nothing, and yams status advised users to flip it. - embeddings enable and disable are removed. enable duplicated embeddings model <name> once its dead keys are gone; a kept no-op would be a shim. status no longer prints an auto-generation line and shows the runtime batch size when set. tune presets write embeddings.runtime.batch_size and drop the delay. - config_migration no longer seeds auto_generate, auto_on_add, generation_delay_ms, or a top-level batch_size. - ConfigResolver and the compatibility policy honour a legacy [embeddings] batch_size silently when the runtime key is absent, so configs in the wild get their value applied without a file rewrite (the migrator exits early for v3 configs, so a rewrite would never run). - status and model hints point at embeddings model <name>. Declared behavior change: `yams config embeddings enable|disable` are unknown subcommands now. Declared behavior change: the legacy [embeddings] batch_size was written by `yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16) but no resolver ever read it. Honouring it as a fallback applies those numbers for the first time; the one consumer whose result moves is the doctor embedding repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32 on a stock config. The embedding service already defaulted to 32. Signed-off-by: trvon <me@trevon.dev> * feat(config): add [tuning] keys for the setters only env could reach Seventeen TuneAdvisor setters were exercised by tests but had no TOML key, so production could reach them only through a deprecated YAMS_* overlay: connection slot sizing, the CPU-high threshold, the ONNX concurrency and reservation knobs, model-eviction thresholds, ingest worker cap, store/embed channel capacities, work-coordinator threads, and connection lifetime. AGENTS.md says tuning lives in typed config; these were half-migrated. Each gets a [tuning] key resolved through ConfigResolver alongside the existing ones, with provenance recorded, and each is included in resetConfiguredOverrides() so a reload that drops a key reverts it (the three with dedicated reset helpers call those). The environment names keep working as compatibility inputs; nothing new is added to the YAMS_* surface. A table-driven test sets all 17 keys, checks every getter, then reloads with an empty [tuning] section and checks every value reverted. docs/architecture/environment-overrides.md lists the keys. The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds in (0, 1), connection_lifetime_s at most 86400) keep their default when handed a value outside the range. The resolver checks the same ranges first and warns, so an ignored value is never recorded as config provenance; test covers all three shapes. Signed-off-by: trvon <me@trevon.dev> * fix(config): report 0 for [tuning] integer keys instead of ignoring it For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back as "unset", and silently fall through to the default while the resolver still recorded config provenance for it. applyUint32 now warns and skips the value: nothing changes for the operator except that the mistake is visible. Two keys mean something by 0 and are exempt: post_ingest_threads (0 = auto, which the config migrator seeds into every config and `yams config` displays as "(auto)") and connection_lifetime_s (0 = no recycling; its setter takes a signed sentinel and accepts 0, and the environment spelling always did). Both keep their provenance and stay silent, so a stock config produces no warning. Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and record no provenance, pool_ipc_max = 48 still applies, and the two zero-valid keys apply with provenance. Signed-off-by: trvon <me@trevon.dev> * refactor(config): one embedding-runtime resolver, one tuning baseline Two resolvers read the same five environment keys and three TOML keys for the embedding runtime with different error semantics: ConfigResolver::resolveEmbeddingConfig used the strict typed readers (warn and keep the default), while resolveEmbeddingRuntimePolicy in config_resolver_policies.cpp used std::stoull inside a bare catch, so YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on the other. The compatibility entry point is now a projection of the typed resolution onto the EmbeddingRuntimePolicy shape that library callers (vector, repair) still consume; it lives in ConfigResolver.cpp because DaemonConfig is declared there. 91 lines of duplicate parsing go away. yams doctor tuning carried its own [tuning] baseline table and it had drifted: worker_poll_ms 75 against the daemon default of 150, pool_io_max sized from hardware against a default of 32, so `doctor tuning --apply` wrote defaults as divergent overrides. The baseline now reads TuneAdvisor's effective values, which is what "baseline" should mean. Tests: the compatibility policy rejects 12abc and agrees with the typed resolver; the doctor proposal equals TuneAdvisor for every key. Signed-off-by: trvon <me@trevon.dev> * refactor(config): stop writing the unread search.reranker_model_path yams init wrote both search.reranker_model (the name the daemon resolves through ConfigResolver::resolveRerankerModel) and search.reranker_model_path, a derived on-disk path nothing in the daemon, search, or vector code reads; yams config search reranker status then echoed it back as if it mattered. init no longer writes the path key and config no longer displays it. The branch that erases a stale reranker_model_path line from an existing config is kept so old files converge. Of the three "aliased key clusters" flagged by the audit, this was the only real drift: the three embedding-dimension keys are already one centralized alias set in config_helpers.cpp (read any, write all), and msl_stack_log_warn_bytes is a metrics counter name, not a config key. portability_allowlist.txt: three init_command.cpp anchors reflowed; no new environment reads. The model command's post-download hint still told operators to set search.reranker_model_path; that line goes too (the two lines above it already name the commands that configure the reranker). Signed-off-by: trvon <me@trevon.dev> * fix(config): validate tuning ranges and pairs Reject unsupported scalar and inverted paired limits before changing runtime overrides or provenance. Preserve explicit zero ONNX reservations with a distinct unset sentinel and avoid false default-backend conflicts. Cover all 27 integer keys, inherited limits, and reservation reset. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
07b9f6b37b
|
fix(topology): bind calibration to routing policy identity
Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
e4e50dc307
|
perf(search): refill chunk-crowded vector results
Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
62b789af6c
|
perf(search): reuse Simeon scores within a request
Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
3b22a30ec5 |
refactor(search): extract auxiliary vector merge (#86)
* refactor(search): extract auxiliary vector merge Signed-off-by: trvon <me@trevon.dev> * fix(search): preserve auxiliary tie behavior Keep the extracted merge comparator identical to the original score-only sort and avoid prescribing equal-score order in characterization tests. Qualify the search detail namespace so the topology test compiles unambiguously. Signed-off-by: trvon <me@trevon.dev> * test(search): avoid auxiliary score tie Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
9883a4382b |
refactor(vector): decouple embedding generator policy (#83)
* refactor(vector): add injectable executor for the embedding daemon bridge await_with_timeout spawned coroutines unconditionally on the daemon global IO context, coupling the vector-layer embedding generator to daemon globals. EmbeddingConfig gains an optional boost::asio::any_io_executor executorOverride; the bridge uses it when present and keeps the global executor as the default — identical behavior for all existing callers (embedding_service, model providers, CLI all construct without an override). Lead 1 plan step (b) of the embedding_generator vector->daemon decomposition (Brick audit follow-up). Validation: full ninja build green; meson test yams:service_manager, vector_submodule, yams:embedding_lifecycle, yams:embedding_recovery — all green. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): split DaemonBackend into own TU Keep daemon client/IPC headers out of the vector-layer generator TU; EmbeddingGenerator builds the backend via createDaemonBackend factory on the existing IEmbeddingBackend seam. Behavior-preserving. Brick audit lead 1: 5 file-level cycles hubbed on embedding_generator.cpp. Signed-off-by: trvon <me@trevon.dev> * revert(vector): keep bridge executor out of public EmbeddingConfig Pre-push gate caught it: the config layer includes the vector header transitively (ConfigResolver.h -> document_chunker.h -> embedding_generator.h) and has no boost asio include path, so <boost/asio/any_io_executor.hpp> in the public header breaks the config build. Executor selection stays in embedding_daemon_backend.cpp (daemon-facing TU by design); per-instance injection needs the config->vector include direction fixed first — recorded as follow-up. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): trust resolved backend policy Remove EmbeddingGenerator's ambient ConfigResolver read. Callers now own backend-policy resolution; EmbeddingService marks its already-resolved runtime policy unconditionally. Keep TuneAdvisor only for the process-wide concurrency gate, where instance-local policy would not coordinate admission. Tests pin typed backend authority against conflicting environment state. Signed-off-by: trvon <me@trevon.dev> * chore(vector): remove stale config include The generator no longer reads config helpers after backend policy moved to callers, so drop the misleading DaemonBackend comment and unused include. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
32c8ccdc5f |
perf(search): favor lexical mixed defaults (#65)
Signed-off-by: trvon <git@trevon.dev> |
||
|
trvon
|
b7eaea8263
|
test: move stress cases to slow suite
Stress-tagged cases ran in the default unit lane because only ~[slow] was filtered. Tag them [slow] and add ~[slow]/[slow] splits to the five binaries (mcp_submodule, work_coordinator, daemon_background, plugin_infrastructure, hybrid_search) so they run in the slow suite. |
||
|
trvon
|
30618eb113
|
refactor(config): consolidate runtime tuning | ||
|
Trevon
|
0dd3985db4 |
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width. * refactor(vector): retire dormant backends Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage. Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation. * refactor(plugins): remove legacy SDK tooling Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture. * build(vector): remove stale FAISS overrides * refactor(ipc): remove dormant FSM snapshots * refactor(ingest): unify bounded KG enqueue * fix(plugins): harden allocation contracts Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0). * fix(storage): define empty object reads Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract. * refactor(storage): trim through string views * test: prune dormant harness helpers Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain. * chore(cpp): initialize analyzer-facing locals * build(simeon): advance analyzer cleanup * refactor(config): drop dead daemon client keys * refactor(ipc): inject daemon client transport Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths. * refactor(retrieval): route CLI through daemon Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection. * refactor(graph): move CLI scope into daemon Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct. * build: isolate test-only runtime controls * refactor(metadata): decouple lock telemetry * fix(services): return on request timeout * fix(retrieval): enforce workspace scope * fix(graph): scope nodes before pagination * fix(grep): drop binary result lines * fix(daemon): bound checkpoint wait * fix(repair): use KG health checks * fix(storage): always verify compressed CRC * perf(vector): skip unchanged PQ snapshots * build(zyp): exclude bench from default build * build: protect Meson files from formatter * refactor: remove unreachable production APIs * refactor(daemon): remove dormant runtime state * fix(vector): harden daemon provider calls Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production. * refactor(storage): remove unusable adapters * fix(cli): unify in-process add requests * fix(tsan): preserve async worker progress * test(cli): isolate daemon-dependent suites * fix(repair): expose queue backoff progress |
||
|
trvon
|
10ff33532c
|
fix(search): cancel disconnected requests
Retire timed-out one-shot sockets and propagate connection cancellation through daemon search execution. Stop scheduling later search stages after cancellation while safely draining in-flight work, and reject KG ingest counts that exceed the wire format. |
||
|
trvon
|
9618370d1f
|
feat(search): trace bounded topology rescue
Keep selector behavior default-off while recording complete eligible tails. Direct Simeon fragment scoring remains an experimental comparison path. |
||
|
Trevon
|
3f37e8f564 |
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols Signed-off-by: trvon <me@trevon.dev> * refactor(vector)!: remove dead transition APIs Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use. BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers. Signed-off-by: trvon <me@trevon.dev> * fix(vector): harden PQ snapshot lifecycle Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics. Signed-off-by: trvon <me@trevon.dev> * feat(topology): expose route coordinates Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero. Signed-off-by: trvon <me@trevon.dev> * feat(xplan): add vector decision gates Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries. Signed-off-by: trvon <me@trevon.dev> * feat(topology): formalize coordinate routing Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment. Signed-off-by: trvon <me@trevon.dev> * fix(search): serialize SymSpell readers SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN. Signed-off-by: trvon <me@trevon.dev> * fix(vector): make mutations atomic Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows. * fix(search)!: surface SymSpell write errors Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms. BREAKING CHANGE: SymSpellSearch mutations now return Result values. * feat(topology): formalize candidate expansion * feat(vector): add coordinate-aware embeddings * feat(topology): preserve protected relation cover * feat(search): add graph candidate rescue * feat(daemon): expose retrieval hydration telemetry * feat(xplan): add rescue and hydration gates * refactor(content): dedupe media handlers Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic. * feat(lifecycle): prune stale corpus data Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open. * feat(search): add topology candidate rescue Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit. * feat(search): expose evidence paths * docs: define mobile host contract * formal: model bounded candidate rescue * fix(prune): inspect extensionless signatures * fix(prune): tolerate stale extensions --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
7df64b1161 |
perf(search): bound topology routing and harden runtime paths (#59)
* fix: address review build issues * perf(search): bound topology routing Cache route membership, shortlist centroids with static ANN, and route SPQ through explicit allowed-document budgets. * test(xplan): measure routed vector work Pin benchmark binary identity and export route, cache, and phase costs for repeated equal-budget comparisons. * test(search): use portable jthread wrapper * fix(daemon): serialize detached cancellation * fix(cli): honor pinned daemon sockets * fix(xplan): support Windows contract imports * fix(list): apply metadata filters before paging * docs(prompts): require memory hydration Treat search results as candidates and hydrate selected artifacts before use. Quote argument hints for strict YAML loaders. * test(cli): keep add on pinned daemon * test(metadata): isolate performance timing * fix(graph): tolerate stale snippet ranges Source files may change before persisted symbol metadata is refreshed. Omit stale snippets instead of aborting graph exploration. * test(daemon): await request readiness Reuse the retrying list probe before asserting post-shutdown error quality under sanitizer load. * ci(release): use hosted cmake Resolve Conan CMake tool requirements from PATH so cold ARM64 caches do not depend on cmake.org. Default BUILD_DIR for failure cleanup. |
||
|
Trevon
|
ae15bf317a |
perf(ingest): harden daemon batching and shutdown (#58)
* fix(daemon): harden model state ownership Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads. * refactor(daemon): consolidate ingest pollers Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration. * refactor(daemon): unify repair execution Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths. * refactor(daemon): unify runtime tuning Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift. * refactor(daemon)!: remove dead IPC surfaces Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact. BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame. * refactor(daemon): hide test hooks Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols. * fix(daemon): avoid replaying successful adds A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again. * bench(ingest): enforce lossless pipeline runs Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift. * bench(search): expose cold seed ingest timings Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements. * perf(ingest): batch deferred content indexing Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window. * feat(metadata): expose insert writer metrics Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics. * fix(daemon): retry transient refused connects The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window. * bench(ingest): attribute submission path costs Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs. * perf(ingest): batch durable document storage Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success. * perf(ingest): coalesce bulk post-processing Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota. * bench(ingest): record post-ingest batch size Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system. * perf(ingest): batch queued storage waves Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock. * perf(ingest): route bulk enrichment batches Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing. * test(daemon): initialize task fixtures * perf(ingest): batch post-ingest dispatch * perf(ingest): widen directory storage waves Borrow from a full wave when one document would otherwise fall through to single-document storage. * perf(storage): bound batch writer fanout Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object. * perf(ingest): align storage batch drains Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work. * fix(ingest): commit metadata state atomically Fold first-version, extracted-title, and embedding-completion state into existing transactions. Read committed extraction state so duplicate-content tasks cannot double-count corpus counters. * bench(ingest): measure production submission Drive pipelined files through DocumentIngestionService. Export dispatch, fingerprint, and enqueue costs through xplan. * fix(xplan): preserve native failures Report child signals and bounded stderr before readiness checks so assertions are not masked as missing traces. * fix(daemon): preserve embedding success hashes * fix(daemon): serialize pooled socket shutdown * fix(daemon): harden batch publication shutdown * fix(storage): use compat jthread shim * fix(daemon): avoid poller template shadowing * fix(search): keep vector pipeline test enabled * fix: harden shutdown and fallback paths Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes. |
||
|
Trevon
|
a3d065dd8d |
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish
Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.
* fix(daemon): stop shutdown log storm
Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.
* refactor(cli): delegate serve to MCP binary
Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.
* refactor(cli): move daemon path helpers
Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.
* refactor(daemon): split embedded client
* perf(search): add lean-guided benchmark gates
Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.
Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.
* refactor(config): move daemon policies
* refactor(daemon): extract plugin runtime
* refactor(daemon): untangle service coordinators
Untangle CLI/MCP from daemon service coordinators and the daemon static lib.
- MetadataRepository uses a daemon-neutral tree-diff callback. The
GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
static lib. Drop ResourceGovernor coupling inside the writer loop; batch
sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
its own enqueuePostIngest; app services no longer call ServiceManager
directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
app services link cleanly without daemon.
Validated:
meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
yams-cli --version, yams-mcp-server init, yams serve --help smoke tests
Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.
* fix(cli): keep stdin add local fallback
No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.
Validated:
meson compile -C build/debug -j4 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
meson compile -C build/release -j4 yams-cli yams-mcp-server
meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output
* feat(bench): add xplan multi-arm experiment harness
Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.
* feat(formal): expand topology/search Lean contracts
Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.
* feat(search,topology): construction purity + graph expansion path
Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.
* feat(topology): add KMeans engine and SGC identity contracts
Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.
* feat(daemon): topology readiness gates and selectable reranker
Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.
* chore(search): drop meta_path tuner knobs
Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].
* feat(bench): add topology_purity_validate plan
Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.
* chore: format add_command and config include order
Mechanical clang-format / include reorder only; no behavior change.
* feat(bench): decision-grade xplan loop for search engine
Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.
* feat(graph): denser product semantic_neighbor topK
Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.
* feat(search): extract topology assist and cross-rerank seams
Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.
* feat(bench): product-path search engine ablation plans
Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.
* fix(ci): repair pre-push fallbacks
* feat(bench): lock-free mid-run builds + multi-corpus gates
xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.
* fix(daemon): keep SearchEngine alive via shared_ptr cache
getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.
Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.
* feat(search): route vectors through cached topology
Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.
Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.
* feat(bench): consolidate search decision plans
Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.
Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.
* feat(search): refine topology vector routing
* fix(search): propagate daemon readiness
* feat(bench): measure generalized memory health
* fix(cli): honor in-process fallbacks
* ci: harden smolvm debug builds
* perf(search): route topology through filtered ANN
* feat(bench): measure read-write pressure
* perf(metadata): index symbol lookups on read pool
* fix(daemon): isolate startup reads and maintenance
* feat(bench): replay isolated generalized corpora
* feat(search): add evidence routing pipeline
Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.
* refactor(search): unify topology evidence path
Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.
* refactor(bench): prune superseded search lanes
Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.
* docs(search): retire stale experiment notes
Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.
* refactor(search): prune dead search surfaces
Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.
* fix(cli): clear build warnings
* fix(xplan): reject stale run identity
* refactor(search)!: remove legacy query stack
Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.
BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.
* refactor(search): type value stage traces
Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.
* refactor(search): isolate benchmark env overlays
Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.
* refactor(search): unify topology routing options
Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.
* fix(search): unify tuner config ownership
Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.
* fix(search): snapshot tuner state per query
Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.
* fix(search): enforce topology routing contract
Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.
* fix(daemon): let SQLite reclaim its file
Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.
* fix(metadata): tolerate missing symbol FTS
Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.
* build(fuzz): drop removed query parser target
The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.
* fix(bench): include writes in drain oracle
A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.
* fix(daemon): vacuum padded WAL databases safely
|
||
|
trvon
|
d112aae8b8
|
perf: coalesce KG writes and document live-mirror drift | ||
|
trvon
|
7e2e5be626
|
refactor(search): remove RM3 from the retrieval product | ||
|
trvon
|
1a11b9d230
|
refactor(search): retire fragment-geometry; fix geom scale-mix |