4 commits

Author	SHA1	Message	Date
Trevon	3272367ea2	fix(search): restore measured scientific ranking (#318) ... * fix(search): make the final lexical tie-break a strict weak ordering The final comparator broke near-ties toward the result with less vector support, and its pairwise |a - b| <= epsilon test was not transitive, which is undefined behavior for the std::sort/partial_sort callers in applySearchTopKLimit. Compare tie buckets floor(score / epsilon) instead, then lexical anchor, keyword and vector evidence (more first), then doc id. Behavior with the tie-break off is unchanged. The comparator moves to final_result_order_internal.h so it can be tested directly. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit ca9b82d6ce9db31b6c9608a7d95b63203649e5a7) Signed-off-by: trvon <git@trevon.dev> * fix(search): keep cross-reranked scores above the unreranked tail applyCrossRerank rewrote the window's scores to a 0..1 min-max blend while the tail kept raw fused scores. Later whole-list sorts (concept boost, top-k partial_sort) compared the two scales, so reranked docs could drop below tail docs or out of the top-k. Map the blended score back onto the window's original range, with the floor raised just above the best tail score when needed. The window's reranked order is unchanged and docTraces.finalScore records the mapped score. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit d66c6d304ca4d32b943e164b0d7720fc7f865db1) Signed-off-by: trvon <git@trevon.dev> * fix(search): judge weak tier-1 queries on the raw bm25 scale The weak-query test compared the top Text/SimeonText component score with weakQueryMinTopTextScore, but lexical batches are min-max normalized, so the top hit is always 1.0. With enough hits every query was strong: the weak-query vector fanout boost and weak-query topology expansion never ran. scoreLexicalBatch now also records absoluteLexicalScore, the hit's score on the existing absolute scale (-bm25 / bm25NormDivisor, same multipliers), and classifyTier1TextSignal uses it. Thresholds keep their values; with the default divisor 25, 0.30 is bm25 <= -7.5 and 0.45 is bm25 <= -11.25. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit bc771c5aab1721170fb9e743013ca0efbf048d61) Signed-off-by: trvon <git@trevon.dev> * fix(search): restore measured scientific fusion Give SCIENTIFIC the measured relative Simeon weight of 0.40 after normalizing the existing text/vector pair. The resulting share is 0.285714, matching the benchmark traces. Other profiles remain unchanged. Keep its one-slot semantic rescue preset instead of raising it to three at the corpus layer. Preserve the existing large-prose recall widening. Cover the preset weights, config propagation, community blend, rich-graph ratios and corpus rescue behavior. Signed-off-by: trvon <git@trevon.dev> * fix(search): order non-finite ranking evidence Treat NaN score/evidence keys as negative infinity and invalid tie epsilon as zero. This keeps comparator equivalence transitive without changing finite-input ordering. Add exhaustive triples covering NaN, infinities and invalid epsilon. Signed-off-by: trvon <git@trevon.dev> * docs(search): clarify lexical scoring limits Distinguish the shared single-hit score conversion from FTS5 BM25 calibration. Note that lexical tie buckets can still reorder numerically separated rerank-window and tail scores. No behavior change. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev>	2026-10-16 09:43:41 -06:00
Trevon	e4b32e98bd	fix: fence enrichment completion and reconcile promotion prerequisites (#157) ... * test(ci): stabilize main matrix (#112) Signed-off-by: trvon <me@trevon.dev> * ci(release): preview experimental promotion (#111) Signed-off-by: trvon <me@trevon.dev> * ci(release): reconcile draft promotion safely (#113) * ci(release): reconcile draft promotion Create promotion visibility independently of release readiness. Leave existing PRs untouched to preserve concurrent maintainer edits. Validate against current main without masking candidate rejection. Signed-off-by: trvon <me@trevon.dev> * ci: isolate hook Git environments Prevent temporary Git fixtures from modifying the calling worktree. Clear repository-local hook variables before the unchanged build gates, and isolate validator and fixture Git subprocesses from caller overrides. Exercise linked-worktree config, ref and index preservation offline. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * fix(ci): reject invalid candidate evidence Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values. Signed-off-by: trvon <me@trevon.dev> * fix: preserve completion and retrieval contracts Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures. Signed-off-by: trvon <me@trevon.dev> * fix: fence asynchronous enrichment completion Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage. Signed-off-by: trvon <me@trevon.dev> * fix: bound semantic graph output reservation Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward. Signed-off-by: trvon <me@trevon.dev> * fix: fence standalone embedding repair completion Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass. Signed-off-by: trvon <me@trevon.dev> * fix: preserve enrichment failure reporting Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged. Signed-off-by: trvon <me@trevon.dev> * test: align explicit hash prefix boundaries Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun. Signed-off-by: trvon <me@trevon.dev> * test: align UI hash validation boundary Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev>	2026-09-12 18:53:12 -06:00
Trevon	864f677269	[09/14] Bound rerank text and reuse search parsing (#149) ... * perf(search): preview-based rerank text and compiled path globs Cross-rerank built each window entry's text with a full getContent() read, then kept only the first rerankSnippetMaxChars characters. The window now hydrates with one batchGetContentPreview() call bounded to that many characters; buildCrossRerankText takes the preview instead of the repository. batchGetContentPreview is virtual so a test can count it: three window results cost one preview batch and zero full reads. Path filtering compiled a std::regex from the glob on every call, and the call sits inside per-document loops in pathSearch, the hash-prefix search, and metadata filtering. GlobMatcher (src/app/services) holds the compiled form; wildcardMatch() reuses it through a small per-thread cache. The glob semantics are unchanged and now pinned by a test: '*' stays within one path segment, '**' crosses, '?' is one character, regex metacharacters are literal. yams::common::wildcard_match was considered and rejected because its '*' crosses separators. portability_allowlist.txt: two search_service.cpp anchors reflowed. Signed-off-by: trvon <me@trevon.dev> * perf(search): parse edge anchor properties once, lower the path query once Graph expansion parsed every visited edge's JSON properties with nlohmann::json::parse to read at most two strings, region and scope, in three places (relation weighting, seed anchoring, and neighbour anchoring). Properties strings repeat heavily across edges and most carry no anchor at all. edgeAnchorHints() now returns immediately when the text cannot contain either key and otherwise parses each distinct string once per thread through a small bounded cache. A process-wide parse counter lets the test pin it: one hundred lookups of the same anchored string cost one parse, and un-anchored or malformed strings cost none. computePathMatchScore lowercased and trimmed the query for every candidate document; pathSearch now does that once and passes the lowered query down. The existing path-search tests cover the scoring. portability_allowlist.txt: two search_service.cpp anchors reflowed. edgeAnchorHints returns its two short strings by value: the per-thread memo clears itself at 1024 entries, so a reference into it could dangle across calls even though today's three callers consume the result immediately. Signed-off-by: trvon <me@trevon.dev> * fix(search): preserve hydration failure semantics Stop reranking before scorer calls or score changes when previews fail. Commit anchor hints atomically and recognize escaped JSON keys. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev>	2026-09-12 10:09:20 -04:00
Trevon	a3d065dd8d	feat(search): make topology-assisted retrieval the default (#57) ... * fix(release): repair package repo publish Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures. * fix(daemon): stop shutdown log storm Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping. * refactor(cli): delegate serve to MCP binary Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF. * refactor(cli): move daemon path helpers Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split. * refactor(daemon): split embedded client * perf(search): add lean-guided benchmark gates Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior. Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in. * refactor(config): move daemon policies * refactor(daemon): extract plugin runtime * refactor(daemon): untangle service coordinators Untangle CLI/MCP from daemon service coordinators and the daemon static lib. - MetadataRepository uses a daemon-neutral tree-diff callback. The GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp, owned by the daemon library. - WriteCoordinator moves to a dedicated build-only yams_write_coordinator static lib. Drop ResourceGovernor coupling inside the writer loop; batch sizing still respects config.maxBatchSize. - AppContext gains enqueuePostIngest callback. ServiceManager wires it to its own enqueuePostIngest; app services no longer call ServiceManager directly. - Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a lease factory backed by the real coordinator. - CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir. - Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp. - Declare missing yams_extraction/yams_ingest deps on yams_app_services so app services link cleanly without daemon. Validated: meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0 yams-cli --version, yams-mcp-server init, yams serve --help smoke tests Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M. * fix(cli): keep stdin add local fallback No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing. Validated: meson compile -C build/debug -j4 yams-cli catch2_cli_commands MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content' MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands meson compile -C build/release -j4 yams-cli yams-mcp-server meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content' ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output * feat(bench): add xplan multi-arm experiment harness Unified plan runner, ablation axes, REPORT/ablation docs, and KPI snapshot so optimization loops stop living in shell matrices. * feat(formal): expand topology/search Lean contracts Boundary models for search provenance, construction gates, and cluster/route objectives so Lean guides correctness while xplan ranks performance levers. * feat(search,topology): construction purity + graph expansion path Raise topology build defaults (min edge 0.25, max component 64) and split oversized CC pieces so hybrid assist is quality-positive on scifact. Add GraphNeighbors expansion with medoid-anchor fallback, structured helpers instead of nested session logic, and BEIR-default xplan plans. Search engine extraction (fanout/classic pipeline) rides along as the integration surface for routing. * feat(topology): add KMeans engine and SGC identity contracts Register kmeans in the factory with deterministic spherical k-means over embeddings, and enforce SGC document-identity invariants in debug builds with matching unit coverage. * feat(daemon): topology readiness gates and selectable reranker Drive topologyReady from published epoch/freshness/rebuild/dirty ingest state (not just KG drain), surface those fields in search stats, and wire onnx|simeon|auto|none cross-reranker selection. * chore(search): drop meta_path tuner knobs Meta-path routing is gone; remove dead TunedParams fields and the ConfigResolver test for [search.meta_path]. * feat(bench): add topology_purity_validate plan Repeats=3 validation matrix for construction-purity winners vs topo_off so MRR deltas can clear run-to-run noise. * chore: format add_command and config include order Mechanical clang-format / include reorder only; no behavior change. * feat(bench): decision-grade xplan loop for search engine Document search-engine measurement in AGENTS; harden xplan with repeats=3 plans, quality×latency primaries, mechanism metrics from debug.jsonl, multi-rep log promotion, and archive superseded topology plans. * feat(graph): denser product semantic_neighbor topK Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product graphs leave fewer degree-0 docs for GraphNeighbors expansion. * feat(search): extract topology assist and cross-rerank seams Pull topology assist into search_topology_stage (seed merge, skip reasons, optional vector seed probe default 0). Extract cross_rerank compute from search_engine. Wire probe via typed config/ConfigResolver. * feat(bench): product-path search engine ablation plans Add topology-off component, vector-weight, and tiered-execution xplan plans (repeats=3) and wire them into the AGENTS measurement loop table. * fix(ci): repair pre-push fallbacks * feat(bench): lock-free mid-run builds + multi-corpus gates xplan skips meson rebuild when the quality binary exists (flock + YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS. * fix(daemon): keep SearchEngine alive via shared_ptr cache getCachedEngine previously published a raw pointer that metrics polling used through rebuild, racing destructor of SearchEngine::Impl (TSAN). Store and return a shared_ptr snapshot so diagnostic readers pin the engine for the duration of getConfig/health checks. Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan. * feat(search): route vectors through cached topology Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics. Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts. * feat(bench): consolidate search decision plans Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts. Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans. * feat(search): refine topology vector routing * fix(search): propagate daemon readiness * feat(bench): measure generalized memory health * fix(cli): honor in-process fallbacks * ci: harden smolvm debug builds * perf(search): route topology through filtered ANN * feat(bench): measure read-write pressure * perf(metadata): index symbol lookups on read pool * fix(daemon): isolate startup reads and maintenance * feat(bench): replay isolated generalized corpora * feat(search): add evidence routing pipeline Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars. * refactor(search): unify topology evidence path Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path. * refactor(bench): prune superseded search lanes Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls. * docs(search): retire stale experiment notes Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans. * refactor(search): prune dead search surfaces Remove unused fusion, reranker, policy, and router abstractions. Bound topology materialization and keep document trace payloads opt-in. * fix(cli): clear build warnings * fix(xplan): reject stale run identity * refactor(search)!: remove legacy query stack Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search. BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search. * refactor(search): type value stage traces Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results. * refactor(search): isolate benchmark env overlays Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition. * refactor(search): unify topology routing options Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys. * fix(search): unify tuner config ownership Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge. * fix(search): snapshot tuner state per query Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot. * fix(search): enforce topology routing contract Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact. * fix(daemon): let SQLite reclaim its file Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism. * fix(metadata): tolerate missing symbol FTS Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent. * build(fuzz): drop removed query parser target The optional AFL build still referenced the deleted legacy query parser API and could no longer compile. * fix(bench): include writes in drain oracle A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight. * fix(daemon): vacuum padded WAL databases safely	2026-07-13 20:26:43 -04:00