Commit graph

19 commits

Author SHA1 Message Date
Trevon
4e97b31fef
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init

GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.

Signed-off-by: trvon <me@trevon.dev>

* chore(warnings): remove or annotate unused parameters, functions and variables

- init_command, s3_signer test, request_handler_lifetime test: comment out
  the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
  __APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
  work_coordinator test: mark platform-conditional or busy-loop
  variables [[maybe_unused]].

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests

- search_admission_controller: count started workers in std::uint32_t so
  the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
  inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
  temporary std::string is created for the call (GCC
  -Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
  which continues the comment (-Wcomment); show the env export on its own
  line.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): drop redundant std::move of a const captured json

The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): mark __int128 use in db_integrity_stamp as an extension

-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): remove always-true unsigned >= 0 comparisons

GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
2026-10-08 14:30:28 -06:00
trvon
6aacda7913
fix(daemon): cancel in-flight request work when shutdown begins
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.

Shutdown now tells in-flight work to stop before joining:

- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
  on every leased connection under the pool mutex, which keeps each
  lease open while it is interrupted. DatabaseManager exposes it for
  both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
  join wait slice through a new joinWithTimeout hook, because SQLite
  drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
  before the interrupt, and publishes it as AppContext::cancellationSignal.
  DocumentService::list and GrepService::grep check it between query
  and hydration steps and between document batches, and return
  OperationCancelled; SearchService checks it on entry and forwards it
  as SearchExecutionContext::hostCancellationSignal, which the engine's
  existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
  componentTimeout is 0, and a canceled search stops waiting for the
  query embedding / concept tasks. Those tasks now own copies of their
  inputs instead of capturing the caller's stack, so abandoning them is
  safe.

Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.

Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:05:07 -06:00
trvon
8eae143259
feat(search): apply lines: qualifiers to result snippets
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.

- lines:<range> now replaces each result's snippet with those lines of
  its stored content (1-based, e.g. "1-3,10"), through the same
  TextBasicHandler `get` uses. It runs after snippet hydration, so
  nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
  content over 1 MiB, or unreadable content keeps its snippet and is
  counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
  now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
  line-range text, ...)" or "pages:1-2 is not applied to search
  results") goes into searchStats["extract_scope"] and queryInfo, which
  the MCP response already carries.

The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.

Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.

The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.

Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
2026-09-24 18:34:46 -06:00
trvon
ac070d8d2a
refactor(search): remove symbol-based result ranking
SymbolEnricher boosted results by looking up code-symbol nodes that no
longer exist since v0.20, so it ran KG queries for the top results and
never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the
symbol_rank_* search stats, and the symbolRank option on the service,
retrieval, MCP and CLI request types. The IPC field stays in the wire
format but is ignored, and --symbol-rank/--no-symbol-rank remain
accepted as hidden no-ops so existing scripts keep working.
2026-09-22 14:40:52 -06:00
Trevon
864f677269
[09/14] Bound rerank text and reuse search parsing (#149)
* perf(search): preview-based rerank text and compiled path globs

Cross-rerank built each window entry's text with a full getContent()
read, then kept only the first rerankSnippetMaxChars characters. The
window now hydrates with one batchGetContentPreview() call bounded to
that many characters; buildCrossRerankText takes the preview instead of
the repository. batchGetContentPreview is virtual so a test can count
it: three window results cost one preview batch and zero full reads.

Path filtering compiled a std::regex from the glob on every call, and
the call sits inside per-document loops in pathSearch, the hash-prefix
search, and metadata filtering. GlobMatcher (src/app/services) holds
the compiled form; wildcardMatch() reuses it through a small
per-thread cache. The glob semantics are unchanged and now pinned by a
test: '*' stays within one path segment, '**' crosses, '?' is one
character, regex metacharacters are literal. yams::common::wildcard_match
was considered and rejected because its '*' crosses separators.

portability_allowlist.txt: two search_service.cpp anchors reflowed.

Signed-off-by: trvon <me@trevon.dev>

* perf(search): parse edge anchor properties once, lower the path query once

Graph expansion parsed every visited edge's JSON properties with
nlohmann::json::parse to read at most two strings, region and scope,
in three places (relation weighting, seed anchoring, and neighbour
anchoring). Properties strings repeat heavily across edges and most
carry no anchor at all. edgeAnchorHints() now returns immediately when
the text cannot contain either key and otherwise parses each distinct
string once per thread through a small bounded cache. A process-wide
parse counter lets the test pin it: one hundred lookups of the same
anchored string cost one parse, and un-anchored or malformed strings
cost none.

computePathMatchScore lowercased and trimmed the query for every
candidate document; pathSearch now does that once and passes the
lowered query down. The existing path-search tests cover the scoring.

portability_allowlist.txt: two search_service.cpp anchors reflowed.

edgeAnchorHints returns its two short strings by value: the per-thread memo
clears itself at 1024 entries, so a reference into it could dangle across
calls even though today's three callers consume the result immediately.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): preserve hydration failure semantics

Stop reranking before scorer calls or score changes when previews fail.
Commit anchor hints atomically and recognize escaped JSON keys.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-12 10:09:20 -04:00
Trevon
3811a2b562
[07/14] Batch tag and topology hydration (#147)
* perf(search): load tag filters once per request, not once per document

pathSearch and searchByHashPrefix filtered candidates through
metadataHasTags(), which for every document called getAllMetadata()
(four retries with backoff on failure), concatenated every metadata
key and value into a string, and logged that string at info level.
A path search over a few thousand documents was a few thousand
metadata round trips and a few thousand info lines.

Both paths now load tags for the whole candidate set with one
batchGetDocumentTags() call (the API GraphComponent already used) and
filter with a pure matcher. The legacy fallback that also accepted a
metadata row with key "tag" is gone: nothing writes that form and the
repository's own tag APIs never honoured it, so search and repository
now agree on what a tag is.

A counting repository subclass pins the shape: one batch call and zero
getAllMetadata calls for both the path and the hash-prefix routes.
portability_allowlist.txt: two search_service.cpp anchors reflowed.

Correction to the claim above that nothing writes the legacy `key = "tag"`
form: the repair downloads pass did (tag values "downloaded", "host:<h>",
"scheme:<s>"), and the search fallback was its only reader. That writer now
uses the same `tag:<name>` keys as every other tag writer, so `--tags
downloaded` keeps matching documents the repair pass tagged. And
batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it
an unbounded candidate set; an oversized list used to fail the statement
and, with this change, would have emptied the result instead of degrading
per document.

Signed-off-by: trvon <me@trevon.dev>

* perf(search): hydrate topology routing candidates in batches

The routing session called getDocumentByHash once per hash in three
places on the product routing path: ranked-candidate admission, cluster
expansion, and the graph-neighbor diagnostic trace (memoized, but still
one statement per distinct hash across four stages). Each now issues
one batchGetDocumentsByHash for its whole set, the same API the
topology input extractor already used, and the docLookupMicros timing
covers the batch. Cluster expansion applies the maxDocs cut before
hydrating so no document is fetched only to be discarded.

A session-level test with one seed and eight graph neighbours under a
counting repository fails on the previous code (17 single lookups, no
batch) and passes now (0 single lookups). The full topology suite is
unchanged.

portability_allowlist.txt: one topology test anchor reflowed.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): bound hydration and preserve errors

Retain legacy tags and honor per-connection SQLite bind limits.
Keep hash chunks in one read snapshot and release it on failure.

Failed lookups are not missing documents or successful empty searches.
Reject partial route admission and preserve tagged path errors.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-09-12 10:09:19 -04:00
trvon
8ad4ab29e0
fix(search): correct code navigation
Honor root-anchored and slash-qualified ignore patterns during live fallback. Center hydrated snippets on query matches and preserve symbol ranking through the daemon boundary.
2026-07-28 22:33:31 -04:00
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
Trevon
a3d065dd8d
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish

Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.

* fix(daemon): stop shutdown log storm

Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.

* refactor(cli): delegate serve to MCP binary

Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.

* refactor(cli): move daemon path helpers

Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.

* refactor(daemon): split embedded client

* perf(search): add lean-guided benchmark gates

Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.

Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.

* refactor(config): move daemon policies

* refactor(daemon): extract plugin runtime

* refactor(daemon): untangle service coordinators

Untangle CLI/MCP from daemon service coordinators and the daemon static lib.

- MetadataRepository uses a daemon-neutral tree-diff callback. The
  GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
  owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
  static lib. Drop ResourceGovernor coupling inside the writer loop; batch
  sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
  its own enqueuePostIngest; app services no longer call ServiceManager
  directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
  concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
  lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
  app services link cleanly without daemon.

Validated:
  meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
  yams-cli --version, yams-mcp-server init, yams serve --help smoke tests

Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.

* fix(cli): keep stdin add local fallback

No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.

Validated:
  meson compile -C build/debug -j4 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
  meson compile -C build/release -j4 yams-cli yams-mcp-server
  meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output

* feat(bench): add xplan multi-arm experiment harness

Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.

* feat(formal): expand topology/search Lean contracts

Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.

* feat(search,topology): construction purity + graph expansion path

Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.

* feat(topology): add KMeans engine and SGC identity contracts

Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.

* feat(daemon): topology readiness gates and selectable reranker

Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.

* chore(search): drop meta_path tuner knobs

Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].

* feat(bench): add topology_purity_validate plan

Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.

* chore: format add_command and config include order

Mechanical clang-format / include reorder only; no behavior change.

* feat(bench): decision-grade xplan loop for search engine

Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.

* feat(graph): denser product semantic_neighbor topK

Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.

* feat(search): extract topology assist and cross-rerank seams

Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.

* feat(bench): product-path search engine ablation plans

Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.

* fix(ci): repair pre-push fallbacks

* feat(bench): lock-free mid-run builds + multi-corpus gates

xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.

* fix(daemon): keep SearchEngine alive via shared_ptr cache

getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.

Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.

* feat(search): route vectors through cached topology

Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.

Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.

* feat(bench): consolidate search decision plans

Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.

Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.

* feat(search): refine topology vector routing

* fix(search): propagate daemon readiness

* feat(bench): measure generalized memory health

* fix(cli): honor in-process fallbacks

* ci: harden smolvm debug builds

* perf(search): route topology through filtered ANN

* feat(bench): measure read-write pressure

* perf(metadata): index symbol lookups on read pool

* fix(daemon): isolate startup reads and maintenance

* feat(bench): replay isolated generalized corpora

* feat(search): add evidence routing pipeline

Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.

* refactor(search): unify topology evidence path

Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.

* refactor(bench): prune superseded search lanes

Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.

* docs(search): retire stale experiment notes

Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.

* refactor(search): prune dead search surfaces

Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.

* fix(cli): clear build warnings

* fix(xplan): reject stale run identity

* refactor(search)!: remove legacy query stack

Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.

BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.

* refactor(search): type value stage traces

Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.

* refactor(search): isolate benchmark env overlays

Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.

* refactor(search): unify topology routing options

Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.

* fix(search): unify tuner config ownership

Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.

* fix(search): snapshot tuner state per query

Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.

* fix(search): enforce topology routing contract

Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.

* fix(daemon): let SQLite reclaim its file

Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.

* fix(metadata): tolerate missing symbol FTS

Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.

* build(fuzz): drop removed query parser target

The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.

* fix(bench): include writes in drain oracle

A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.

* fix(daemon): vacuum padded WAL databases safely
2026-07-13 20:26:43 -04:00
trvon
d67d5d63ac
Remediate Fold SAST findings
Centralize duplicated helpers and harden numeric, timing, IPC, SQLite retry, config, plugin, metrics, and async content-store paths identified during Fold SAST triage.\n\nAdd targeted tests for shared utilities and preserve per-task YAMS checkpoints.
2026-05-27 22:41:44 -04:00
trvon
2d5c6625eb
fix: download logic improvements and ci test fixes. Stagging download fixes seperate from search features 2026-05-04 13:29:17 -04:00
trvon
a4fdb70e6c
feat: introducing epochs for search rebuilding 2026-04-14 14:24:15 -04:00
trvon
55a7f5afcd
feat: search / grep freshness and query intent router improvements. Test improvements 2026-04-14 13:37:41 -04:00
trvon
c475f101b0
feat: Improving search speed for hybrid engine. Trying best to balance speed and precision 2026-04-10 17:30:21 -04:00
trvon
13be8fdfa0
feat: checkpoint for updated unit test for daemon communication / protobuf errors. Expanding untested surface with optimizations for communication and ingestion. Additional work for ci apt / rpm install improvements, systemd configuration for installs and improved validation testing of packaging logic to reduce gpg errors. 2026-04-08 11:39:15 -04:00
trvon
943393af6d
fix: fixing search regression when fuzzy search triggered and testing improvements 2026-03-21 09:00:54 -04:00
0xkite
56cb6a66c5
Recreate last 7 commits (signed) 2026-02-24 14:32:58 -05:00
trvon
901337b135
fix: windows test fixes 2026-02-20 22:51:41 -05:00
trvon
014e12c975
fix: testing improvements and bug fixes in daemon logic for lifecycle management 2026-02-17 15:55:29 -05:00