* refactor(tuning): prune dormant runtime knobs
Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.
* refactor(vector): retire dormant backends
Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.
Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.
* refactor(plugins): remove legacy SDK tooling
Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.
* build(vector): remove stale FAISS overrides
* refactor(ipc): remove dormant FSM snapshots
* refactor(ingest): unify bounded KG enqueue
* fix(plugins): harden allocation contracts
Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).
* fix(storage): define empty object reads
Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.
* refactor(storage): trim through string views
* test: prune dormant harness helpers
Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.
* chore(cpp): initialize analyzer-facing locals
* build(simeon): advance analyzer cleanup
* refactor(config): drop dead daemon client keys
* refactor(ipc): inject daemon client transport
Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.
* refactor(retrieval): route CLI through daemon
Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.
* refactor(graph): move CLI scope into daemon
Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.
* build: isolate test-only runtime controls
* refactor(metadata): decouple lock telemetry
* fix(services): return on request timeout
* fix(retrieval): enforce workspace scope
* fix(graph): scope nodes before pagination
* fix(grep): drop binary result lines
* fix(daemon): bound checkpoint wait
* fix(repair): use KG health checks
* fix(storage): always verify compressed CRC
* perf(vector): skip unchanged PQ snapshots
* build(zyp): exclude bench from default build
* build: protect Meson files from formatter
* refactor: remove unreachable production APIs
* refactor(daemon): remove dormant runtime state
* fix(vector): harden daemon provider calls
Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.
* refactor(storage): remove unusable adapters
* fix(cli): unify in-process add requests
* fix(tsan): preserve async worker progress
* test(cli): isolate daemon-dependent suites
* fix(repair): expose queue backoff progress
Retire timed-out one-shot sockets and propagate connection cancellation through daemon search execution. Stop scheduling later search stages after cancellation while safely draining in-flight work, and reject KG ingest counts that exceed the wire format.
* fix(graph): handle qualified C++ symbols
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector)!: remove dead transition APIs
Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.
BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): harden PQ snapshot lifecycle
Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): expose route coordinates
Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.
Signed-off-by: trvon <me@trevon.dev>
* feat(xplan): add vector decision gates
Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): formalize coordinate routing
Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): serialize SymSpell readers
SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make mutations atomic
Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.
* fix(search)!: surface SymSpell write errors
Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.
BREAKING CHANGE: SymSpellSearch mutations now return Result values.
* feat(topology): formalize candidate expansion
* feat(vector): add coordinate-aware embeddings
* feat(topology): preserve protected relation cover
* feat(search): add graph candidate rescue
* feat(daemon): expose retrieval hydration telemetry
* feat(xplan): add rescue and hydration gates
* refactor(content): dedupe media handlers
Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.
* feat(lifecycle): prune stale corpus data
Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.
* feat(search): add topology candidate rescue
Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.
* feat(search): expose evidence paths
* docs: define mobile host contract
* formal: model bounded candidate rescue
* fix(prune): inspect extensionless signatures
* fix(prune): tolerate stale extensions
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): harden model state ownership
Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.
* refactor(daemon): consolidate ingest pollers
Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.
* refactor(daemon): unify repair execution
Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.
* refactor(daemon): unify runtime tuning
Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.
* refactor(daemon)!: remove dead IPC surfaces
Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.
BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.
* refactor(daemon): hide test hooks
Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.
* fix(daemon): avoid replaying successful adds
A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.
* bench(ingest): enforce lossless pipeline runs
Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.
* bench(search): expose cold seed ingest timings
Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.
* perf(ingest): batch deferred content indexing
Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.
* feat(metadata): expose insert writer metrics
Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.
* fix(daemon): retry transient refused connects
The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.
* bench(ingest): attribute submission path costs
Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.
* perf(ingest): batch durable document storage
Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.
* perf(ingest): coalesce bulk post-processing
Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.
* bench(ingest): record post-ingest batch size
Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.
* perf(ingest): batch queued storage waves
Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.
* perf(ingest): route bulk enrichment batches
Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.
* test(daemon): initialize task fixtures
* perf(ingest): batch post-ingest dispatch
* perf(ingest): widen directory storage waves
Borrow from a full wave when one document would otherwise fall through to single-document storage.
* perf(storage): bound batch writer fanout
Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.
* perf(ingest): align storage batch drains
Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.
* fix(ingest): commit metadata state atomically
Fold first-version, extracted-title, and embedding-completion state
into existing transactions.
Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.
* bench(ingest): measure production submission
Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.
* fix(xplan): preserve native failures
Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.
* fix(daemon): preserve embedding success hashes
* fix(daemon): serialize pooled socket shutdown
* fix(daemon): harden batch publication shutdown
* fix(storage): use compat jthread shim
* fix(daemon): avoid poller template shadowing
* fix(search): keep vector pipeline test enabled
* fix: harden shutdown and fallback paths
Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
* fix(release): repair package repo publish
Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.
* fix(daemon): stop shutdown log storm
Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.
* refactor(cli): delegate serve to MCP binary
Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.
* refactor(cli): move daemon path helpers
Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.
* refactor(daemon): split embedded client
* perf(search): add lean-guided benchmark gates
Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.
Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.
* refactor(config): move daemon policies
* refactor(daemon): extract plugin runtime
* refactor(daemon): untangle service coordinators
Untangle CLI/MCP from daemon service coordinators and the daemon static lib.
- MetadataRepository uses a daemon-neutral tree-diff callback. The
GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
static lib. Drop ResourceGovernor coupling inside the writer loop; batch
sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
its own enqueuePostIngest; app services no longer call ServiceManager
directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
app services link cleanly without daemon.
Validated:
meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
yams-cli --version, yams-mcp-server init, yams serve --help smoke tests
Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.
* fix(cli): keep stdin add local fallback
No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.
Validated:
meson compile -C build/debug -j4 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
meson compile -C build/release -j4 yams-cli yams-mcp-server
meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output
* feat(bench): add xplan multi-arm experiment harness
Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.
* feat(formal): expand topology/search Lean contracts
Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.
* feat(search,topology): construction purity + graph expansion path
Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.
* feat(topology): add KMeans engine and SGC identity contracts
Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.
* feat(daemon): topology readiness gates and selectable reranker
Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.
* chore(search): drop meta_path tuner knobs
Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].
* feat(bench): add topology_purity_validate plan
Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.
* chore: format add_command and config include order
Mechanical clang-format / include reorder only; no behavior change.
* feat(bench): decision-grade xplan loop for search engine
Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.
* feat(graph): denser product semantic_neighbor topK
Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.
* feat(search): extract topology assist and cross-rerank seams
Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.
* feat(bench): product-path search engine ablation plans
Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.
* fix(ci): repair pre-push fallbacks
* feat(bench): lock-free mid-run builds + multi-corpus gates
xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.
* fix(daemon): keep SearchEngine alive via shared_ptr cache
getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.
Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.
* feat(search): route vectors through cached topology
Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.
Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.
* feat(bench): consolidate search decision plans
Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.
Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.
* feat(search): refine topology vector routing
* fix(search): propagate daemon readiness
* feat(bench): measure generalized memory health
* fix(cli): honor in-process fallbacks
* ci: harden smolvm debug builds
* perf(search): route topology through filtered ANN
* feat(bench): measure read-write pressure
* perf(metadata): index symbol lookups on read pool
* fix(daemon): isolate startup reads and maintenance
* feat(bench): replay isolated generalized corpora
* feat(search): add evidence routing pipeline
Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.
* refactor(search): unify topology evidence path
Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.
* refactor(bench): prune superseded search lanes
Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.
* docs(search): retire stale experiment notes
Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.
* refactor(search): prune dead search surfaces
Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.
* fix(cli): clear build warnings
* fix(xplan): reject stale run identity
* refactor(search)!: remove legacy query stack
Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.
BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.
* refactor(search): type value stage traces
Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.
* refactor(search): isolate benchmark env overlays
Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.
* refactor(search): unify topology routing options
Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.
* fix(search): unify tuner config ownership
Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.
* fix(search): snapshot tuner state per query
Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.
* fix(search): enforce topology routing contract
Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.
* fix(daemon): let SQLite reclaim its file
Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.
* fix(metadata): tolerate missing symbol FTS
Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.
* build(fuzz): drop removed query parser target
The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.
* fix(bench): include writes in drain oracle
A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.
* fix(daemon): vacuum padded WAL databases safely
Add focused API, CLI, doctor, downloader, search, and vector
infrastructure coverage.
Cover retry seams, path-scoped search fallback, renderer fail-soft
paths, router helpers, and model infrastructure without changing
production behavior.
Keep coverage gates stable by skipping Zyp on Darwin coverage builds
and ignoring SIGPIPE in daemon protocol tests.
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter
Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close
* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler
* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly
ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.
* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict
setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.
* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build
* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs
* fix(build): restore hooks and shrink srht
Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.
Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.
Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.
* refactor(storage): drop dead code, add coverage
* refactor(cli): polish grep/search graph UX
* fix(daemon): stabilize shutdown and CI paths
- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work
* test(daemon): add db vacuum coverage
* test(daemon): add WAL startup recovery
- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown
* refactor(grep): consolidate indexed content scan
* fix(grep): remove pre-scan candidate truncation
* refactor(runtime): split startup, harden storage
* feat(bench): add manifest retrieval corpus
* fix(bench): run fusion experiment on simeon
* fix(cli): sharpen search graph hints
* fix(build): refresh version metadata
* fix(ci): stabilize sync search and srht packaging
* fix(ci): unblock windows and shrink artifacts
* fix(ci): stabilize linux runtime checks
* fix(build): shrink srht manifest
* feat: search work and moving to experimental
* WIP
* feat(search): diagnostic logging pipeline — graph build, concepts, reranker gate
- New [concepts] per-50-query log showing extracted query concepts, type
distribution, and used_gliner flag.
- New [PIQ-entity] per-1000-dispatch + per-document completion log with
entity type counts (e.g., 'protein=3, gene=1, chemical=2').
- New [graph_rerank-gate] one-shot log showing why enableGraphRerank
is zeroed: shows shortQueryBudgeted, corpusWarming, bypassWarmingGate,
lexicalReady, and post-ingest inflight.
- New [graph_rerank] per-50-query log with window size, matched
candidates, applied count, concept count, and elapsed time.
- retrieval_quality_bench.cpp: add YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE=1
so benchmark queries are not gated by freshness checks.
- resolve merge conflicts from experimental branch.
* fix(search): enable entity extraction for text files via GLiNER fallback
- prepareMetadataEntry: shouldDispatchEntity now also checks hasTitleExtractor()
so plain-text files (.txt, .md) get entity extraction even when no binary
entity provider supports their extension.
- processEntityExtractionStage: when no ExternalEntityProvider matches,
fall back to processTitleExtractionStage which runs GLiNER on text.
- refreshStageAvailability: entity stage is now considered capable when
hasTitleExtractor() is true, so the entity poller gets concurrency budget
even if entityProviders_ is empty.
- kCombinedEntityTypes: add biomedical entity types (protein, gene, cell,
disease, chemical, drug, pathway, biological_process, biomarker, anatomy,
organism) so GLiNER extracts high-value biomedical entities instead of
only generic NL types (person, organization, concept).
* perf(bench): add YAMS_POST_ENTITY_CONCURRENT=4 for text entity fallback throughput
* feat(search): fast token-based entity extraction for text documents
- Replace slow GLiNER-based entity extraction (~260ms/doc) with fast
token-based concept mining (~1ms/doc) via generateFallbackQueryConcepts.
Uses sub-phrase mining + IDF-weighted token salience — the same function
already used for query-time concept fallback.
- Store entities with full KG scaffolding: KGNode with entity type/label,
KGAlias for query-time resolution, DeferredEdge from entity→document,
and DeferredDocEntity linking entity to document for getDocEntitiesForDocument.
- Add biomedical entity types to kCombinedEntityTypes in title extraction
(protein, gene, cell, disease, chemical, drug, pathway, etc.) so GLiNER
can output high-value biomedical entities when used.
- Add include for query_expansion.h to access generateFallbackQueryConcepts.
- bench: add YAMS_POST_ENTITY_CONCURRENT=4 for entity poller throughput.
Benchmark: 22K+ entities extracted across 5183 docs, ~41% high-value
biomedical. MRR 0.4281 (beats 0.4273 baseline by +0.2%).
* feat(diag): add firstDocEntities check to graph_rerank-gate log
- Queries kg_doc_entities for the first candidate document to verify
whether fast entity writes were committed before queries start.
- Confirmed: firstDocEntities=0 — entities stuck in WriteCoordinator
async queue when benchmark queries begin.
* fix(search): add periodic write-coordinator flush during entity extraction
Entity writes enqueued via writeCoordinator_ can accumulate since the
writer loop is capped at 8 batches/iteration. For 5K+ documents this
creates a backlog that outlasts the post-ingest drain wait — entities
are not committed to kg_doc_entities before benchmark queries start.
Add a periodic flush (every 128 batches, 500ms timeout) in the entity
extraction batch processor to drain the WC queue incrementally.
Diagnostic confirms firstDocEntities=0 persists — a synchronous
committed flush or batched WC enqueue is needed.
* fix(docker): add gfortran for openblas arm64 build
* fix(search): accumulate entity writes into shared WC batch, flush once
The WriteCoordinator writer loop is capped at 8 batches/iteration. With
5K+ per-doc entity writes, entities outlast the benchmark's post-ingest
drain and firstDocEntities=0 when the graph scorer runs.
- processEntityExtractionStage: add optional DeferredKGBatch* parameter.
Text-fallback appends entity nodes/aliases/edges/doc-entities to the
shared batch instead of creating per-doc WC batches.
- processEntityExtractionBatch: creates one DeferredKGBatch, passes it to
all stage calls in the batch, enqueues to WC once at end.
- Single-shot WC flush: after all stages go idle (totalInFlight==0),
calls writeCoordinator_->flush(15s) to drain remaining entity writes
before benchmark queries start.
- DeferredKGBatch forward-declared in PostIngestQueue header.
* ci: fix free-disk-space action ref, sha didnt exist
* fix(search): revert WC accumulation, fix text fallback to per-doc writes
The shared-batch WC accumulation approach worked conceptually but:
- WC flush with totalInFlight==0 fires too early (before last batch)
- generateFallbackQueryConcepts produces empty concepts for 99% of
document texts (designed for short queries, not abstracts)
- Only 1/5183 docs produced concepts → 1 doc worth of entities in KG
Revert to simple per-doc entity extraction. The enable=/disable=
graph reranker gate is still controlled by shortQueryBudgeted=1
which zeroes the graph reranker on the first query (intent=Code).
firstDocEntities=0 persists — entities need a different creation
strategy for document text (phrase mining too conservative).
* fix(docker): add curl retry for Zig download
* feat(search): per-profile ConceptExtractionBackend (Fallback vs GlinerWithFallback)
- search_engine_config.h: new ConceptExtractionBackend enum with Fallback
(sub-phrase mining + IDF tokens, <1ms) and GlinerWithFallback (GLiNER
first, fall back on empty/timeout). Default: GlinerWithFallback.
- search_tuner.h: SCIENTIFIC profile sets Fallback (GLiNER returns empty
for 99% of short biomedical claims). Other profiles keep default.
Wired through TunedParams.applyTo().
- search_engine.cpp: gate GLiNER concept extraction on the config backend.
When Fallback, conceptFuture is never created and concepts come directly
from generateFallbackQueryConcepts. Added [concept-backend] diagnostic.
- search_engine_builder.cpp: builder log now includes concept_backend
in the override-state startup message.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC profile uses Fallback.
Note: builder correctly sets Fallback but query-time workingConfig still
shows GlinerWithFallback (int=1). This is a config-sync issue between
builder-time profile application and runtime config snapshot — the
tuner or a post-override config copy may reset the field. To be fixed.
* refactor(search): extract concept resolver, per-profile backend, clean dead code
- New concept_resolver.h/cpp: extract enrichWithFallbackConcepts from the
search engine monolith into a testable unit. Generates IDF-weighted
token concepts via generateFallbackQueryConcepts, deduplicates, and
appends to existing concepts.
- search_engine_config.h: new ConceptExtractionBackend enum (Fallback vs
GlinerWithFallback). SCIENTIFIC profile sets Fallback since GLiNER
extracts 0 concepts for 99% of short biomedical claims.
- search_tuner.h: wire conceptExtractionBackend through TunedParams,
applyTo, seedTunedParamsFromConfig (the one-line fix that resolved
the config-sync bug where profile Fallback was overridden at query
policy resolution time).
- search_engine.cpp: materializeConcepts lambda restructured to always
reach the fallback even when GLiNER future is invalid. Removed
~80 lines of diagnostic bloat ([concept-backend], [workingConfig-*],
per-50 [concepts]). GLiNER gate checks config backend.
- PostIngestQueue.cpp: removed dead fast-token entity extraction
(~150 lines of concept generation + KG scaffolding that produced
0 entities for 99% of documents). Reverted entity-stage gating
(entityCapable || hasTitleExtractor) and shouldDispatchEntity to
extension-only checks. Removed DeferredKGBatch forward decl.
- gliner_query_extractor: removed unused createGlinerExtractionFunc-
FromProvider fallback factory (~150 lines).
- ServiceManager.cpp: removed fallback call site for removed factory.
- builder log: removed concept_backend diagnostic line noise.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC uses Fallback.
Net: ~400 lines removed, ~120 lines added (concept_resolver unit +
config wiring). GLiNER concept extraction savings: ~30-80ms/query
× 300 queries = 9-24s total for SCIENTIFIC profiles.
* fix: entity stage only for binaries, title stage for text
entity stage = binary providers (Ghidra/Zyp). title stage = GLiNER for all
text. No overlap — title already stores NL entities to KG.
* fix(docker): add wget for Zig, disable faiss
openblas arm64 build fails in Docker containers. YAMS_DISABLE_FAISS=1
env var skips faiss (and its openblas transitive dep). Zig download
now uses wget -c for resume on flaky ziglang.org connections.
* ci: add experimental branch to tests trigger
* ci: pin free-disk-space to v1.3.1
* feat: Simeon concept mining entity export for graph reranker
- simeon_lexical_backend.h: EntityCallback in Config. Called with
(docId, entityText, confidence) per concept match per doc.
- simeon_lexical_backend.cpp: after mine_concepts, iterate docs
via collect_doc_concepts + entity_callback.
- SearchEngineManager.cpp: enable concept_mining_enabled, wire
entity_callback batch-inserting DocEntity via KG store. Fix
kgStore capture-before-move race.
Note: simeon submodule has collect_doc_concepts in concept_mining.hpp
+ concept_mining.cpp (must commit to submodule separately).
* ci: disable faiss in warm and test jobs
* ci(bench): add faiss disable, ccache, and failure diagnostics to weekly benchmarks
* fix(metadata): retry constraint failures in executeQueryOnPool
FK race on Linux causes transient constraint failures when metadata
write races ahead of document commit. Previously treated as terminal.
Now retried with backoff like lock errors.
* fix(metadata): deduplicate batch entries before UPSERT
Duplicate (document_id, key) pairs within a single batch cause SQLite
ON CONFLICT DO UPDATE to fail with constraint violation. Dedup keeps
the last write per key.
* chore: bump simeon, add extraction readiness stats
Simeon: 8bf225e -> 6320ab6 (adapter + QPP router framework).
CorpusStats: add contentExtracted*, ftsIndexed*, titleCoverage,
languageCoverage, and KG topology richness fields.
* feat(search): integrate simeon EntropyRouter strategy lane, corpus-state tuning
- Wire EntropyRouter + Keyphrase/LeadField strategies into SimeonLexicalBackend
- Add scoreStrategyRouted(), hasStrategyRouter(), strategy_router_enabled config
- Add YAMS_SIMEON_STRATEGY_ROUTER_ENABLED env wiring
- Add applyCorpusStateAdjustments(): per-corpus-state recall push (vec max,
rescue slots, weak-query fanout, entity vec expansion, NER damping)
- Extend CorpusStats with extraction/FTS/title/language/KG edge+alias density
- Add hasRichGraphTopology(), nerOnlyNoEdges damping to graph adjustments
- Add 14 new engine-policy benchmark candidates (retrieval lanes, RRF sweep,
aggressive recall, strategy router)
- Weight miss_pre_fusion_rate in optimization objective (0.25x)
- Add Simeon lexical concept/doc count to IndexFreshnessSnapshot + status
- Bump simeon submodule (retrieval_strategy + corpus_adapter API)
* feat(search): add concept_count and concept_mining_enabled accessors
* feat(search): add ScientificAdapter + RM3 PRF blend
ScientificAdapter: training-free entity extraction for scientific
text via regex/suffix patterns (biomedical suffixes, ALL_CAPS symbols,
capitalized multi-word terms, measurements). Replaces TextAdapter
in strategy router build, providing entity lists for KeyphraseStrategy.
RM3: add score_with_prf blend path to scoreStrategyRouted at 0.30
weight when rm3_enabled=true. New benchmark candidate
scientific_strategy_rm3.
Strategy router + ScientificAdapter: mrr=0.4514 ndcg=0.4789 recall=0.5849
