When the daemon holds background work back for a busy host, a user
looking at a stale topology or an unfinished repair scan needs to see
why. Status had no host signal at all.
Carry the governor's host sample through the metrics snapshot and
StatusResponse: source (psi, loadavg), whether the host counts as busy,
the PSI cpu/io/memory "some avg10" values or the per-CPU load average,
and which deferrable work is currently held back. The proto transport
piggybacks the fields on request_counts like the other string status
fields; the native serializer appends them as optional tail fields.
`yams daemon status` (brief and detailed) shows a Host Load row, e.g.
"Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and
`yams status --json` gains a host_pressure object. Document the new
[tuning.resource] keys next to the other typed tuning sections.
Signed-off-by: trvon <me@trevon.dev>
README, AGENTS.md, the yams skill guide and the engineering prompt
still promised symbol extraction, calls/defined_in edges, and graph
--impact for callers. Point callers and blast radius at yams grep and
describe the relations the graph still holds.
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays
LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.
Removed (CANDIDATE rows of the report, verbatim):
YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
YAMS_SEARCH_GRAPH_ENABLE_PATHS
YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
YAMS_SEARCH_GRAPH_HOP_DECAY
YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
YAMS_SEARCH_GRAPH_MAX_HOPS
YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
YAMS_SEARCH_GRAPH_MAX_PATHS
YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
YAMS_SEARCH_GRAPH_TEXT_WEIGHT
YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
YAMS_SEARCH_RERANK_TOP_K
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
YAMS_SEARCH_TIERED_MIN_CANDIDATES
YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE
Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.
A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.
The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): drop unread embeddings.* keys and the no-op toggles
yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.
- embeddings enable and disable are removed. enable duplicated
embeddings model <name> once its dead keys are gone; a kept no-op
would be a shim. status no longer prints an auto-generation line and
shows the runtime batch size when set. tune presets write
embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
[embeddings] batch_size silently when the runtime key is absent, so
configs in the wild get their value applied without a file rewrite (the
migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.
Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.
Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.
Signed-off-by: trvon <me@trevon.dev>
* feat(config): add [tuning] keys for the setters only env could reach
Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.
Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.
A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.
The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.
Signed-off-by: trvon <me@trevon.dev>
* fix(config): report 0 for [tuning] integer keys instead of ignoring it
For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.
Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.
Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): one embedding-runtime resolver, one tuning baseline
Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.
yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.
Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): stop writing the unread search.reranker_model_path
yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.
Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.
portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.
The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).
Signed-off-by: trvon <me@trevon.dev>
* fix(config): validate tuning ranges and pairs
Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.
Cover all 27 integer keys, inherited limits, and reservation reset.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* chore(repo): land the retrieval contract and drop root cruft
Lands work that had been sitting uncommitted in the working tree:
- docs/architecture/retrieval_contracts.md plus its executable form,
tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py,
now registered in meson as retrieval_contract_unit (suite: policy).
.gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore.
- AGENTS.md: the Development Memory section, and the product default
recorded as hybrid_assist + shadow with topology-off as the control,
which is what search_topology_stage already does.
docs/benchmarks/index.md gains the matching search_shadow_cost plan.
- .gitleaks.toml, and a Secret scan workflow that actually runs it on
push and pull requests (nothing in .github invoked gitleaks before).
Deletes three root files with zero references anywhere in the tree:
bench_profile.json (ad-hoc profiling leftover), CppProperties.json
(Visual Studio config; the repo standardises on Meson), and spdlog.pc
(a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would
mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan).
docs/plans/ (two p2p designs already shipped in 5a53cdfa and d012d95b)
was ingested into YAMS with task=desloppification source=plan, hydrated
via yams cat and byte-compared before removal, per the new AGENTS.md
rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a
plans directory.
Also reflows two stale line anchors in raw_environment_allowlist.txt
for tools/yams-cli/main.cpp (shifted by 11c39157) so the policy suite
passes again; no new environment writes.
Not included: the external/yams-repo submodule pointer in the working
tree was a stale local clone two commits behind, not an intended bump.
Signed-off-by: trvon <me@trevon.dev>
* test(policy): make the configuration-authority ratchet bidirectional
check_configuration_authority.py only reported keys and TOML readers
present in code but missing from the allowlists, so the reviewed
YAMS_* surface could grow but never shrink: production_environment_keys.txt
even said removals "may remain grandfathered". An allowlist that only
accretes stops describing the code.
The policy now also fails on allowlist entries with no remaining
occurrence in src/, include/, or tools/, naming the lines to delete.
Key removals in later commits therefore have to update the allowlist in
the same change, and the count in the success banner becomes a real
measure of the environment surface (432 today).
stale_entries() exposes the two stale sets for tooling; three unit
tests cover stale key, stale reader, and the returned names.
Signed-off-by: trvon <me@trevon.dev>
* tools(policy): report where every configuration key is referenced
Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in
production_environment_keys.txt and each dotted TOML key literally
resolved in ConfigResolver.cpp, count whole-token references per tree
bucket (src, test, bench, docs, plugin; build output and the allowlists
themselves excluded) and assign a verdict: KEEP when anything outside
product code exercises or documents the key, CANDIDATE when only
product code mentions it, NOT_IN_SRC when the allowlist is stale.
The report is evidence, not a decision. From here on a key deletion
must paste its CANDIDATE rows into the change description; the first
audit pass hand-listed keys that turned out to have bench and test
consumers, which is exactly the mistake a mechanical inventory avoids.
Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys
(103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit.
The report counts only files git tracks (git ls-files) when the root is a
work tree, so an ignored local bench harness cannot turn a CANDIDATE into a
KEEP on one machine and not another; outside a work tree it falls back to
scanning the directory, which is what the synthetic-tree test exercises.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): handle qualified C++ symbols
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector)!: remove dead transition APIs
Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.
BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): harden PQ snapshot lifecycle
Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): expose route coordinates
Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.
Signed-off-by: trvon <me@trevon.dev>
* feat(xplan): add vector decision gates
Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): formalize coordinate routing
Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): serialize SymSpell readers
SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make mutations atomic
Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.
* fix(search)!: surface SymSpell write errors
Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.
BREAKING CHANGE: SymSpellSearch mutations now return Result values.
* feat(topology): formalize candidate expansion
* feat(vector): add coordinate-aware embeddings
* feat(topology): preserve protected relation cover
* feat(search): add graph candidate rescue
* feat(daemon): expose retrieval hydration telemetry
* feat(xplan): add rescue and hydration gates
* refactor(content): dedupe media handlers
Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.
* feat(lifecycle): prune stale corpus data
Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.
* feat(search): add topology candidate rescue
Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.
* feat(search): expose evidence paths
* docs: define mobile host contract
* formal: model bounded candidate rescue
* fix(prune): inspect extensionless signatures
* fix(prune): tolerate stale extensions
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(release): repair package repo publish
Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.
* fix(daemon): stop shutdown log storm
Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.
* refactor(cli): delegate serve to MCP binary
Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.
* refactor(cli): move daemon path helpers
Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.
* refactor(daemon): split embedded client
* perf(search): add lean-guided benchmark gates
Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.
Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.
* refactor(config): move daemon policies
* refactor(daemon): extract plugin runtime
* refactor(daemon): untangle service coordinators
Untangle CLI/MCP from daemon service coordinators and the daemon static lib.
- MetadataRepository uses a daemon-neutral tree-diff callback. The
GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
static lib. Drop ResourceGovernor coupling inside the writer loop; batch
sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
its own enqueuePostIngest; app services no longer call ServiceManager
directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
app services link cleanly without daemon.
Validated:
meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
yams-cli --version, yams-mcp-server init, yams serve --help smoke tests
Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.
* fix(cli): keep stdin add local fallback
No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.
Validated:
meson compile -C build/debug -j4 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
meson compile -C build/release -j4 yams-cli yams-mcp-server
meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output
* feat(bench): add xplan multi-arm experiment harness
Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.
* feat(formal): expand topology/search Lean contracts
Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.
* feat(search,topology): construction purity + graph expansion path
Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.
* feat(topology): add KMeans engine and SGC identity contracts
Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.
* feat(daemon): topology readiness gates and selectable reranker
Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.
* chore(search): drop meta_path tuner knobs
Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].
* feat(bench): add topology_purity_validate plan
Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.
* chore: format add_command and config include order
Mechanical clang-format / include reorder only; no behavior change.
* feat(bench): decision-grade xplan loop for search engine
Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.
* feat(graph): denser product semantic_neighbor topK
Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.
* feat(search): extract topology assist and cross-rerank seams
Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.
* feat(bench): product-path search engine ablation plans
Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.
* fix(ci): repair pre-push fallbacks
* feat(bench): lock-free mid-run builds + multi-corpus gates
xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.
* fix(daemon): keep SearchEngine alive via shared_ptr cache
getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.
Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.
* feat(search): route vectors through cached topology
Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.
Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.
* feat(bench): consolidate search decision plans
Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.
Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.
* feat(search): refine topology vector routing
* fix(search): propagate daemon readiness
* feat(bench): measure generalized memory health
* fix(cli): honor in-process fallbacks
* ci: harden smolvm debug builds
* perf(search): route topology through filtered ANN
* feat(bench): measure read-write pressure
* perf(metadata): index symbol lookups on read pool
* fix(daemon): isolate startup reads and maintenance
* feat(bench): replay isolated generalized corpora
* feat(search): add evidence routing pipeline
Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.
* refactor(search): unify topology evidence path
Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.
* refactor(bench): prune superseded search lanes
Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.
* docs(search): retire stale experiment notes
Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.
* refactor(search): prune dead search surfaces
Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.
* fix(cli): clear build warnings
* fix(xplan): reject stale run identity
* refactor(search)!: remove legacy query stack
Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.
BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.
* refactor(search): type value stage traces
Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.
* refactor(search): isolate benchmark env overlays
Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.
* refactor(search): unify topology routing options
Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.
* fix(search): unify tuner config ownership
Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.
* fix(search): snapshot tuner state per query
Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.
* fix(search): enforce topology routing contract
Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.
* fix(daemon): let SQLite reclaim its file
Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.
* fix(metadata): tolerate missing symbol FTS
Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.
* build(fuzz): drop removed query parser target
The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.
* fix(bench): include writes in drain oracle
A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.
* fix(daemon): vacuum padded WAL databases safely
* feat: search work and moving to experimental
* WIP
* feat(search): diagnostic logging pipeline — graph build, concepts, reranker gate
- New [concepts] per-50-query log showing extracted query concepts, type
distribution, and used_gliner flag.
- New [PIQ-entity] per-1000-dispatch + per-document completion log with
entity type counts (e.g., 'protein=3, gene=1, chemical=2').
- New [graph_rerank-gate] one-shot log showing why enableGraphRerank
is zeroed: shows shortQueryBudgeted, corpusWarming, bypassWarmingGate,
lexicalReady, and post-ingest inflight.
- New [graph_rerank] per-50-query log with window size, matched
candidates, applied count, concept count, and elapsed time.
- retrieval_quality_bench.cpp: add YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE=1
so benchmark queries are not gated by freshness checks.
- resolve merge conflicts from experimental branch.
* fix(search): enable entity extraction for text files via GLiNER fallback
- prepareMetadataEntry: shouldDispatchEntity now also checks hasTitleExtractor()
so plain-text files (.txt, .md) get entity extraction even when no binary
entity provider supports their extension.
- processEntityExtractionStage: when no ExternalEntityProvider matches,
fall back to processTitleExtractionStage which runs GLiNER on text.
- refreshStageAvailability: entity stage is now considered capable when
hasTitleExtractor() is true, so the entity poller gets concurrency budget
even if entityProviders_ is empty.
- kCombinedEntityTypes: add biomedical entity types (protein, gene, cell,
disease, chemical, drug, pathway, biological_process, biomarker, anatomy,
organism) so GLiNER extracts high-value biomedical entities instead of
only generic NL types (person, organization, concept).
* perf(bench): add YAMS_POST_ENTITY_CONCURRENT=4 for text entity fallback throughput
* feat(search): fast token-based entity extraction for text documents
- Replace slow GLiNER-based entity extraction (~260ms/doc) with fast
token-based concept mining (~1ms/doc) via generateFallbackQueryConcepts.
Uses sub-phrase mining + IDF-weighted token salience — the same function
already used for query-time concept fallback.
- Store entities with full KG scaffolding: KGNode with entity type/label,
KGAlias for query-time resolution, DeferredEdge from entity→document,
and DeferredDocEntity linking entity to document for getDocEntitiesForDocument.
- Add biomedical entity types to kCombinedEntityTypes in title extraction
(protein, gene, cell, disease, chemical, drug, pathway, etc.) so GLiNER
can output high-value biomedical entities when used.
- Add include for query_expansion.h to access generateFallbackQueryConcepts.
- bench: add YAMS_POST_ENTITY_CONCURRENT=4 for entity poller throughput.
Benchmark: 22K+ entities extracted across 5183 docs, ~41% high-value
biomedical. MRR 0.4281 (beats 0.4273 baseline by +0.2%).
* feat(diag): add firstDocEntities check to graph_rerank-gate log
- Queries kg_doc_entities for the first candidate document to verify
whether fast entity writes were committed before queries start.
- Confirmed: firstDocEntities=0 — entities stuck in WriteCoordinator
async queue when benchmark queries begin.
* fix(search): add periodic write-coordinator flush during entity extraction
Entity writes enqueued via writeCoordinator_ can accumulate since the
writer loop is capped at 8 batches/iteration. For 5K+ documents this
creates a backlog that outlasts the post-ingest drain wait — entities
are not committed to kg_doc_entities before benchmark queries start.
Add a periodic flush (every 128 batches, 500ms timeout) in the entity
extraction batch processor to drain the WC queue incrementally.
Diagnostic confirms firstDocEntities=0 persists — a synchronous
committed flush or batched WC enqueue is needed.
* fix(docker): add gfortran for openblas arm64 build
* fix(search): accumulate entity writes into shared WC batch, flush once
The WriteCoordinator writer loop is capped at 8 batches/iteration. With
5K+ per-doc entity writes, entities outlast the benchmark's post-ingest
drain and firstDocEntities=0 when the graph scorer runs.
- processEntityExtractionStage: add optional DeferredKGBatch* parameter.
Text-fallback appends entity nodes/aliases/edges/doc-entities to the
shared batch instead of creating per-doc WC batches.
- processEntityExtractionBatch: creates one DeferredKGBatch, passes it to
all stage calls in the batch, enqueues to WC once at end.
- Single-shot WC flush: after all stages go idle (totalInFlight==0),
calls writeCoordinator_->flush(15s) to drain remaining entity writes
before benchmark queries start.
- DeferredKGBatch forward-declared in PostIngestQueue header.
* ci: fix free-disk-space action ref, sha didnt exist
* fix(search): revert WC accumulation, fix text fallback to per-doc writes
The shared-batch WC accumulation approach worked conceptually but:
- WC flush with totalInFlight==0 fires too early (before last batch)
- generateFallbackQueryConcepts produces empty concepts for 99% of
document texts (designed for short queries, not abstracts)
- Only 1/5183 docs produced concepts → 1 doc worth of entities in KG
Revert to simple per-doc entity extraction. The enable=/disable=
graph reranker gate is still controlled by shortQueryBudgeted=1
which zeroes the graph reranker on the first query (intent=Code).
firstDocEntities=0 persists — entities need a different creation
strategy for document text (phrase mining too conservative).
* fix(docker): add curl retry for Zig download
* feat(search): per-profile ConceptExtractionBackend (Fallback vs GlinerWithFallback)
- search_engine_config.h: new ConceptExtractionBackend enum with Fallback
(sub-phrase mining + IDF tokens, <1ms) and GlinerWithFallback (GLiNER
first, fall back on empty/timeout). Default: GlinerWithFallback.
- search_tuner.h: SCIENTIFIC profile sets Fallback (GLiNER returns empty
for 99% of short biomedical claims). Other profiles keep default.
Wired through TunedParams.applyTo().
- search_engine.cpp: gate GLiNER concept extraction on the config backend.
When Fallback, conceptFuture is never created and concepts come directly
from generateFallbackQueryConcepts. Added [concept-backend] diagnostic.
- search_engine_builder.cpp: builder log now includes concept_backend
in the override-state startup message.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC profile uses Fallback.
Note: builder correctly sets Fallback but query-time workingConfig still
shows GlinerWithFallback (int=1). This is a config-sync issue between
builder-time profile application and runtime config snapshot — the
tuner or a post-override config copy may reset the field. To be fixed.
* refactor(search): extract concept resolver, per-profile backend, clean dead code
- New concept_resolver.h/cpp: extract enrichWithFallbackConcepts from the
search engine monolith into a testable unit. Generates IDF-weighted
token concepts via generateFallbackQueryConcepts, deduplicates, and
appends to existing concepts.
- search_engine_config.h: new ConceptExtractionBackend enum (Fallback vs
GlinerWithFallback). SCIENTIFIC profile sets Fallback since GLiNER
extracts 0 concepts for 99% of short biomedical claims.
- search_tuner.h: wire conceptExtractionBackend through TunedParams,
applyTo, seedTunedParamsFromConfig (the one-line fix that resolved
the config-sync bug where profile Fallback was overridden at query
policy resolution time).
- search_engine.cpp: materializeConcepts lambda restructured to always
reach the fallback even when GLiNER future is invalid. Removed
~80 lines of diagnostic bloat ([concept-backend], [workingConfig-*],
per-50 [concepts]). GLiNER gate checks config backend.
- PostIngestQueue.cpp: removed dead fast-token entity extraction
(~150 lines of concept generation + KG scaffolding that produced
0 entities for 99% of documents). Reverted entity-stage gating
(entityCapable || hasTitleExtractor) and shouldDispatchEntity to
extension-only checks. Removed DeferredKGBatch forward decl.
- gliner_query_extractor: removed unused createGlinerExtractionFunc-
FromProvider fallback factory (~150 lines).
- ServiceManager.cpp: removed fallback call site for removed factory.
- builder log: removed concept_backend diagnostic line noise.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC uses Fallback.
Net: ~400 lines removed, ~120 lines added (concept_resolver unit +
config wiring). GLiNER concept extraction savings: ~30-80ms/query
× 300 queries = 9-24s total for SCIENTIFIC profiles.
* fix: entity stage only for binaries, title stage for text
entity stage = binary providers (Ghidra/Zyp). title stage = GLiNER for all
text. No overlap — title already stores NL entities to KG.
* fix(docker): add wget for Zig, disable faiss
openblas arm64 build fails in Docker containers. YAMS_DISABLE_FAISS=1
env var skips faiss (and its openblas transitive dep). Zig download
now uses wget -c for resume on flaky ziglang.org connections.
* ci: add experimental branch to tests trigger
* ci: pin free-disk-space to v1.3.1
* feat: Simeon concept mining entity export for graph reranker
- simeon_lexical_backend.h: EntityCallback in Config. Called with
(docId, entityText, confidence) per concept match per doc.
- simeon_lexical_backend.cpp: after mine_concepts, iterate docs
via collect_doc_concepts + entity_callback.
- SearchEngineManager.cpp: enable concept_mining_enabled, wire
entity_callback batch-inserting DocEntity via KG store. Fix
kgStore capture-before-move race.
Note: simeon submodule has collect_doc_concepts in concept_mining.hpp
+ concept_mining.cpp (must commit to submodule separately).
* ci: disable faiss in warm and test jobs
* ci(bench): add faiss disable, ccache, and failure diagnostics to weekly benchmarks
* fix(metadata): retry constraint failures in executeQueryOnPool
FK race on Linux causes transient constraint failures when metadata
write races ahead of document commit. Previously treated as terminal.
Now retried with backoff like lock errors.
* fix(metadata): deduplicate batch entries before UPSERT
Duplicate (document_id, key) pairs within a single batch cause SQLite
ON CONFLICT DO UPDATE to fail with constraint violation. Dedup keeps
the last write per key.
* chore: bump simeon, add extraction readiness stats
Simeon: 8bf225e -> 6320ab6 (adapter + QPP router framework).
CorpusStats: add contentExtracted*, ftsIndexed*, titleCoverage,
languageCoverage, and KG topology richness fields.
* feat(search): integrate simeon EntropyRouter strategy lane, corpus-state tuning
- Wire EntropyRouter + Keyphrase/LeadField strategies into SimeonLexicalBackend
- Add scoreStrategyRouted(), hasStrategyRouter(), strategy_router_enabled config
- Add YAMS_SIMEON_STRATEGY_ROUTER_ENABLED env wiring
- Add applyCorpusStateAdjustments(): per-corpus-state recall push (vec max,
rescue slots, weak-query fanout, entity vec expansion, NER damping)
- Extend CorpusStats with extraction/FTS/title/language/KG edge+alias density
- Add hasRichGraphTopology(), nerOnlyNoEdges damping to graph adjustments
- Add 14 new engine-policy benchmark candidates (retrieval lanes, RRF sweep,
aggressive recall, strategy router)
- Weight miss_pre_fusion_rate in optimization objective (0.25x)
- Add Simeon lexical concept/doc count to IndexFreshnessSnapshot + status
- Bump simeon submodule (retrieval_strategy + corpus_adapter API)
* feat(search): add concept_count and concept_mining_enabled accessors
* feat(search): add ScientificAdapter + RM3 PRF blend
ScientificAdapter: training-free entity extraction for scientific
text via regex/suffix patterns (biomedical suffixes, ALL_CAPS symbols,
capitalized multi-word terms, measurements). Replaces TextAdapter
in strategy router build, providing entity lists for KeyphraseStrategy.
RM3: add score_with_prf blend path to scoreStrategyRouted at 0.30
weight when rm3_enabled=true. New benchmark candidate
scientific_strategy_rm3.
Strategy router + ScientificAdapter: mrr=0.4514 ndcg=0.4789 recall=0.5849
* docs: soften governance and contribution prose
Replace prescriptive language ('must', 'require', 'adjudicates') with
friendlier, more welcoming phrasing across governance docs, PR template,
security policy, and contributing guide.
* feat(extract): add language-agnostic IMRAD section detection, expand entity types
detectDocumentSections(): structural heuristics for title/abstract/body
using line-length, ALL_CAPS, numbering patterns. No word matching.
Works across all languages and scripts.
deriveTitle(): add IMRAD detection step before first-line fallback.
isHighValueGraphType(): use ScientificAdapter suffix detection (40+
biomedical suffixes) instead of hardcoded 11 types.
* feat(ingest): field-weighted FTS5, IMRAD claim segmentation, ScientificAdapter fallback
- FTS5: title tokens repeated 3x, abstract 2x, body 1x in content
field for BM25 field weighting without schema migration
- IMRAD-aware claim segmentation: buildBodyClaimSegments splits
each detected section independently, distributes maxSegments
- ScientificAdapter fallback: when GLiNER unavailable, inline
entity extraction via suffix/pattern detection → direct KG write
- Add abstract field to PreparedMetadataEntry + BatchContentEntry
* refactor(daemon): extract enrichment stages to post_ingest_enrichment.cpp
Split PostIngestQueue.cpp (3145 lines) into:
- PostIngestQueue.cpp (~3084 lines): coordinator, lifecycle, queue
management, extraction pipeline, gradient limiters
- post_ingest_enrichment.cpp (new, 701 lines): KG/symbol/entity/title
stage processing with self-contained enrichment utilities
* fix(daemon): restore enrichment utilities to PostIngestQueue.cpp
Processing methods still in PostIngestQueue.cpp reference entity/NL
utilities (isUsefulNlEntity, buildBodyClaimSegments, etc.) that were
removed during decomposition. Restore them to anonymous namespace.
* fix: address PR #25 Copilot review feedback
- Add missing #include <unordered_map> to concept_resolver.cpp
- Batch entity writes per doc in SearchEngineManager (reduces write amp)
- Remove non-existent CorpusAdapter API claims from README
- Replace absolute paths with repo-relative in daemon-download-jobs.md
- Revert plugin-zyp default to false (noisy warnings)
- Fix grammar in extraction-repair-pipeline.md heading
- Add gitignore entries for .wrangler, .claude, .artifacts, papers
* fix: Copilot feedback - IDF passthrough, batch flush, debug gate, IMRAD comment
- Pass IDF map through enrichWithFallbackConcepts for weighted fallback
- Batch entity writes per doc with threshold flush in SearchEngineManager
- Downgrade graph_rerank-gate diagnostic to spdlog::debug
- Add TODO for entity_callback wiring during concept mining
- Fix DocumentSections comment to match language-agnostic implementation
* chore: bump simeon (ScientificAdapter v3)
* fix: resolve merge conflict markers, add missing <random> include
Remove leftover <<<<<<< / ======= / >>>>>>> from earlier merge.
Add #include <random> to ingestion_throughput_bench.cpp.
* chore: trigger CI after simeon submodule push
* feat: search work and moving to experimental
* WIP
* feat(search): diagnostic logging pipeline — graph build, concepts, reranker gate
- New [concepts] per-50-query log showing extracted query concepts, type
distribution, and used_gliner flag.
- New [PIQ-entity] per-1000-dispatch + per-document completion log with
entity type counts (e.g., 'protein=3, gene=1, chemical=2').
- New [graph_rerank-gate] one-shot log showing why enableGraphRerank
is zeroed: shows shortQueryBudgeted, corpusWarming, bypassWarmingGate,
lexicalReady, and post-ingest inflight.
- New [graph_rerank] per-50-query log with window size, matched
candidates, applied count, concept count, and elapsed time.
- retrieval_quality_bench.cpp: add YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE=1
so benchmark queries are not gated by freshness checks.
- resolve merge conflicts from experimental branch.
* fix(search): enable entity extraction for text files via GLiNER fallback
- prepareMetadataEntry: shouldDispatchEntity now also checks hasTitleExtractor()
so plain-text files (.txt, .md) get entity extraction even when no binary
entity provider supports their extension.
- processEntityExtractionStage: when no ExternalEntityProvider matches,
fall back to processTitleExtractionStage which runs GLiNER on text.
- refreshStageAvailability: entity stage is now considered capable when
hasTitleExtractor() is true, so the entity poller gets concurrency budget
even if entityProviders_ is empty.
- kCombinedEntityTypes: add biomedical entity types (protein, gene, cell,
disease, chemical, drug, pathway, biological_process, biomarker, anatomy,
organism) so GLiNER extracts high-value biomedical entities instead of
only generic NL types (person, organization, concept).
* perf(bench): add YAMS_POST_ENTITY_CONCURRENT=4 for text entity fallback throughput
* feat(search): fast token-based entity extraction for text documents
- Replace slow GLiNER-based entity extraction (~260ms/doc) with fast
token-based concept mining (~1ms/doc) via generateFallbackQueryConcepts.
Uses sub-phrase mining + IDF-weighted token salience — the same function
already used for query-time concept fallback.
- Store entities with full KG scaffolding: KGNode with entity type/label,
KGAlias for query-time resolution, DeferredEdge from entity→document,
and DeferredDocEntity linking entity to document for getDocEntitiesForDocument.
- Add biomedical entity types to kCombinedEntityTypes in title extraction
(protein, gene, cell, disease, chemical, drug, pathway, etc.) so GLiNER
can output high-value biomedical entities when used.
- Add include for query_expansion.h to access generateFallbackQueryConcepts.
- bench: add YAMS_POST_ENTITY_CONCURRENT=4 for entity poller throughput.
Benchmark: 22K+ entities extracted across 5183 docs, ~41% high-value
biomedical. MRR 0.4281 (beats 0.4273 baseline by +0.2%).
* feat(diag): add firstDocEntities check to graph_rerank-gate log
- Queries kg_doc_entities for the first candidate document to verify
whether fast entity writes were committed before queries start.
- Confirmed: firstDocEntities=0 — entities stuck in WriteCoordinator
async queue when benchmark queries begin.
* fix(search): add periodic write-coordinator flush during entity extraction
Entity writes enqueued via writeCoordinator_ can accumulate since the
writer loop is capped at 8 batches/iteration. For 5K+ documents this
creates a backlog that outlasts the post-ingest drain wait — entities
are not committed to kg_doc_entities before benchmark queries start.
Add a periodic flush (every 128 batches, 500ms timeout) in the entity
extraction batch processor to drain the WC queue incrementally.
Diagnostic confirms firstDocEntities=0 persists — a synchronous
committed flush or batched WC enqueue is needed.
* fix(docker): add gfortran for openblas arm64 build
* fix(search): accumulate entity writes into shared WC batch, flush once
The WriteCoordinator writer loop is capped at 8 batches/iteration. With
5K+ per-doc entity writes, entities outlast the benchmark's post-ingest
drain and firstDocEntities=0 when the graph scorer runs.
- processEntityExtractionStage: add optional DeferredKGBatch* parameter.
Text-fallback appends entity nodes/aliases/edges/doc-entities to the
shared batch instead of creating per-doc WC batches.
- processEntityExtractionBatch: creates one DeferredKGBatch, passes it to
all stage calls in the batch, enqueues to WC once at end.
- Single-shot WC flush: after all stages go idle (totalInFlight==0),
calls writeCoordinator_->flush(15s) to drain remaining entity writes
before benchmark queries start.
- DeferredKGBatch forward-declared in PostIngestQueue header.
* ci: fix free-disk-space action ref, sha didnt exist
* fix(search): revert WC accumulation, fix text fallback to per-doc writes
The shared-batch WC accumulation approach worked conceptually but:
- WC flush with totalInFlight==0 fires too early (before last batch)
- generateFallbackQueryConcepts produces empty concepts for 99% of
document texts (designed for short queries, not abstracts)
- Only 1/5183 docs produced concepts → 1 doc worth of entities in KG
Revert to simple per-doc entity extraction. The enable=/disable=
graph reranker gate is still controlled by shortQueryBudgeted=1
which zeroes the graph reranker on the first query (intent=Code).
firstDocEntities=0 persists — entities need a different creation
strategy for document text (phrase mining too conservative).
* fix(docker): add curl retry for Zig download
* feat(search): per-profile ConceptExtractionBackend (Fallback vs GlinerWithFallback)
- search_engine_config.h: new ConceptExtractionBackend enum with Fallback
(sub-phrase mining + IDF tokens, <1ms) and GlinerWithFallback (GLiNER
first, fall back on empty/timeout). Default: GlinerWithFallback.
- search_tuner.h: SCIENTIFIC profile sets Fallback (GLiNER returns empty
for 99% of short biomedical claims). Other profiles keep default.
Wired through TunedParams.applyTo().
- search_engine.cpp: gate GLiNER concept extraction on the config backend.
When Fallback, conceptFuture is never created and concepts come directly
from generateFallbackQueryConcepts. Added [concept-backend] diagnostic.
- search_engine_builder.cpp: builder log now includes concept_backend
in the override-state startup message.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC profile uses Fallback.
Note: builder correctly sets Fallback but query-time workingConfig still
shows GlinerWithFallback (int=1). This is a config-sync issue between
builder-time profile application and runtime config snapshot — the
tuner or a post-override config copy may reset the field. To be fixed.
* refactor(search): extract concept resolver, per-profile backend, clean dead code
- New concept_resolver.h/cpp: extract enrichWithFallbackConcepts from the
search engine monolith into a testable unit. Generates IDF-weighted
token concepts via generateFallbackQueryConcepts, deduplicates, and
appends to existing concepts.
- search_engine_config.h: new ConceptExtractionBackend enum (Fallback vs
GlinerWithFallback). SCIENTIFIC profile sets Fallback since GLiNER
extracts 0 concepts for 99% of short biomedical claims.
- search_tuner.h: wire conceptExtractionBackend through TunedParams,
applyTo, seedTunedParamsFromConfig (the one-line fix that resolved
the config-sync bug where profile Fallback was overridden at query
policy resolution time).
- search_engine.cpp: materializeConcepts lambda restructured to always
reach the fallback even when GLiNER future is invalid. Removed
~80 lines of diagnostic bloat ([concept-backend], [workingConfig-*],
per-50 [concepts]). GLiNER gate checks config backend.
- PostIngestQueue.cpp: removed dead fast-token entity extraction
(~150 lines of concept generation + KG scaffolding that produced
0 entities for 99% of documents). Reverted entity-stage gating
(entityCapable || hasTitleExtractor) and shouldDispatchEntity to
extension-only checks. Removed DeferredKGBatch forward decl.
- gliner_query_extractor: removed unused createGlinerExtractionFunc-
FromProvider fallback factory (~150 lines).
- ServiceManager.cpp: removed fallback call site for removed factory.
- builder log: removed concept_backend diagnostic line noise.
- search_tuner_catch2_test.cpp: verify SCIENTIFIC uses Fallback.
Net: ~400 lines removed, ~120 lines added (concept_resolver unit +
config wiring). GLiNER concept extraction savings: ~30-80ms/query
× 300 queries = 9-24s total for SCIENTIFIC profiles.
* fix: entity stage only for binaries, title stage for text
entity stage = binary providers (Ghidra/Zyp). title stage = GLiNER for all
text. No overlap — title already stores NL entities to KG.
* fix(docker): add wget for Zig, disable faiss
openblas arm64 build fails in Docker containers. YAMS_DISABLE_FAISS=1
env var skips faiss (and its openblas transitive dep). Zig download
now uses wget -c for resume on flaky ziglang.org connections.
* ci: add experimental branch to tests trigger
* ci: pin free-disk-space to v1.3.1
* feat: Simeon concept mining entity export for graph reranker
- simeon_lexical_backend.h: EntityCallback in Config. Called with
(docId, entityText, confidence) per concept match per doc.
- simeon_lexical_backend.cpp: after mine_concepts, iterate docs
via collect_doc_concepts + entity_callback.
- SearchEngineManager.cpp: enable concept_mining_enabled, wire
entity_callback batch-inserting DocEntity via KG store. Fix
kgStore capture-before-move race.
Note: simeon submodule has collect_doc_concepts in concept_mining.hpp
+ concept_mining.cpp (must commit to submodule separately).
* ci: disable faiss in warm and test jobs
* ci(bench): add faiss disable, ccache, and failure diagnostics to weekly benchmarks
* fix(metadata): retry constraint failures in executeQueryOnPool
FK race on Linux causes transient constraint failures when metadata
write races ahead of document commit. Previously treated as terminal.
Now retried with backoff like lock errors.
* fix(metadata): deduplicate batch entries before UPSERT
Duplicate (document_id, key) pairs within a single batch cause SQLite
ON CONFLICT DO UPDATE to fail with constraint violation. Dedup keeps
the last write per key.
* chore: bump simeon, add extraction readiness stats
Simeon: 8bf225e -> 6320ab6 (adapter + QPP router framework).
CorpusStats: add contentExtracted*, ftsIndexed*, titleCoverage,
languageCoverage, and KG topology richness fields.
* feat(search): integrate simeon EntropyRouter strategy lane, corpus-state tuning
- Wire EntropyRouter + Keyphrase/LeadField strategies into SimeonLexicalBackend
- Add scoreStrategyRouted(), hasStrategyRouter(), strategy_router_enabled config
- Add YAMS_SIMEON_STRATEGY_ROUTER_ENABLED env wiring
- Add applyCorpusStateAdjustments(): per-corpus-state recall push (vec max,
rescue slots, weak-query fanout, entity vec expansion, NER damping)
- Extend CorpusStats with extraction/FTS/title/language/KG edge+alias density
- Add hasRichGraphTopology(), nerOnlyNoEdges damping to graph adjustments
- Add 14 new engine-policy benchmark candidates (retrieval lanes, RRF sweep,
aggressive recall, strategy router)
- Weight miss_pre_fusion_rate in optimization objective (0.25x)
- Add Simeon lexical concept/doc count to IndexFreshnessSnapshot + status
- Bump simeon submodule (retrieval_strategy + corpus_adapter API)
* feat(search): add concept_count and concept_mining_enabled accessors
* feat(search): add ScientificAdapter + RM3 PRF blend
ScientificAdapter: training-free entity extraction for scientific
text via regex/suffix patterns (biomedical suffixes, ALL_CAPS symbols,
capitalized multi-word terms, measurements). Replaces TextAdapter
in strategy router build, providing entity lists for KeyphraseStrategy.
RM3: add score_with_prf blend path to scoreStrategyRouted at 0.30
weight when rm3_enabled=true. New benchmark candidate
scientific_strategy_rm3.
Strategy router + ScientificAdapter: mrr=0.4514 ndcg=0.4789 recall=0.5849
Topology ablation bench: read YAMS_BENCH_DAEMON_DATA_DIR so multiple
daemon spawns can share a stable data_dir. Lets the topology tuner's
persisted MAB state carry over between bench reps; without this, every
fresh DaemonHarness spawn cold-starts UCB1 and picks the alphabetically-
first arm. Pairs with the TopologyTuner persistence (already on HEAD).
Bench also gains per-surface vector-leg counters (vec_unique_doc_counts,
topology_weak_routing_applied, vector_tier_skip_reason, embedding_status,
vector_db_dim/query_embedding_dim) and a modeOf reducer for categorical
metrics — Phase F1-era richer attribution carried forward.
Docs: AGENTS.md / daemon_architecture / retrieval_precision_optimization /
PROMPT-eng-codex updates from the parallel sessions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
