Prune CI artifacts and stream metadata scans. Preserve relocated CLI
loader dependencies, reject incomparable path roots, and make deadline
and corruption fixtures test their stated contracts.
Keep unresolved Windows publication and hook failures diagnostic-only.
Signed-off-by: trvon <me@trevon.dev>
Apply the existing configured timeout scale to the integration executable. Observe periodic backfill attempts while retaining all publication assertions; no forced backfill or production scheduling change. Full TSan memory-sync suite passed.
Signed-off-by: trvon <me@trevon.dev>
Scope vector fixture overrides across every disable alias and verify restoration. Prune excluded repository directories before walking them. Exercise CLI serve delegation and distinguish outer spawn failure from missing child diagnostics using isolated stubs.
Signed-off-by: trvon <me@trevon.dev>
* test(ci): stabilize main matrix (#112)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): preview experimental promotion (#111)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): reconcile draft promotion safely (#113)
* ci(release): reconcile draft promotion
Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.
Signed-off-by: trvon <me@trevon.dev>
* ci: isolate hook Git environments
Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): reject invalid candidate evidence
Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve completion and retrieval contracts
Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence asynchronous enrichment completion
Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.
Signed-off-by: trvon <me@trevon.dev>
* fix: bound semantic graph output reservation
Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence standalone embedding repair completion
Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve enrichment failure reporting
Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test: align explicit hash prefix boundaries
Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.
Signed-off-by: trvon <me@trevon.dev>
* test: align UI hash validation boundary
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): one table for repair operation wire codes
The code published under repair_current_operation_code was defined in
three places: a name->code chain in RepairService.cpp, an enum->name
switch beside it, and a code->label switch in the CLI. The two daemon
tables agreed with each other but the CLI copy had already drifted in
wording, and nothing tied the three together.
metric_keys.h now carries the single code<->name table next to the
metric key it describes, with constexpr lookups in both directions.
RepairService's name->code chain delegates to it; the enum->name
switch stays because it is keyed by the private enum. The CLI copy
goes in the next commit with the status renderer.
Signed-off-by: trvon <me@trevon.dev>
* refactor(cli): extract the daemon status renderer
showStatus() in daemon_command.cpp was 1,762 lines: two RPCs, a
bootstrap-file fallback, and two ~400/1,100-line render bodies that
wrote straight to std::cout from inside the command class, reachable
only through a live daemon. The render bodies read nothing from the
command except the expected data directory, the explicit-socket flag,
and a memory-sync fetch they triggered mid-render.
The bodies now live in src/cli/commands/daemon_status_render.cpp as
free functions over StatusResponse plus a small DaemonStatusRenderContext
(configured data dir, explicit-socket flag, optional MemorySyncResponse)
writing to an ostream. The readiness/label helpers that only they used
(humanizeToken, classifyReadiness, isEphemeralDataDir,
daemonDataDirFromStatus, paintStatus, neutralText) move with them.
DaemonCommand keeps the RPCs and the bootstrap fallback; the memory-sync
fetch becomes fetchMemorySyncStatus() and runs just before rendering
instead of in the middle of it, so the section prints in the same place
with the same bounded 3 s timeout.
The move is verbatim: after the mechanical substitutions
(std::cout -> os, expectedDataDir() -> ctx.configuredDataDir, the
memory-sync call, the explicit-socket flag, and the repair label
function) the two bodies match the originals character for character
modulo whitespace. The CLI's private repair-operation switch is gone;
the label now comes from the shared table in metric_keys.h with
underscores rendered as spaces, which reproduces the previous wording
("stuck docs", "block refs", "path tree").
tests/unit/cli/daemon_status_render_catch2_test.cpp drives both
renderers and the memory-sync section from synthetic responses:
overview rows, blockers with init progress, last error, data-dir
mismatch (and its suppression under --socket), temporary data dir,
the active repair operation label, every detailed section, and the
started/not-started memory-sync gate.
daemon_command.cpp: 4,082 -> 2,341 lines; daemon_status_render.cpp:
1,642 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(app): one home for hash predicates, path aliasing, and metadata qualifiers
Three small helpers had grown parallel copies with quietly different
rules:
- "does this look like a hash" existed four times: search_service
(8-64 hex), document_service (6-63), cat_command (6-64), plus a
never-defined utils::looksLikeHash declaration in services.hpp. The
8 vs 6 threshold is deliberate (D3): a bare query token must reach
8 hex characters before it stops being text, while an explicit hash
argument may be 6. yams/common/hash_predicates.h now names the two
predicates apart (looksLikeHashQueryToken, looksLikePartialHashArgument)
over one isHexDigits; document_service's 6..63 bound was deliberate
for the name resolver (a full 64-hex name must not resolve as a prefix
and delete by hash), so its two resolver sites now guard size() != 64
explicitly, as the partial-hash resolution site already did.
- macOS /private aliasing had three incompatible policies: search
appended the alias of /var only, grep canonicalized /var and /tmp
to the /private spelling, and the delete resolver displayed
/private/var as /var. yams/common/fs_utils.h now carries one alias
table ({/var,/tmp} <-> /private/...) with three lexical policies over
it (otherSpelling, canonicalSpelling, displaySpelling) and the
platform-gated wrappers callers use. Declared behavior change, macOS
only: search patterns under /tmp now also match their /private/tmp
spelling, and the resolver displays /private/tmp/... as /tmp/....
- Inline key=value metadata filters were parsed twice: SearchService's
extractStructuredMetadataQuery (exactly one '=', no quoting/grouping
characters, deduplicated by key) and SearchEngine's
parseMetadataFiltersFromQuery (any '=' token, duplicates kept). The
stricter, tested parser moves into yams/search/query_qualifiers.hpp
and both call it. Declared behavior change for the hybrid metadata
component: tokens like name="x" or a=b=c stay in the query instead
of becoming filters, and repeated keys collapse to the last value.
Tests: hash_predicates_catch2_test.cpp pins both thresholds and the
64-character bound; path_alias_catch2_test.cpp pins the alias table,
the three policies, and the platform gating; query_qualifiers_catch2_test.cpp
gains the structured-metadata cases (split, dedupe, malformed tokens,
filters-only). search_service, document_service, grep_service,
service_utilities, and resolve_submodule pass unchanged. Portability
allowlist anchors reflowed for the four files whose lines moved.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): flush status before optional collection
Render useful primary status before waiting for memory-sync details.
Keep both brief and detailed paths behind the tested ordering seam.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): give the CLI fixture the shared corpus scope it needs
94e948c6 made createMemorySyncService fail closed unless
memory_sync.corpus_scope is "shared" and updated the service-manager
test, but not the p2p CLI fixture, whose eight cases have failed at
construction ever since ("replication requires
memory_sync.corpus_scope=shared"). The fixture now sets the scope; the
cases run again and pass.
Signed-off-by: trvon <me@trevon.dev>
* refactor(memory_sync): move MemorySyncLoop bodies into a translation unit
include/yams/memory_sync/memory_sync.h was a 3,527-line header-only
class: 120 inline method definitions, 2,900 lines of implementation,
pulled into every translation unit that includes daemon.h or the p2p
protocol header. The memory-sync subsystem had no source directory at
all.
MemorySyncLoop's bodies now live in src/memory_sync/memory_sync.cpp,
compiled once into a new yams_memory_sync static library that follows
the yams_wal pattern. The header keeps the declarations, the nested
types, the constructor, one-line accessors, and the YAMS_TESTING hooks
(those must stay inline because the library is not built with the
testing define). The daemon library and its declared dependency link
the new library, as do the five memory-sync test binaries.
Zero semantic change: the move is mechanical (a scripted split that
copies each body verbatim, drops default arguments and static from the
definitions, and qualifies nested return types). All seven memory-sync
test binaries and the service-manager suite pass unchanged.
memory_sync.h: 3,527 -> 569 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): move TuneAdvisor method bodies out of the header
TuneAdvisor.h was 3,345 lines, almost all of it the bodies of ~300
static accessors and setters, included by 60 translation units. Each
of them recompiled the whole policy surface on every touch, and any
change to one knob rebuilt most of the daemon.
The bodies now live in src/daemon/components/TuneAdvisor.cpp (which
previously held only the system-memory probe). The header keeps the
declarations, the static inline override slots (unchanged, so the
ODR story for process-wide tuning state is exactly what it was), the
YAMS_TESTING hooks, and the two methods whose bodies are
preprocessor-conditional on YAMS_TESTING; those must stay inline
because the library is not built with the testing define and moving
them would silently switch tests to the production branch.
Mechanical move by the same scripted split used for memory_sync.h:
bodies copied verbatim, static and default arguments dropped from the
definitions, nested return types qualified. The one template
(readConfiguredOverridesSnapshot) stays in the header because three
other translation units instantiate it.
The header is included well outside src/daemon (vector, repair, app
services, MCP, the CLI doctor, the ONNX plugin), and those targets do
not link yams_daemon. They used to get the bodies for free from the
header, so the move needs a home they can all link: the new
yams_tune_advisor static library (src/daemon/tune_advisor/meson.build)
builds TuneAdvisor.cpp alone, and yams_daemon, yams_vector,
yams_repair, yams_app_services, yams_mcp, yams_cli, the ONNX plugin,
and the standalone snapshot test depend on it.
Executables compiled with -DYAMS_TESTING link a second build of the
same file, yams_tune_advisor_testing, ahead of the production archive.
Under that define compatibilityEnvironment() reads the process
environment live instead of the startup snapshot, which every
env-driven tuning test relies on; before the split each such TU
instantiated the inline bodies itself with the define. Meson emits
dependencies in list order and the first archive defining a symbol
wins, so the order is load-bearing: test_deps, the benchmark
dependency lists, and the mobile ABI test all put the testing archive
first, and tests/meson.build says so. The testing archive is built
under the same predicate that enters tests/ (a plain debug configure
included), so `meson setup --buildtype=debug` keeps working. The
standalone snapshot test keeps the production build on purpose.
No behavior change; a full debug compile is clean and the tuning
suites pass unchanged.
TuneAdvisor.h: 3,345 -> 1,170 lines.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): preview-based rerank text and compiled path globs
Cross-rerank built each window entry's text with a full getContent()
read, then kept only the first rerankSnippetMaxChars characters. The
window now hydrates with one batchGetContentPreview() call bounded to
that many characters; buildCrossRerankText takes the preview instead of
the repository. batchGetContentPreview is virtual so a test can count
it: three window results cost one preview batch and zero full reads.
Path filtering compiled a std::regex from the glob on every call, and
the call sits inside per-document loops in pathSearch, the hash-prefix
search, and metadata filtering. GlobMatcher (src/app/services) holds
the compiled form; wildcardMatch() reuses it through a small
per-thread cache. The glob semantics are unchanged and now pinned by a
test: '*' stays within one path segment, '**' crosses, '?' is one
character, regex metacharacters are literal. yams::common::wildcard_match
was considered and rejected because its '*' crosses separators.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): parse edge anchor properties once, lower the path query once
Graph expansion parsed every visited edge's JSON properties with
nlohmann::json::parse to read at most two strings, region and scope,
in three places (relation weighting, seed anchoring, and neighbour
anchoring). Properties strings repeat heavily across edges and most
carry no anchor at all. edgeAnchorHints() now returns immediately when
the text cannot contain either key and otherwise parses each distinct
string once per thread through a small bounded cache. A process-wide
parse counter lets the test pin it: one hundred lookups of the same
anchored string cost one parse, and un-anchored or malformed strings
cost none.
computePathMatchScore lowercased and trimmed the query for every
candidate document; pathSearch now does that once and passes the
lowered query down. The existing path-search tests cover the scoring.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
edgeAnchorHints returns its two short strings by value: the per-thread memo
clears itself at 1024 entries, so a reference into it could dangle across
calls even though today's three callers consume the result immediately.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): preserve hydration failure semantics
Stop reranking before scorer calls or score changes when previews fail.
Commit anchor hints atomically and recognize escaped JSON keys.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): load tag filters once per request, not once per document
pathSearch and searchByHashPrefix filtered candidates through
metadataHasTags(), which for every document called getAllMetadata()
(four retries with backoff on failure), concatenated every metadata
key and value into a string, and logged that string at info level.
A path search over a few thousand documents was a few thousand
metadata round trips and a few thousand info lines.
Both paths now load tags for the whole candidate set with one
batchGetDocumentTags() call (the API GraphComponent already used) and
filter with a pure matcher. The legacy fallback that also accepted a
metadata row with key "tag" is gone: nothing writes that form and the
repository's own tag APIs never honoured it, so search and repository
now agree on what a tag is.
A counting repository subclass pins the shape: one batch call and zero
getAllMetadata calls for both the path and the hash-prefix routes.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
Correction to the claim above that nothing writes the legacy `key = "tag"`
form: the repair downloads pass did (tag values "downloaded", "host:<h>",
"scheme:<s>"), and the search fallback was its only reader. That writer now
uses the same `tag:<name>` keys as every other tag writer, so `--tags
downloaded` keeps matching documents the repair pass tagged. And
batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it
an unbounded candidate set; an oversized list used to fail the statement
and, with this change, would have emptied the result instead of degrading
per document.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hydrate topology routing candidates in batches
The routing session called getDocumentByHash once per hash in three
places on the product routing path: ranked-candidate admission, cluster
expansion, and the graph-neighbor diagnostic trace (memoized, but still
one statement per distinct hash across four stages). Each now issues
one batchGetDocumentsByHash for its whole set, the same API the
topology input extractor already used, and the docLookupMicros timing
covers the batch. Cluster expansion applies the maxDocs cut before
hydrating so no document is fetched only to be discarded.
A session-level test with one seed and eight graph neighbours under a
counting repository fails on the previous code (17 single lookups, no
batch) and passes now (0 single lookups). The full topology suite is
unchanged.
portability_allowlist.txt: one topology test anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): bound hydration and preserve errors
Retain legacy tags and honor per-connection SQLite bind limits.
Keep hash chunks in one read snapshot and release it on failure.
Failed lookups are not missing documents or successful empty searches.
Reject partial route admission and preserve tagged path errors.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): count uncached prepares in Database::CacheStats
Database::prepare() compiles a statement every call; prepareCached()
reuses one. The cache stats reported hits and misses but nothing about
the bypass, so a hot path that re-prepared fixed SQL on every row was
invisible. CacheStats gains uncachedPrepares, incremented under the
existing cache mutex and carried through move construction and
assignment.
This is the measurement primitive for the following hot-path changes:
a test can warm a path, then assert that N more calls add zero uncached
prepares.
portability_allowlist.txt: one database.cpp anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(metadata): cache KG statements and batch the scorer's neighbor fetch
Measured with the new uncachedPrepares counter on a single-connection
pool. Before: 64 warmed neighbors() calls plus 32 document lookups cost
306 uncached statement compilations, and one SimpleKGScorer::score()
over eight query terms cost 620. After: 0 and 1.
Three things were compiling SQL per call on the search path:
- The sqlite KG store used Database::prepare() for fixed SQL in
neighbors, getDocumentIdByHash/ByPath, ensureBlobNode,
ensureDocumentNode, ensurePathNode (both lookups), selectKgNodeIdByKey,
getNodeById, resolveAliasExact/Fuzzy, getEdgesFrom/To,
getDocEntitiesForDocument, and searchNodesByLabel. All now use
prepareCached (85 -> 70 raw prepares in the file; the rest are
dynamic IN-list or maintenance queries).
- ConnectionPool::isConnectionValid ran prepare("SELECT 1") on every
acquire and again on every release, so each pooled call paid two
compilations before doing any work. The probe is now cached.
- Database::tableExists compiled its sqlite_master probe every time;
the fuzzy alias resolver calls it per query token.
SimpleKGScorer builds its 1-hop neighbor union with one
getEdgesFromBatch call instead of neighbors() per query node.
kMaxCacheSize rises from 64 to 128: 90 cached call sites now share a
connection when KG and repository work interleave, and the cache
evicts first-found, not least-recently-used.
Tests pin both numbers: the store test asserts zero uncached prepares
across 96 warmed lookups; the scorer test asserts fewer than eight on
a second run (previously one per query node from neighbors() alone).
portability_allowlist.txt: two anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(metadata): bound batched edge query parameters
Respect connection-specific SQLite limits and deduplicate chunk inputs.
Preserve directional ranking and a single read snapshot across chunks;
roll back the savepoint on query failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays
LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.
Removed (CANDIDATE rows of the report, verbatim):
YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
YAMS_SEARCH_GRAPH_ENABLE_PATHS
YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
YAMS_SEARCH_GRAPH_HOP_DECAY
YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
YAMS_SEARCH_GRAPH_MAX_HOPS
YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
YAMS_SEARCH_GRAPH_MAX_PATHS
YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
YAMS_SEARCH_GRAPH_TEXT_WEIGHT
YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
YAMS_SEARCH_RERANK_TOP_K
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
YAMS_SEARCH_TIERED_MIN_CANDIDATES
YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE
Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.
A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.
The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): drop unread embeddings.* keys and the no-op toggles
yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.
- embeddings enable and disable are removed. enable duplicated
embeddings model <name> once its dead keys are gone; a kept no-op
would be a shim. status no longer prints an auto-generation line and
shows the runtime batch size when set. tune presets write
embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
[embeddings] batch_size silently when the runtime key is absent, so
configs in the wild get their value applied without a file rewrite (the
migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.
Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.
Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.
Signed-off-by: trvon <me@trevon.dev>
* feat(config): add [tuning] keys for the setters only env could reach
Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.
Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.
A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.
The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.
Signed-off-by: trvon <me@trevon.dev>
* fix(config): report 0 for [tuning] integer keys instead of ignoring it
For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.
Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.
Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): one embedding-runtime resolver, one tuning baseline
Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.
yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.
Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): stop writing the unread search.reranker_model_path
yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.
Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.
portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.
The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).
Signed-off-by: trvon <me@trevon.dev>
* fix(config): validate tuning ranges and pairs
Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.
Cover all 27 integer keys, inherited limits, and reservation reset.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* chore(repo): land the retrieval contract and drop root cruft
Lands work that had been sitting uncommitted in the working tree:
- docs/architecture/retrieval_contracts.md plus its executable form,
tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py,
now registered in meson as retrieval_contract_unit (suite: policy).
.gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore.
- AGENTS.md: the Development Memory section, and the product default
recorded as hybrid_assist + shadow with topology-off as the control,
which is what search_topology_stage already does.
docs/benchmarks/index.md gains the matching search_shadow_cost plan.
- .gitleaks.toml, and a Secret scan workflow that actually runs it on
push and pull requests (nothing in .github invoked gitleaks before).
Deletes three root files with zero references anywhere in the tree:
bench_profile.json (ad-hoc profiling leftover), CppProperties.json
(Visual Studio config; the repo standardises on Meson), and spdlog.pc
(a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would
mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan).
docs/plans/ (two p2p designs already shipped in 5a53cdfa and d012d95b)
was ingested into YAMS with task=desloppification source=plan, hydrated
via yams cat and byte-compared before removal, per the new AGENTS.md
rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a
plans directory.
Also reflows two stale line anchors in raw_environment_allowlist.txt
for tools/yams-cli/main.cpp (shifted by 11c39157) so the policy suite
passes again; no new environment writes.
Not included: the external/yams-repo submodule pointer in the working
tree was a stale local clone two commits behind, not an intended bump.
Signed-off-by: trvon <me@trevon.dev>
* test(policy): make the configuration-authority ratchet bidirectional
check_configuration_authority.py only reported keys and TOML readers
present in code but missing from the allowlists, so the reviewed
YAMS_* surface could grow but never shrink: production_environment_keys.txt
even said removals "may remain grandfathered". An allowlist that only
accretes stops describing the code.
The policy now also fails on allowlist entries with no remaining
occurrence in src/, include/, or tools/, naming the lines to delete.
Key removals in later commits therefore have to update the allowlist in
the same change, and the count in the success banner becomes a real
measure of the environment surface (432 today).
stale_entries() exposes the two stale sets for tooling; three unit
tests cover stale key, stale reader, and the returned names.
Signed-off-by: trvon <me@trevon.dev>
* tools(policy): report where every configuration key is referenced
Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in
production_environment_keys.txt and each dotted TOML key literally
resolved in ConfigResolver.cpp, count whole-token references per tree
bucket (src, test, bench, docs, plugin; build output and the allowlists
themselves excluded) and assign a verdict: KEEP when anything outside
product code exercises or documents the key, CANDIDATE when only
product code mentions it, NOT_IN_SRC when the allowlist is stale.
The report is evidence, not a decision. From here on a key deletion
must paste its CANDIDATE rows into the change description; the first
audit pass hand-listed keys that turned out to have bench and test
consumers, which is exactly the mistake a mechanical inventory avoids.
Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys
(103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit.
The report counts only files git tracks (git ls-files) when the root is a
work tree, so an ignored local bench harness cannot turn a CANDIDATE into a
KEEP on one machine and not another; outside a work tree it falls back to
scanning the directory, which is what the synthetic-tree test exercises.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(embedding): complete embeddings by derivation token
cbc4959c added derivation tokens (begin/complete with a per-attempt
generation, invalidated by document_content triggers) but nothing
produced them: beginDocumentEmbeddingDerivation had no callers and the
completion op's derivations field was never written, while every ingest
still paid for the triggers and the per-hash NOT EXISTS guard.
processEmbedJob now mints one token per document it will embed, before
any input snapshot is read, and completes by token instead of by hash.
A document whose content changes while it is being embedded keeps
has_embedding = 0 and is re-derived later, instead of publishing vectors
for text the corpus no longer holds. Repair jobs travel the same
channel, so they get the same guarantee. The recipe binds the
preparation policy, embedding space, provider version, and dimension
through the embeddingDerivationRecipe helper that cbc4959c also left
unused.
Repository: batchBeginDocumentEmbeddingDerivations and
batchCompleteDocumentEmbeddingDerivations do a whole job in one
transaction each (the single-token completion loop in WriteCoordinator
opened one transaction per document). WriteCoordinator uses the batch
form and takes its stats lock once.
A small EmbeddingDerivationLedger maps job hashes to their tokens so
completion is issued for exactly the documents that produced vectors.
Behavior note: a re-embed (skipExisting = false) of an already-embedded
document resets its readiness at mint time; if that job then fails, the
document reads as not embedded until the next attempt. That is the
attempt semantics the token table was built for.
Not included: the legacy hash completion path is now unproduced and is
removed in a follow-up so the coalescer change is reviewed on its own.
tests/scripts/portability_allowlist.txt: 13 EmbeddingService.cpp
line anchors reflowed; no new environment reads.
The startup reconcile (ServiceManager, "Reconciling document_embeddings_status
from vector rows") used to mark every document with vector rows as embedded,
which would have undone the stale-token guarantee on the next daemon start: a
document whose content changed under an in-flight embed has vector rows for the
old text. reconcileDocumentEmbeddingStatusByHashes now leaves a document with an
open derivation (document_embedding_derivations.completed = 0) at has_embedding =
0, so it is re-derived rather than published; the reconciled-count query applies
the same predicate. Test: an open derivation stays unembedded across a reconcile
and flips once its token completes.
Signed-off-by: trvon <me@trevon.dev>
* fix(embedding): require current recipe to skip
Re-derive legacy or mismatched recipes rather than trusting readiness
alone. Treat derivation lookup errors as failed gathering.
Cover service-level recipe changes and legacy readiness, and characterize
atomic batch completion rollback and retry.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Only prune a nonempty segment after every byte is proven zero. A zero header alone is not evidence that recovery data is absent. Scan with bounded memory and preserve the file on read failure.
Signed-off-by: trvon <me@trevon.dev>
WALManager::Config defaulted walDirectory to "./wal", so any manager built
from a default config wrote segments into the process working directory.
The repo root had accumulated 43,577 such files (42,499 zero-byte, 1,078
zero-filled 1 MiB mmap preallocations from killed writers; 1.1 GB, no
entries). The daemon already passes <dataDir>/wal explicitly.
- walDirectory has no default; initialize() returns InvalidArgument when
it is empty. The default constructor is removed (it had no callers).
- initialize() still derives the sequence from every segment name, then
deletes plain segments that never received an entry: zero bytes, or a
zero magic word at offset 0 (a preallocation left by an unclean exit).
Compressed segments and anything with real bytes are untouched.
Tests: unconfigured directory is rejected; 50 empty + 1 preallocated
segments are pruned, a non-empty one survives, and the sequence still
advances past the pruned names.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): rebuild stale memberships
Incremental reuse can retain deleted members outside the dirty region.
Rebuild coherent artifacts from current metadata when prior members vanish;
retain final validation, pending dirty work, and unready-corpus deferral.
Cover deleted seeds, partial clusters, empty corpora, dry runs, and late
deletion without changing the live corpus or claiming a speedup.
Signed-off-by: trvon <me@trevon.dev>
* Add cstdint and optional includes to test file
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: trvon <me@trevon.dev>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
dispatchNonEmbeddingStages hands one shared_ptr<vector<byte>> to the KG,
symbol, and entity channels. The entity stage moved out of that buffer, so
whichever consumer ran after it saw zero bytes and silently extracted
nothing. The KG batch consumer already read through a pointer; the entity
stage now does the same, keeping an owned fallback only for the
store-reload path.
Also delete processSymbolExtractionStage, the legacy single-item symbol
handler: it had no callers, and it moved out of the shared buffer as well
before copying the bytes a second time into contentUtf8.
Adds a YAMS_TESTING hook and a Catch2 case that runs two entity jobs over
one shared buffer and asserts the buffer survives.
* refactor(search): extract auxiliary vector merge
Signed-off-by: trvon <me@trevon.dev>
* fix(search): preserve auxiliary tie behavior
Keep the extracted merge comparator identical to the original score-only sort
and avoid prescribing equal-score order in characterization tests. Qualify the
search detail namespace so the topology test compiles unambiguously.
Signed-off-by: trvon <me@trevon.dev>
* test(search): avoid auxiliary score tie
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
* refactor(vector): add injectable executor for the embedding daemon bridge
await_with_timeout spawned coroutines unconditionally on the daemon
global IO context, coupling the vector-layer embedding generator to
daemon globals. EmbeddingConfig gains an optional
boost::asio::any_io_executor executorOverride; the bridge uses it when
present and keeps the global executor as the default — identical
behavior for all existing callers (embedding_service, model providers,
CLI all construct without an override).
Lead 1 plan step (b) of the embedding_generator vector->daemon
decomposition (Brick audit follow-up).
Validation: full ninja build green; meson test yams:service_manager,
vector_submodule, yams:embedding_lifecycle, yams:embedding_recovery —
all green.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): split DaemonBackend into own TU
Keep daemon client/IPC headers out of the vector-layer generator TU;
EmbeddingGenerator builds the backend via createDaemonBackend factory
on the existing IEmbeddingBackend seam. Behavior-preserving.
Brick audit lead 1: 5 file-level cycles hubbed on
embedding_generator.cpp.
Signed-off-by: trvon <me@trevon.dev>
* revert(vector): keep bridge executor out of public EmbeddingConfig
Pre-push gate caught it: the config layer includes the vector header
transitively (ConfigResolver.h -> document_chunker.h ->
embedding_generator.h) and has no boost asio include path, so
<boost/asio/any_io_executor.hpp> in the public header breaks the
config build.
Executor selection stays in embedding_daemon_backend.cpp (daemon-facing
TU by design); per-instance injection needs the config->vector include
direction fixed first — recorded as follow-up.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): trust resolved backend policy
Remove EmbeddingGenerator's ambient ConfigResolver read. Callers now own
backend-policy resolution; EmbeddingService marks its already-resolved
runtime policy unconditionally. Keep TuneAdvisor only for the process-wide
concurrency gate, where instance-local policy would not coordinate admission.
Tests pin typed backend authority against conflicting environment state.
Signed-off-by: trvon <me@trevon.dev>
* chore(vector): remove stale config include
The generator no longer reads config helpers after backend policy moved to
callers, so drop the misleading DaemonBackend comment and unused include.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
