Codifies the rule after the 2026-10 incident where p2p and topology work
pinned the CPU of a memory-constrained host. Lists the components that do
not consult the governor yet.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
Squash-merged promotion (#203) and back-merge (#205) PRs dropped main's
commits from experimental, so Release Candidate rejected the next
candidate. Record which PR kinds need a merge commit, the back-merge
check, the BEGIN_COMMIT_OVERRIDE hazard behind the 9,908-line notes, and
Release-As for pinning a version.
Signed-off-by: trvon <me@trevon.dev>
A stacked PR (#187) merged into its parent branch after the parent (#186)
had already merged into experimental, so its commits missed experimental
and the promotion PR until #189 carried them over. Record the merge order,
a git cherry check for stranded commits, and the no-squash rule for
submodule PRs.
README, AGENTS.md, the yams skill guide and the engineering prompt
still promised symbol extraction, calls/defined_in edges, and graph
--impact for callers. Point callers and blast radius at yams grep and
describe the relations the graph still holds.
* chore(repo): land the retrieval contract and drop root cruft
Lands work that had been sitting uncommitted in the working tree:
- docs/architecture/retrieval_contracts.md plus its executable form,
tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py,
now registered in meson as retrieval_contract_unit (suite: policy).
.gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore.
- AGENTS.md: the Development Memory section, and the product default
recorded as hybrid_assist + shadow with topology-off as the control,
which is what search_topology_stage already does.
docs/benchmarks/index.md gains the matching search_shadow_cost plan.
- .gitleaks.toml, and a Secret scan workflow that actually runs it on
push and pull requests (nothing in .github invoked gitleaks before).
Deletes three root files with zero references anywhere in the tree:
bench_profile.json (ad-hoc profiling leftover), CppProperties.json
(Visual Studio config; the repo standardises on Meson), and spdlog.pc
(a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would
mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan).
docs/plans/ (two p2p designs already shipped in 5a53cdfa and d012d95b)
was ingested into YAMS with task=desloppification source=plan, hydrated
via yams cat and byte-compared before removal, per the new AGENTS.md
rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a
plans directory.
Also reflows two stale line anchors in raw_environment_allowlist.txt
for tools/yams-cli/main.cpp (shifted by 11c39157) so the policy suite
passes again; no new environment writes.
Not included: the external/yams-repo submodule pointer in the working
tree was a stale local clone two commits behind, not an intended bump.
Signed-off-by: trvon <me@trevon.dev>
* test(policy): make the configuration-authority ratchet bidirectional
check_configuration_authority.py only reported keys and TOML readers
present in code but missing from the allowlists, so the reviewed
YAMS_* surface could grow but never shrink: production_environment_keys.txt
even said removals "may remain grandfathered". An allowlist that only
accretes stops describing the code.
The policy now also fails on allowlist entries with no remaining
occurrence in src/, include/, or tools/, naming the lines to delete.
Key removals in later commits therefore have to update the allowlist in
the same change, and the count in the success banner becomes a real
measure of the environment surface (432 today).
stale_entries() exposes the two stale sets for tooling; three unit
tests cover stale key, stale reader, and the returned names.
Signed-off-by: trvon <me@trevon.dev>
* tools(policy): report where every configuration key is referenced
Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in
production_environment_keys.txt and each dotted TOML key literally
resolved in ConfigResolver.cpp, count whole-token references per tree
bucket (src, test, bench, docs, plugin; build output and the allowlists
themselves excluded) and assign a verdict: KEEP when anything outside
product code exercises or documents the key, CANDIDATE when only
product code mentions it, NOT_IN_SRC when the allowlist is stale.
The report is evidence, not a decision. From here on a key deletion
must paste its CANDIDATE rows into the change description; the first
audit pass hand-listed keys that turned out to have bench and test
consumers, which is exactly the mistake a mechanical inventory avoids.
Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys
(103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit.
The report counts only files git tracks (git ls-files) when the root is a
work tree, so an ignored local bench harness cannot turn a CANDIDATE into a
KEEP on one machine and not another; outside a work tree it falls back to
scanning the directory, which is what the synthetic-tree test exercises.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Stop shipping the experiment harness and compatibility wrappers while preserving them as gitignored local tooling. Remove stale Meson registrations and mark recorded benchmark numbers as historical.
* fix(graph): handle qualified C++ symbols
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector)!: remove dead transition APIs
Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.
BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): harden PQ snapshot lifecycle
Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): expose route coordinates
Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.
Signed-off-by: trvon <me@trevon.dev>
* feat(xplan): add vector decision gates
Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): formalize coordinate routing
Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): serialize SymSpell readers
SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make mutations atomic
Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.
* fix(search)!: surface SymSpell write errors
Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.
BREAKING CHANGE: SymSpellSearch mutations now return Result values.
* feat(topology): formalize candidate expansion
* feat(vector): add coordinate-aware embeddings
* feat(topology): preserve protected relation cover
* feat(search): add graph candidate rescue
* feat(daemon): expose retrieval hydration telemetry
* feat(xplan): add rescue and hydration gates
* refactor(content): dedupe media handlers
Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.
* feat(lifecycle): prune stale corpus data
Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.
* feat(search): add topology candidate rescue
Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.
* feat(search): expose evidence paths
* docs: define mobile host contract
* formal: model bounded candidate rescue
* fix(prune): inspect extensionless signatures
* fix(prune): tolerate stale extensions
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(release): repair package repo publish
Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.
* fix(daemon): stop shutdown log storm
Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.
* refactor(cli): delegate serve to MCP binary
Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.
* refactor(cli): move daemon path helpers
Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.
* refactor(daemon): split embedded client
* perf(search): add lean-guided benchmark gates
Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.
Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.
* refactor(config): move daemon policies
* refactor(daemon): extract plugin runtime
* refactor(daemon): untangle service coordinators
Untangle CLI/MCP from daemon service coordinators and the daemon static lib.
- MetadataRepository uses a daemon-neutral tree-diff callback. The
GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
static lib. Drop ResourceGovernor coupling inside the writer loop; batch
sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
its own enqueuePostIngest; app services no longer call ServiceManager
directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
app services link cleanly without daemon.
Validated:
meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
yams-cli --version, yams-mcp-server init, yams serve --help smoke tests
Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.
* fix(cli): keep stdin add local fallback
No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.
Validated:
meson compile -C build/debug -j4 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
meson compile -C build/release -j4 yams-cli yams-mcp-server
meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output
* feat(bench): add xplan multi-arm experiment harness
Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.
* feat(formal): expand topology/search Lean contracts
Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.
* feat(search,topology): construction purity + graph expansion path
Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.
* feat(topology): add KMeans engine and SGC identity contracts
Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.
* feat(daemon): topology readiness gates and selectable reranker
Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.
* chore(search): drop meta_path tuner knobs
Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].
* feat(bench): add topology_purity_validate plan
Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.
* chore: format add_command and config include order
Mechanical clang-format / include reorder only; no behavior change.
* feat(bench): decision-grade xplan loop for search engine
Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.
* feat(graph): denser product semantic_neighbor topK
Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.
* feat(search): extract topology assist and cross-rerank seams
Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.
* feat(bench): product-path search engine ablation plans
Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.
* fix(ci): repair pre-push fallbacks
* feat(bench): lock-free mid-run builds + multi-corpus gates
xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.
* fix(daemon): keep SearchEngine alive via shared_ptr cache
getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.
Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.
* feat(search): route vectors through cached topology
Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.
Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.
* feat(bench): consolidate search decision plans
Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.
Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.
* feat(search): refine topology vector routing
* fix(search): propagate daemon readiness
* feat(bench): measure generalized memory health
* fix(cli): honor in-process fallbacks
* ci: harden smolvm debug builds
* perf(search): route topology through filtered ANN
* feat(bench): measure read-write pressure
* perf(metadata): index symbol lookups on read pool
* fix(daemon): isolate startup reads and maintenance
* feat(bench): replay isolated generalized corpora
* feat(search): add evidence routing pipeline
Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.
* refactor(search): unify topology evidence path
Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.
* refactor(bench): prune superseded search lanes
Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.
* docs(search): retire stale experiment notes
Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.
* refactor(search): prune dead search surfaces
Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.
* fix(cli): clear build warnings
* fix(xplan): reject stale run identity
* refactor(search)!: remove legacy query stack
Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.
BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.
* refactor(search): type value stage traces
Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.
* refactor(search): isolate benchmark env overlays
Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.
* refactor(search): unify topology routing options
Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.
* fix(search): unify tuner config ownership
Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.
* fix(search): snapshot tuner state per query
Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.
* fix(search): enforce topology routing contract
Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.
* fix(daemon): let SQLite reclaim its file
Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.
* fix(metadata): tolerate missing symbol FTS
Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.
* build(fuzz): drop removed query parser target
The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.
* fix(bench): include writes in drain oracle
A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.
* fix(daemon): vacuum padded WAL databases safely
* perf(daemon): coalesce metadata writes
* refactor(metadata): split repository ops
* fix(metadata): harden repository invariants
* feat: test coverage and continued refactor work in metadata code
* fix(graph): handle path-like explore queries
- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH
* fix(daemon): quiesce shutdown background work
- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain
* fix(daemon): cancel repair work on shutdown
Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.
* perf(metadata): reduce sqlite lock contention
* perf(metadata): flatten batch content retries
* fix(daemon): align shutdown wait budgets
A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.
* perf(metadata): bound feedback write waits
* perf(metadata): prep batch content before tx
* fix(daemon): honor no-embeddings tasks
Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.
* perf(daemon): batch fallback metadata writes
* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements
* perf(metadata): flatten insert retries
* fix(metadata): clear CI portability blockers
Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.
* perf(metadata): batch writes and steady bench
* fix(daemon): skip inactive post-ingest pollers
* perf(metadata): tighten tag and path caches
* perf(metadata): cache small batch upserts
* fix(daemon): harden embedded host teardown
* feat(docs): updating docs policies and language
* perf(metadata): tune doc update retries
* perf(metadata): bound download retries
* perf(metadata): bound download metadata waits
* bench(metadata): scope post-ingest content path
* fix(daemon): export embedded test hooks
* fix(daemon): harden test helper linkage
* fix(build): add missing standard includes
Topology ablation bench: read YAMS_BENCH_DAEMON_DATA_DIR so multiple
daemon spawns can share a stable data_dir. Lets the topology tuner's
persisted MAB state carry over between bench reps; without this, every
fresh DaemonHarness spawn cold-starts UCB1 and picks the alphabetically-
first arm. Pairs with the TopologyTuner persistence (already on HEAD).
Bench also gains per-surface vector-leg counters (vec_unique_doc_counts,
topology_weak_routing_applied, vector_tier_skip_reason, embedding_status,
vector_db_dim/query_embedding_dim) and a modeOf reducer for categorical
metrics — Phase F1-era richer attribution carried forward.
Docs: AGENTS.md / daemon_architecture / retrieval_precision_optimization /
PROMPT-eng-codex updates from the parallel sessions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SocketServer:
- Increase connection handler wait timeout from 100ms to 2s
- Increase accept loop wait timeout from 500ms to 3s
- Prevents race conditions where handlers are cancelled mid-operation
WorkCoordinator:
- Replace thread detach with extended join attempt
- Prevents use-after-free from detached threads accessing freed resources
Daemon Command:
- Increase shutdown wait timeout from 3s to 6s
- Increase default waitForDaemonStop timeout from 5s to 10s
- Allows time for graceful SocketServer shutdown
These changes address SIGABRT and SIGTERM issues during daemon shutdown
by ensuring async operations complete before resources are freed.
