Startup ordering defects were only observable by waiting minutes on a live
daemon, and the one that aborted it took a live restart loop to find. Give the
startup contracts a deterministic harness that runs in milliseconds.
+ startup_fuzz_support.h drives the real ServiceManagerFsm and the real
DeferralGate with virtual time: legal phase advances, injected failure,
shutdown and terminal reset, idle/busy, pressure/calm, starvation deadlines
and cancellation
+ oracles fail closed on shutdown resurrection without reset, failure-to-ready,
eligibility disagreeing with fsm.isReady(), admission outside the virtual
deadline, and ineligible work touching the admission gate; fuzzed IEEE-754
components must never yield non-finite or out-of-range scores
+ catch2_startup_replay runs named regressions plus 1024 seeded traces per run
and reports the root seed and action offset for replay
+ fuzz_startup builds the same harness for libFuzzer/AFL, with a seed generator
that refuses to overwrite different existing files
- the fuzzing lane no longer defaults to TSan: harnesses link
-fsanitize=fuzzer,address,undefined and the two runtimes cannot share a
process; combining them now fails with an explanation instead of producing a
binary that cannot start
- documented the lane invocation, target naming and the coverage limits,
including that governor call ordering inside the poller is not covered
Scope: contract-valid single-owner schedules. Executor interleavings, late
callbacks, concurrent FSM instances and filesystem/model faults are not modeled.
Signed-off-by: trvon <git@trevon.dev>
Semantic-neighbor maintenance aborted the daemon during startup:
std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine
threshold could exceed 1.0 for a legal pair of identical vectors. The same
work was also admitted while the daemon was still initializing, so the
300-second starvation guard forced a pass before readiness.
+ semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the
existing embed policy header, shared by the streamed and corpus branches;
non-finite vectors or scores produce no neighbor and legal scores are bounded
before ranking and threshold selection
+ startupMaintenanceEligible() requires ServiceManagerState::Ready, not
stopping and idle, and is evaluated before
ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no
longer manufacture missing readiness
+ StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter
line reporting state, phase and phase_elapsed_ms instead of a debug-level
state number; a 53-minute startup previously logged nothing actionable
+ regression through the public backfill and corpus-rebuild APIs using a 64-D
vector whose old float inverse norm rounds its self-cosine above 1
- removed the duplicated inverse-norm and cosine lambdas from EmbeddingService
- renamed __test_setModelProvider/__test_setKgStore to testing_* because they
used reserved identifiers, across 11 files
- re-anchored 13 pre-existing raw-environment allowlist entries to their new
line numbers; no new exception was added
Startup is still blocked by corpus-sized memory-sync scans; this change stops
the abort and names the blocking phase, it does not make startup fast.
Signed-off-by: trvon <git@trevon.dev>
* test(search,daemon,storage): assert behaviour instead of frozen literals
Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.
Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.
Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.
The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.
Signed-off-by: trvon <git@trevon.dev>
* ci(tests): summarise Meson failures without flooding the log
A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.
Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.
Signed-off-by: trvon <git@trevon.dev>
* build(deb): validate candidate identity, mapping and upgrade ordering
The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.
Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): bound whole-sweep candidate state
Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision.
Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default.
The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Let RouteOnly and Augment expand every routed hit along its cluster's
walk order. The snapshot cache loader resolves each cluster's
`orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`,
`TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with
`orderKind` None or a permutation that fails `validateMemberPermutation`
resolves to nothing. The store's `orderUnavailableReason` is carried on
the session result and emitted as `topology_order_unavailable_reason`
only when set.
When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex`
after the road beam: the selected routes first, then the road-expanded
clusters, keeping only clusters with an order and counting the rest in
`roadWalkClustersUnordered`. The index borrows its member views from the
snapshot it keeps alive; the certificate is never touched.
The vector stage walks after the certified and road groups. Every routed
hit so far (certified route hits and road-beam hits) anchors +-window
positions along each order it belongs to, bounded to that order
(`walkTopologyRoadOrders`). Anchoring on route hits too means the walk
rescues the ordinal neighbours of the semantically closest members even
with `road_hops` = 0; the plan's "around road hits" reads as every hit a
road-era routed query produced. The walked set, deduplicated against the
hits, is one extra routed query whose quota is its own size (ranking
ordinal neighbours by similarity and truncating would reintroduce the
filter the walk bypasses); its rows and distance evaluations land in the
routed diagnostics like road candidates. Shadow and Narrow never walk.
Config: `search.topology.road_walk_window` (default 0, off) on
`SearchEngineConfig::topologyRoadWalkWindow` (copied by
`applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by
SearchEngineManager. `makeTopologyRoutingOptions` requests the window
only for Augment and a gate-open RouteOnly, like the beam. The window is
excluded from `topologyRoutingPolicyFingerprint` (comment and test).
Defaults leave every stage byte-identical: the PR 2 goldens (692), the
RouteOnly cells (417) and the road beam cells (904) are unchanged.
Debug keys appear only when the window > 0:
`topology_road_walk_window_applied` (0 when nothing was walkable),
`topology_road_walk_anchors`, `topology_road_walk_candidates_added`,
`topology_road_walk_clusters_unordered`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes
`search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and
YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and
`topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3
can set the order kind per arm and build roads at all).
Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp
(new catch2_search_vector_stage_road_walk): two reciprocal chains A
(six members) and B (four) with a road, built with
`TopologyBuildConfig::roadOrder` Time from eventTime so the walk order
differs from the similarity ranking; loader resolution including the
invalid-permutation drop and the cache carry; fingerprint equality for
any window and the option gating per policy; session-level certificate,
beam and work invariance with the index listing routes then road
clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an
unordered cluster counted and skipped, no index without any order, the
order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0
equal to the road beam cell with no walk key, window 1 adding the
immediate order neighbours of all four hits, window 2 the whole
clusters, rows visited growing while router work is unchanged,
certified keys unchanged, unordered B contributing nothing, no order
anywhere equal to the beam cell plus counters, hops 0 walking the
selected route alone; Augment adding the walked documents as rescues;
Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads, the bench reads its keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
Give every cluster with at least two members a walk order:
`ClusterArtifact::orderedMembers`, a uint32 permutation over the
hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each
deterministic with ties resolved to the smaller member hash:
- Time: ascending `TopologyDocumentInput::eventTime`; members without a
time last, by hash.
- Session: grouped by `sessionId`; groups ordered by earliest event time
(groups with no time after those with one), then session id; inside a
group by time then hash; unsessioned members last, by time then hash.
- SimilarityChain: greedy nearest-neighbor tour by cosine over the
members' embeddings, starting from the cluster representative
(`medoid`, else the smallest-hash usable member); members without a
usable embedding last, by hash.
The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML
`topology.road_order` = none|time|session|similarity_chain through the
topology.* resolver block and `TopologyManager::setRoadOrder`, threaded
into the build config like `roadMaxPerCluster`. Orders are computed after
boundary spill so they cover replicas, at full build in every engine and
for the rebuilt clusters of a connected-engine incremental update;
untouched clusters keep theirs (a kind change reaches them on the next
full rebuild, like the representative rule) and `none` clears every
order. Orders are independent of `roadMaxPerCluster`.
Orders persist in the road sidecar, now version 3: after the roads, u32
count, then per ordered cluster u32 cluster-id table index, u32 kind,
u32 length and the permutation as u32 indices. A sidecar without orders
is still written as v2, so roads-only sidecars are byte-identical to the
previous layout; v1 and v2 decode with no orders, and the envelope gains
`member_order_count`. The sidecar is written when the batch has roads or
at least one ordered cluster. On load each order is validated on its own
(`attachTopologyMemberOrders`: cluster exists, kind not none,
permutation covers the members exactly once); an invalid entry drops
only that cluster's order and appends `<token>:<clusterId>` to the new
load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`.
Roads never lose anything to an order error and vice versa.
The main snapshot codec, the routing construction, representation and
road fingerprints never see orders, so the default `none` leaves the
snapshot payload, the sidecar and every fingerprint byte-identical. No
search consumer yet; the road walk lands separately.
Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp
(permutation validator tokens, time order with missing times, session
grouping and group tie-break, hand-computed greedy chain from the medoid
on a five-point fixture with fallback start, unusable members and exact
ties, every engine stores valid permutations of every kind including over
spill replicas, default none byte-identical payload/sidecar/fingerprints,
sidecar v3 round trip plus hand-built v2 bytes and corrupt-section
rejection, per-cluster attach validation with reasons, store round trip
with orders-only sidecars, invalid order dropped with reason while roads
and other orders stay, incremental update keeps untouched orders and
recomputes rebuilt ones), plus resolver and manager setter coverage.
Signed-off-by: trvon <me@trevon.dev>
Let the vector stage follow cluster roads from the selected route
clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the
snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of
both stored orientations of every road, per source sorted by weight
desc then target cluster id asc; it is empty when the batch carries no
roads, whether never built or dropped with `roadUnavailableReason`.
The session runs a beam after the certificate is materialized: for
`roadHops` hops it scores every unvisited road neighbour of the
frontier by road weight times the router's query-centroid similarity
((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster
id asc) and dedupes across hops. Each centroid evaluation is cached per
cluster and counted once in `routeRepresentativeDistanceEvaluations`
and `routeExactRepresentativeDistanceEvaluations`, so routed work stays
comparable with the no-road arm. The result lands only in
`TopologyRoutingSessionResult::roadExpandedClusterIds` and
`roadExpandedDocumentHashGroups` (one member group per cluster,
excluding certificate members). `certificate.allowedDocumentHashes`,
admission and `topologyRoutingPolicyFingerprint` are untouched; the
fingerprint comment and a test pin that the road knobs stay out of it.
`VectorStageRequest` carries the road groups with their own quotas
(`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly
and Augment query them after the certified groups; the certified part
of the routed result is identical with roads on or off. Shadow and
Narrow never see road groups, and `makeTopologyRoutingOptions` requests
hops only for Augment and a gate-open RouteOnly, so a gated-off
RouteOnly running as Shadow takes no roads with it.
Config: `search.topology.road_hops` (default 0, off) and
`search.topology.road_beam_width` (default 4, read when hops > 0) on
`SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`,
copied by `applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy` and applied by
SearchEngineManager. Defaults leave every stage byte-identical: the PR 2
goldens (692) and the RouteOnly cells (417) are unchanged.
Debug keys appear only when hops > 0: `topology_road_hops_applied`,
`topology_road_clusters_added`, `topology_road_candidates_added` (road
documents the stage appended), `topology_road_work_centroid_evaluations`,
`topology_road_skip_reason` (roads_absent, no_selected_routes,
route_low_confidence) and, under stage trace,
`topology_road_expanded_cluster_ids`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH
write the two TOML keys.
Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp
(new catch2_search_vector_stage_road_beam): a four-cluster fixture with
hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A;
adjacency construction and cache carry; fingerprint equality for hops
0 vs 2 and any beam width; session-level certificate invariance with
hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0
equals the route-only cell with no road keys, hops 1 gives A u B, hops 2
with beam width 1 gives A u B u C via the heavier road, work counters
grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and
Narrow take no road groups. ConfigResolver covers both TOML keys.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads are added, the bench reads its two keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
A paused or admission-blocked poller slept for the full max idle delay
instead of waiting on its registered timer, so resumeStage, resumeAll,
enabling the knowledge graph, and adding entity or title providers took
effect only after the delay elapsed. The poller now waits through
awaitPressureLimitedPollerWake and the setters signal the stage timer.
Tests: pressure_limited_poller and post_ingest_queue_unit.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
The governor only measured the daemon itself (RSS, own CPU), so it could
not tell that the machine was busy. On a loaded host the daemon still
started background work at full width, and that work ran far less
efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU
seconds on an idle 48-core host and 96 CPU seconds with the host 2x
oversubscribed (stress-ng), and took 4x longer to finish.
Sample a cheap host-wide signal on the existing governor tick, at most
once per second: Linux PSI "some avg10" for cpu, io and memory, falling
back to the one-minute load average per online CPU; on macOS the load
average plus the kernel VM pressure level. A busy host is a flag with a
release band (signals must drop to 75% of the threshold to clear) so
work does not flap around a threshold.
Host load stays a separate dimension from the governor's pressure
level. Folding it into the level would reuse the Warning response,
which blocks model loads and trims foreground concurrency; a busy build
on the same machine should not stop a search from loading its model.
Background scheduling consumes the flag in a follow-up change.
Thresholds are typed config under [tuning.resource]: host_pressure,
host_cpu_pressure_pct (40), host_io_pressure_pct (30),
host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in
the runtime tuning status with their provenance.
Signed-off-by: trvon <me@trevon.dev>
~/.local/state/yams/sessions grows without bound: tool integrations
create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files
out of 1,764 on one machine) and nothing ever removes them.
Add expireIdleSessions() next to the session store and run it from
RepairService's idle maintenance once a day, only while no client is
connected. A session file is deleted only when nobody has created,
opened, closed or modified it for [daemon.maintenance]
session_expiry_days (typed config, default 30, 0 disables), and it is
kept when it is:
- the current session,
- watching a directory that still exists,
- still tagged on documents (they stay reachable through `session
merge` / `discard`; without a metadata repository ownership is
unknown and nothing expires),
- not valid JSON (not ours to judge).
Tests build a store shaped like the audited one and check each rule,
plus the RepairService wiring against a real metadata repository and
that a connected client defers the scan.
Signed-off-by: trvon <me@trevon.dev>
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:
- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
the audited machine) that `yams init` / `yams config grammar` built
and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
every start with two warnings (the ABI host's skip plus the plugin
manager's "load failed"), and the refusal flipped the plugin host FSM
through Failed.
Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.
The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.
Signed-off-by: trvon <me@trevon.dev>
vectors.db had no reclaim path at all. On a real install it had grown to
39.7 GB with 48% of its pages on the freelist (about 19 GB of dead
space), while yams.db already had a startup auto-VACUUM with a size,
free-page-ratio and free-disk check.
Move that decision into a shared sqlite_vacuum helper so both databases
follow the same rules, and let RepairService's background loop check
vectors.db once a day (15 minutes after start, typed config
[daemon.maintenance] vector_vacuum_interval_hours, 0 disables).
VACUUM holds the vectors.db write lock for the whole rewrite, so it only
starts while the daemon is idle: no client connections, resource
pressure below Warning, no queued or in-flight embedding writes, no
VectorIndexCoordinator rebuild or bulk load, and no on-demand repair.
It runs on its own connection with file-backed temp storage and a short
busy timeout, so a live writer makes it back off instead of waiting, and
a progress handler rolls it back as soon as any of those signals change
or shutdown starts. Readers on the owner's connection are unaffected in
WAL mode. Besides the data-dir free-space rule it also requires the temp
directory to hold a copy of the live pages.
Tests cover the thresholds (including the audited 39.7 GB / 48% case),
a real vacuum that shrinks a file, abort and live-writer back-off, the
admission truth table, and RepairService vacuuming a live VectorDatabase
file only when idle, which stays writable afterwards.
Signed-off-by: trvon <me@trevon.dev>
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
- config_resolver: several [tuning] defaults derive from core count (e.g.
onnx_max_concurrent is 7 on a 48-thread host), so a reload is now checked
against the values observed before the override instead of != sentinel.
- service_manager: write the P2P trust files with explicit permissions so
a umask of 002 (Debian user-private groups) no longer trips the
group-writable check before the symbolic-link check under test.
Graph rerank recomputes connected components over mutual
semantic_neighbor edges among the candidates on every query, reading
each candidate's edges from the KG. The resident topology snapshot
already holds corpus-wide clusters built from those edges.
Add search.topology.graph_community_source (reciprocal_edges |
topology_snapshot). topology_snapshot scores candidates that share a
snapshot cluster with the same normalization and falls back to the edge
walk when no snapshot is loaded; graph_community_source in debug stats
reports which ran. The default stays reciprocal_edges until a benchmark
gate shows the snapshot source is at least as good. The retrieval bench
can set it for ablations.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
sgcHops, sgcNormalize, bqCandidateLimit and bqPrefixDimension existed
only as struct fields, so no production path could enable them. Resolve
topology.sgc_hops / topology.sgc_normalize into the TopologyManager
build config (hops capped at 4 to bound over-smoothing) and
search.topology.bq_candidate_limit / bq_prefix_dim into the routing
options, request, and fingerprint. The snapshot cache rebuilds only the
route index when the BQ prefix changes. All default to off.
Drop the snapshot interner, which was allocated but never used.
Scope vector fixture overrides across every disable alias and verify restoration. Prune excluded repository directories before walking them. Exercise CLI serve delegation and distinguish outer spawn failure from missing child diagnostics using isolated stubs.
Signed-off-by: trvon <me@trevon.dev>
* test(ci): stabilize main matrix (#112)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): preview experimental promotion (#111)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): reconcile draft promotion safely (#113)
* ci(release): reconcile draft promotion
Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.
Signed-off-by: trvon <me@trevon.dev>
* ci: isolate hook Git environments
Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): reject invalid candidate evidence
Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve completion and retrieval contracts
Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence asynchronous enrichment completion
Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.
Signed-off-by: trvon <me@trevon.dev>
* fix: bound semantic graph output reservation
Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence standalone embedding repair completion
Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve enrichment failure reporting
Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test: align explicit hash prefix boundaries
Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.
Signed-off-by: trvon <me@trevon.dev>
* test: align UI hash validation boundary
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hold Simeon concept mining to the corpus byte budget
The BM25 pass streams one chunked document at a time and stops when
processed text would exceed max_corpus_bytes. The concept-mining pass
that follows re-read every document's raw text into one vector and
kept it all resident for mine_concepts(), with no budget at all: on a
large corpus the build's peak memory was the whole corpus, regardless
of the cap the first pass had just enforced. This is the likeliest
single contributor to the daemon's multi-gigabyte footprint.
The concept pass now applies the same budget to raw bytes as it loads.
When the corpus exceeds it, mining is skipped with a warning and the
BM25 index is kept; the partially loaded text is released. A corpus
within budget is mined exactly as before.
Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass
under 64 KiB. Over budget, the build completes with zero concepts
(previously 81 were mined from ~480 KiB of resident text); with a
4 MiB budget the same corpus mines concepts.
Signed-off-by: trvon <me@trevon.dev>
* perf(ingest): bound the pending KG job overflow
When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob
parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob
can carry the document's content bytes, so a stalled or slow KG stage
turned into unbounded ingest memory on top of the ~36k entries the
symbol, entity and KG channels already hold.
The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max
(typed key on TuneAdvisor, default 16384, no environment twin per
AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for
this ingest, the drop is counted through the existing kgDropped bus
counter (shared with shutdown cancellations), and a throttled warning
names the cap. Nothing re-derives the per-document entity graph for a
dropped job today; the structural repair pass rebuilds document nodes
and orphans but not that stage, so an operator who sees the warning
should raise the cap or re-ingest. Bounded memory with a visible drop
beats unbounded memory with an invisible stall.
Test: with the channel detached and a live coordinator (so the drain
coroutine neither cancels nor consumes), three enqueues against a cap
of two leave two pending and one dropped; before the cap all three
were pending.
Signed-off-by: trvon <me@trevon.dev>
* fix(ingest): preserve durable KG admission state
Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays
LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.
Removed (CANDIDATE rows of the report, verbatim):
YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
YAMS_SEARCH_GRAPH_ENABLE_PATHS
YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
YAMS_SEARCH_GRAPH_HOP_DECAY
YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
YAMS_SEARCH_GRAPH_MAX_HOPS
YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
YAMS_SEARCH_GRAPH_MAX_PATHS
YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
YAMS_SEARCH_GRAPH_TEXT_WEIGHT
YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
YAMS_SEARCH_RERANK_TOP_K
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
YAMS_SEARCH_TIERED_MIN_CANDIDATES
YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE
Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.
A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.
The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): drop unread embeddings.* keys and the no-op toggles
yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.
- embeddings enable and disable are removed. enable duplicated
embeddings model <name> once its dead keys are gone; a kept no-op
would be a shim. status no longer prints an auto-generation line and
shows the runtime batch size when set. tune presets write
embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
[embeddings] batch_size silently when the runtime key is absent, so
configs in the wild get their value applied without a file rewrite (the
migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.
Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.
Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.
Signed-off-by: trvon <me@trevon.dev>
* feat(config): add [tuning] keys for the setters only env could reach
Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.
Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.
A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.
The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.
Signed-off-by: trvon <me@trevon.dev>
* fix(config): report 0 for [tuning] integer keys instead of ignoring it
For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.
Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.
Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): one embedding-runtime resolver, one tuning baseline
Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.
yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.
Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): stop writing the unread search.reranker_model_path
yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.
Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.
portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.
The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).
Signed-off-by: trvon <me@trevon.dev>
* fix(config): validate tuning ranges and pairs
Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.
Cover all 27 integer keys, inherited limits, and reservation reset.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(embedding): complete embeddings by derivation token
cbc4959c added derivation tokens (begin/complete with a per-attempt
generation, invalidated by document_content triggers) but nothing
produced them: beginDocumentEmbeddingDerivation had no callers and the
completion op's derivations field was never written, while every ingest
still paid for the triggers and the per-hash NOT EXISTS guard.
processEmbedJob now mints one token per document it will embed, before
any input snapshot is read, and completes by token instead of by hash.
A document whose content changes while it is being embedded keeps
has_embedding = 0 and is re-derived later, instead of publishing vectors
for text the corpus no longer holds. Repair jobs travel the same
channel, so they get the same guarantee. The recipe binds the
preparation policy, embedding space, provider version, and dimension
through the embeddingDerivationRecipe helper that cbc4959c also left
unused.
Repository: batchBeginDocumentEmbeddingDerivations and
batchCompleteDocumentEmbeddingDerivations do a whole job in one
transaction each (the single-token completion loop in WriteCoordinator
opened one transaction per document). WriteCoordinator uses the batch
form and takes its stats lock once.
A small EmbeddingDerivationLedger maps job hashes to their tokens so
completion is issued for exactly the documents that produced vectors.
Behavior note: a re-embed (skipExisting = false) of an already-embedded
document resets its readiness at mint time; if that job then fails, the
document reads as not embedded until the next attempt. That is the
attempt semantics the token table was built for.
Not included: the legacy hash completion path is now unproduced and is
removed in a follow-up so the coalescer change is reviewed on its own.
tests/scripts/portability_allowlist.txt: 13 EmbeddingService.cpp
line anchors reflowed; no new environment reads.
The startup reconcile (ServiceManager, "Reconciling document_embeddings_status
from vector rows") used to mark every document with vector rows as embedded,
which would have undone the stale-token guarantee on the next daemon start: a
document whose content changed under an in-flight embed has vector rows for the
old text. reconcileDocumentEmbeddingStatusByHashes now leaves a document with an
open derivation (document_embedding_derivations.completed = 0) at has_embedding =
0, so it is re-derived rather than published; the reconciled-count query applies
the same predicate. Test: an open derivation stays unembedded across a reconcile
and flips once its token completes.
Signed-off-by: trvon <me@trevon.dev>
* fix(embedding): require current recipe to skip
Re-derive legacy or mismatched recipes rather than trusting readiness
alone. Treat derivation lookup errors as failed gathering.
Cover service-level recipe changes and legacy readiness, and characterize
atomic batch completion rollback and retry.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
dispatchNonEmbeddingStages hands one shared_ptr<vector<byte>> to the KG,
symbol, and entity channels. The entity stage moved out of that buffer, so
whichever consumer ran after it saw zero bytes and silently extracted
nothing. The KG batch consumer already read through a pointer; the entity
stage now does the same, keeping an owned fallback only for the
store-reload path.
Also delete processSymbolExtractionStage, the legacy single-item symbol
handler: it had no callers, and it moved out of the shared buffer as well
before copying the bytes a second time into contentUtf8.
Adds a YAMS_TESTING hook and a Catch2 case that runs two entity jobs over
one shared buffer and asserts the buffer survives.
* feat(storage): expose typed disk pressure policy
Signed-off-by: trvon <git@trevon.dev>
* fix(mcp): preserve embedded stdio buffers
Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order.
* refactor(daemon): scope shared P2P protected-file accessors for cross-TU use
The protected-file accessors (readProtectedP2pPrivateKey,
readProtectedP2pTrustFile, writeExclusiveP2pPrivateKey) are shared by the
P2P identity-key paths and the memory-sync bootstrap. Keep their
platform-specific definitions in p2p_integration.cpp unchanged and scope
them into yams::daemon::service_manager_detail, declared by the private,
non-installed header p2p_protected_file.h. Qualify the internal
callsites. No behavior change.
Prepares moving ServiceManager::initializeMemorySync into the
memory-sync translation unit (Brick audit cohesion note).
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): place initializeMemorySync with memory-sync TU
Brick structural audit of PR #79 flagged that
ServiceManager::initializeMemorySync lives in p2p_integration.cpp while
the rest of the memory-sync surface lives in memory_sync.cpp. Move it —
together with its exclusive helper ensureAuthenticatedDirectStore — into
memory_sync.cpp.
The shared protected-file accessors remain defined in
p2p_integration.cpp (see the preceding service_manager_detail commit);
moved function bodies are unchanged and ServiceManager.h is untouched.
Brick evidence (experimental pre-79 af609a1e vs post-79 4d942793):
ServiceManager.cpp functions 84 -> 59 with 25 moved and 0 vanished;
ServiceManager.cpp outgoing call/import edges 1292 -> 954, distinct
targets 59 -> 41.
Validation: full ninja build green; meson test yams:service_manager,
yams:service_manager_shutdown, yams:service_manager_degraded — 3/3 OK.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): break simeon_embedding_backend.h include cycle with embedding_generator.h
simeon_embedding_backend.h only declares functions taking EmbeddingConfig
by reference, yet included the full embedding_generator.h definition,
forcing every consumer of the simeon backend header to transitively pull
the generator header (and feeding the embedding_generator <->
simeon_embedding_backend 2-cycle reported by the Brick structural survey).
Replace the include with forward declarations for IEmbeddingBackend and
EmbeddingConfig. Consumers that use the full definitions
(simeon_embedding_backend.cpp, search_engine.cpp,
ingestion_throughput_bench.cpp) now include embedding_generator.h
directly.
Brick caveat: the tool's lexical lane cannot see the header-include edge
removal (0.65-confidence interface-dispatch misattribution keeps a
file-level cycle in its graph); the include-cycle break is verified at
source level. Brick survey pre/post artifacts kept in
worktree build/brick/architecture-iter{,-after}.json.
Validation: full ninja build green; meson test yams:vector_submodule,
yams:simeon_model_provider, yams:embedding_lifecycle,
yams:embedding_recovery, yams:service_manager — 5/5 OK.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): extract computeCosineSimilarity from VectorDatabase into vector_utils
The pure cosine-similarity helper was stranded as a static member of the
VectorDatabase storage class, forcing sqlite_vec_backend.cpp to include
vector_database.h for a pure math function (Brick deferred lead #2 from
the PR #79 audit loop).
Move the implementation to vector_utils as a free function
(yams::vector::vector_utils::computeCosineSimilarity, semantics copied
verbatim: 0.0 on size mismatch or zero norm), keep
VectorDatabase::computeCosineSimilarity as a forwarding static for the
external caller in doctor dedupe, and switch the four backend callsites
to the free function. sqlite_vec_backend.cpp no longer includes
vector_database.h.
Validation: full ninja build green; meson test vector_submodule (incl.
simeon_backend_catch2_test) and yams:service_manager — green.
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): extract client transport types to break sandbox probe header cycle
ClientTransportMode and ClientConfig moved verbatim from daemon_client.h
into a new shared header client_config.h (Brick deferred lead #3 from the
PR #79 audit loop: daemon_client <-> sandbox_detection type-level cycle).
sandbox_detection.h now depends only on client_config.h; its
implementation adds the explicit daemon_client.h include it needs for
the DaemonClient discovery statics. All existing consumers are
unaffected: daemon_client.h includes client_config.h so ClientConfig
remains visible transitively.
Remaining runtime dependency (sandbox_detection.cpp calling
DaemonClient::resolveSocketPathConfigFirst / isDaemonRunning statics,
which forward to socket_utils and a file-local ping helper) is recorded
as the follow-up for full decoupling; the header-level cycle is broken.
Validation: full ninja build green; meson test yams:service_manager,
yams:multiplex_client, yams:jsonrpc_client, vector_submodule — all green.
Signed-off-by: trvon <me@trevon.dev>
* test: refresh portability allowlist line anchors after include shifts
The Brick-audit include additions (search_engine.cpp,
ingestion_throughput_bench.cpp) and removal (sqlite_vec_backend.cpp)
shifted line numbers, making seven portability allowlist entries stale
(same reviewed raw getenv/path usages, new line numbers). Refresh the
anchors; no code or policy change.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): include <fstream> for all POSIX /proc probe paths
The /proc readers compile in the non-Windows/non-Apple branch (BSD and
other POSIX targets), not just under __linux__; guard the <fstream>
include with the same condition so those targets compile.
Copilot review finding on PR #80 (src/daemon/components/DaemonMetrics.cpp:7).
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): apply storage.disk_pressure outside applyMemorySync
ConfigResolver::applyMemorySync also applied [storage.disk_pressure],
coupling unrelated config parsing to the memory_sync apply path and
producing a misleading refusal message for invalid disk-pressure
policies. Move the call to its natural site in daemon_main before the
memory_sync apply, with its own accurate refusal message. Same startup
semantics (exit 1 on invalid policy), correct attribution.
Copilot review finding on PR #80 (src/daemon/components/ConfigResolver.cpp:1658).
Signed-off-by: trvon <me@trevon.dev>
* test(daemon): update applyMemorySync no-op expectation after disk-pressure decoupling
applyMemorySync no longer applies [storage.disk_pressure] (moved to its
own call site in daemon_main), so a section set without memory_sync
entries is now a no-op rather than a disk-pressure rejection. The
invalid-disk-pressure rejection is still asserted directly against
applyStorageDiskPressure in the preceding checks.
Caught by the pre-push CI gate (asan+tsan: daemon_component_submodule).
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Signed-off-by: trvon <me@trevon.dev>
Keep `p2p status` text output machine-parseable by removing ANSI severity
colorization, and stop connect/disconnect/forget from throwing on malformed
daemon responses (safe .value defaults, uint64 counters). Bound handshake and
delta timeouts to kP2pMaxOperationTimeout up front, and ceiling direct
sync_interval_ms to the 5-minute reconnect bound so invalid config fails early
instead of aborting daemon startup. Degrade memory-sync status to peerCount=0
on a transient peer-registry read instead of failing the whole status IPC. Add
an explicit `p2p forget` operation (proto FORGET=9 through dispatcher, manager,
and registry) so operators can rotate a peer's pinned key while preserving
fail-closed enrollment.
Signed-off-by: trvon <git@trevon.dev>
Require authenticated direct stores and explicit operator enrollment while hardening private keys and trust files across Unix and Windows. Sign local control artifacts, bind them to a protected store generation, and serialize async initialization shutdown.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
