Semantic-neighbor maintenance aborted the daemon during startup:
std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine
threshold could exceed 1.0 for a legal pair of identical vectors. The same
work was also admitted while the daemon was still initializing, so the
300-second starvation guard forced a pass before readiness.
+ semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the
existing embed policy header, shared by the streamed and corpus branches;
non-finite vectors or scores produce no neighbor and legal scores are bounded
before ranking and threshold selection
+ startupMaintenanceEligible() requires ServiceManagerState::Ready, not
stopping and idle, and is evaluated before
ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no
longer manufacture missing readiness
+ StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter
line reporting state, phase and phase_elapsed_ms instead of a debug-level
state number; a 53-minute startup previously logged nothing actionable
+ regression through the public backfill and corpus-rebuild APIs using a 64-D
vector whose old float inverse norm rounds its self-cosine above 1
- removed the duplicated inverse-norm and cosine lambdas from EmbeddingService
- renamed __test_setModelProvider/__test_setKgStore to testing_* because they
used reserved identifiers, across 11 files
- re-anchored 13 pre-existing raw-environment allowlist entries to their new
line numbers; no new exception was added
Startup is still blocked by corpus-sized memory-sync scans; this change stops
the abort and names the blocking phase, it does not make startup fast.
Signed-off-by: trvon <git@trevon.dev>
Dispatch explicit committed memory reads on admitted workers, checking the
governor before queueing and before I/O. Require the coordinator instead of
using the synchronous strand fallback; keep cached APIs and wire opcodes
unchanged and preserve task-record validation.
Use the paged content adapter, distinguish typed local admission deferral
from hard errors and retain known deferred totals/ages after incomplete
scans. Forward exact topology totals independently of diagnostic samples
and keep attempted cycles distinct from complete scans and successful apply.
Add real daemon IPC tests for authenticated off-resident reads, pressure
health, exact 70-edge status/retry accounting and authenticated cold document
retrieval through the existing MCP get handler, DaemonClient IPC and CAS.
Reanchor existing dispatcher environment-read entries without new exceptions.
Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader
fault/readiness/P2P convergence and patched resource measurements remain open.
Signed-off-by: trvon <git@trevon.dev>
Spill exclusive native-filesystem sweep observations and committed winners
into bounded SQLite workspaces on the corpus filesystem. Preserve complete
operation/vector fork observations and validate payloads before publishing
an immutable generation; keep resident caches separate from full authority.
Add fallible committed/pinned reads, bounded adapter bookkeeping, cooperative
filesystem traversal, secure scratch allocation/cleanup and pinned lifetime
limits. Retain exact durable erase intents and condition own-record erasure
on the full current envelope, including a post-reconciliation check.
Preserve the existing shared-store, cache, value and scan limits. Reanchor
only the existing filesystem environment-read entries and register SQLite
and layout-independent test hooks in the memory-sync library.
Bootstrap import/finalize/restart, broader ingress/scratch faults and final
SDK ABI/security/performance signoff remain separate work.
Signed-off-by: trvon <git@trevon.dev>
Publish owning data-directory snapshots under a mutex and route bootstrap
and fallback publications through the same synchronization boundary.
Keep polling completion locked through its final notification so shutdown
cannot destroy the condition variable while the notifier still uses it.
Exercise both publication windows with deterministic event-based tests and
use the repository thread-compatibility wrapper.
Preserve Failed in status rendering instead of reporting initialization.
Reanchor the existing metrics environment-read entry; add no exceptions.
This does not close the separate queued-collector and embedding-borrower
lifetime audits or establish universal bounded callback shutdown.
Signed-off-by: trvon <git@trevon.dev>
Remove the bare Linux libtsan lookup that mixes GCC's shared runtime with
Clang's instrumented host. Add an ELF linkage test that requires actual
TSan instrumentation and rejects the other compiler's runtime.
Include cstring explicitly for retrieval-session memory operations.
The regression was exercised against the actual instrumented shared object;
no sanitizer instrumentation, interceptors, or checks are disabled.
Signed-off-by: trvon <git@trevon.dev>
* test(search,daemon,storage): assert behaviour instead of frozen literals
Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.
Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.
Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.
The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.
Signed-off-by: trvon <git@trevon.dev>
* ci(tests): summarise Meson failures without flooding the log
A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.
Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.
Signed-off-by: trvon <git@trevon.dev>
* build(deb): validate candidate identity, mapping and upgrade ordering
The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.
Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): bound whole-sweep candidate state
Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision.
Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default.
The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(storage): bound filesystem listPage residency
listPage() walked the whole op store and retained every matching key before
slicing one page. At 1025 records and a 512-record page it held 1025 keys
where 513 is the working set, and the retained set grew with the store.
Traverse once into a max-heap holding the smallest eligible limit+1 keys.
The walk itself stays complete, so a traversal error still returns an error
instead of a partial page. Exclusive lexical cursor, ordering, Windows key
decoding and the SIZE_MAX limit stay as they were.
Add a YAMS_TESTING-only observation seam that counts real walks, entries
visited and peak retained keys, so the traversal cost can be measured
instead of inferred. The new traversal shifts two reviewed getenv allowlist
line numbers.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep backfill progress past an oversized vector
A record above the resident limit failed publication, the domain cursor
never advanced, and every later cycle retried the same row. Valid vectors
behind it never reconciled, so one oversized value could pin a domain
indefinitely. The daemon log showed this repeating every ~100 s.
Classify the row only after a publish fails with InvalidArgument and the
same serialized record is independently confirmed to exceed the immutable
limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep
the local copy, and revisit it when a new source generation or service
limit appears. A generic backend InvalidArgument stays retryable, and
healthy rows still serialize only once.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): reuse a bounded complete listing across publishes
Every store invalidated the cached listing pages, so a publish followed by
syncFully walked the whole op store again, once per 512-record page. On the
profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of
it system time, with listPage at 55% of user samples.
Keep a complete key-only prefix snapshot, built only from normal contiguous
pages starting at an empty cursor on the concrete filesystem backend. The
snapshot is capped by key count and key bytes, served only in lexical order,
and dropped as incomplete unless the walk reached the end. Any mutation
invalidates it before the backend call; a successful store or remove then
applies the exact key to matching snapshots, and a failure drops them
because the write may have committed. Initialising the async path disables
cache reuse permanently rather than racing a future.
Cache maintenance is best effort: it runs after the backend result is known
and any allocation failure clears the cache and keeps the committed result.
Opaque cursors are untouched, so only the built-in filesystem backend gets
snapshot treatment.
Active publishes now cost the same number of walks as a warm sweep instead
of four, six and nine at 511, 513 and 1025 records.
Signed-off-by: trvon <git@trevon.dev>
* test(bench): measure memory-sync listing cost across arms
Add a standalone harness that drives a real filesystem op store through
MemorySyncLoop, one scenario per process so no cache state survives a
repeat, and reports wall, user, system and peak RSS per run. It uses
production APIs only, so the same source builds against a baseline and a
candidate tree and neither arm needs a measurement seam ported into it.
The driver seeds one corpus per size, interleaves the two arms inside each
repeat, and writes results.jsonl plus a median report. Enumeration counts
are not re-derived here; the unit suite owns those.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(search): make the final lexical tie-break a strict weak ordering
The final comparator broke near-ties toward the result with less vector
support, and its pairwise |a - b| <= epsilon test was not transitive, which
is undefined behavior for the std::sort/partial_sort callers in
applySearchTopKLimit.
Compare tie buckets floor(score / epsilon) instead, then lexical anchor,
keyword and vector evidence (more first), then doc id. Behavior with the
tie-break off is unchanged. The comparator moves to
final_result_order_internal.h so it can be tested directly.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit ca9b82d6ce9db31b6c9608a7d95b63203649e5a7)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): keep cross-reranked scores above the unreranked tail
applyCrossRerank rewrote the window's scores to a 0..1 min-max blend while
the tail kept raw fused scores. Later whole-list sorts (concept boost,
top-k partial_sort) compared the two scales, so reranked docs could drop
below tail docs or out of the top-k.
Map the blended score back onto the window's original range, with the
floor raised just above the best tail score when needed. The window's
reranked order is unchanged and docTraces.finalScore records the mapped
score.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit d66c6d304ca4d32b943e164b0d7720fc7f865db1)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): judge weak tier-1 queries on the raw bm25 scale
The weak-query test compared the top Text/SimeonText component score with
weakQueryMinTopTextScore, but lexical batches are min-max normalized, so the
top hit is always 1.0. With enough hits every query was strong: the
weak-query vector fanout boost and weak-query topology expansion never ran.
scoreLexicalBatch now also records absoluteLexicalScore, the hit's score on
the existing absolute scale (-bm25 / bm25NormDivisor, same multipliers),
and classifyTier1TextSignal uses it. Thresholds keep their values; with the
default divisor 25, 0.30 is bm25 <= -7.5 and 0.45 is bm25 <= -11.25.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit bc771c5aab1721170fb9e743013ca0efbf048d61)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): restore measured scientific fusion
Give SCIENTIFIC the measured relative Simeon weight of 0.40 after normalizing the existing text/vector pair. The resulting share is 0.285714, matching the benchmark traces. Other profiles remain unchanged.
Keep its one-slot semantic rescue preset instead of raising it to three at the corpus layer. Preserve the existing large-prose recall widening. Cover the preset weights, config propagation, community blend, rich-graph ratios and corpus rescue behavior.
Signed-off-by: trvon <git@trevon.dev>
* fix(search): order non-finite ranking evidence
Treat NaN score/evidence keys as negative infinity and invalid tie epsilon as zero. This keeps comparator equivalence transitive without changing finite-input ordering. Add exhaustive triples covering NaN, infinities and invalid epsilon.
Signed-off-by: trvon <git@trevon.dev>
* docs(search): clarify lexical scoring limits
Distinguish the shared single-hit score conversion from FTS5 BM25 calibration. Note that lexical tie buckets can still reorder numerically separated rerank-window and tail scores. No behavior change.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
Ingest resolves symlinks before it writes a path (populatePathDerivedFields
calls weakly_canonical), and buildGraphCwdScopePathRanges resolves the scope
the same way. The graph dispatcher fixture wrote node keys and labels from an
unresolved temp_directory_path(). On macOS that is /var/folders/..., a
symlink to /private/var/folders/..., so the scoped listByType sections found
0 nodes there and passed on Linux. The fixture now resolves its temp dir,
like the stored paths it stands in for.
Signed-off-by: trvon <me@trevon.dev>
#296 sets XDG_STATE_HOME before Catch2 starts so each test process gets a
private state directory. The support library links no yams config, so it
calls setenv/_putenv_s directly, and raw_environment_policy has failed the
Tests job on every platform since. Record both sites as reviewed.
Signed-off-by: trvon <me@trevon.dev>
Co-authored-by: trvon <me@trevon.dev>
Benchmark seeding wrote only mutual neighbor pairs. The connected
topology engine keeps every reciprocal edge at or above min_edge_score
inside its component, so with mutual-only seeding no raw neighbor
crosses a cluster boundary and the engine builds no roads. The Eval 2
smoke showed roads_absent on every connected-engine query.
The daemon stores one-way kNN edges (GraphComponent treats them as
ordinary non-mutual neighbors). YAMS_BENCH_SEED_SEMANTIC_DIRECTED=1
seeds them too, as single directed edges. Default off keeps existing
plans unchanged.
The pair loop moves into buildSemanticNeighborSeedEdges in the bench
support library, sorted by (source, target), with a unit test for both
modes. A topology test pins the mutual-only case: no roads on the
connected engine until a one-way edge exists.
Signed-off-by: trvon <me@trevon.dev>
Let RouteOnly and Augment expand every routed hit along its cluster's
walk order. The snapshot cache loader resolves each cluster's
`orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`,
`TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with
`orderKind` None or a permutation that fails `validateMemberPermutation`
resolves to nothing. The store's `orderUnavailableReason` is carried on
the session result and emitted as `topology_order_unavailable_reason`
only when set.
When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex`
after the road beam: the selected routes first, then the road-expanded
clusters, keeping only clusters with an order and counting the rest in
`roadWalkClustersUnordered`. The index borrows its member views from the
snapshot it keeps alive; the certificate is never touched.
The vector stage walks after the certified and road groups. Every routed
hit so far (certified route hits and road-beam hits) anchors +-window
positions along each order it belongs to, bounded to that order
(`walkTopologyRoadOrders`). Anchoring on route hits too means the walk
rescues the ordinal neighbours of the semantically closest members even
with `road_hops` = 0; the plan's "around road hits" reads as every hit a
road-era routed query produced. The walked set, deduplicated against the
hits, is one extra routed query whose quota is its own size (ranking
ordinal neighbours by similarity and truncating would reintroduce the
filter the walk bypasses); its rows and distance evaluations land in the
routed diagnostics like road candidates. Shadow and Narrow never walk.
Config: `search.topology.road_walk_window` (default 0, off) on
`SearchEngineConfig::topologyRoadWalkWindow` (copied by
`applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by
SearchEngineManager. `makeTopologyRoutingOptions` requests the window
only for Augment and a gate-open RouteOnly, like the beam. The window is
excluded from `topologyRoutingPolicyFingerprint` (comment and test).
Defaults leave every stage byte-identical: the PR 2 goldens (692), the
RouteOnly cells (417) and the road beam cells (904) are unchanged.
Debug keys appear only when the window > 0:
`topology_road_walk_window_applied` (0 when nothing was walkable),
`topology_road_walk_anchors`, `topology_road_walk_candidates_added`,
`topology_road_walk_clusters_unordered`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes
`search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and
YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and
`topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3
can set the order kind per arm and build roads at all).
Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp
(new catch2_search_vector_stage_road_walk): two reciprocal chains A
(six members) and B (four) with a road, built with
`TopologyBuildConfig::roadOrder` Time from eventTime so the walk order
differs from the similarity ranking; loader resolution including the
invalid-permutation drop and the cache carry; fingerprint equality for
any window and the option gating per policy; session-level certificate,
beam and work invariance with the index listing routes then road
clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an
unordered cluster counted and skipped, no index without any order, the
order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0
equal to the road beam cell with no walk key, window 1 adding the
immediate order neighbours of all four hits, window 2 the whole
clusters, rows visited growing while router work is unchanged,
certified keys unchanged, unordered B contributing nothing, no order
anywhere equal to the beam cell plus counters, hops 0 walking the
selected route alone; Augment adding the walked documents as rescues;
Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads, the bench reads its keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): typed event time and session for topology inputs
Add `eventTime` (epoch seconds, UTC) and `sessionId` to
`TopologyDocumentInput`. The extractor fills them from two documented
metadata keys: `event_time` (integer epoch seconds, or ISO-8601
`YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive
datetimes are UTC) takes precedence over the document createdTime; an
unparsable value is skipped with a debug log, never an error, and a
zero createdTime leaves the field unset. `session_id` is copied
verbatim and empty when absent. The keys are only consulted when the
extraction loads metadata; otherwise createdTime is the only source.
`parseTopologyEventTime` is public for tests and future callers; the
calendar math is Hinnant's days_from_civil with range-checked fields.
Extraction stats count the event time source and invalid values.
Nothing downstream reads the fields yet. Artifacts, codecs and both
routing fingerprints are unchanged, which the new test pins.
Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp):
parser accept/reject tables, metadata precedence over createdTime,
invalid ignored, session_id, includeMetadata=false fallback, and
byte-identical snapshot payload plus equal construction and road
fingerprints with and without the typed fields.
Signed-off-by: trvon <me@trevon.dev>
* bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar
Every exported corpus document now carries `metadata.event_time`
(ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM`
haystack date, first-seen date per session) and `metadata.session_id`.
Document ids, titles, texts, queries and qrels are unchanged in the
default mode.
`--round-docs` adds one document per session round (`<sid>__r<n>`): a
user turn opens a round and the replies that follow join it; a leading
assistant turn is its own round. Round docs share the session's
`event_time`/`session_id` and add `round_index`; qrels stay at the
session level.
`--full-haystack` also writes `haystacks.jsonl`, one line per question
with `haystack_session_ids` and the normalized `haystack_dates`, so
per-question membership survives the global corpus deduplication
(the 2026-10-10 preflight lost it).
The converter is split into `convert` / `write_outputs`; `--self-test`
runs both on a three-session synthetic fixture without a download and
is the regression check for the default output.
Signed-off-by: trvon <me@trevon.dev>
* bench(retrieval): carry BEIR corpus metadata onto ingested documents
`retrieval_quality_bench` ingests the BEIR corpus as a directory, which
cannot carry per-file metadata, so `event_time`, `session_id` and
`round_index` from corpus.jsonl never reached the topology extractor.
`BEIRDocument::metadata` now holds the corpus line's `metadata` object
(strings as is, numbers and booleans stringified; both the generic and
the SciFact loader read it), and `applyBenchmarkDocumentMetadata`
copies it onto the stored documents matched by exact file path through
the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can
repeat it). The bench runs it after the ingest wait, for cold and
reused corpora alike, and logs how many documents were updated or not
found.
Tests: loader parses and coerces metadata; apply helper writes the
keys, skips documents without metadata, counts missing ones and is
idempotent against a real MetadataRepository.
The portability allowlist entries for the two touched bench files are
re-pointed at their shifted lines; no raw environment read was added.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Let the vector stage follow cluster roads from the selected route
clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the
snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of
both stored orientations of every road, per source sorted by weight
desc then target cluster id asc; it is empty when the batch carries no
roads, whether never built or dropped with `roadUnavailableReason`.
The session runs a beam after the certificate is materialized: for
`roadHops` hops it scores every unvisited road neighbour of the
frontier by road weight times the router's query-centroid similarity
((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster
id asc) and dedupes across hops. Each centroid evaluation is cached per
cluster and counted once in `routeRepresentativeDistanceEvaluations`
and `routeExactRepresentativeDistanceEvaluations`, so routed work stays
comparable with the no-road arm. The result lands only in
`TopologyRoutingSessionResult::roadExpandedClusterIds` and
`roadExpandedDocumentHashGroups` (one member group per cluster,
excluding certificate members). `certificate.allowedDocumentHashes`,
admission and `topologyRoutingPolicyFingerprint` are untouched; the
fingerprint comment and a test pin that the road knobs stay out of it.
`VectorStageRequest` carries the road groups with their own quotas
(`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly
and Augment query them after the certified groups; the certified part
of the routed result is identical with roads on or off. Shadow and
Narrow never see road groups, and `makeTopologyRoutingOptions` requests
hops only for Augment and a gate-open RouteOnly, so a gated-off
RouteOnly running as Shadow takes no roads with it.
Config: `search.topology.road_hops` (default 0, off) and
`search.topology.road_beam_width` (default 4, read when hops > 0) on
`SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`,
copied by `applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy` and applied by
SearchEngineManager. Defaults leave every stage byte-identical: the PR 2
goldens (692) and the RouteOnly cells (417) are unchanged.
Debug keys appear only when hops > 0: `topology_road_hops_applied`,
`topology_road_clusters_added`, `topology_road_candidates_added` (road
documents the stage appended), `topology_road_work_centroid_evaluations`,
`topology_road_skip_reason` (roads_absent, no_selected_routes,
route_low_confidence) and, under stage trace,
`topology_road_expanded_cluster_ids`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH
write the two TOML keys.
Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp
(new catch2_search_vector_stage_road_beam): a four-cluster fixture with
hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A;
adjacency construction and cache carry; fingerprint equality for hops
0 vs 2 and any beam width; session-level certificate invariance with
hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0
equals the route-only cell with no road keys, hops 1 gives A u B, hops 2
with beam width 1 gives A u B u C via the heavier road, work counters
grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and
Narrow take no road groups. ConfigResolver covers both TOML keys.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads are added, the bench reads its two keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
* test(search): vector stage completes into its own state on component timeout
Executor-backed SearchEngine with a vector hydration step that blocks until the
test releases it. With a small componentTimeout the search returns without the
vector component; the test then releases the worker and runs a second search
from the same call site, so the second search's locals reuse the first frame's
stack region. A stage that still writes into the first frame races with the
second search (TSan data race) until the stage state is heap-owned by the
posted work. A second case pins that a component meeting its deadline is still
collected with its diagnostics.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): vector-stage lifetime on component timeout
The posted vector work still reached into the search frame after a
componentTimeout: its injected query wrapper captured `this`, workingConfig
and four frame-local relaxed-retry atomics, and the frame adopted the stage
outcome unconditionally after collectIf, even when the component had timed out
and the worker was still running (a data race on the outcome, and a stale write
into whatever next occupies that stack region once the frame returns).
Give the wrapper a heap home: VectorStageState {SearchEngineConfig config;
relaxedRetryEnabled; RelaxedVectorRetryStats relaxedRetry; VectorStageOutcome
outcome} is captured by shared_ptr in the posted lambda and the query wrapper is
built from it by makeQueryVectorWithRelaxedRetry, which holds the repository
and vector database by value, so the work depends on neither the frame nor the
engine. The sequential path uses the same factory bound to frame locals; the
relaxed-retry atomics become a plain struct only the frame thread touches.
collectAndRecord now returns the ComponentStatus. The frame adopts the stage
state only when the component finished (Success or Failed); after TimedOut or
Canceled the late completion lands in state only the worker references and is
dropped with it, so the diagnostics merge and route-work application are
no-ops. Behavior with no timeout is unchanged.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): move route work helpers to the vector pipeline
Pure move of evaluateTopologyRouteWork, observeTopologyRouteWork and
allocateBalancedRouteQuotas from the search_engine.cpp anonymous
namespace into search_vector_pipeline_internal.h so the vector candidate
stage can be extracted into its own translation unit. The portability
allowlist line numbers follow the move.
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): extract runVectorCandidateStage
Move the fanout.vector lambda body out of SearchEngine::searchWithResponse
into src/search/search_vector_stage.{h,cpp}. VectorStageRequest carries
the policy flags, allowed sets and groups (shared_ptr<const>, aliasing a
certificate snapshot), quotas, limits, backend capabilities and the
injected relaxed-retry query function. VectorStageOutcome carries the
results, every counter the lambda used to write through captured
references, the observed route work and separate globalDiag / routedDiag.
The posted work owns its outcome through a shared_ptr captured by value
and the frame adopts it after the component is collected; the relaxed
retry closure is still the frame's. SearchEngine merges globalDiag into
the reported vector diagnostics, merges routedDiag only when the policy
is not Shadow (Shadow's routed query is counterfactual and feeds route
work only), and applies the observed work to the session certificate
after adoption. Query order inside the stage is unchanged, so the
last-writer relaxed_vector_primary_hit_count stays identical.
No behaviour change: the vector stage characterization goldens are the
acceptance test.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Pin the current behaviour of the vector candidate stage in
search_engine.cpp before it is extracted into search_vector_stage.{h,cpp}.
A new catch2_search_vector_stage binary runs the matrix policy
{Shadow, Augment, Narrow commit, Narrow fallback} x backend
{SimeonPqAdc, ExactScan, Vec0L2} with stage trace off and on, plus one
executor-backed fanout case, and compares each cell against explicit
goldens: fused hash/rank/score and vector contribution, raw vector
ComponentResults (trace_component_hits_json), the stage's debug keys
without timings, certificate.work / admission.work via the
topology_route_work_* and admission keys, and the merged vector_search_*
diagnostics. Quirks are pinned as-is: relaxed_vector_primary_hit_count is
last-writer, Vec0 keeps global results unchanged under Shadow/Augment,
Narrow commits only on Simeon PQ because ExactScan and Vec0 never observe
an ANN candidate budget (work_observation_unavailable), and Vec0 leaks a
partial ann_candidate_budget_actual into the unobserved work record. A
hidden [.dump] case regenerates the goldens.
Move the shared topology fixtures (TopologySearchFixture,
buildTwoClusterTopologyBatch, topologyRoutingTestConfig,
runTopologySearch, configureCertifiedTopologyRoute, TopologyVectorEnvPin)
into tests/unit/search/topology_search_test_support.h so both tests use
the same corpus. runTopologySearch takes an optional executor and
configureCertifiedTopologyRoute an options struct; existing call sites are
unchanged. The portability allowlist entry for the raw YAMS_TEST_TMPDIR
read moves with the helper. No product behaviour changes.
Signed-off-by: trvon <me@trevon.dev>
SessionService reads the current session from XDG_STATE_HOME. On a build
host whose user had an auto-created session ("auto: /data") with watch
enabled, every test that booted a ServiceManager inherited it, the session
watcher walked /data during init, and six daemon tests timed out. With the
state dir isolated the same six pass.
The shared test main now points XDG_STATE_HOME at a per-process temp dir
when the caller did not set one, and removes it at exit. A small test checks
the process sees no host session.
Signed-off-by: trvon <me@trevon.dev>
* feat(profiling): Tracy zones on p2p and memory-sync background paths
Zone the memory-sync cycle (publish, sync, reconcile, apply, backfill,
delete-outbox drain), the p2p reconnect cycle, accept loop, per-session
worker and frame read, and the delta send/apply/bootstrap stages. Plot the
retained session count and pending erase intents. No behavior change; the
macros are no-ops without Tracy.
Signed-off-by: trvon <me@trevon.dev>
* test(infra): per-test-case CPU and peak RSS accounting for Catch2 binaries
A Catch2 listener in the shared test main records getrusage (GetProcessTimes
on Windows) CPU per case and prints the five heaviest cases plus the run's
peak RSS to stderr at run end. YAMS_TEST_RUSAGE=1 adds one line per case.
The measurement helpers are header-only and unit tested.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make vector_database.cpp compile with Tracy enabled
The Profiling lane did not build: eight YAMS_PLOT sites in
vector_database.cpp used -> on a Result<T>, and two passed an int that is
ambiguous between Tracy's PlotData overloads. Use .value() and int64_t
casts. No behavior change outside TRACY_ENABLE builds.
Signed-off-by: trvon <me@trevon.dev>
* build: enable Tracy for every yams library in the Profiling lane
Define yams_tracy_args/yams_tracy_deps once in the root meson.build
(Tracy found, optional rocprofiler hooks, link check) and apply them to
every library and executable under src/, replacing the per-library copies
in search, topology, vector, daemon and tests/benchmarks. Empty when
enable-profiling is off.
Make YAMS_PLOT accept any numeric type and fix a Result deref in
search_service so the newly instrumented TUs compile with Tracy.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
getStats ran the aggregate query on every call. The result is now cached
with the connection's total_changes() and data_version and reused while
both are unchanged; any write, restore, or copy invalidates it. A failed
generation probe disables reuse rather than the aggregate fallback.
Tests: reference_counter (generation reuse, invalidation, restore and copy paths).
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover --topology-clusters --scope-cwd outside src/include
Follow-up to #280. --topology-clusters --scope-cwd still built its
scope from <cwd>/src and <cwd>/include, so a corpus laid out any other
way had no scoped members:
- The scope path set must hold every stored path under the cwd, also
when the cwd is reached through a symlink, and not a sibling that
shares the prefix.
- GraphTopologySupport must build the set from its own metadata repo.
The existing cluster-scope case built its path set by hand from the
old src/include contract. It now builds the set from /repo, so
/repo/tests/ is in scope and a document outside /repo is not.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): scope --topology-clusters --scope-cwd to the whole cwd
Follow-up to #280. Topology views built their --scope-cwd path set from
<cwd>/src and <cwd>/include only, a leftover from the code-symbol graph
removed in v0.20. Corpora laid out any other way reported no scoped
members.
Build the set from the same roots as the --list-type ranges: the cwd,
lexical and symlink-resolved. Keep only paths really under a root, since
the prefix query matches with LIKE. The topology view no longer prunes
build/, packages and IDE paths, so it matches --list-type.
GraphTopologySupport now builds the set from its own metadata repo
instead of the CLI app context. Remove buildGraphCodeScopePathSet and
buildGraphCurrentScopePathSet, which have no other callers, and use one
"Scoped to paths under the current directory" note for both views.
Signed-off-by: trvon <me@trevon.dev>
* test(cli): cover graph --name on a document with a path:file: node
Follow-up to #280. yams graph --name <path> tried path:file:<path>
node keys before the document lookup. v0.20 ingestion writes
path:logical: and path:<snapshot>: keys, so the probe misses on a
fresh corpus; after `yams doctor repair --graph` (or NL entity
extraction) the key exists, and --name printed a raw traversal of the
file node instead of the document's related documents.
Smoke cases through the in-process CLI:
- --name on a file with a path:file: node returns its hash and its
semantic neighbours (red).
- A relative --name resolves under the cwd even though a newer copy
with the same relative path exists in another checkout.
- A bare file name still resolves by name.
The portability allowlist entry for this file moves with its getenv
line.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): resolve graph --name through the document lookup only
Follow-up to #280. yams graph --name first traversed path:file:<path>
node keys. v0.20 ingestion writes path:logical:<path> and
path:<snapshot>:<path>, which only chain to blob versions; semantic
neighbours hang off doc:<hash>. On a fresh corpus the probe always
missed. After `yams doctor repair --graph` or NL entity extraction it
hit and printed a raw file-node traversal instead of the related
documents.
Drop the probe and go straight to the document lookup, which starts at
doc:<hash> since #280. Resolving to path:logical: instead would show
the version chain, not related documents; --node-key still traverses
any node.
Send --name as the path ingestion stores: relative to the client cwd,
lexically normalized, symlinks resolved. The daemon resolved relative
names against its own cwd and fell back to a suffix match, which picks
the newest copy across checkouts. Fall back to the name as given when
it does not resolve, so bare file names and suffixes still work.
build_graph_file_node_candidates had no other callers; its tests are
replaced by tests for buildGraphDocumentNameCandidates.
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover --limit on the document graph lookup
Follow-up to #280. yams graph <hash> and --name ignored --limit: the
daemon capped the graph query at 100 nodes and the same-directory
fallback at 20 documents.
Smoke cases through the in-process CLI:
- --limit 1 returns one related document by name and by hash, and the
first of the uncapped list.
- --limit 1 caps the same-directory fallback for a document without
graph edges.
- Without --limit the related lists are unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): pass --limit through the document graph lookup
Follow-up to #280. yams graph <hash> and --name sent --depth but not
--limit; DocumentService fixed the graph query at 100 nodes and the
same-directory fallback at 20 documents.
Add GetRequest.graph_limit (proto field 27) and
RetrieveDocumentRequest::graphLimit, and pass them from the CLI through
the dispatcher. A limit caps the related list, graph neighbours, path
history and the same-directory fallback alike. The graph query never
asks for fewer than 100 nodes, since path and blob nodes count toward
that limit.
The CLI sends a limit only when --limit is given; 0 keeps the old caps,
so default output is unchanged. The binary GetRequest layout appends
the field and still reads payloads without it.
Unit tests cover the proto round trip, the dispatcher mapping and the
DocumentService caps.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(vector): pin vec0 ranking to cosine order on non-unit rows
vec0 ranks by L2 distance, which matches cosine order only for unit
vectors. Insert rows whose L2 order disagrees with cosine order and
assert vec0 returns cosine order, agrees with the exact cosine scan,
skips zero rows and rejects a zero query. Pins a vector-enabled env
so the cases run under the CI test env.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): rank vec0 by cosine on unit-normalized copies
vec0 ranks by L2 distance, which matches cosine order only when both
sides are unit length. Stored rows reach the vectors table at whatever
magnitude the provider, config, import or sync produced, and the vec0
aux table copied them as-is, so vec0 selected and ordered rows
differently from the Simeon PQ and exact cosine paths.
The vec0 table is a derived index rebuilt from vectors.embedding, so
write the unit direction into it and leave the source blob, which
export/sync and exact scoring read, untouched. Normalize the query the
same way. Zero rows are left out of vec0 and never match; a zero or
non-finite query is rejected as the exact scan rejects it. Returned
rows are ordered by their cosine score with the PQ tie-break.
No schema change. The aux table is rebuilt on first use in each
process (ready dims are in-memory), so existing non-unit rows are
normalized on the next rebuild without a repair step.
Signed-off-by: trvon <me@trevon.dev>
* test(topology): pin the k-means stored centroid to the unit direction
Spherical k-means assigns against unit centroids but stored the shorter
arithmetic mean. Assert the stored centroid is unit length along the
member mean direction, with and without boundary spill. Add a SOAR case
showing that with unit centroids the spill follows angular distance,
and that a short mean centroid reorders the candidates.
Reformat the touched k-means test file to the lint formatter.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): store the unit k-means centroid
The k-means engine clusters by cosine against normalized centroids,
then stored the unnormalized member mean as centroidEmbedding. Routing
reads the centroid by cosine and does not care, but SOAR boundary spill
measures Euclidean residuals against it. A loose cluster's short mean
adds a radial term to those residuals, so spill candidates were ranked
in a different space from the one the clusters were built and routed
in.
Store the unit mean direction for k-means, which is the converged
Lloyd centroid. Connected-component and Louvain keep the arithmetic
mean, as their tests and contract require. Correct the stale field
comment.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
A CompressedStorageEngine wrapping an engine without a base path used a
fixed /tmp/yams-compressed-wrapper as its own base, and the StorageEngine
base creates objects/ and temp/ there. With the daemon now running as a
systemd user unit for each user (UMask=0077), whichever user's daemon
started first owned that directory, and every other user's daemon with
compression enabled (the yams init default) failed at startup with
"Failed to create storage directory: Permission denied" and never became
ready. Found by the package validation: two lingering users, reboot, the
second daemon stuck initializing.
Suffix the scratch directory with the real UID on POSIX.
Signed-off-by: trvon <me@trevon.dev>
The packaged service becomes a systemd user unit that runs each user's
own daemon on the CLI's default per-user socket, with that user's
config and data. There is no system daemon left to discover, and the
daemon always runs as the same user as the CLI, so both workarounds for
the system-service design go:
- Revert "feat(cli): find the packaged system daemon socket without
configuration" (client-side /run/yams/yams-daemon.sock discovery, the
systemd refusal in `daemon stop|restart`, the spawn refusal and the
status mismatch suppression, plus its tests).
- Revert "fix(cli): send file content to a daemon that runs as another
user" (peer-UID probe and content upload in `yams add`).
Per-user socket resolution is back to: explicit flag/env/config, then
$XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback).
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin cached compiled CoreML models directly in the model
directory. With the plugin now able to run on either the system or the
bundled ONNX Runtime, a cache compiled by one version was reused by the
other: on macOS, after a run on Homebrew's 1.30 the bundled 1.23 failed
every embedding batch with 'Non-zero status code returned while running
..._CoreML_... node'. The default cache is now
<model dir>/coreml-ort-<runtime version>; YAMS_COREML_CACHE_DIR still
overrides it.
Verified on macOS arm64: a run on the system runtime creates
coreml-ort-1.30.0, a run with Homebrew's copy hidden uses the bundled
runtime, creates coreml-ort-1.23.0 and embeds new documents without
CoreML errors.
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
`yams add <path>` passes the path to the daemon, which reads the file
itself. The packaged system daemon runs as its own account with
ProtectHome= and PrivateTmp=, so adding ~/notes.txt failed with
"Permission denied" and a /tmp file was accepted but never stored (the
daemon's /tmp is private).
When SO_PEERCRED/getpeereid shows the daemon's UID differs from ours,
the CLI now reads single files itself and sends their bytes (up to
15 MiB, under the 16 MiB IPC frame) named by their absolute path.
Directory adds stay daemon-side walks (include/exclude patterns and
ignore files), so for such a daemon they now fail up front with an
explanation instead of queueing a walk that finds nothing. Same-user
daemons keep the path-based flow.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(init): align fresh-install embedding dimensions and vector sentinel
A fresh `yams init` emitted dimension-conflict warnings and created a
vector store whose dimension did not match the configured embedding model:
Embedding policy: embedding dimension conflict:
embeddings.embedding_dim='1024' overrides compatibility key
vector_database.embedding_dim='384' (and vector_index.dimension='384')
Root causes:
- getLatestConfigDefaults() hardcoded vector_database.embedding_dim and
vector_index.dimension to 384 while [embeddings] used 1024 for the
default Simeon backend, so the generated config was self-inconsistent.
- init resolved the vector dimension only by scanning the optional ONNX
model list, leaving it at 0 for the default "simeon-default", and never
wrote vectors_sentinel.json.
- updateV2Config searched for [vector_database] keys (enabled/model/
model_path) that do not exist in the v3 schema and never updated the
dimension keys.
Fixes:
- Default vector_database.embedding_dim and vector_index.dimension to 1024.
- Add resolveModelDimension() and stamp vectors_sentinel.json from the
vector DB's effective dimension only when this init created the store,
so a forced re-init cannot overwrite a real stored dimension.
- Rewrite updateV2Config to set enable/embedding_dim for [vector_database],
enable/preferred_model/backend/embedding_dim for [embeddings], and
dimension for [vector_index] consistently; disable the enable flags when
semantic search is declined.
- Non-interactive init now defaults to semantic search enabled, matching
the interactive default.
- Keep compatibility dimension keys aligned during v1/v2 migration so an
existing 384-d install never receives a 1024-d sibling default.
* fix(init): prompt for the tuning profile once
`execute()` asked for the tuning profile before the idempotency check and
`handleAlreadyInitialized()` asked again for an already-initialized instance,
so `yams init` showed the tuning prompt twice. Collect the profile once on the
fresh path (after the idempotency check); the already-initialized path keeps
its single prompt.
Add interactive regression tests with a scripted stdin helper:
- already-initialized init prompts for tuning exactly once and persists the
chosen profile;
- declining semantic search disables both `[vector_database]` and
`[embeddings]`, writes no dimension sentinel, and still prompts for tuning
once (also covers the fresh interactive path).
---------
Co-authored-by: trvon <git@trevon.dev>
Full isolation still let get_runtime_dir() consume the ambient XDG_RUNTIME_DIR, so yams-daemon.status.json landed outside the temp root. Relocate XDG_RUNTIME_DIR (POSIX) and APPDATA/LOCALAPPDATA/USERPROFILE (Windows) under the run root, and scrub YAMS_EMBED_BACKEND, YAMS_PREFERRED_MODEL, YAMS_RERANKER_MODEL and YAMS_EMBED_DIM, which ConfigResolver ranks above TOML, so the generated config is deterministic. Apply the same to ingestion_e2e_bench and extend the isolation test with a decoy runtime dir and model overrides.
Signed-off-by: trvon <me@trevon.dev>
repair_ability_bench built its harness without any isolation, so the daemon
resolved config from ambient YAMS_CONFIG, ~/.config/yams and HOME and could
index real files. multi_client_ingestion_bench isolated state but not config,
HOME or the XDG cache/data dirs, and ingestion_e2e_bench left HOME untouched.
Use DaemonHarnessOptions::isolateEnvironment in repair_ability_bench (with an
explicit simeon embeddings config so the embed fault kind stays repairable)
and both multi_client_ingestion_bench option builders; an explicit
YAMS_BENCH_CONFIG still wins. In ingestion_e2e_bench set HOME and
XDG_DATA_HOME to the fixture root and unset YAMS_STORAGE and the daemon
socket aliases. Shift the affected portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the pre-September upload code, which passed
--force, while the workflow still pins wrangler 4.69.0. That version's
'r2 object put' has no --force option (it overwrites without prompting)
and exits with 'Unknown argument: force', so the v0.20.3 release failed
its first R2 upload and no apt, yum or Arch repository was published.
Immutable objects are still checked with 'get' before any write.
Signed-off-by: trvon <me@trevon.dev>
WranglerObjectStore passes str(Path("/wrangler")) to the command, which
is \\wrangler on Windows, so the Windows lane failed the literal
"/wrangler" comparison.
Signed-off-by: trvon <me@trevon.dev>
When the daemon holds background work back for a busy host, a user
looking at a stale topology or an unfinished repair scan needs to see
why. Status had no host signal at all.
Carry the governor's host sample through the metrics snapshot and
StatusResponse: source (psi, loadavg), whether the host counts as busy,
the PSI cpu/io/memory "some avg10" values or the per-CPU load average,
and which deferrable work is currently held back. The proto transport
piggybacks the fields on request_counts like the other string status
fields; the native serializer appends them as optional tail fields.
`yams daemon status` (brief and detailed) shows a Host Load row, e.g.
"Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and
`yams status --json` gains a host_pressure object. Document the new
[tuning.resource] keys next to the other typed tuning sections.
Signed-off-by: trvon <me@trevon.dev>
Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
v0.20 stopped producing code-symbol nodes and calls/references/
inherits/implements edges, and migration 40 deleted the existing ones.
GraphContextService::impact() and affectedTests() had already been
reduced to returning a "code-symbol graph was removed" warning, so
`yams graph --impact`, `--affected-tests` and `--test-pattern` (and the
MCP graph actions `impact` and `affected_tests`) could only ever print
that warning.
Remove the dead surface:
- CLI: the three flags, their execute/render helpers, and the --impact
mention in the --global help text.
- MCP: the two actions, their schema entries and the changed_files /
test_pattern request fields. Unknown graph actions now fail with
InvalidArgument instead of silently running a default query.
- Service: impact()/affectedTests() and their request/response structs.
The GraphImpact/GraphAffectedTests IPC messages stay for wire
compatibility. The dispatcher now answers them directly with the same
removal warning, without touching the metadata repository.
Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph
node labels and aliases and walk edges of any relation, so they still
work on document, path, version and entity graphs.
Tests: the service-level impact/affected-tests case moves to the
dispatcher graph suite; the sandbox smoke test now expects --impact to
be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and
blast-radius questions at `yams grep`.
Signed-off-by: trvon <me@trevon.dev>
Mode::None is the initial value of mode_ and means "no subcommand
parsed". execute() only reached it through a catch-all default, which
also hid any future Mode enumerator that someone forgot to dispatch.
Give None its own case (same InvalidArgument error as before) and drop
the default so -Wswitch flags an unhandled mode at compile time. The
enumerator stays: removing it would make an unparsed command silently
default to Init.
Add a CLI test that `yams session` with no subcommand is rejected; the
None path stays a defensive guard because require_subcommand() stops
that case at parse time.
Signed-off-by: trvon <me@trevon.dev>
#216 and #220 each shifted the two reviewed getenv reads in
RequestDispatcher.cpp and each updated the allowlist on its own; merged
together the entries point at the old lines and the portability check fails.
Signed-off-by: trvon <me@trevon.dev>
~/.local/state/yams/sessions grows without bound: tool integrations
create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files
out of 1,764 on one machine) and nothing ever removes them.
Add expireIdleSessions() next to the session store and run it from
RepairService's idle maintenance once a day, only while no client is
connected. A session file is deleted only when nobody has created,
opened, closed or modified it for [daemon.maintenance]
session_expiry_days (typed config, default 30, 0 disables), and it is
kept when it is:
- the current session,
- watching a directory that still exists,
- still tagged on documents (they stay reachable through `session
merge` / `discard`; without a metadata repository ownership is
unknown and nothing expires),
- not valid JSON (not ours to judge).
Tests build a store shaped like the audited one and check each rule,
plus the RepairService wiring against a real metadata repository and
that a connected client defers the scan.
Signed-off-by: trvon <me@trevon.dev>
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:
- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
the audited machine) that `yams init` / `yams config grammar` built
and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
every start with two warnings (the ABI host's skip plus the plugin
manager's "load failed"), and the refusal flipped the plugin host FSM
through Failed.
Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.
The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.
Signed-off-by: trvon <me@trevon.dev>
During a search storm on a large corpus, daemon shutdown blocked in
Phase 6.7 (SearchEngineManager::clearEngine) until the 120 s watchdog
forced an exit. Dropping the engine destroys SimeonLexicalBackend, whose
destructor joins the BM25 build thread, and the build only looked at its
stop token once every 1024 documents, each needing a metadata fetch.
Stop was also requested only from that destructor, so the build had no
head start while the earlier shutdown phases ran.
SimeonLexicalBackend::requestStop() now flags the build without joining
it and makes later buildAsync() calls no-ops. The build checks the flag
(alongside the jthread token) before every document fetch in the corpus
and fragment passes and before fragment geometry. SearchEngine and
SearchEngineManager expose requestShutdown(), and ServiceManager calls
it in Phase 4, so the build thread has normally exited by the time
clearEngine() destroys the engine.
simeon's own finalize, concept-mining and PMI-learning steps take no
stop token, so a build already inside one of them still runs it to
completion; making those interruptible needs a change in the simeon
submodule.
Signed-off-by: trvon <me@trevon.dev>
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.
Shutdown now tells in-flight work to stop before joining:
- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
on every leased connection under the pool mutex, which keeps each
lease open while it is interrupted. DatabaseManager exposes it for
both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
join wait slice through a new joinWithTimeout hook, because SQLite
drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
before the interrupt, and publishes it as AppContext::cancellationSignal.
DocumentService::list and GrepService::grep check it between query
and hydration steps and between document batches, and return
OperationCancelled; SearchService checks it on entry and forwards it
as SearchExecutionContext::hostCancellationSignal, which the engine's
existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
componentTimeout is 0, and a canceled search stops waiting for the
query embedding / concept tasks. Those tasks now own copies of their
inputs instead of capturing the caller's stack, so abandoning them is
safe.
Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.
Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.
Signed-off-by: trvon <me@trevon.dev>
