Startup ordering defects were only observable by waiting minutes on a live
daemon, and the one that aborted it took a live restart loop to find. Give the
startup contracts a deterministic harness that runs in milliseconds.
+ startup_fuzz_support.h drives the real ServiceManagerFsm and the real
DeferralGate with virtual time: legal phase advances, injected failure,
shutdown and terminal reset, idle/busy, pressure/calm, starvation deadlines
and cancellation
+ oracles fail closed on shutdown resurrection without reset, failure-to-ready,
eligibility disagreeing with fsm.isReady(), admission outside the virtual
deadline, and ineligible work touching the admission gate; fuzzed IEEE-754
components must never yield non-finite or out-of-range scores
+ catch2_startup_replay runs named regressions plus 1024 seeded traces per run
and reports the root seed and action offset for replay
+ fuzz_startup builds the same harness for libFuzzer/AFL, with a seed generator
that refuses to overwrite different existing files
- the fuzzing lane no longer defaults to TSan: harnesses link
-fsanitize=fuzzer,address,undefined and the two runtimes cannot share a
process; combining them now fails with an explanation instead of producing a
binary that cannot start
- documented the lane invocation, target naming and the coverage limits,
including that governor call ordering inside the poller is not covered
Scope: contract-valid single-owner schedules. Executor interleavings, late
callbacks, concurrent FSM instances and filesystem/model faults are not modeled.
Signed-off-by: trvon <git@trevon.dev>
Semantic-neighbor maintenance aborted the daemon during startup:
std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine
threshold could exceed 1.0 for a legal pair of identical vectors. The same
work was also admitted while the daemon was still initializing, so the
300-second starvation guard forced a pass before readiness.
+ semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the
existing embed policy header, shared by the streamed and corpus branches;
non-finite vectors or scores produce no neighbor and legal scores are bounded
before ranking and threshold selection
+ startupMaintenanceEligible() requires ServiceManagerState::Ready, not
stopping and idle, and is evaluated before
ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no
longer manufacture missing readiness
+ StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter
line reporting state, phase and phase_elapsed_ms instead of a debug-level
state number; a 53-minute startup previously logged nothing actionable
+ regression through the public backfill and corpus-rebuild APIs using a 64-D
vector whose old float inverse norm rounds its self-cosine above 1
- removed the duplicated inverse-norm and cosine lambdas from EmbeddingService
- renamed __test_setModelProvider/__test_setKgStore to testing_* because they
used reserved identifiers, across 11 files
- re-anchored 13 pre-existing raw-environment allowlist entries to their new
line numbers; no new exception was added
Startup is still blocked by corpus-sized memory-sync scans; this change stops
the abort and names the blocking phase, it does not make startup fast.
Signed-off-by: trvon <git@trevon.dev>
Dispatch explicit committed memory reads on admitted workers, checking the
governor before queueing and before I/O. Require the coordinator instead of
using the synchronous strand fallback; keep cached APIs and wire opcodes
unchanged and preserve task-record validation.
Use the paged content adapter, distinguish typed local admission deferral
from hard errors and retain known deferred totals/ages after incomplete
scans. Forward exact topology totals independently of diagnostic samples
and keep attempted cycles distinct from complete scans and successful apply.
Add real daemon IPC tests for authenticated off-resident reads, pressure
health, exact 70-edge status/retry accounting and authenticated cold document
retrieval through the existing MCP get handler, DaemonClient IPC and CAS.
Reanchor existing dispatcher environment-read entries without new exceptions.
Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader
fault/readiness/P2P convergence and patched resource measurements remain open.
Signed-off-by: trvon <git@trevon.dev>
Use pinned, governed pages for metadata, content, vectors and topology.
Preflight identity/content/model prerequisites before writes and continue
independent records without softening authentication, capacity or I/O errors.
Apply topology deletions, nodes and edges in one real KG batch with bounded
transaction-local reads. Preserve ordinary-read isolation, report exact edge
deferrals independently of bounded diagnostic samples and avoid empty commits.
Retract absent local topology only from exact committed own observations and
conditionally erase the same envelope, including off-resident/stale-winner
regressions. Probe local KG presence with fallible scalar SQL rather than
materializing properties; retain SQL errors as errors, not absent records.
The public C++ KG/batch interfaces gain virtual capabilities. Defaults provide
source-level hard errors only: installed/old-binary ABI compatibility is NOT
established and providers must be rebuilt/reviewed before promotion.
Signed-off-by: trvon <git@trevon.dev>
Expose owner-validated committed views, bounded exclusive-prefix pages and
fallible exact historical payload reads without per-value reconciliation.
Keep cached reads nonblocking and separate from explicit committed I/O.
Add governed adapter indexes, revocable escaped-handle lifetime checks and
typed local pressure/contention deferral. Release the loop lock only across
audited native payload fetches and recheck cancellation/quarantine afterward.
Preserve the process-wide recursive callback serializer while releasing the
lifecycle gate before its drain on both restart and stop. Recheck workers and
callback generations after the gap; retain all cancellation and join guards.
Wire the optional local-yield callback through the config factory alongside
the constructor change.
Add native SDK concurrency, pressure, lifetime, off-resident and callback-order
regressions. Arbitrary blocking backend/callback shutdown is not signed off.
Signed-off-by: trvon <git@trevon.dev>
Spill exclusive native-filesystem sweep observations and committed winners
into bounded SQLite workspaces on the corpus filesystem. Preserve complete
operation/vector fork observations and validate payloads before publishing
an immutable generation; keep resident caches separate from full authority.
Add fallible committed/pinned reads, bounded adapter bookkeeping, cooperative
filesystem traversal, secure scratch allocation/cleanup and pinned lifetime
limits. Retain exact durable erase intents and condition own-record erasure
on the full current envelope, including a post-reconciliation check.
Preserve the existing shared-store, cache, value and scan limits. Reanchor
only the existing filesystem environment-read entries and register SQLite
and layout-independent test hooks in the memory-sync library.
Bootstrap import/finalize/restart, broader ingress/scratch faults and final
SDK ABI/security/performance signoff remain separate work.
Signed-off-by: trvon <git@trevon.dev>
Publish owning data-directory snapshots under a mutex and route bootstrap
and fallback publications through the same synchronization boundary.
Keep polling completion locked through its final notification so shutdown
cannot destroy the condition variable while the notifier still uses it.
Exercise both publication windows with deterministic event-based tests and
use the repository thread-compatibility wrapper.
Preserve Failed in status rendering instead of reporting initialization.
Reanchor the existing metrics environment-read entry; add no exceptions.
This does not close the separate queued-collector and embedding-borrower
lifetime audits or establish universal bounded callback shutdown.
Signed-off-by: trvon <git@trevon.dev>
Remove the bare Linux libtsan lookup that mixes GCC's shared runtime with
Clang's instrumented host. Add an ELF linkage test that requires actual
TSan instrumentation and rejects the other compiler's runtime.
Include cstring explicitly for retrieval-session memory operations.
The regression was exercised against the actual instrumented shared object;
no sanitizer instrumentation, interceptors, or checks are disabled.
Signed-off-by: trvon <git@trevon.dev>
A complete sweep fails closed when its retained state cannot hold the
corpus, returning ResourceExhausted ("complete-sweep candidate budget
exhausted"). That is correct: a truncated sweep must never commit a fork
decision. What was missing is a failure state. Every caller retried
immediately, and each attempt re-listed the whole op store, so a corpus
larger than maxTrackedIdentities pinned a core indefinitely.
Observed on a 65,612-object op store with 24,689 index records against
maxTrackedIdentities=8192: one thread in
publish -> reconcileBeforeWrite -> syncFully -> sync
-> FilesystemBackend::listPage("index/")
-> visitFilesystemKeys -> recursive_directory_iterator
92% of it in kernel time, 7h26m CPU over 7h29m wall in a single run, and
17,486 "backfill domain failed: complete-sweep candidate budget
exhausted" lines. The listing snapshot cache could not help: it never
completes, because the sweep aborts before reaching the end.
Remember the unsatisfiable sweep and gate retries behind bounded
exponential backoff (5s..600s) so one attempt per window replaces one
attempt per caller. A single retry is still admitted, because one
exhaustion can be a store that is about to fit (an erase or quarantine
landing between attempts); the latch starts on consecutive failures and
clears when a sweep completes.
Only the sweep's own budget sites set sweepBoundExhausted_. A
resource-governor denial reports ResourceExhausted too, but it is
transient backpressure and must keep retrying freely, otherwise pressure
that has already lifted suppresses convergence for a whole window.
Fail-closed behaviour is unchanged: the wrapper never converts a failure
into success, nothing commits until a sweep completes, and the memoized
error carries the original code and message. Cancellation is never
memoized, so shutdown is not delayed.
Signed-off-by: trvon <git@trevon.dev>
* test(search,daemon,storage): assert behaviour instead of frozen literals
Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.
Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.
Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.
The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.
Signed-off-by: trvon <git@trevon.dev>
* ci(tests): summarise Meson failures without flooding the log
A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.
Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.
Signed-off-by: trvon <git@trevon.dev>
* build(deb): validate candidate identity, mapping and upgrade ordering
The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.
Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Graph explore matched candidates across the whole knowledge graph by
label, then filtered by scope. For a bare basename with many foreign
matches the global ORDER BY label LIMIT filled every slot before scope
filtering, so the correct in-scope file never appeared. For a single
absolute path, logical, historical and blob representations of the same
file consumed the symbol budget and the reported files.
Seed an explicit single-term file-like target through scoped direct-key
lookup instead of relying on the saturated global pool, and rank the
remaining candidates by query coverage. Collapse whole-file hydration
representatives to one entry per path while keeping every contributing
node identity for relationship collection, so distinct symbols, text
segments and explicit source spans are still emitted. Raw lookup is
unchanged and still returns all nodes.
Truncation accounting now covers clipped snippets and omitted unreadable
sources. Explore hints keep an external target and add --global when the
target resolves outside the current directory; local hints stay scoped.
Tests cover scoped lookup under a saturated foreign label pool, file and
version budget diversity, direct-key seeding, relationship retention
across duplicate file representations, segment provenance, the
relationship cap, disabled relation and zero-symbol budgets, and
external hint scope.
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): bound whole-sweep candidate state
Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision.
Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default.
The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* perf(memory-sync): verify each envelope's signature once
Every publish and every backfill cycle reconciles the whole op store, and
reconciliation authenticated every index envelope it read. On a store with
44k objects that put 73% of the daemon's remaining CPU inside Ed25519
verification, re-checking signatures whose bytes had not changed.
Remember positively verified envelopes for the life of a loop, keyed by the
digest of the whole serialized envelope including the signature, so a swapped
signature is a different key and gets verified. Trust set, corpus, epoch and
signature policy are fixed for a loop, so rebuilding the loop is the
invalidation: key rotation, revocation and allowlist changes all arrive as a
new loop. The set is capped by a new typed limit and cleared on overflow or on
allocation failure, so exhausting it costs a verification, never an
acceptance. Nothing is memoized for verifyDigest, and cache lookup happens
only after the envelope digest has already been checked against its key.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): stop pinning document backfill on an oversized record
The document sweep is an ordered id cursor that only advances after a
document publishes. A record larger than the resident publish limit failed
with InvalidArgument, which is neither of the unrecoverable codes, so the
failure was rethrown and the cursor stayed put. The same document was then
retried every five seconds forever, and every retry re-reconciled and
re-authenticated the whole op store, which is what kept a live daemon at 40%
of a core with its status reporting idle.
Classify the rejection from the serialized record bytes rather than from the
error text, and skip that document so the pass reaches the ones behind it.
The vector sweep already behaves this way. The local record is untouched, so
a corrected value still publishes on a later pass once the store changes.
MetadataSyncAdapter gains a size helper for that classification; it reuses
the same record builder publish() serializes.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(storage): bound filesystem listPage residency
listPage() walked the whole op store and retained every matching key before
slicing one page. At 1025 records and a 512-record page it held 1025 keys
where 513 is the working set, and the retained set grew with the store.
Traverse once into a max-heap holding the smallest eligible limit+1 keys.
The walk itself stays complete, so a traversal error still returns an error
instead of a partial page. Exclusive lexical cursor, ordering, Windows key
decoding and the SIZE_MAX limit stay as they were.
Add a YAMS_TESTING-only observation seam that counts real walks, entries
visited and peak retained keys, so the traversal cost can be measured
instead of inferred. The new traversal shifts two reviewed getenv allowlist
line numbers.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep backfill progress past an oversized vector
A record above the resident limit failed publication, the domain cursor
never advanced, and every later cycle retried the same row. Valid vectors
behind it never reconciled, so one oversized value could pin a domain
indefinitely. The daemon log showed this repeating every ~100 s.
Classify the row only after a publish fails with InvalidArgument and the
same serialized record is independently confirmed to exceed the immutable
limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep
the local copy, and revisit it when a new source generation or service
limit appears. A generic backend InvalidArgument stays retryable, and
healthy rows still serialize only once.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): reuse a bounded complete listing across publishes
Every store invalidated the cached listing pages, so a publish followed by
syncFully walked the whole op store again, once per 512-record page. On the
profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of
it system time, with listPage at 55% of user samples.
Keep a complete key-only prefix snapshot, built only from normal contiguous
pages starting at an empty cursor on the concrete filesystem backend. The
snapshot is capped by key count and key bytes, served only in lexical order,
and dropped as incomplete unless the walk reached the end. Any mutation
invalidates it before the backend call; a successful store or remove then
applies the exact key to matching snapshots, and a failure drops them
because the write may have committed. Initialising the async path disables
cache reuse permanently rather than racing a future.
Cache maintenance is best effort: it runs after the backend result is known
and any allocation failure clears the cache and keeps the committed result.
Opaque cursors are untouched, so only the built-in filesystem backend gets
snapshot treatment.
Active publishes now cost the same number of walks as a warm sweep instead
of four, six and nine at 511, 513 and 1025 records.
Signed-off-by: trvon <git@trevon.dev>
* test(bench): measure memory-sync listing cost across arms
Add a standalone harness that drives a real filesystem op store through
MemorySyncLoop, one scenario per process so no cache state survives a
repeat, and reports wall, user, system and peak RSS per run. It uses
production APIs only, so the same source builds against a baseline and a
candidate tree and neither arm needs a measurement seam ported into it.
The driver seeds one corpus per size, interleaves the two arms inside each
repeat, and writes results.jsonl plus a median report. Enumeration counts
are not re-derived here; the unit suite owns those.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(search): make the final lexical tie-break a strict weak ordering
The final comparator broke near-ties toward the result with less vector
support, and its pairwise |a - b| <= epsilon test was not transitive, which
is undefined behavior for the std::sort/partial_sort callers in
applySearchTopKLimit.
Compare tie buckets floor(score / epsilon) instead, then lexical anchor,
keyword and vector evidence (more first), then doc id. Behavior with the
tie-break off is unchanged. The comparator moves to
final_result_order_internal.h so it can be tested directly.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit ca9b82d6ce9db31b6c9608a7d95b63203649e5a7)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): keep cross-reranked scores above the unreranked tail
applyCrossRerank rewrote the window's scores to a 0..1 min-max blend while
the tail kept raw fused scores. Later whole-list sorts (concept boost,
top-k partial_sort) compared the two scales, so reranked docs could drop
below tail docs or out of the top-k.
Map the blended score back onto the window's original range, with the
floor raised just above the best tail score when needed. The window's
reranked order is unchanged and docTraces.finalScore records the mapped
score.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit d66c6d304ca4d32b943e164b0d7720fc7f865db1)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): judge weak tier-1 queries on the raw bm25 scale
The weak-query test compared the top Text/SimeonText component score with
weakQueryMinTopTextScore, but lexical batches are min-max normalized, so the
top hit is always 1.0. With enough hits every query was strong: the
weak-query vector fanout boost and weak-query topology expansion never ran.
scoreLexicalBatch now also records absoluteLexicalScore, the hit's score on
the existing absolute scale (-bm25 / bm25NormDivisor, same multipliers),
and classifyTier1TextSignal uses it. Thresholds keep their values; with the
default divisor 25, 0.30 is bm25 <= -7.5 and 0.45 is bm25 <= -11.25.
Signed-off-by: trvon <me@trevon.dev>
(cherry picked from commit bc771c5aab1721170fb9e743013ca0efbf048d61)
Signed-off-by: trvon <git@trevon.dev>
* fix(search): restore measured scientific fusion
Give SCIENTIFIC the measured relative Simeon weight of 0.40 after normalizing the existing text/vector pair. The resulting share is 0.285714, matching the benchmark traces. Other profiles remain unchanged.
Keep its one-slot semantic rescue preset instead of raising it to three at the corpus layer. Preserve the existing large-prose recall widening. Cover the preset weights, config propagation, community blend, rich-graph ratios and corpus rescue behavior.
Signed-off-by: trvon <git@trevon.dev>
* fix(search): order non-finite ranking evidence
Treat NaN score/evidence keys as negative infinity and invalid tie epsilon as zero. This keeps comparator equivalence transitive without changing finite-input ordering. Add exhaustive triples covering NaN, infinities and invalid epsilon.
Signed-off-by: trvon <git@trevon.dev>
* docs(search): clarify lexical scoring limits
Distinguish the shared single-hit score conversion from FTS5 BM25 calibration. Note that lexical tie buckets can still reorder numerically separated rerank-window and tail scores. No behavior change.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
aggregateChunkVectorScores collected every chunk score per document even
when the strategy was MAX, which only needs the best record. Skip the
list for MAX; the other strategies are unchanged.
Test: search_vector_pipeline.
Signed-off-by: trvon <me@trevon.dev>
* test(topology): cover incremental rebuilds for louvain and kmeans
An incremental TopologyManager rebuild with the louvain or kmeans engine
publishes a snapshot that holds only the dirty documents. The store then
drops topology.cluster_id from every untouched document. Add a failing
case for both engines, and a regression case that the connected engine
still rebuilds only the seed's prior cluster.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): rebuild the corpus when a dirty region needs it
TopologyManager logged requiresWiderRebuild and then published the
engine's region update anyway. Louvain and KMeans have no incremental
path: their dirty region is the seed set and their update rebuilds only
those documents, so the published snapshot held just the dirty docs and
the store cleared the topology keys of every other document.
Honor the flag: re-extract the full corpus and call buildArtifacts, and
report the run as a full rebuild with one fallback. The connected engine
sets the flag only when a region exceeds its seed or document budget;
under budget its incremental path is unchanged. Over budget it used to
drop seeds past the cap, and now rebuilds the corpus.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(search): rename a road beam local that collides with the Windows far macro
windows.h defines far and near as empty macros, so `const auto far = ...`
in search_vector_stage_road_beam_catch2_test.cpp does not compile with MSVC
(C2513), and the Windows Tests job has failed since the road beam tests
landed. Rename it to deep.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory_sync): skip rescans of an unchanged direct op store
The sync worker rescanned the whole op store every cycle: a full walk of the
sharded objects tree, then a read, hash, decode and signature check of every
index envelope. On an idle 3-node mesh (2 CPUs per node, 5,256 objects) that
cost about 1.1 s of CPU per 5 s cycle on each node, mostly in getdents/openat.
In direct transport the daemon is the op store's only writer. The loop now
counts its own backend writes and reuses the last complete scan until it
writes again or a writer is quarantined. Shared-store and temporary namespaces
keep rescanning every cycle.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory_sync): keep exclusive-backend listings until the loop writes
FilesystemBackend::listPage walks every shard directory of the op store for
any prefix. After the rescan skip, each idle direct-transport cycle still
listed the store twice: replaying ready erases and inspecting the erase
outbox. The loop's write-counting backend now keeps listing pages on an
exclusive backend and drops them on any store, remove or clear.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
The daemon closes a connection that has served one request and then stays
quiet for 250 ms. A pooled client can write its next request as that
happens and read EOF. The streaming transport retries once on a fresh
connection, but the read loop had already passed the EOF to the caller's
onError. SuccessHandler and most other handlers keep that error, so
ping() returned "[ipc:eof] End of file" after the retry got a pong.
cli_responsiveness_under_load failed this way on macOS CI (1 of 4 pings,
one idle close logged).
The transport now holds each attempt's error and forwards it only when it
returns that attempt's result, not when it retries.
Signed-off-by: trvon <me@trevon.dev>
Ingest resolves symlinks before it writes a path (populatePathDerivedFields
calls weakly_canonical), and buildGraphCwdScopePathRanges resolves the scope
the same way. The graph dispatcher fixture wrote node keys and labels from an
unresolved temp_directory_path(). On macOS that is /var/folders/..., a
symlink to /private/var/folders/..., so the scoped listByType sections found
0 nodes there and passed on Linux. The fixture now resolves its temp dir,
like the stored paths it stands in for.
Signed-off-by: trvon <me@trevon.dev>
makeAdjacency filled each neighbor list in PairWeightMap (unordered_map)
order, and the local-move loop took the first best community from an
unordered_map. Both orders differ between libc++ and libstdc++, so the same
corpus produced different communities on macOS and Linux, and
topology_representative_rule failed on macOS. Neighbor lists are now sorted
by index and equal-gain moves go to the smaller community id. The louvain
characterization is regenerated; macOS and Linux now give the same result.
Signed-off-by: trvon <me@trevon.dev>
#296 sets XDG_STATE_HOME before Catch2 starts so each test process gets a
private state directory. The support library links no yams config, so it
calls setenv/_putenv_s directly, and raw_environment_policy has failed the
Tests job on every platform since. Record both sites as reviewed.
Signed-off-by: trvon <me@trevon.dev>
Co-authored-by: trvon <me@trevon.dev>
Benchmark seeding wrote only mutual neighbor pairs. The connected
topology engine keeps every reciprocal edge at or above min_edge_score
inside its component, so with mutual-only seeding no raw neighbor
crosses a cluster boundary and the engine builds no roads. The Eval 2
smoke showed roads_absent on every connected-engine query.
The daemon stores one-way kNN edges (GraphComponent treats them as
ordinary non-mutual neighbors). YAMS_BENCH_SEED_SEMANTIC_DIRECTED=1
seeds them too, as single directed edges. Default off keeps existing
plans unchanged.
The pair loop moves into buildSemanticNeighborSeedEdges in the bench
support library, sorted by (source, target), with a unit test for both
modes. A topology test pins the mutual-only case: no roads on the
connected engine until a one-way edge exists.
Signed-off-by: trvon <me@trevon.dev>
Let RouteOnly and Augment expand every routed hit along its cluster's
walk order. The snapshot cache loader resolves each cluster's
`orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`,
`TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with
`orderKind` None or a permutation that fails `validateMemberPermutation`
resolves to nothing. The store's `orderUnavailableReason` is carried on
the session result and emitted as `topology_order_unavailable_reason`
only when set.
When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex`
after the road beam: the selected routes first, then the road-expanded
clusters, keeping only clusters with an order and counting the rest in
`roadWalkClustersUnordered`. The index borrows its member views from the
snapshot it keeps alive; the certificate is never touched.
The vector stage walks after the certified and road groups. Every routed
hit so far (certified route hits and road-beam hits) anchors +-window
positions along each order it belongs to, bounded to that order
(`walkTopologyRoadOrders`). Anchoring on route hits too means the walk
rescues the ordinal neighbours of the semantically closest members even
with `road_hops` = 0; the plan's "around road hits" reads as every hit a
road-era routed query produced. The walked set, deduplicated against the
hits, is one extra routed query whose quota is its own size (ranking
ordinal neighbours by similarity and truncating would reintroduce the
filter the walk bypasses); its rows and distance evaluations land in the
routed diagnostics like road candidates. Shadow and Narrow never walk.
Config: `search.topology.road_walk_window` (default 0, off) on
`SearchEngineConfig::topologyRoadWalkWindow` (copied by
`applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by
SearchEngineManager. `makeTopologyRoutingOptions` requests the window
only for Augment and a gate-open RouteOnly, like the beam. The window is
excluded from `topologyRoutingPolicyFingerprint` (comment and test).
Defaults leave every stage byte-identical: the PR 2 goldens (692), the
RouteOnly cells (417) and the road beam cells (904) are unchanged.
Debug keys appear only when the window > 0:
`topology_road_walk_window_applied` (0 when nothing was walkable),
`topology_road_walk_anchors`, `topology_road_walk_candidates_added`,
`topology_road_walk_clusters_unordered`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes
`search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and
YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and
`topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3
can set the order kind per arm and build roads at all).
Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp
(new catch2_search_vector_stage_road_walk): two reciprocal chains A
(six members) and B (four) with a road, built with
`TopologyBuildConfig::roadOrder` Time from eventTime so the walk order
differs from the similarity ranking; loader resolution including the
invalid-permutation drop and the cache carry; fingerprint equality for
any window and the option gating per policy; session-level certificate,
beam and work invariance with the index listing routes then road
clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an
unordered cluster counted and skipped, no index without any order, the
order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0
equal to the road beam cell with no walk key, window 1 adding the
immediate order neighbours of all four hits, window 2 the whole
clusters, rows visited growing while router work is unchanged,
certified keys unchanged, unordered B contributing nothing, no order
anywhere equal to the beam cell plus counters, hops 0 walking the
selected route alone; Augment adding the walked documents as rescues;
Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads, the bench reads its keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
Give every cluster with at least two members a walk order:
`ClusterArtifact::orderedMembers`, a uint32 permutation over the
hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each
deterministic with ties resolved to the smaller member hash:
- Time: ascending `TopologyDocumentInput::eventTime`; members without a
time last, by hash.
- Session: grouped by `sessionId`; groups ordered by earliest event time
(groups with no time after those with one), then session id; inside a
group by time then hash; unsessioned members last, by time then hash.
- SimilarityChain: greedy nearest-neighbor tour by cosine over the
members' embeddings, starting from the cluster representative
(`medoid`, else the smallest-hash usable member); members without a
usable embedding last, by hash.
The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML
`topology.road_order` = none|time|session|similarity_chain through the
topology.* resolver block and `TopologyManager::setRoadOrder`, threaded
into the build config like `roadMaxPerCluster`. Orders are computed after
boundary spill so they cover replicas, at full build in every engine and
for the rebuilt clusters of a connected-engine incremental update;
untouched clusters keep theirs (a kind change reaches them on the next
full rebuild, like the representative rule) and `none` clears every
order. Orders are independent of `roadMaxPerCluster`.
Orders persist in the road sidecar, now version 3: after the roads, u32
count, then per ordered cluster u32 cluster-id table index, u32 kind,
u32 length and the permutation as u32 indices. A sidecar without orders
is still written as v2, so roads-only sidecars are byte-identical to the
previous layout; v1 and v2 decode with no orders, and the envelope gains
`member_order_count`. The sidecar is written when the batch has roads or
at least one ordered cluster. On load each order is validated on its own
(`attachTopologyMemberOrders`: cluster exists, kind not none,
permutation covers the members exactly once); an invalid entry drops
only that cluster's order and appends `<token>:<clusterId>` to the new
load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`.
Roads never lose anything to an order error and vice versa.
The main snapshot codec, the routing construction, representation and
road fingerprints never see orders, so the default `none` leaves the
snapshot payload, the sidecar and every fingerprint byte-identical. No
search consumer yet; the road walk lands separately.
Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp
(permutation validator tokens, time order with missing times, session
grouping and group tie-break, hand-computed greedy chain from the medoid
on a five-point fixture with fallback start, unusable members and exact
ties, every engine stores valid permutations of every kind including over
spill replicas, default none byte-identical payload/sidecar/fingerprints,
sidecar v3 round trip plus hand-built v2 bytes and corrupt-section
rejection, per-cluster attach validation with reasons, store round trip
with orders-only sidecars, invalid order dropped with reason while roads
and other orders stay, incremental update keeps untouched orders and
recomputes rebuilt ones), plus resolver and manager setter coverage.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): typed event time and session for topology inputs
Add `eventTime` (epoch seconds, UTC) and `sessionId` to
`TopologyDocumentInput`. The extractor fills them from two documented
metadata keys: `event_time` (integer epoch seconds, or ISO-8601
`YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive
datetimes are UTC) takes precedence over the document createdTime; an
unparsable value is skipped with a debug log, never an error, and a
zero createdTime leaves the field unset. `session_id` is copied
verbatim and empty when absent. The keys are only consulted when the
extraction loads metadata; otherwise createdTime is the only source.
`parseTopologyEventTime` is public for tests and future callers; the
calendar math is Hinnant's days_from_civil with range-checked fields.
Extraction stats count the event time source and invalid values.
Nothing downstream reads the fields yet. Artifacts, codecs and both
routing fingerprints are unchanged, which the new test pins.
Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp):
parser accept/reject tables, metadata precedence over createdTime,
invalid ignored, session_id, includeMetadata=false fallback, and
byte-identical snapshot payload plus equal construction and road
fingerprints with and without the typed fields.
Signed-off-by: trvon <me@trevon.dev>
* bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar
Every exported corpus document now carries `metadata.event_time`
(ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM`
haystack date, first-seen date per session) and `metadata.session_id`.
Document ids, titles, texts, queries and qrels are unchanged in the
default mode.
`--round-docs` adds one document per session round (`<sid>__r<n>`): a
user turn opens a round and the replies that follow join it; a leading
assistant turn is its own round. Round docs share the session's
`event_time`/`session_id` and add `round_index`; qrels stay at the
session level.
`--full-haystack` also writes `haystacks.jsonl`, one line per question
with `haystack_session_ids` and the normalized `haystack_dates`, so
per-question membership survives the global corpus deduplication
(the 2026-10-10 preflight lost it).
The converter is split into `convert` / `write_outputs`; `--self-test`
runs both on a three-session synthetic fixture without a download and
is the regression check for the default output.
Signed-off-by: trvon <me@trevon.dev>
* bench(retrieval): carry BEIR corpus metadata onto ingested documents
`retrieval_quality_bench` ingests the BEIR corpus as a directory, which
cannot carry per-file metadata, so `event_time`, `session_id` and
`round_index` from corpus.jsonl never reached the topology extractor.
`BEIRDocument::metadata` now holds the corpus line's `metadata` object
(strings as is, numbers and booleans stringified; both the generic and
the SciFact loader read it), and `applyBenchmarkDocumentMetadata`
copies it onto the stored documents matched by exact file path through
the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can
repeat it). The bench runs it after the ingest wait, for cold and
reused corpora alike, and logs how many documents were updated or not
found.
Tests: loader parses and coerces metadata; apply helper writes the
keys, skips documents without metadata, counts missing ones and is
idempotent against a real MetadataRepository.
The portability allowlist entries for the two touched bench files are
re-pointed at their shifted lines; no raw environment read was added.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Let the vector stage follow cluster roads from the selected route
clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the
snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of
both stored orientations of every road, per source sorted by weight
desc then target cluster id asc; it is empty when the batch carries no
roads, whether never built or dropped with `roadUnavailableReason`.
The session runs a beam after the certificate is materialized: for
`roadHops` hops it scores every unvisited road neighbour of the
frontier by road weight times the router's query-centroid similarity
((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster
id asc) and dedupes across hops. Each centroid evaluation is cached per
cluster and counted once in `routeRepresentativeDistanceEvaluations`
and `routeExactRepresentativeDistanceEvaluations`, so routed work stays
comparable with the no-road arm. The result lands only in
`TopologyRoutingSessionResult::roadExpandedClusterIds` and
`roadExpandedDocumentHashGroups` (one member group per cluster,
excluding certificate members). `certificate.allowedDocumentHashes`,
admission and `topologyRoutingPolicyFingerprint` are untouched; the
fingerprint comment and a test pin that the road knobs stay out of it.
`VectorStageRequest` carries the road groups with their own quotas
(`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly
and Augment query them after the certified groups; the certified part
of the routed result is identical with roads on or off. Shadow and
Narrow never see road groups, and `makeTopologyRoutingOptions` requests
hops only for Augment and a gate-open RouteOnly, so a gated-off
RouteOnly running as Shadow takes no roads with it.
Config: `search.topology.road_hops` (default 0, off) and
`search.topology.road_beam_width` (default 4, read when hops > 0) on
`SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`,
copied by `applyTopologyPolicyFrom`), resolved through
`ConfigResolver::TopologyRoutingPolicy` and applied by
SearchEngineManager. Defaults leave every stage byte-identical: the PR 2
goldens (692) and the RouteOnly cells (417) are unchanged.
Debug keys appear only when hops > 0: `topology_road_hops_applied`,
`topology_road_clusters_added`, `topology_road_candidates_added` (road
documents the stage appended), `topology_road_work_centroid_evaluations`,
`topology_road_skip_reason` (roads_absent, no_selected_routes,
route_low_confidence) and, under stage trace,
`topology_road_expanded_cluster_ids`.
Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH
write the two TOML keys.
Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp
(new catch2_search_vector_stage_road_beam): a four-cluster fixture with
hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A;
adjacency construction and cache carry; fingerprint equality for hops
0 vs 2 and any beam width; session-level certificate invariance with
hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0
equals the route-only cell with no road keys, hops 1 gives A u B, hops 2
with beam width 1 gives A u B u C via the heavier road, work counters
grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and
Narrow take no road groups. ConfigResolver covers both TOML keys.
The line-keyed portability allowlist is re-keyed for the two shifted
files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw
environment reads are added, the bench reads its two keys through
yams::config::getenv_optional.
Signed-off-by: trvon <me@trevon.dev>
Maintain cluster roads across incremental rebuilds instead of clearing
them. `ConnectedComponentTopologyEngine::updateArtifacts` now keeps the
roads between clusters the update did not touch, drops every road with
an endpoint in a removed or rebuilt cluster (reused ids included), and
recomputes the rebuilt clusters' roads from the dirty region's raw
neighbor lists against the merged memberships, so a region document's
neighbor in an untouched cluster yields that road in both orientations.
Each source then re-ranks its top-K over the union with PR 6's order and
tie-break (weight desc, target id asc).
Only the region's neighbor lists are available during an update, so a
cross-boundary edge is observed from one side; the out-of-region target
is credited with the mirror observation its own list would have added.
That is exact on a symmetric neighbor graph and the stated approximation
otherwise. An untouched source can only re-rank the roads its last full
build kept, so the result is derived, not full.
Provenance on the batch: `roadsIncrementalPasses` counts passes since
the last full accumulation (0 = full roads) and `roadsFullBuildEpoch`
is the `topologyEpoch` of the snapshot that accumulation produced. A
full build leaves both at 0; an update sets the epoch from `existing`
when its roads were full and carries it forward otherwise. The fields
ride in the road sidecar header, now version 2 (u64 epoch, u32 passes
after the snapshot id); version 1 sidecars decode with both at 0, the
envelope format string is unchanged, and the main v1 payload and the
routing construction fingerprint do not see them. The store copies them
both ways. Roads off (`road_max_per_cluster = 0`, the default) still
yields no roads and zero provenance, so default behavior is unchanged.
`topology_build_utils.h` splits `accumulateClusterRoads` into pair-weight
accumulation (with the optional observed-document set that drives
mirroring), orientation and ranking, and adds `deriveIncrementalRoads`
over those pieces; the full-build path passes no observed set and is
unchanged.
Louvain and k-means `updateArtifacts` rebuild the supplied input from
scratch, so their roads are full roads for that batch with zero passes;
the test states and checks that property for them.
Tests: tests/unit/topology/topology_roads_incremental_catch2_test.cpp
(roads off through an update, full-build provenance, kept/retired/
recomputed roads with mirrored weights on a four-cluster graph, top-K
re-ranking over kept and recomputed roads, derivation without prior
roads, a property test over seeded planted-cluster symmetric graphs
with cross-edge rewiring and member moves comparing incremental roads to
a from-scratch build after each of several dirty regions, a tight-top-K
property for rebuilt sources and derived weights, and the alternate
engines), plus sidecar v2 round trip, hand-built v1 decode, future
version rejection and store round trip of the provenance in
topology_road_sidecar_catch2_test.cpp.
Signed-off-by: trvon <me@trevon.dev>
Persist `TopologyArtifactBatch::roads` in a sidecar knowledge-graph node,
`topology:roads:<snapshotId>` of type `topology_road_snapshot`, instead
of growing the main snapshot codec. The sidecar has its own binary magic
('Y','A','M','R'), version 1, and a Zstd JSON envelope with
`format = "topology_roads_zstd_v1"`. The main snapshot stays a plain v1
payload, byte-identical with roads on or off, so memory-sync replicas on
older peers keep decoding it, and an unknown node type is simply never
read by an old store. `deserializeTopologyBatchCompressed` now rejects an
envelope whose `format` it does not know instead of treating it as legacy
raw JSON.
Header layout (little-endian): u32 magic, u32 version, u32 maxPerCluster,
u32 reserved, string snapshotId, u32 cluster-id table count + strings,
u32 road count, then per road u32 from index, u32 to index, f64 weight.
`TopologyArtifactBatch::roadMaxPerCluster` records the bound the builder
applied so the header can carry it; both engines set it.
The store writes the sidecar right after the snapshot node and before
the `topology:snapshot:latest` pointer, so the pointer remains the commit
point: an injected sidecar failure leaves the previous snapshot current.
Retention deletes each pruned snapshot's sidecar with it, and re-storing
a snapshot id without roads removes its stale sidecar. Loading is
optional and soft: a missing sidecar yields empty roads with no error; an
undecodable or invalid one (snapshot mismatch, endpoint not a cluster,
self loop, negative or non-finite weight, more roads per source than the
declared top-K) drops the roads and records
`TopologyArtifactBatch::roadUnavailableReason`, surfaced on the routing
session and as the `topology_road_unavailable_reason` debug key, while
the snapshot still loads and routes.
`topologyRoadFingerprint(batch)` gives roads their own order-independent
identity for cache provenance; the routing construction fingerprint is
unchanged and stays road-free through the persisted path.
Lean (formal/topology/Yams/Topology/Artifacts.lean): `ClusterRoad`,
`WellFormedRoads` (endpoints in clusters, no self loop, non-negative
weight, bounded per source), `constructionIdentity`, and lemmas that the
construction identity and `WellFormedBatch` are independent of roads and
that an empty road layer is well formed. Checked with lake (lean 4.31.0).
Tests: tests/unit/topology/topology_road_sidecar_catch2_test.cpp (codec
round trip, foreign/truncated payload rejection, main payload identical
on/off, validation reasons, store round trip, missing sidecar, every
corrupt-sidecar case with routing still admitted, write-before-pointer,
retention, fingerprints through the persisted path) and the extended
fingerprint invariance test in topology_roads_catch2_test.cpp.
`road_to` knowledge-graph edges are deferred until a consumer needs them.
Signed-off-by: trvon <me@trevon.dev>
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
* test(search): vector stage completes into its own state on component timeout
Executor-backed SearchEngine with a vector hydration step that blocks until the
test releases it. With a small componentTimeout the search returns without the
vector component; the test then releases the worker and runs a second search
from the same call site, so the second search's locals reuse the first frame's
stack region. A stage that still writes into the first frame races with the
second search (TSan data race) until the stage state is heap-owned by the
posted work. A second case pins that a component meeting its deadline is still
collected with its diagnostics.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): vector-stage lifetime on component timeout
The posted vector work still reached into the search frame after a
componentTimeout: its injected query wrapper captured `this`, workingConfig
and four frame-local relaxed-retry atomics, and the frame adopted the stage
outcome unconditionally after collectIf, even when the component had timed out
and the worker was still running (a data race on the outcome, and a stale write
into whatever next occupies that stack region once the frame returns).
Give the wrapper a heap home: VectorStageState {SearchEngineConfig config;
relaxedRetryEnabled; RelaxedVectorRetryStats relaxedRetry; VectorStageOutcome
outcome} is captured by shared_ptr in the posted lambda and the query wrapper is
built from it by makeQueryVectorWithRelaxedRetry, which holds the repository
and vector database by value, so the work depends on neither the frame nor the
engine. The sequential path uses the same factory bound to frame locals; the
relaxed-retry atomics become a plain struct only the frame thread touches.
collectAndRecord now returns the ComponentStatus. The frame adopts the stage
state only when the component finished (Success or Failed); after TimedOut or
Canceled the late completion lands in state only the worker references and is
dropped with it, so the diagnostics merge and route-work application are
no-ops. Behavior with no timeout is unchanged.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): move route work helpers to the vector pipeline
Pure move of evaluateTopologyRouteWork, observeTopologyRouteWork and
allocateBalancedRouteQuotas from the search_engine.cpp anonymous
namespace into search_vector_pipeline_internal.h so the vector candidate
stage can be extracted into its own translation unit. The portability
allowlist line numbers follow the move.
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): extract runVectorCandidateStage
Move the fanout.vector lambda body out of SearchEngine::searchWithResponse
into src/search/search_vector_stage.{h,cpp}. VectorStageRequest carries
the policy flags, allowed sets and groups (shared_ptr<const>, aliasing a
certificate snapshot), quotas, limits, backend capabilities and the
injected relaxed-retry query function. VectorStageOutcome carries the
results, every counter the lambda used to write through captured
references, the observed route work and separate globalDiag / routedDiag.
The posted work owns its outcome through a shared_ptr captured by value
and the frame adopts it after the component is collected; the relaxed
retry closure is still the frame's. SearchEngine merges globalDiag into
the reported vector diagnostics, merges routedDiag only when the policy
is not Shadow (Shadow's routed query is counterfactual and feeds route
work only), and applies the observed work to the session certificate
after adoption. Query order inside the stage is unchanged, so the
last-writer relaxed_vector_primary_hit_count stays identical.
No behaviour change: the vector stage characterization goldens are the
acceptance test.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Pin the current behaviour of the vector candidate stage in
search_engine.cpp before it is extracted into search_vector_stage.{h,cpp}.
A new catch2_search_vector_stage binary runs the matrix policy
{Shadow, Augment, Narrow commit, Narrow fallback} x backend
{SimeonPqAdc, ExactScan, Vec0L2} with stage trace off and on, plus one
executor-backed fanout case, and compares each cell against explicit
goldens: fused hash/rank/score and vector contribution, raw vector
ComponentResults (trace_component_hits_json), the stage's debug keys
without timings, certificate.work / admission.work via the
topology_route_work_* and admission keys, and the merged vector_search_*
diagnostics. Quirks are pinned as-is: relaxed_vector_primary_hit_count is
last-writer, Vec0 keeps global results unchanged under Shadow/Augment,
Narrow commits only on Simeon PQ because ExactScan and Vec0 never observe
an ANN candidate budget (work_observation_unavailable), and Vec0 leaks a
partial ann_candidate_budget_actual into the unobserved work record. A
hidden [.dump] case regenerates the goldens.
Move the shared topology fixtures (TopologySearchFixture,
buildTwoClusterTopologyBatch, topologyRoutingTestConfig,
runTopologySearch, configureCertifiedTopologyRoute, TopologyVectorEnvPin)
into tests/unit/search/topology_search_test_support.h so both tests use
the same corpus. runTopologySearch takes an optional executor and
configureCertifiedTopologyRoute an options struct; existing call sites are
unchanged. The portability allowlist entry for the raw YAMS_TEST_TMPDIR
read moves with the helper. No product behaviour changes.
Signed-off-by: trvon <me@trevon.dev>
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores
The vector and topology catch-up sweeps restart a full pass as soon as
one ends, so an idle node reads and re-checks every node, edge, vector,
and committed topology record forever. On a 3-node mesh limited to 2
CPUs each this held every node at ~24% of a core with nothing changing.
New cases drive the production backfill through ServiceManager:
- after a completed pass with no writes, further cycles serve no sweep
item and take no graph-store connection;
- a node, edge, and vector written after a pass are published by the
next pass while unchanged records keep their committed operation;
- a node deleted after the sweeps went idle is retracted with its edge.
Store seams: KnowledgeGraphStore::changeGeneration() and
VectorDatabase::changeGeneration() move on writes and batch commits,
not on reads.
Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32,
acquisitions 157 == 97); kg_store and vector_smoke do not compile.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): run memory_sync catch-up sweeps only when a store changed
The vector and topology catch-up sweeps in publishMemorySyncBackfill
restarted a full pass as soon as one ended. An idle node therefore read
and re-checked every node, edge, vector, and committed topology record
forever, and every published record cost each peer a delta exchange and
an apply. On a 3-node mesh limited to 2 CPUs each this held every node
at ~24% of a core with nothing changing; on a real corpus the first
pass never ends.
Each store now exposes a change generation: KnowledgeGraphStore counts
write transactions (single writes through the write pool and write
batch commits), VectorDatabase counts inserts, updates, and deletes. A
sweep records the generation a pass began at and, once the pass
completes, does not start another until the generation differs. Writes
behind a running pass's cursor move the generation past the pass's own
reading, so the following pass still covers them; deletes move it too,
so the retraction phase still runs after a rebuild drops records. With
nothing changed, a cycle costs each sweep one atomic read.
No config knob: the gate is a property of the stores, not a tunable.
Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases),
catch2_kg_store, catch2_vector_submodule.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
SessionService reads the current session from XDG_STATE_HOME. On a build
host whose user had an auto-created session ("auto: /data") with watch
enabled, every test that booted a ServiceManager inherited it, the session
watcher walked /data during init, and six daemon tests timed out. With the
state dir isolated the same six pass.
The shared test main now points XDG_STATE_HOME at a per-process temp dir
when the caller did not set one, and removes it at exit. A small test checks
the process sees no host session.
Signed-off-by: trvon <me@trevon.dev>
* feat(profiling): Tracy zones on p2p and memory-sync background paths
Zone the memory-sync cycle (publish, sync, reconcile, apply, backfill,
delete-outbox drain), the p2p reconnect cycle, accept loop, per-session
worker and frame read, and the delta send/apply/bootstrap stages. Plot the
retained session count and pending erase intents. No behavior change; the
macros are no-ops without Tracy.
Signed-off-by: trvon <me@trevon.dev>
* test(infra): per-test-case CPU and peak RSS accounting for Catch2 binaries
A Catch2 listener in the shared test main records getrusage (GetProcessTimes
on Windows) CPU per case and prints the five heaviest cases plus the run's
peak RSS to stderr at run end. YAMS_TEST_RUSAGE=1 adds one line per case.
The measurement helpers are header-only and unit tested.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make vector_database.cpp compile with Tracy enabled
The Profiling lane did not build: eight YAMS_PLOT sites in
vector_database.cpp used -> on a Result<T>, and two passed an int that is
ambiguous between Tracy's PlotData overloads. Use .value() and int64_t
casts. No behavior change outside TRACY_ENABLE builds.
Signed-off-by: trvon <me@trevon.dev>
* build: enable Tracy for every yams library in the Profiling lane
Define yams_tracy_args/yams_tracy_deps once in the root meson.build
(Tracy found, optional rocprofiler hooks, link check) and apply them to
every library and executable under src/, replacing the per-library copies
in search, topology, vector, daemon and tests/benchmarks. Empty when
enable-profiling is off.
Make YAMS_PLOT accept any numeric type and fix a Result deref in
search_service so the newly instrumented TUs compile with Tracy.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
The mesh harness ran nodes with the build host's resources, so a daemon
that pins a small machine's CPU while idle never showed up. `up` and `run`
take --cpus and --memory (docker cpus/mem_limit per node). The new `idle`
command samples docker stats with no client traffic, writes idle.csv and
idle.json, optionally captures top -H of the hottest node, and exits 5
when a node's mean CPU exceeds --max-cpu.
Local/manual tooling; nothing runs in CI.
Signed-off-by: trvon <me@trevon.dev>
getStats ran the aggregate query on every call. The result is now cached
with the connection's total_changes() and data_version and reused while
both are unchanged; any write, restore, or copy invalidates it. A failed
generation probe disables reuse rather than the aggregate fallback.
Tests: reference_counter (generation reuse, invalidation, restore and copy paths).
Signed-off-by: trvon <me@trevon.dev>
A paused or admission-blocked poller slept for the full max idle delay
instead of waiting on its registered timer, so resumeStage, resumeAll,
enabling the knowledge graph, and adding entity or title providers took
effect only after the delay elapsed. The poller now waits through
awaitPressureLimitedPollerWake and the setters signal the stage timer.
Tests: pressure_limited_poller and post_ingest_queue_unit.
Signed-off-by: trvon <me@trevon.dev>
TuningManager treated any open non-health connection as activity, so a
client that kept its socket open held the daemon in busy mode and kept
background scaling up. RequestHandler now counts executing non-health
requests with an RAII guard, SocketServer wires the counter into
DaemonStats, and the idle/busy decisions read it.
Tests: request_handler_status_unary, socket_server_teardown, and two slow
real-service tests (tuning_registry_activity, socket_proxy_request_activity).
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover --topology-clusters --scope-cwd outside src/include
Follow-up to #280. --topology-clusters --scope-cwd still built its
scope from <cwd>/src and <cwd>/include, so a corpus laid out any other
way had no scoped members:
- The scope path set must hold every stored path under the cwd, also
when the cwd is reached through a symlink, and not a sibling that
shares the prefix.
- GraphTopologySupport must build the set from its own metadata repo.
The existing cluster-scope case built its path set by hand from the
old src/include contract. It now builds the set from /repo, so
/repo/tests/ is in scope and a document outside /repo is not.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): scope --topology-clusters --scope-cwd to the whole cwd
Follow-up to #280. Topology views built their --scope-cwd path set from
<cwd>/src and <cwd>/include only, a leftover from the code-symbol graph
removed in v0.20. Corpora laid out any other way reported no scoped
members.
Build the set from the same roots as the --list-type ranges: the cwd,
lexical and symlink-resolved. Keep only paths really under a root, since
the prefix query matches with LIKE. The topology view no longer prunes
build/, packages and IDE paths, so it matches --list-type.
GraphTopologySupport now builds the set from its own metadata repo
instead of the CLI app context. Remove buildGraphCodeScopePathSet and
buildGraphCurrentScopePathSet, which have no other callers, and use one
"Scoped to paths under the current directory" note for both views.
Signed-off-by: trvon <me@trevon.dev>
* test(cli): cover graph --name on a document with a path:file: node
Follow-up to #280. yams graph --name <path> tried path:file:<path>
node keys before the document lookup. v0.20 ingestion writes
path:logical: and path:<snapshot>: keys, so the probe misses on a
fresh corpus; after `yams doctor repair --graph` (or NL entity
extraction) the key exists, and --name printed a raw traversal of the
file node instead of the document's related documents.
Smoke cases through the in-process CLI:
- --name on a file with a path:file: node returns its hash and its
semantic neighbours (red).
- A relative --name resolves under the cwd even though a newer copy
with the same relative path exists in another checkout.
- A bare file name still resolves by name.
The portability allowlist entry for this file moves with its getenv
line.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): resolve graph --name through the document lookup only
Follow-up to #280. yams graph --name first traversed path:file:<path>
node keys. v0.20 ingestion writes path:logical:<path> and
path:<snapshot>:<path>, which only chain to blob versions; semantic
neighbours hang off doc:<hash>. On a fresh corpus the probe always
missed. After `yams doctor repair --graph` or NL entity extraction it
hit and printed a raw file-node traversal instead of the related
documents.
Drop the probe and go straight to the document lookup, which starts at
doc:<hash> since #280. Resolving to path:logical: instead would show
the version chain, not related documents; --node-key still traverses
any node.
Send --name as the path ingestion stores: relative to the client cwd,
lexically normalized, symlinks resolved. The daemon resolved relative
names against its own cwd and fell back to a suffix match, which picks
the newest copy across checkouts. Fall back to the name as given when
it does not resolve, so bare file names and suffixes still work.
build_graph_file_node_candidates had no other callers; its tests are
replaced by tests for buildGraphDocumentNameCandidates.
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover --limit on the document graph lookup
Follow-up to #280. yams graph <hash> and --name ignored --limit: the
daemon capped the graph query at 100 nodes and the same-directory
fallback at 20 documents.
Smoke cases through the in-process CLI:
- --limit 1 returns one related document by name and by hash, and the
first of the uncapped list.
- --limit 1 caps the same-directory fallback for a document without
graph edges.
- Without --limit the related lists are unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): pass --limit through the document graph lookup
Follow-up to #280. yams graph <hash> and --name sent --depth but not
--limit; DocumentService fixed the graph query at 100 nodes and the
same-directory fallback at 20 documents.
Add GetRequest.graph_limit (proto field 27) and
RetrieveDocumentRequest::graphLimit, and pass them from the CLI through
the dispatcher. A limit caps the related list, graph neighbours, path
history and the same-directory fallback alike. The graph query never
asks for fewer than 100 nodes, since path and blob nodes count toward
that limit.
The CLI sends a limit only when --limit is given; 0 keeps the old caps,
so default output is unchanged. The binary GetRequest layout appends
the field and still reads payloads without it.
Unit tests cover the proto round trip, the dispatcher mapping and the
DocumentService caps.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover document lookup and --scope-cwd listing (#280)
Red tests for the two graph CLI bugs in #280:
- GetResponse graph fields and file name must survive the proto wire.
- DocumentService graph lookup by hash and by name must return the
same documents as traversing the doc:<hash> node.
- The document graph JSON must name the document when only name is set.
- KG path-range scoping must match document nodes by label and
snapshot path nodes by $.path.
- listByType --scope-cwd must keep document and path nodes under the
client cwd (including via a symlinked cwd) outside src/ and include/.
The existing code-roots dispatcher case now places its out-of-scope
file outside the scope root, since the scope is the whole cwd.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280)
yams graph <hash> and --name returned graphEnabled=false with no
related documents while --node-key doc:<hash> listed neighbours:
- The GetResponse proto binding dropped graphEnabled, related and
fileName, so socket clients never saw graph data.
- DocumentService started traversal at blob:<hash>, which has no
outgoing edges, and filtered out semantic_neighbor. Start at the
doc:<hash> node and allow semantic_neighbor.
- GraphQueryService did not hydrate documentHash for doc: nodes, so
reached documents were skipped.
- The presenter read only fileName for "document"; fall back to name.
--list-type <t> --scope-cwd returned nothing:
- The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the
whole cwd, lexical and symlink-resolved.
- The KG range predicate ignored document labels and $.path on
snapshot path nodes. Match both.
Signed-off-by: trvon <me@trevon.dev>
* docs(skills): correct graph document lookup examples
--format dot is only rendered for node traversal, so show it with
--node-key doc:<hash>. Describe --scope-cwd as cwd-scoped.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
- Include <version> before testing __cpp_lib_* macros. Without it every
library gate read 0 in a file that included this header first (verified
with Apple Clang 21 and GCC 16 at -std=c++23).
- Detect [[likely]]/[[unlikely]] with __has_cpp_attribute. __cpp_attributes
is 200809L on every compiler, so YAMS_HAS_LIKELY_UNLIKELY was always 0.
- YAMS_HAS_REFLECTION now requires __cpp_impl_reflection and
__cpp_lib_reflection >= 202506L (P2996R13). __cpp_reflection is not a
standard macro.
- Remove YAMS_HAS_PROFILES and YAMS_PROFILE. __cpp_profiles does not
exist and C++26 did not adopt Profiles; nothing outside the test used it.
- YAMS_CPP_VERSION reads _MSVC_LANG when defined.
- get_feature_summary() reports the language mode instead of tying
"C++23" to constexpr vector/string, which are C++20 features.
- Correct the per-gate toolchain notes (flat_map: libstdc++ 15 / libc++ 20 /
MSVC 19.51; move_only_function: not in libc++; expected: Clang 18 with
libstdc++ reports 0) and drop the std::expected migration guidance:
Result<T> stays.
Tests: core_cpp23_features checks each gate against the compiler's own
feature-test macros with the header included first. New
core_cpp23_features_fallback forces every gate to 0 so the fallback paths,
including the magic_numbers runtime path, compile in every lane.
YAMS_CONSTEXPR_IF_SUPPORTED's inline fallback was invalid on a lambda; the
tests now use it on a function.
Signed-off-by: trvon <me@trevon.dev>
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(search): pin augment routed-candidate accounting
Under the augment vector policy a document reachable only through a
topology route reaches the fused results through the candidate-rescue
vector merge, but the routing session and the
topology_weak_query_added_candidates stat report zero additions, because
the product stage always collects route membership and the session skips
the add loop in that mode.
Pin the routed-only fixture (x2 in the route, outside lexical Tier-1 and
the global vector top-1) for augment, and characterize shadow (identical
to topology-disabled) and uncertified narrow (global, no additions).
Signed-off-by: trvon <me@trevon.dev>
* fix(search): count augment route members as added candidates
#57 unified topology onto one evidence path: the product assist stage
always collects route membership, and augment, shadow and narrow all
consume the certificate's allowed set (augment vector-scores it through
the candidate-rescue merge, shadow measures it, narrow filters to it).
The session skips the old add loop in that mode, so under augment
addedCandidates, duplicateCandidates and
topology_weak_query_added_candidates stayed 0 even when routed
documents were rescued into the fused results.
Under augment only, count materialized allowed-set members that are
new to the existing candidate pool as added candidates and the rest as
duplicates. The allowed set, its maxDocs bound, the routing policy
fingerprint, applied and skip reasons are unchanged; narrow and shadow
report no additions as before.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(vector): pin vec0 ranking to cosine order on non-unit rows
vec0 ranks by L2 distance, which matches cosine order only for unit
vectors. Insert rows whose L2 order disagrees with cosine order and
assert vec0 returns cosine order, agrees with the exact cosine scan,
skips zero rows and rejects a zero query. Pins a vector-enabled env
so the cases run under the CI test env.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): rank vec0 by cosine on unit-normalized copies
vec0 ranks by L2 distance, which matches cosine order only when both
sides are unit length. Stored rows reach the vectors table at whatever
magnitude the provider, config, import or sync produced, and the vec0
aux table copied them as-is, so vec0 selected and ordered rows
differently from the Simeon PQ and exact cosine paths.
The vec0 table is a derived index rebuilt from vectors.embedding, so
write the unit direction into it and leave the source blob, which
export/sync and exact scoring read, untouched. Normalize the query the
same way. Zero rows are left out of vec0 and never match; a zero or
non-finite query is rejected as the exact scan rejects it. Returned
rows are ordered by their cosine score with the PQ tie-break.
No schema change. The aux table is rebuilt on first use in each
process (ready dims are in-memory), so existing non-unit rows are
normalized on the next rebuild without a repair step.
Signed-off-by: trvon <me@trevon.dev>
* test(topology): pin the k-means stored centroid to the unit direction
Spherical k-means assigns against unit centroids but stored the shorter
arithmetic mean. Assert the stored centroid is unit length along the
member mean direction, with and without boundary spill. Add a SOAR case
showing that with unit centroids the spill follows angular distance,
and that a short mean centroid reorders the candidates.
Reformat the touched k-means test file to the lint formatter.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): store the unit k-means centroid
The k-means engine clusters by cosine against normalized centroids,
then stored the unnormalized member mean as centroidEmbedding. Routing
reads the centroid by cosine and does not care, but SOAR boundary spill
measures Euclidean residuals against it. A loose cluster's short mean
adds a radial term to those residuals, so spill candidates were ranked
in a different space from the one the clusters were built and routed
in.
Store the unit mean direction for k-means, which is the converged
Lloyd centroid. Connected-component and Louvain keep the arithmetic
mean, as their tests and contract require. Correct the stale field
comment.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(wal): pin v1 entry encoding to a Clang-written fixture
store_block_entry_clang_v1.bin is one StoreBlock entry serialized by the
current wal_entry.cpp built with Clang 21, which zero-fills the 4 bytes of
trailing Header padding. The new cases decode and verify it, re-encode the
same logical entry and require identical bytes, and round-trip fresh
entries through serialize/deserialize/verifyChecksum.
GCC 16 leaves that padding uninitialized in stableHeaderBytes(), so these
cases fail there.
Signed-off-by: trvon <me@trevon.dev>
* fix(wal): encode entry header field by field, never via padding
stableHeaderBytes() built the checksummed header with `Header stable{}`
plus member assignments and then memcpy'd sizeof(Header) bytes. That is
aggregate initialization, so the 4 bytes of trailing padding (offsets
44..47) are not guaranteed to be zero. Clang 21 memsets the object first;
GCC 16 at -O0 stores only the members, so stack garbage reached both the
serialized entry and the CRC, and verifyChecksum() failed on GCC builds.
Encode and decode the header field by field into a zero-filled 48-byte
array at the existing offsets, with static_asserts pinning every offset
and size. The checksum is CRC32 over that canonical encoding plus the
payload, which is exactly what Clang builds have always written, so their
WAL files verify unchanged. deserialize() also accepts an entry whose CRC
covers nonzero padding as stored on disk (written by earlier GCC builds)
and normalizes its in-memory checksum. WALFile and WALManager recovery
read headers through the new WALEntry::decodeHeader().
TransactionData (uint64 + uint32) has 4 bytes of trailing padding and was
copied whole into the payload, relying on the same unguaranteed zeroing
(GCC 16 and Clang 21 happen to zero it today); it is now encoded field by
field too. The remaining payload structs are padding-free, which
appendObjectBytes/readObject now enforce with
has_unique_object_representations_v.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
