Since v0.7.3 'yams get' has requested compressed transfers
(acceptCompressed), and the daemon has answered with a framed payload: a
CompressionHeader followed by a Zstandard body. Only the MCP server
decoded it; 'yams get' wrote the frame itself to stdout or -o, so users
saw the 'CNRK' header and a zstd frame instead of the document. 'yams cat'
was unaffected because it does not request compression.
- Move the frame format into compression/framed_payload
(encodeFramedPayload/decodeFramedPayload). DocumentService encodes with
it; 'yams get' and MCP decode with it.
- The decoder checks the header, body size and CRC, and the decoded size
and CRC. A payload that fails is an error; MCP used to fall back to
returning the framed bytes as content.
Verified end to end: 'yams get' output (stdout and -o) is byte-identical
to the added file while the transfer stays compressed (51.75x).
Signed-off-by: trvon <me@trevon.dev>
Every store added two metadata rows per snapshot to the document,
snapshot_id:<id> and snapshot_time:<id>, and nothing removed them.
Content-addressed documents are shared across paths and passes, so they
piled up: on one 93 GB database they were 43.1M of 44.0M metadata rows,
one file carried 52,367 snapshots, and 'yams list' timed out hydrating
2.69M rows for its 100 documents. Meanwhile findDocumentsBySnapshot read
only the latest snapshot_id, so a re-stored document dropped out of every
earlier snapshot.
- Migration 42 creates document_snapshots(document_id, snapshot_id,
snapshot_time), keyed on (document_id, snapshot_id). It only creates the
table, so startup stays fast.
- The insert transaction records membership; stores keep only the latest
snapshot_id/snapshot_time keys.
- findDocumentsBySnapshot and file history read membership, falling back
to legacy keys through one shared parser (legacySnapshotEntries).
- getMetadataForDocuments skips legacy keys via idx_metadata_doc_key: on
the 93 GB database, 0.21 s instead of 18.5 s for list's 100 documents.
- RepairService moves legacy keys in the background, 1000 snapshots per
batch, 200 ms apart, only below Warning pressure.
Signed-off-by: trvon <me@trevon.dev>
Read and write pools share the slow-holder threshold, but the warning
always said 'slow write-connection hold'. A slow read-only 'yams list'
(tag client_list, 42 s) was reported as a write hold, which pointed the
investigation at a nonexistent write. Say read or write from the pool's
readOnly config.
Signed-off-by: trvon <me@trevon.dev>
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon
resolves the Simeon scoring mode from reranker policy (ConfigResolver), so
the stat said 0 for every outer-MaxSim run and the scifact ablation could
not tell which scorer ran. setCrossReranker() now takes the mode label the
wiring resolved; the engine reports it as cross_rerank_scoring_mode and
derives the outer-MaxSim stat from it.
Graph rerank recomputes connected components over mutual
semantic_neighbor edges among the candidates on every query, reading
each candidate's edges from the KG. The resident topology snapshot
already holds corpus-wide clusters built from those edges.
Add search.topology.graph_community_source (reciprocal_edges |
topology_snapshot). topology_snapshot scores candidates that share a
snapshot cluster with the same normalization and falls back to the edge
walk when no snapshot is loaded; graph_community_source in debug stats
reports which ran. The default stays reciprocal_edges until a benchmark
gate shows the snapshot source is at least as good. The retrieval bench
can set it for ablations.
Every kNN edge was also written in reverse, so each semantic_neighbor
edge looked reciprocal and topology's reciprocalOnly filtering, SGC
smoothing over reciprocal edges, and graph reciprocity metrics filtered
nothing. Write only the edge from a document to each of its k nearest
neighbours; a reverse edge now exists exactly when the pair is mutual.
Semantic maintenance pruned one-way edges on the assumption that edges
were symmetric; with directed kNN edges that would erase every
non-mutual neighbour, so it now removes only self-loops (and no longer
reports a fabricated community size for the scoped pass).
Edge reads truncated by insertion order: getEdgesBidirectional sorted by
id and getEdgesTo had no order, so a LIMIT kept arbitrary or oldest
edges. Both now return strongest first, and the topology extractor reads
a document's own neighbours separately from its incoming edges so a
hub's many incoming edges cannot evict them.
Existing databases keep their mirrored edges until the semantic
neighbour graph is rebuilt.
SymbolEnricher boosted results by looking up code-symbol nodes that no
longer exist since v0.20, so it ran KG queries for the top results and
never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the
symbol_rank_* search stats, and the symbolRank option on the service,
retrieval, MCP and CLI request types. The IPC field stays in the wire
format but is ignored, and --symbol-rank/--no-symbol-rank remain
accepted as hidden no-ops so existing scripts keep working.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
Report topology_route_bq_used / topology_route_bq_candidates in search
debug stats so benchmarks can tell BQ-shortlisted routing apart from
ANN and exhaustive routing. Let retrieval_quality_bench write
bq_candidate_limit, bq_prefix_dim, sgc_hops and
simeon_rerank_outer_maxsim into its per-run config from YAMS_BENCH_*
harness settings, read through getenv_optional.
storeBatch wrote per-document keys before the snapshot node, so a
failure in between left documents pointing at a snapshot that was never
persisted. Write the snapshot node first and the latest pointer last so
a partial store leaves the previous snapshot authoritative.
Remove only the two per-document keys still written, in one transaction
via a new removeMetadataBatch, and stop the membership slow path from
reading structural keys that migration 41 prunes.
The metadata store cached the latest batch in an unguarded mutable
optional that search threads read and wrote, and handed out a full copy
on every loadLatest, so search kept the snapshot resident twice.
Publish the latest batch and its membership index as one immutable
shared snapshot under a mutex, add loadLatestShared to hand it out
without copying, and feed the routing snapshot cache from it. Historical
loads no longer displace the resident latest snapshot, and a reader that
raced a write never installs the older batch.
- Use v39 metadata DB template for legacy symbol metadata test cases in
metadata_schema_catch2_test.cpp and kg_store_catch2_test.cpp to avoid
running against schema v40 where symbol tables are dropped.
- Use yams::test::make_temp_sqlite_path instead of raw getenv in
metadata_schema_catch2_test.cpp and prune the stale portability allowlist entry.
- Add site directory to SKIP_PARTS in check_repository_metadata.py to prevent
local docs builds from triggering retired marker warnings.
Signed-off-by: trvon <me@trevon.dev>
- Add Migration 41 to drop and recreate metadata_value_counts triggers with
'WHEN ... AND key NOT LIKE topology.%' condition, eliminating ~650,000 trigger
executions per topology rebuild in a 72k document corpus.
- Purge legacy continuous float/structural keys from metadata table
(persistence_score, cohesion_score, bridge_score, role, cluster_level,
parent_cluster_id, overlap_cluster_ids_json) in Migration 41.
- In MetadataKgTopologyArtifactStore::storeBatch, replace N individual
getDocumentByHash calls with single batchGetDocumentsByHash and write
only essential snapshot_id and cluster_id to metadata table.
- In MetadataKgTopologyArtifactStore::loadMemberships, add in-memory fast
path serving memberships directly from cached latest snapshot batch.
- Update serializeTopologyBatchCompressed to handle uncompressed tiny batches.
- Add Catch2 unit tests in tests/unit/topology/topology_metadata_store_catch2_test.cpp
validating trigger bypass, metadata pruning, and fast-path membership loading.
Signed-off-by: trvon <me@trevon.dev>
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py.
- Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile.
- Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests.
- Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test
fixtures to run symbol-specific test cases against schema v39.
Signed-off-by: trvon <me@trevon.dev>
Remove legacy Tree-sitter symbol extractor plugin (plugins/symbol_extractor_treesitter),
the unused plugin-symbols build option, and associated tests and benchmark targets:
- Drop plugins/symbol_extractor_treesitter/ (grammar_loader, symbol_extractor, plugin_api).
- Remove plugin-symbols Meson option and packaging references.
- Remove dedicated symbol plugin tests and benchmark targets (symbol_extraction_bench,
symbol_extractor_all_langs_catch2_test, symbol_extractor_edge_cases_catch2_test,
symbol_extractor_solidity_catch2_test, symbol_extractor_query_test,
symbol_extractor_validation_test).
- Clean up dead symbol extraction plugin helper and test from hnsw_entity_search_bench.
- Prune obsolete portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
Drop legacy Tree-sitter symbol extraction subsystem, associated database
tables, triggers, and AST edge types:
- Migration 40 drops symbol_metadata, document_symbol_extraction_state,
symbol_metadata_fts, associated triggers, and prunes historical bloated
topology snapshots while preserving active snapshot topology,
semantic_neighbor edges, and non-symbol observed_as provenance edges.
Exclude nl_entity:% nodes from kg_nodes / kg_edges pruning to preserve
canonicalized method-type technology entities.
Provides complete downSQL restoring symbol tables, indexes, FTS virtual table,
and sync triggers for clean v39 rollbacks.
- In EntityGraphService, ensure locally resolved documentDbId propagates to
WriteBatch for knowledgeGraphDocumentId to ensure accurate enrichment
token completion, and only populate batch->documentIdToDelete for snapshot
documents with a resolved document database ID.
- In SymbolEnricher and SearchService, restore one-directional symbol-contains-query
matching with non-empty checks to prevent short substrings matching broad queries.
- In tests/unit/metadata/symbol_metadata_schema_test.cpp, migrate test DB up to
version 39 so tests pass against their target version prior to v40 table drops.
- Remove Tree-sitter grammar download and compile workflows from CLI init
- Default plugin-symbols to false.
- Fix macOS daemon client and CLI executable resolution using
_NSGetExecutablePath with candidate fallbacks.
- Add migration validation tests and symbol enrichment unit tests.
Signed-off-by: trvon <me@trevon.dev>
InternalEventBus::get_or_create_channel returns an existing channel for a name and silently
discards the requested capacity, so the first requester sizes a channel for the whole process.
Several call sites depend on that silence, and five of them survived an earlier fix precisely
because nothing reported it.
Record each distinct (name, actual, requested) disagreement and warn once. The warning is emitted
outside the lock so a sink cannot re-enter the bus while mu_ is held. remove_channel now drops a
name's records too, so the list describes live disagreements and an assertion cannot become
order-dependent. The spdlog include is guarded with __has_include, matching
pressure_limited_poller.h, since this is an installed public header.
The recorded capacity is stored beside the type-erased channel pointer rather than read back through
static_pointer_cast<SpscQueue<T>>. A name is NOT guaranteed to map to one T: "model.events" is
registered as SpscQueue<ModelReadyEvent> at plugins/onnx/model_provider.cpp:1076 and as
SpscQueue<ModelLoadFailedEvent> at :1089/:1266/:1503, so reading capacity() through the cast would be
a wrong-type member access on a reachable production path. The cast itself is still unvalidated, but
that risk is unchanged from what callers already accepted for a doubly-registered name.
On the test side the drain helpers no longer pin a capacity: they drain and remove, so each case's
own declared capacity applies. The two embed_jobs sites that want a small ring drain first, so they
actually receive 2048 instead of silently inheriting a larger registered ring, and a
ScopedChannelRemoval guard drops that ring at scope exit. The throughput case asserts that no
post_ingest disagreement was recorded, guarding the drain-then-remove behaviour its ring depends on.
Evidence: with the pre-fix pinning helper restored, the diagnostic reports "channel 'post_ingest'
already exists with capacity 32 but 64 was requested" and the new assertion fails (10 assertions, 1
failed); with the fix the case passes 10/10 and the binary is 307/307. The 4-batch outcome itself is
load-dependent and did not reproduce on this machine, so the guard's value is that it detects the
precondition deterministically rather than waiting for a loaded runner.
Two defects in an earlier revision of this commit were found by the entry-review workflow (Gemini
3.8 Flash + Claude Opus, adjudicated) before the push, and are fixed here: one embed_jobs site never
established the 2048 its comment claimed, so the comment and this message were false; and the
capacity read went through the cast while claiming a one-T-per-name invariant that is already false
in production.
This does not reduce the mismatch count in this binary (9 -> 10 -> 9 distinct after these fixes, with
the false direction gone). The
remaining records are deliberate test-local rings plus production disagreements that need separate
fixes: embed_jobs (8192 vs 1000) and post_ingest (1000 vs 4096 in RepairService). The diagnostic also
surfaced a family that was not previously known: PostIngestQueue derives symbol_extraction,
entity_extraction, title_extraction and kg_jobs from its per-instance capacity_ via
boundedStageChannelCapacity, while those channels are process-wide singletons, so the
first-constructed queue sizes them for every later queue (32 vs 512 and 32 vs 64 both observed).
Signed-off-by: trvon <me@trevon.dev>
The case "keeps multi-doc batches when extraction concurrency is low"
asserts an exact batch count (extractionBatches == 3) and produced 4 on a
loaded macOS runner (16+16+15+1).
The binding constraint was not the coalescing window: the drain helper
created the shared post_ingest channel with capacity 32 before any queue
was constructed, and get_or_create_channel ignores the requested capacity
on a name hit, so the queue declared capacity 64 but got a 31-slot ring.
With 48 tasks the producer had to block on the channel until the poller
drained, which made batch boundaries a function of runner load.
Drain and then drop the channel so each case capacity applies, and add the
bus remove_channel() that makes that possible. The exact-count assertions
are kept, not relaxed.
Signed-off-by: trvon <me@trevon.dev>
Directory enumeration builds descendant paths internally, so a short
root can still produce child paths past MAX_PATH and fail partway
through the walk. Run list() and getStats() iteration in the extended
namespace unconditionally on Windows while keeping key derivation on
de-prefixed paths.
Found by the hosted Windows lane: the long-key listing case failed at
REQUIRE(listed) with the backend returning a filesystem error.
Signed-off-by: trvon <me@trevon.dev>
Add the extended Win32 prefix per filesystem operation and only when
the path reaches MAX_PATH, instead of rewriting the whole storage root.
Short paths keep their ordinary spelling, so listing, canonicalisation
and iteration semantics are unchanged.
Derive list() keys lexically from de-prefixed entries, and canonicalise
containment in the extended namespace but compare it in the ordinary one,
so prefixed and plain spellings are never mixed.
Keep the existing publication regression and cover drive/UNC spelling
plus long-key CRUD.
Signed-off-by: trvon <me@trevon.dev>
PostIngestQueue::processed() counts prepared tasks before their content is
committed, so waiting on it and then reading repository state is racy.
Wait on the committed observable in the three affected cases and
characterize the counter ordering with a gated-insert regression.
Signed-off-by: trvon <me@trevon.dev>
Reserve slots before unlocking for connection creation. Route replacement
and maintenance through the same capacity accounting and wake waiters
when reservations settle.
Reproduce concurrent over-admission with bounded creation barriers.
Reanchor existing portability exceptions without adding exemptions.
Signed-off-by: trvon <me@trevon.dev>
* test(ci): stabilize main matrix (#112)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): preview experimental promotion (#111)
Signed-off-by: trvon <me@trevon.dev>
* ci(release): reconcile draft promotion safely (#113)
* ci(release): reconcile draft promotion
Create promotion visibility independently of release readiness.
Leave existing PRs untouched to preserve concurrent maintainer edits.
Validate against current main without masking candidate rejection.
Signed-off-by: trvon <me@trevon.dev>
* ci: isolate hook Git environments
Prevent temporary Git fixtures from modifying the calling worktree.
Clear repository-local hook variables before the unchanged build gates,
and isolate validator and fixture Git subprocesses from caller overrides.
Exercise linked-worktree config, ref and index preservation offline.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(ci): reject invalid candidate evidence
Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve completion and retrieval contracts
Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence asynchronous enrichment completion
Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage.
Signed-off-by: trvon <me@trevon.dev>
* fix: bound semantic graph output reservation
Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward.
Signed-off-by: trvon <me@trevon.dev>
* fix: fence standalone embedding repair completion
Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass.
Signed-off-by: trvon <me@trevon.dev>
* fix: preserve enrichment failure reporting
Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged.
Signed-off-by: trvon <me@trevon.dev>
* test: align explicit hash prefix boundaries
Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun.
Signed-off-by: trvon <me@trevon.dev>
* test: align UI hash validation boundary
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): one table for repair operation wire codes
The code published under repair_current_operation_code was defined in
three places: a name->code chain in RepairService.cpp, an enum->name
switch beside it, and a code->label switch in the CLI. The two daemon
tables agreed with each other but the CLI copy had already drifted in
wording, and nothing tied the three together.
metric_keys.h now carries the single code<->name table next to the
metric key it describes, with constexpr lookups in both directions.
RepairService's name->code chain delegates to it; the enum->name
switch stays because it is keyed by the private enum. The CLI copy
goes in the next commit with the status renderer.
Signed-off-by: trvon <me@trevon.dev>
* refactor(cli): extract the daemon status renderer
showStatus() in daemon_command.cpp was 1,762 lines: two RPCs, a
bootstrap-file fallback, and two ~400/1,100-line render bodies that
wrote straight to std::cout from inside the command class, reachable
only through a live daemon. The render bodies read nothing from the
command except the expected data directory, the explicit-socket flag,
and a memory-sync fetch they triggered mid-render.
The bodies now live in src/cli/commands/daemon_status_render.cpp as
free functions over StatusResponse plus a small DaemonStatusRenderContext
(configured data dir, explicit-socket flag, optional MemorySyncResponse)
writing to an ostream. The readiness/label helpers that only they used
(humanizeToken, classifyReadiness, isEphemeralDataDir,
daemonDataDirFromStatus, paintStatus, neutralText) move with them.
DaemonCommand keeps the RPCs and the bootstrap fallback; the memory-sync
fetch becomes fetchMemorySyncStatus() and runs just before rendering
instead of in the middle of it, so the section prints in the same place
with the same bounded 3 s timeout.
The move is verbatim: after the mechanical substitutions
(std::cout -> os, expectedDataDir() -> ctx.configuredDataDir, the
memory-sync call, the explicit-socket flag, and the repair label
function) the two bodies match the originals character for character
modulo whitespace. The CLI's private repair-operation switch is gone;
the label now comes from the shared table in metric_keys.h with
underscores rendered as spaces, which reproduces the previous wording
("stuck docs", "block refs", "path tree").
tests/unit/cli/daemon_status_render_catch2_test.cpp drives both
renderers and the memory-sync section from synthetic responses:
overview rows, blockers with init progress, last error, data-dir
mismatch (and its suppression under --socket), temporary data dir,
the active repair operation label, every detailed section, and the
started/not-started memory-sync gate.
daemon_command.cpp: 4,082 -> 2,341 lines; daemon_status_render.cpp:
1,642 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(app): one home for hash predicates, path aliasing, and metadata qualifiers
Three small helpers had grown parallel copies with quietly different
rules:
- "does this look like a hash" existed four times: search_service
(8-64 hex), document_service (6-63), cat_command (6-64), plus a
never-defined utils::looksLikeHash declaration in services.hpp. The
8 vs 6 threshold is deliberate (D3): a bare query token must reach
8 hex characters before it stops being text, while an explicit hash
argument may be 6. yams/common/hash_predicates.h now names the two
predicates apart (looksLikeHashQueryToken, looksLikePartialHashArgument)
over one isHexDigits; document_service's 6..63 bound was deliberate
for the name resolver (a full 64-hex name must not resolve as a prefix
and delete by hash), so its two resolver sites now guard size() != 64
explicitly, as the partial-hash resolution site already did.
- macOS /private aliasing had three incompatible policies: search
appended the alias of /var only, grep canonicalized /var and /tmp
to the /private spelling, and the delete resolver displayed
/private/var as /var. yams/common/fs_utils.h now carries one alias
table ({/var,/tmp} <-> /private/...) with three lexical policies over
it (otherSpelling, canonicalSpelling, displaySpelling) and the
platform-gated wrappers callers use. Declared behavior change, macOS
only: search patterns under /tmp now also match their /private/tmp
spelling, and the resolver displays /private/tmp/... as /tmp/....
- Inline key=value metadata filters were parsed twice: SearchService's
extractStructuredMetadataQuery (exactly one '=', no quoting/grouping
characters, deduplicated by key) and SearchEngine's
parseMetadataFiltersFromQuery (any '=' token, duplicates kept). The
stricter, tested parser moves into yams/search/query_qualifiers.hpp
and both call it. Declared behavior change for the hybrid metadata
component: tokens like name="x" or a=b=c stay in the query instead
of becoming filters, and repeated keys collapse to the last value.
Tests: hash_predicates_catch2_test.cpp pins both thresholds and the
64-character bound; path_alias_catch2_test.cpp pins the alias table,
the three policies, and the platform gating; query_qualifiers_catch2_test.cpp
gains the structured-metadata cases (split, dedupe, malformed tokens,
filters-only). search_service, document_service, grep_service,
service_utilities, and resolve_submodule pass unchanged. Portability
allowlist anchors reflowed for the four files whose lines moved.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): flush status before optional collection
Render useful primary status before waiting for memory-sync details.
Keep both brief and detailed paths behind the tested ordering seam.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): preview-based rerank text and compiled path globs
Cross-rerank built each window entry's text with a full getContent()
read, then kept only the first rerankSnippetMaxChars characters. The
window now hydrates with one batchGetContentPreview() call bounded to
that many characters; buildCrossRerankText takes the preview instead of
the repository. batchGetContentPreview is virtual so a test can count
it: three window results cost one preview batch and zero full reads.
Path filtering compiled a std::regex from the glob on every call, and
the call sits inside per-document loops in pathSearch, the hash-prefix
search, and metadata filtering. GlobMatcher (src/app/services) holds
the compiled form; wildcardMatch() reuses it through a small
per-thread cache. The glob semantics are unchanged and now pinned by a
test: '*' stays within one path segment, '**' crosses, '?' is one
character, regex metacharacters are literal. yams::common::wildcard_match
was considered and rejected because its '*' crosses separators.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): parse edge anchor properties once, lower the path query once
Graph expansion parsed every visited edge's JSON properties with
nlohmann::json::parse to read at most two strings, region and scope,
in three places (relation weighting, seed anchoring, and neighbour
anchoring). Properties strings repeat heavily across edges and most
carry no anchor at all. edgeAnchorHints() now returns immediately when
the text cannot contain either key and otherwise parses each distinct
string once per thread through a small bounded cache. A process-wide
parse counter lets the test pin it: one hundred lookups of the same
anchored string cost one parse, and un-anchored or malformed strings
cost none.
computePathMatchScore lowercased and trimmed the query for every
candidate document; pathSearch now does that once and passes the
lowered query down. The existing path-search tests cover the scoring.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
edgeAnchorHints returns its two short strings by value: the per-thread memo
clears itself at 1024 entries, so a reference into it could dangle across
calls even though today's three callers consume the result immediately.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): preserve hydration failure semantics
Stop reranking before scorer calls or score changes when previews fail.
Commit anchor hints atomically and recognize escaped JSON keys.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* perf(search): load tag filters once per request, not once per document
pathSearch and searchByHashPrefix filtered candidates through
metadataHasTags(), which for every document called getAllMetadata()
(four retries with backoff on failure), concatenated every metadata
key and value into a string, and logged that string at info level.
A path search over a few thousand documents was a few thousand
metadata round trips and a few thousand info lines.
Both paths now load tags for the whole candidate set with one
batchGetDocumentTags() call (the API GraphComponent already used) and
filter with a pure matcher. The legacy fallback that also accepted a
metadata row with key "tag" is gone: nothing writes that form and the
repository's own tag APIs never honoured it, so search and repository
now agree on what a tag is.
A counting repository subclass pins the shape: one batch call and zero
getAllMetadata calls for both the path and the hash-prefix routes.
portability_allowlist.txt: two search_service.cpp anchors reflowed.
Correction to the claim above that nothing writes the legacy `key = "tag"`
form: the repair downloads pass did (tag values "downloaded", "host:<h>",
"scheme:<s>"), and the search fallback was its only reader. That writer now
uses the same `tag:<name>` keys as every other tag writer, so `--tags
downloaded` keeps matching documents the repair pass tagged. And
batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it
an unbounded candidate set; an oversized list used to fail the statement
and, with this change, would have emptied the result instead of degrading
per document.
Signed-off-by: trvon <me@trevon.dev>
* perf(search): hydrate topology routing candidates in batches
The routing session called getDocumentByHash once per hash in three
places on the product routing path: ranked-candidate admission, cluster
expansion, and the graph-neighbor diagnostic trace (memoized, but still
one statement per distinct hash across four stages). Each now issues
one batchGetDocumentsByHash for its whole set, the same API the
topology input extractor already used, and the docLookupMicros timing
covers the batch. Cluster expansion applies the maxDocs cut before
hydrating so no document is fetched only to be discarded.
A session-level test with one seed and eight graph neighbours under a
counting repository fails on the previous code (17 single lookups, no
batch) and passes now (0 single lookups). The full topology suite is
unchanged.
portability_allowlist.txt: one topology test anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): bound hydration and preserve errors
Retain legacy tags and honor per-connection SQLite bind limits.
Keep hash chunks in one read snapshot and release it on failure.
Failed lookups are not missing documents or successful empty searches.
Reject partial route admission and preserve tagged path errors.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): count uncached prepares in Database::CacheStats
Database::prepare() compiles a statement every call; prepareCached()
reuses one. The cache stats reported hits and misses but nothing about
the bypass, so a hot path that re-prepared fixed SQL on every row was
invisible. CacheStats gains uncachedPrepares, incremented under the
existing cache mutex and carried through move construction and
assignment.
This is the measurement primitive for the following hot-path changes:
a test can warm a path, then assert that N more calls add zero uncached
prepares.
portability_allowlist.txt: one database.cpp anchor reflowed.
Signed-off-by: trvon <me@trevon.dev>
* perf(metadata): cache KG statements and batch the scorer's neighbor fetch
Measured with the new uncachedPrepares counter on a single-connection
pool. Before: 64 warmed neighbors() calls plus 32 document lookups cost
306 uncached statement compilations, and one SimpleKGScorer::score()
over eight query terms cost 620. After: 0 and 1.
Three things were compiling SQL per call on the search path:
- The sqlite KG store used Database::prepare() for fixed SQL in
neighbors, getDocumentIdByHash/ByPath, ensureBlobNode,
ensureDocumentNode, ensurePathNode (both lookups), selectKgNodeIdByKey,
getNodeById, resolveAliasExact/Fuzzy, getEdgesFrom/To,
getDocEntitiesForDocument, and searchNodesByLabel. All now use
prepareCached (85 -> 70 raw prepares in the file; the rest are
dynamic IN-list or maintenance queries).
- ConnectionPool::isConnectionValid ran prepare("SELECT 1") on every
acquire and again on every release, so each pooled call paid two
compilations before doing any work. The probe is now cached.
- Database::tableExists compiled its sqlite_master probe every time;
the fuzzy alias resolver calls it per query token.
SimpleKGScorer builds its 1-hop neighbor union with one
getEdgesFromBatch call instead of neighbors() per query node.
kMaxCacheSize rises from 64 to 128: 90 cached call sites now share a
connection when KG and repository work interleave, and the cache
evicts first-found, not least-recently-used.
Tests pin both numbers: the store test asserts zero uncached prepares
across 96 warmed lookups; the scorer test asserts fewer than eight on
a second run (previously one per query node from neighbors() alone).
portability_allowlist.txt: two anchors reflowed.
Signed-off-by: trvon <me@trevon.dev>
* fix(metadata): bound batched edge query parameters
Respect connection-specific SQLite limits and deduplicate chunk inputs.
Preserve directional ranking and a single read snapshot across chunks;
roll back the savepoint on query failure.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays
LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays
gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as
test-only benchmark levers, but tests/scripts/report_config_key_usage.py
shows 45 of them are set by no test, no bench arm, no script, and no
doc: they were levers nobody pulled, each shadowing a SearchEngineConfig
field that benches and tests already set directly. The entire
YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene
note that graph-vector fusion is a parked lever.
Removed (CANDIDATE rows of the report, verbatim):
YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE
YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD
YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION
YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION
YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE
YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS
YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT
YAMS_SEARCH_GRAPH_ENABLE_PATHS
YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY
YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS
YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS
YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K
YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE
YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY
YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL
YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER
YAMS_SEARCH_GRAPH_HOP_DECAY
YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW
YAMS_SEARCH_GRAPH_MAX_HOPS
YAMS_SEARCH_GRAPH_MAX_NEIGHBORS
YAMS_SEARCH_GRAPH_MAX_PATHS
YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE
YAMS_SEARCH_GRAPH_TEXT_WEIGHT
YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION
YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING
YAMS_SEARCH_GRAPH_VECTOR_WEIGHT
YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS
YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY
YAMS_SEARCH_RERANK_TOP_K
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE
YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK
YAMS_SEARCH_TIERED_MIN_CANDIDATES
YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH
YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS
YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE
YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK
YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER
YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS
YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE
Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and
YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of
YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE
(different clamps, later block won when both were set); only the
current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by
two identical blocks and is now read once.
A test builds the environment over a recording lookup and asserts none
of the retired names is ever consulted, and that the surviving names
are consulted exactly once. production_environment_keys.txt shrinks by
45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale.
The search_latency_graph_rerank task in .vscode/tasks.json set two of the
retired overlays; those assignments are dropped so the task does not claim a
configuration the code no longer honours.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): drop unread embeddings.* keys and the no-op toggles
yams config embeddings enable/disable/tune wrote embeddings.auto_generate,
embeddings.generation_delay_ms, and embeddings.batch_size, and the
migrator seeded them (plus embeddings.auto_on_add) into every new
config.toml. No daemon code reads any of them: the post-ingest queue
always dispatches embed jobs when an embed queue exists, and the only
batch size the resolvers read is embeddings.runtime.batch_size. The
toggle was a switch wired to nothing, and yams status advised users to
flip it.
- embeddings enable and disable are removed. enable duplicated
embeddings model <name> once its dead keys are gone; a kept no-op
would be a shim. status no longer prints an auto-generation line and
shows the runtime batch size when set. tune presets write
embeddings.runtime.batch_size and drop the delay.
- config_migration no longer seeds auto_generate, auto_on_add,
generation_delay_ms, or a top-level batch_size.
- ConfigResolver and the compatibility policy honour a legacy
[embeddings] batch_size silently when the runtime key is absent, so
configs in the wild get their value applied without a file rewrite (the
migrator exits early for v3 configs, so a rewrite would never run).
- status and model hints point at embeddings model <name>.
Declared behavior change: `yams config embeddings enable|disable` are
unknown subcommands now.
Declared behavior change: the legacy [embeddings] batch_size was written by
`yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16)
but no resolver ever read it. Honouring it as a fallback applies those numbers
for the first time; the one consumer whose result moves is the doctor embedding
repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32
on a stock config. The embedding service already defaulted to 32.
Signed-off-by: trvon <me@trevon.dev>
* feat(config): add [tuning] keys for the setters only env could reach
Seventeen TuneAdvisor setters were exercised by tests but had no TOML
key, so production could reach them only through a deprecated YAMS_*
overlay: connection slot sizing, the CPU-high threshold, the ONNX
concurrency and reservation knobs, model-eviction thresholds, ingest
worker cap, store/embed channel capacities, work-coordinator threads,
and connection lifetime. AGENTS.md says tuning lives in typed config;
these were half-migrated.
Each gets a [tuning] key resolved through ConfigResolver alongside the
existing ones, with provenance recorded, and each is included in
resetConfiguredOverrides() so a reload that drops a key reverts it
(the three with dedicated reset helpers call those). The environment
names keep working as compatibility inputs; nothing new is added to the
YAMS_* surface.
A table-driven test sets all 17 keys, checks every getter, then reloads
with an empty [tuning] section and checks every value reverted.
docs/architecture/environment-overrides.md lists the keys.
The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds
in (0, 1), connection_lifetime_s at most 86400) keep their default when
handed a value outside the range. The resolver checks the same ranges first
and warns, so an ignored value is never recorded as config provenance; test
covers all three shapes.
Signed-off-by: trvon <me@trevon.dev>
* fix(config): report 0 for [tuning] integer keys instead of ignoring it
For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below
every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back
as "unset", and silently fall through to the default while the resolver
still recorded config provenance for it. applyUint32 now warns and skips
the value: nothing changes for the operator except that the mistake is
visible.
Two keys mean something by 0 and are exempt: post_ingest_threads (0 =
auto, which the config migrator seeds into every config and
`yams config` displays as "(auto)") and connection_lifetime_s (0 = no
recycling; its setter takes a signed sentinel and accepts 0, and the
environment spelling always did). Both keep their provenance and stay
silent, so a stock config produces no warning.
Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and
record no provenance, pool_ipc_max = 48 still applies, and the two
zero-valid keys apply with provenance.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): one embedding-runtime resolver, one tuning baseline
Two resolvers read the same five environment keys and three TOML keys
for the embedding runtime with different error semantics:
ConfigResolver::resolveEmbeddingConfig used the strict typed readers
(warn and keep the default), while resolveEmbeddingRuntimePolicy in
config_resolver_policies.cpp used std::stoull inside a bare catch, so
YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on
the other. The compatibility entry point is now a projection of the
typed resolution onto the EmbeddingRuntimePolicy shape that library
callers (vector, repair) still consume; it lives in ConfigResolver.cpp
because DaemonConfig is declared there. 91 lines of duplicate parsing
go away.
yams doctor tuning carried its own [tuning] baseline table and it had
drifted: worker_poll_ms 75 against the daemon default of 150,
pool_io_max sized from hardware against a default of 32, so
`doctor tuning --apply` wrote defaults as divergent overrides. The
baseline now reads TuneAdvisor's effective values, which is what
"baseline" should mean.
Tests: the compatibility policy rejects 12abc and agrees with the typed
resolver; the doctor proposal equals TuneAdvisor for every key.
Signed-off-by: trvon <me@trevon.dev>
* refactor(config): stop writing the unread search.reranker_model_path
yams init wrote both search.reranker_model (the name the daemon resolves
through ConfigResolver::resolveRerankerModel) and
search.reranker_model_path, a derived on-disk path nothing in the
daemon, search, or vector code reads; yams config search reranker
status then echoed it back as if it mattered. init no longer writes the
path key and config no longer displays it. The branch that erases a
stale reranker_model_path line from an existing config is kept so old
files converge.
Of the three "aliased key clusters" flagged by the audit, this was the
only real drift: the three embedding-dimension keys are already one
centralized alias set in config_helpers.cpp (read any, write all), and
msl_stack_log_warn_bytes is a metrics counter name, not a config key.
portability_allowlist.txt: three init_command.cpp anchors reflowed; no
new environment reads.
The model command's post-download hint still told operators to set
search.reranker_model_path; that line goes too (the two lines above it
already name the commands that configure the reranker).
Signed-off-by: trvon <me@trevon.dev>
* fix(config): validate tuning ranges and pairs
Reject unsupported scalar and inverted paired limits before changing
runtime overrides or provenance. Preserve explicit zero ONNX reservations
with a distinct unset sentinel and avoid false default-backend conflicts.
Cover all 27 integer keys, inherited limits, and reservation reset.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(embedding): complete embeddings by derivation token
cbc4959c added derivation tokens (begin/complete with a per-attempt
generation, invalidated by document_content triggers) but nothing
produced them: beginDocumentEmbeddingDerivation had no callers and the
completion op's derivations field was never written, while every ingest
still paid for the triggers and the per-hash NOT EXISTS guard.
processEmbedJob now mints one token per document it will embed, before
any input snapshot is read, and completes by token instead of by hash.
A document whose content changes while it is being embedded keeps
has_embedding = 0 and is re-derived later, instead of publishing vectors
for text the corpus no longer holds. Repair jobs travel the same
channel, so they get the same guarantee. The recipe binds the
preparation policy, embedding space, provider version, and dimension
through the embeddingDerivationRecipe helper that cbc4959c also left
unused.
Repository: batchBeginDocumentEmbeddingDerivations and
batchCompleteDocumentEmbeddingDerivations do a whole job in one
transaction each (the single-token completion loop in WriteCoordinator
opened one transaction per document). WriteCoordinator uses the batch
form and takes its stats lock once.
A small EmbeddingDerivationLedger maps job hashes to their tokens so
completion is issued for exactly the documents that produced vectors.
Behavior note: a re-embed (skipExisting = false) of an already-embedded
document resets its readiness at mint time; if that job then fails, the
document reads as not embedded until the next attempt. That is the
attempt semantics the token table was built for.
Not included: the legacy hash completion path is now unproduced and is
removed in a follow-up so the coalescer change is reviewed on its own.
tests/scripts/portability_allowlist.txt: 13 EmbeddingService.cpp
line anchors reflowed; no new environment reads.
The startup reconcile (ServiceManager, "Reconciling document_embeddings_status
from vector rows") used to mark every document with vector rows as embedded,
which would have undone the stale-token guarantee on the next daemon start: a
document whose content changed under an in-flight embed has vector rows for the
old text. reconcileDocumentEmbeddingStatusByHashes now leaves a document with an
open derivation (document_embedding_derivations.completed = 0) at has_embedding =
0, so it is re-derived rather than published; the reconciled-count query applies
the same predicate. Test: an open derivation stays unembedded across a reconcile
and flips once its token completes.
Signed-off-by: trvon <me@trevon.dev>
* fix(embedding): require current recipe to skip
Re-derive legacy or mismatched recipes rather than trusting readiness
alone. Treat derivation lookup errors as failed gathering.
Cover service-level recipe changes and legacy readiness, and characterize
atomic batch completion rollback and retry.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): extract auxiliary vector merge
Signed-off-by: trvon <me@trevon.dev>
* fix(search): preserve auxiliary tie behavior
Keep the extracted merge comparator identical to the original score-only sort
and avoid prescribing equal-score order in characterization tests. Qualify the
search detail namespace so the topology test compiles unambiguously.
Signed-off-by: trvon <me@trevon.dev>
* test(search): avoid auxiliary score tie
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
* chore: remove dead post-ingest helper
Remove shouldPrepareSequentially, the sole Fold unused-function finding
confirmed dead by clean ASan and TSan builds. All other 45 findings have
live callers or uses that Fold's bounded model missed and remain intact.
Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run).
Signed-off-by: trvon <me@trevon.dev>
* chore(plugins): document Fold ownership
Document 6 proved memory-leak findings as intentional ownership:
process-lifetime Ort::Env singletons and host-freed C-ABI result data.
Refresh portability allowlist anchors shifted by the comments.
Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run).
Signed-off-by: trvon <me@trevon.dev>
* fix(storage): treat reserve boundary as emergency
Classify available bytes equal to the configured emergency reserve as Emergency, matching the hard-minimum contract and operator message that the reserve has been reached. Add a regression assertion at the exact boundary.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
* feat(storage): expose typed disk pressure policy
Signed-off-by: trvon <git@trevon.dev>
* fix(mcp): preserve embedded stdio buffers
Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order.
* refactor(daemon): scope shared P2P protected-file accessors for cross-TU use
The protected-file accessors (readProtectedP2pPrivateKey,
readProtectedP2pTrustFile, writeExclusiveP2pPrivateKey) are shared by the
P2P identity-key paths and the memory-sync bootstrap. Keep their
platform-specific definitions in p2p_integration.cpp unchanged and scope
them into yams::daemon::service_manager_detail, declared by the private,
non-installed header p2p_protected_file.h. Qualify the internal
callsites. No behavior change.
Prepares moving ServiceManager::initializeMemorySync into the
memory-sync translation unit (Brick audit cohesion note).
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): place initializeMemorySync with memory-sync TU
Brick structural audit of PR #79 flagged that
ServiceManager::initializeMemorySync lives in p2p_integration.cpp while
the rest of the memory-sync surface lives in memory_sync.cpp. Move it —
together with its exclusive helper ensureAuthenticatedDirectStore — into
memory_sync.cpp.
The shared protected-file accessors remain defined in
p2p_integration.cpp (see the preceding service_manager_detail commit);
moved function bodies are unchanged and ServiceManager.h is untouched.
Brick evidence (experimental pre-79 af609a1e vs post-79 4d942793):
ServiceManager.cpp functions 84 -> 59 with 25 moved and 0 vanished;
ServiceManager.cpp outgoing call/import edges 1292 -> 954, distinct
targets 59 -> 41.
Validation: full ninja build green; meson test yams:service_manager,
yams:service_manager_shutdown, yams:service_manager_degraded — 3/3 OK.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): break simeon_embedding_backend.h include cycle with embedding_generator.h
simeon_embedding_backend.h only declares functions taking EmbeddingConfig
by reference, yet included the full embedding_generator.h definition,
forcing every consumer of the simeon backend header to transitively pull
the generator header (and feeding the embedding_generator <->
simeon_embedding_backend 2-cycle reported by the Brick structural survey).
Replace the include with forward declarations for IEmbeddingBackend and
EmbeddingConfig. Consumers that use the full definitions
(simeon_embedding_backend.cpp, search_engine.cpp,
ingestion_throughput_bench.cpp) now include embedding_generator.h
directly.
Brick caveat: the tool's lexical lane cannot see the header-include edge
removal (0.65-confidence interface-dispatch misattribution keeps a
file-level cycle in its graph); the include-cycle break is verified at
source level. Brick survey pre/post artifacts kept in
worktree build/brick/architecture-iter{,-after}.json.
Validation: full ninja build green; meson test yams:vector_submodule,
yams:simeon_model_provider, yams:embedding_lifecycle,
yams:embedding_recovery, yams:service_manager — 5/5 OK.
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector): extract computeCosineSimilarity from VectorDatabase into vector_utils
The pure cosine-similarity helper was stranded as a static member of the
VectorDatabase storage class, forcing sqlite_vec_backend.cpp to include
vector_database.h for a pure math function (Brick deferred lead #2 from
the PR #79 audit loop).
Move the implementation to vector_utils as a free function
(yams::vector::vector_utils::computeCosineSimilarity, semantics copied
verbatim: 0.0 on size mismatch or zero norm), keep
VectorDatabase::computeCosineSimilarity as a forwarding static for the
external caller in doctor dedupe, and switch the four backend callsites
to the free function. sqlite_vec_backend.cpp no longer includes
vector_database.h.
Validation: full ninja build green; meson test vector_submodule (incl.
simeon_backend_catch2_test) and yams:service_manager — green.
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): extract client transport types to break sandbox probe header cycle
ClientTransportMode and ClientConfig moved verbatim from daemon_client.h
into a new shared header client_config.h (Brick deferred lead #3 from the
PR #79 audit loop: daemon_client <-> sandbox_detection type-level cycle).
sandbox_detection.h now depends only on client_config.h; its
implementation adds the explicit daemon_client.h include it needs for
the DaemonClient discovery statics. All existing consumers are
unaffected: daemon_client.h includes client_config.h so ClientConfig
remains visible transitively.
Remaining runtime dependency (sandbox_detection.cpp calling
DaemonClient::resolveSocketPathConfigFirst / isDaemonRunning statics,
which forward to socket_utils and a file-local ping helper) is recorded
as the follow-up for full decoupling; the header-level cycle is broken.
Validation: full ninja build green; meson test yams:service_manager,
yams:multiplex_client, yams:jsonrpc_client, vector_submodule — all green.
Signed-off-by: trvon <me@trevon.dev>
* test: refresh portability allowlist line anchors after include shifts
The Brick-audit include additions (search_engine.cpp,
ingestion_throughput_bench.cpp) and removal (sqlite_vec_backend.cpp)
shifted line numbers, making seven portability allowlist entries stale
(same reviewed raw getenv/path usages, new line numbers). Refresh the
anchors; no code or policy change.
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): include <fstream> for all POSIX /proc probe paths
The /proc readers compile in the non-Windows/non-Apple branch (BSD and
other POSIX targets), not just under __linux__; guard the <fstream>
include with the same condition so those targets compile.
Copilot review finding on PR #80 (src/daemon/components/DaemonMetrics.cpp:7).
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): apply storage.disk_pressure outside applyMemorySync
ConfigResolver::applyMemorySync also applied [storage.disk_pressure],
coupling unrelated config parsing to the memory_sync apply path and
producing a misleading refusal message for invalid disk-pressure
policies. Move the call to its natural site in daemon_main before the
memory_sync apply, with its own accurate refusal message. Same startup
semantics (exit 1 on invalid policy), correct attribution.
Copilot review finding on PR #80 (src/daemon/components/ConfigResolver.cpp:1658).
Signed-off-by: trvon <me@trevon.dev>
* test(daemon): update applyMemorySync no-op expectation after disk-pressure decoupling
applyMemorySync no longer applies [storage.disk_pressure] (moved to its
own call site in daemon_main), so a section set without memory_sync
entries is now a no-op rather than a disk-pressure rejection. The
invalid-disk-pressure rejection is still asserted directly against
applyStorageDiskPressure in the preceding checks.
Caught by the pre-push CI gate (asan+tsan: daemon_component_submodule).
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
Signed-off-by: trvon <me@trevon.dev>
Keep `p2p status` text output machine-parseable by removing ANSI severity
colorization, and stop connect/disconnect/forget from throwing on malformed
daemon responses (safe .value defaults, uint64 counters). Bound handshake and
delta timeouts to kP2pMaxOperationTimeout up front, and ceiling direct
sync_interval_ms to the 5-minute reconnect bound so invalid config fails early
instead of aborting daemon startup. Degrade memory-sync status to peerCount=0
on a transient peer-registry read instead of failing the whole status IPC. Add
an explicit `p2p forget` operation (proto FORGET=9 through dispatcher, manager,
and registry) so operators can rotate a peer's pinned key while preserving
fail-closed enrollment.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
