With -Denable-mobile-bindings=true, libyams_mobile no longer compiled:
- diff/cat called IDocumentService::retrieveDocument, which was replaced
by retrieve(); they now fetch stored bytes through retrieve() with
acceptCompressed left false, so content comes back uncompressed, and
map service errors through map_error_code instead of NOT_FOUND.
- repair awaited DaemonClient::call() as a future and read response
fields that no longer exist; it now goes through daemon_call() and
folds RepairResponse::operationResults into the C result.
- YAMS_PROFILE_ZONE is not a defined macro; use YAMS_ZONE_SCOPED_N.
- doctor built its version with std::string(char); emit JSON instead.
- yams_mobile_string_view_destroy, _cat_result_destroy and
_restore_result_destroy were declared but never defined.
The repair/diff/cat/restore/model/doctor declarations sat outside the
extern "C" block and inside #ifdef __cplusplus, so C and FFI hosts could
not see them and C++ builds exported mangled names. Move them into the
C block.
Daemon-mode contexts handed their own worker_pool executor to the
process-wide connection pool keyed by socket path. The pool outlived
yams_mobile_context_destroy, so a second context reused sockets bound to
a freed thread pool (ASAN heap-use-after-free in AsioConnection) and the
daemon was left with connections that never drained, which then tripped
a use-after-free in daemon shutdown. Contexts now use single-use
connections and disconnect the client before joining the worker pool.
Tests: add an embedded cat/diff test (exact bytes, line diff, identical
docs, not-found mapping) and a daemon test that creates and destroys two
contexts and checks the server returns to its baseline connection count.
This proves the Linux x86_64 host build and host-side behavior only. It
does not show that an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
v0.20 stopped producing code-symbol nodes and calls/references/
inherits/implements edges, and migration 40 deleted the existing ones.
GraphContextService::impact() and affectedTests() had already been
reduced to returning a "code-symbol graph was removed" warning, so
`yams graph --impact`, `--affected-tests` and `--test-pattern` (and the
MCP graph actions `impact` and `affected_tests`) could only ever print
that warning.
Remove the dead surface:
- CLI: the three flags, their execute/render helpers, and the --impact
mention in the --global help text.
- MCP: the two actions, their schema entries and the changed_files /
test_pattern request fields. Unknown graph actions now fail with
InvalidArgument instead of silently running a default query.
- Service: impact()/affectedTests() and their request/response structs.
The GraphImpact/GraphAffectedTests IPC messages stay for wire
compatibility. The dispatcher now answers them directly with the same
removal warning, without touching the metadata repository.
Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph
node labels and aliases and walk edges of any relation, so they still
work on document, path, version and entity graphs.
Tests: the service-level impact/affected-tests case moves to the
dispatcher graph suite; the sandbox smoke test now expects --impact to
be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and
blast-radius questions at `yams grep`.
Signed-off-by: trvon <me@trevon.dev>
ClientConfig::acceptCompressed was written by RetrievalService's config
mapping and compared in the CLI client-pool equality operator, but no
DaemonClient or transport code ever read it, so it could not change
behavior. Compression is negotiated per request through
GetRequest::acceptCompressed, which RetrievalOptions::acceptCompressed
still feeds via makeGetRequest.
Drop the field, the mapping, the equality term (configs that differed
only in this flag now share a pooled client, which is what their
behavior already was), and the mapping-test assertion. The per-request
flag and its tests are unchanged.
Signed-off-by: trvon <me@trevon.dev>
RetrievalService::cat had no callers anywhere in the tree (CLI, MCP,
services, tests, benchmarks). Content reads go through get/getByName*
instead. Remove the method and its declaration.
The daemon CatRequest/CatResponse protocol messages stay: DaemonClient::cat
is still exercised by the daemon socket integration test and the
multi-client ingestion benchmark, the dispatcher still handles it, and
removing the message would break wire compatibility with older clients.
Signed-off-by: trvon <me@trevon.dev>
YamsDaemon::stop() stopped the SocketServer and destroyed it while the
IOCoordinator threads were still running, and only then stopped and
destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for
connections to drain, and the handle_connection coroutine frames keep a
raw pointer to the server in their cleanup guard. A connection that
outlived the drain window either resumed on an I/O thread after the
server was freed, or had its frame destroyed together with the
io_context, and both paths decremented the connection counters of the
freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The
ServiceManager-init failure path stopped the server the same way and
left the I/O threads running.
Tear the IPC layer down in one place, teardownSocketServer(): stop the
server (which still releases its acceptors and accept strand while the
io_context runs), stop and join the I/O threads so no handler can
resume, destroy the IOCoordinator so the io_context destroys the pending
handler frames while the server is valid, and only then destroy the
server. All daemon stop and startup-failure paths use it.
The unit test stops the io_context under the connection handlers so
they are deterministically pending when the teardown runs; the daemon
integration test parks the I/O threads past the drain window.
Signed-off-by: trvon <me@trevon.dev>
SocketServer::stop() waits at most ~2s for connection handlers to drain
and then frees the connection slot semaphores. A proxy connection that is
still draining releases its slot from a guard in the handler coroutine
frame, so when that frame later resumes, or is destroyed with the
io_context, it writes to the freed semaphore (ASAN heap-use-after-free in
SemaphoreGuard::~SemaphoreGuard).
Hold the semaphores through shared_ptr and give each handler frame its
own reference. The handler is now a captureless coroutine lambda whose
arguments are copied into the frame, so it no longer reads captures from
a closure that lives only for one accept-loop iteration.
Signed-off-by: trvon <me@trevon.dev>
With 5 x 10 MB rotation the daemon log covered about three hours on the
audited machine, mostly one of a few lines repeated:
- "Failed to submit extraction ...: entity_graph_jobs channel full"
(11k lines) from GraphComponent, echoed per batch by PostIngestQueue's
"KG batch failed", plus "Entity channel full, dropping job";
- the per-document "Title+NL extraction for <hash> in N ms" info line;
- session-watcher scan/indexing warnings retried every pass.
Add yams::common::LogRateLimiter, a small lock-free helper that admits
one message per interval and reports how many it held back ("(N similar
messages suppressed)"), and use it at those sites (60 s for the queue
paths, 10 min for the session watcher, including its first-failure
warning per path). The Title+NL timing line is per-document progress
and moves to debug. The drop counters are unchanged, so status still
shows the full numbers.
Signed-off-by: trvon <me@trevon.dev>
`yams daemon status` printed "Vector Index: disabled by configuration"
whenever the daemon had not attempted vector init yet. That includes
every start where vectors are enabled but the metadata integrity check
is still running, so a slow start looked like a misconfiguration.
The daemon now publishes readiness key vector_db_disabled (only when
true, so clients that list every non-ready key do not show it) when
vectors are disabled by configuration. The renderer says "disabled / by
configuration" only in that case; otherwise it shows "pending" with what
it is waiting on (e.g. "waiting for database (Checking Integrity)"). The
detailed view's Vector DB row gets the same distinction instead of "Not
initialized".
Signed-off-by: trvon <me@trevon.dev>
~/.local/state/yams/sessions grows without bound: tool integrations
create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files
out of 1,764 on one machine) and nothing ever removes them.
Add expireIdleSessions() next to the session store and run it from
RepairService's idle maintenance once a day, only while no client is
connected. A session file is deleted only when nobody has created,
opened, closed or modified it for [daemon.maintenance]
session_expiry_days (typed config, default 30, 0 disables), and it is
kept when it is:
- the current session,
- watching a directory that still exists,
- still tagged on documents (they stay reachable through `session
merge` / `discard`; without a metadata repository ownership is
unknown and nothing expires),
- not valid JSON (not ours to judge).
Tests build a store shaped like the audited one and check each rule,
plus the RepairService wiring against a real metadata repository and
that a connected client defers the scan.
Signed-off-by: trvon <me@trevon.dev>
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:
- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
the audited machine) that `yams init` / `yams config grammar` built
and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
every start with two warnings (the ABI host's skip plus the plugin
manager's "load failed"), and the refusal flipped the plugin host FSM
through Failed.
Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.
The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.
Signed-off-by: trvon <me@trevon.dev>
vectors.db had no reclaim path at all. On a real install it had grown to
39.7 GB with 48% of its pages on the freelist (about 19 GB of dead
space), while yams.db already had a startup auto-VACUUM with a size,
free-page-ratio and free-disk check.
Move that decision into a shared sqlite_vacuum helper so both databases
follow the same rules, and let RepairService's background loop check
vectors.db once a day (15 minutes after start, typed config
[daemon.maintenance] vector_vacuum_interval_hours, 0 disables).
VACUUM holds the vectors.db write lock for the whole rewrite, so it only
starts while the daemon is idle: no client connections, resource
pressure below Warning, no queued or in-flight embedding writes, no
VectorIndexCoordinator rebuild or bulk load, and no on-demand repair.
It runs on its own connection with file-backed temp storage and a short
busy timeout, so a live writer makes it back off instead of waiting, and
a progress handler rolls it back as soon as any of those signals change
or shutdown starts. Readers on the owner's connection are unaffected in
WAL mode. Besides the data-dir free-space rule it also requires the temp
directory to hold a copy of the live pages.
Tests cover the thresholds (including the audited 39.7 GB / 48% case),
a real vacuum that shrinks a file, abort and live-writer back-off, the
admission truth table, and RepairService vacuuming a live VectorDatabase
file only when idle, which stays writable afterwards.
Signed-off-by: trvon <me@trevon.dev>
The clean-shutdown stamp is written through yams.db.integrity.tmp and
consumed by renaming it to yams.db.integrity.claim.<pid>.<n>. Both are
removed on the normal path, but a crash between the rename and the unlink
leaves them behind, and nothing ever cleaned them up, so they accumulate in
the data directory across crashes.
sweepStaleDbIntegrityStampArtifacts() removes them conservatively:
- only the exact .tmp name and .claim.<pid>.<n> names are considered; the
live stamp and any other file are never touched;
- a file must be older than a minimum age, so an in-flight publish or
consume is never raced;
- a claim is removed only when its pid is provably not running (kill(pid, 0)
reports ESRCH / OpenProcess reports no such process); an unparsable owner
keeps the file.
It runs from RepairService::runRecoveryArtifactMaintenance() alongside the
corrupt-DB retention cleanup, so it only happens in the background when
maintenance is allowed, with stampArtifactMinAge (1h) in
RepairService::Config.
Signed-off-by: trvon <me@trevon.dev>
Corrupt-DB handling could destroy the only copy of documents:
- removeCorruptDbFiles() deleted every readable yams.db.corrupt-* after a
startup or `repair --orphans --remove-corrupt` salvage, including DBs whose
salvageFromCorruptDb() had failed or only partly copied, and DBs that were
never salvaged because their document count was <= the live DB's.
- A corrupt DB whose count was <= the live DB's was otherwise kept forever
with only a "skipping" log, with no path that ever decided it was safe to
drop.
- quarantineAndRecreate() renamed onto yams.db.corrupt-<second>, so two
quarantines in the same second overwrote the first copy.
Deletion is now decided by evidence, not by which code path ran:
corruptDbSalvageConfirmed() opens the corrupt DB read-only, attaches the
live DB and checks that every document hash in it is present in the live
DB (salvage copies documents keyed by hash). removeSalvagedCorruptDbs()
removes an artifact only when that check succeeds and the artifact is older
than a retention window (quarantine timestamp from its name, else mtime);
unreadable, unconfirmed or young artifacts are kept and reported with the
reason.
Startup salvage no longer deletes anything. RepairService owns cleanup:
runRecoveryArtifactMaintenance() runs from the background loop when
maintenanceAllowed(), every recoveryArtifactSweepInterval (6h), with
corruptDbRetention (7 days) in RepairService::Config. An explicit
`repair --orphans --remove-corrupt` waives the retention window but still
requires confirmation. yams doctor explains the retention policy next to
the artifacts it lists.
The "yams.db" / ".corrupt-" naming is centralized in db_recovery.h
(kMetadataDbFileName, kCorruptDbMarker, corruptDbPrefix(),
isCorruptDbFileName(), listCorruptDbs()) and used by salvage, recovery,
RepairService and the doctor checks instead of four hand-rolled scans.
Signed-off-by: trvon <me@trevon.dev>
When a stale-WAL checkpoint at startup (ServiceManager::recoverStaleWalIfPresent)
or the shutdown checkpoint (DatabaseManager::shutdown) failed with a
"malformed"/"corrupt" error, both paths deleted yams.db-wal and yams.db-shm.
A WAL that SQLite cannot checkpoint may still contain committed transactions
that never reached the main file, so deleting it discarded them with no copy
left for manual recovery.
Both paths now call quarantineSqliteSidecars(), which renames the sidecars
next to the database as yams.db-wal.quarantine-<UTC ts> (and -shm), never
over an earlier quarantine, and logs where each file went. SQLite no longer
treats them as sidecars, so the database still continues from its last
checkpoint, but the bytes are preserved. The WAL is moved first; if it cannot
be moved the SHM is left in place so SQLite sees a consistent pair.
yams doctor now lists quarantined sidecars alongside corrupt DB artifacts so
they are not forgotten.
Signed-off-by: trvon <me@trevon.dev>
During a search storm on a large corpus, daemon shutdown blocked in
Phase 6.7 (SearchEngineManager::clearEngine) until the 120 s watchdog
forced an exit. Dropping the engine destroys SimeonLexicalBackend, whose
destructor joins the BM25 build thread, and the build only looked at its
stop token once every 1024 documents, each needing a metadata fetch.
Stop was also requested only from that destructor, so the build had no
head start while the earlier shutdown phases ran.
SimeonLexicalBackend::requestStop() now flags the build without joining
it and makes later buildAsync() calls no-ops. The build checks the flag
(alongside the jthread token) before every document fetch in the corpus
and fragment passes and before fragment geometry. SearchEngine and
SearchEngineManager expose requestShutdown(), and ServiceManager calls
it in Phase 4, so the build thread has normally exited by the time
clearEngine() destroys the engine.
simeon's own finalize, concept-mining and PMI-learning steps take no
stop token, so a build already inside one of them still runs it to
completion; making those interruptible needs a change in the simeon
submodule.
Signed-off-by: trvon <me@trevon.dev>
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.
Shutdown now tells in-flight work to stop before joining:
- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
on every leased connection under the pool mutex, which keeps each
lease open while it is interrupted. DatabaseManager exposes it for
both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
join wait slice through a new joinWithTimeout hook, because SQLite
drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
before the interrupt, and publishes it as AppContext::cancellationSignal.
DocumentService::list and GrepService::grep check it between query
and hydration steps and between document batches, and return
OperationCancelled; SearchService checks it on entry and forwards it
as SearchExecutionContext::hostCancellationSignal, which the engine's
existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
componentTimeout is 0, and a canceled search stops waiting for the
query embedding / concept tasks. Those tasks now own copies of their
inputs instead of capturing the caller's stack, so abandoning them is
safe.
Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.
Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.
Signed-off-by: trvon <me@trevon.dev>
When the Phase 5 WorkCoordinator join times out, the log only printed a
tid_hash per stuck worker, so the handler that kept the daemon from
quiescing (and from publishing the clean-shutdown integrity stamp) could
not be identified without attaching a debugger.
Give each worker a "current job" slot. WorkCoordinator::JobScope records
a static job name and start time on the calling worker, restoring the
enclosing job on exit; offload_to_worker and WorkCoordinator::post now
require a job name and run the callable inside a scope. The join timeout
detail line reports each stuck worker's job name and age, and
activeJobs() exposes the same snapshot for tests and diagnostics.
Signed-off-by: trvon <me@trevon.dev>
The TLS listener refuses a connection before the session handler runs
when the handshake fails, the peer pin is not allowed, or capacity is
exhausted; it logged that only at debug level, so the most common inbound
failure (an unenrolled peer) was still invisible. P2pListener::Options
gains an onRejected observer; P2pManager records those refusals with
stage 'transport', through the same path as later-stage failures.
Signed-off-by: trvon <me@trevon.dev>
P2pManager::handleInbound discarded runInboundSession's outcome, so a
rejected or failed inbound peer left no log line and no counter; the
InboundSessionStage it computed was never read.
- Count inbound sessions and failures, keep the last failure's stage,
error and time, and log a warning naming the stage.
- Expose them through P2pManager::inboundStats, the memory-sync status
(MemorySyncResponse, protobuf fields 17-21) and 'yams p2p status'
(text and --json).
Signed-off-by: trvon <me@trevon.dev>
Compressed GetResponse payloads were decoded only by 'yams get' (since
#214) and the MCP server, each with its own copy of the decoder. Any other
client of the daemon (library users, future commands) got the framed
bytes. In-process callers of DocumentService::retrieve defaulted to
acceptCompressed=true, so the daemon's CatRequest handler and the mobile
bindings would also have returned a frame instead of the document.
- DaemonClient::get decodes a compressed response once it is complete and
returns the document; the compression fields stay for display, and a
payload that does not decode is an error. The transfer stays compressed
on the wire.
- RetrieveDocumentRequest::acceptCompressed defaults to false; only
callers whose receiver decodes should opt in.
- Remove the now-redundant decoders from 'yams get' and the MCP get
handler.
- The integration test now requires a compressed request to deliver the
stored document, identical to a plain one.
Signed-off-by: trvon <me@trevon.dev>
Since v0.7.3 'yams get' has requested compressed transfers
(acceptCompressed), and the daemon has answered with a framed payload: a
CompressionHeader followed by a Zstandard body. Only the MCP server
decoded it; 'yams get' wrote the frame itself to stdout or -o, so users
saw the 'CNRK' header and a zstd frame instead of the document. 'yams cat'
was unaffected because it does not request compression.
- Move the frame format into compression/framed_payload
(encodeFramedPayload/decodeFramedPayload). DocumentService encodes with
it; 'yams get' and MCP decode with it.
- The decoder checks the header, body size and CRC, and the decoded size
and CRC. A payload that fails is an error; MCP used to fall back to
returning the framed bytes as content.
Verified end to end: 'yams get' output (stdout and -o) is byte-identical
to the added file while the transfer stays compressed (51.75x).
Signed-off-by: trvon <me@trevon.dev>
Every store added two metadata rows per snapshot to the document,
snapshot_id:<id> and snapshot_time:<id>, and nothing removed them.
Content-addressed documents are shared across paths and passes, so they
piled up: on one 93 GB database they were 43.1M of 44.0M metadata rows,
one file carried 52,367 snapshots, and 'yams list' timed out hydrating
2.69M rows for its 100 documents. Meanwhile findDocumentsBySnapshot read
only the latest snapshot_id, so a re-stored document dropped out of every
earlier snapshot.
- Migration 42 creates document_snapshots(document_id, snapshot_id,
snapshot_time), keyed on (document_id, snapshot_id). It only creates the
table, so startup stays fast.
- The insert transaction records membership; stores keep only the latest
snapshot_id/snapshot_time keys.
- findDocumentsBySnapshot and file history read membership, falling back
to legacy keys through one shared parser (legacySnapshotEntries).
- getMetadataForDocuments skips legacy keys via idx_metadata_doc_key: on
the 93 GB database, 0.21 s instead of 18.5 s for list's 100 documents.
- RepairService moves legacy keys in the background, 1000 snapshots per
batch, 200 ms apart, only below Warning pressure.
Signed-off-by: trvon <me@trevon.dev>
A single removal that failed returned before the watcher advanced its
snapshot, so every later pass saw the same files as changed, re-ran
addDirectory on all of them, and retried the removal forever. Removal
used force=false, and a path indexed more than once fails with 'Multiple
documents match', so one such path wedged the watcher permanently. A
live daemon re-indexed 1,793 files every 2 s this way and logged a
warning on each pass.
- Hold back only the paths that fail: a changed file whose indexing
failed stays changed, a removal that failed stays pending, and the
snapshot advances for everything else.
- Remove a deleted path with force: the file is gone, so every document
stored at it is stale.
- Warn once per failing path and retry quietly after that.
- Honor the watched directory's .gitignore (the unused gitignorePatterns
and gitignoreMtime state), and skip .git, so ignored build output is
not tracked, restatted, or removed.
Signed-off-by: trvon <me@trevon.dev>
Without a trusted clean-shutdown stamp, startup runs a full PRAGMA
quick_check before the database is ready. On a 93 GB database that took
over 11 minutes, and status showed 'Database: Opening' while the log
repeated 'still opening database', so a healthy scan looked like a hang.
Add the checking_integrity database phase, set it around the blocking
check, render it as 'Checking integrity ... full scan after unclean stop'
in 'yams daemon status', and name it in the startup watchdog log. The
check itself is unchanged: it still blocks readiness, and still recovers
confirmed corruption.
Signed-off-by: trvon <me@trevon.dev>
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
repairMissingEmbeddings built its database path as dataPath/'vectors.db'
with create_if_missing, so an unresolved (empty) data directory created a
stray vectors.db in the process working directory. The dispatcher passes
an empty path when the service manager has not resolved its data dir.
Return an error instead.
Tests that exercised repair or placed plugins under the unresolved data
directory now give the service manager a temp dir; the dispatcher suite
no longer leaves vectors.db or dispatcher_external_* in the working
directory.
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon
resolves the Simeon scoring mode from reranker policy (ConfigResolver), so
the stat said 0 for every outer-MaxSim run and the scifact ablation could
not tell which scorer ran. setCrossReranker() now takes the mode label the
wiring resolved; the engine reports it as cross_rerank_scoring_mode and
derives the outer-MaxSim stat from it.
Graph rerank recomputes connected components over mutual
semantic_neighbor edges among the candidates on every query, reading
each candidate's edges from the KG. The resident topology snapshot
already holds corpus-wide clusters built from those edges.
Add search.topology.graph_community_source (reciprocal_edges |
topology_snapshot). topology_snapshot scores candidates that share a
snapshot cluster with the same normalization and falls back to the edge
walk when no snapshot is loaded; graph_community_source in debug stats
reports which ran. The default stays reciprocal_edges until a benchmark
gate shows the snapshot source is at least as good. The retrieval bench
can set it for ablations.
Topology wrote back only an opaque compressed snapshot node, so yams
graph, explore and graph expansion could not traverse clusters, and
every rebuild added another full snapshot node that was never removed.
Each stored snapshot now replaces the cluster graph in one KG write
batch before the latest pointer flips: topology:cluster:<id> nodes with
size, level, persistence, cohesion and density; document member_of
edges weighted by persistence, medoid_of edges, overlaps edges for
boundary-spill memberships, and subcluster_of hierarchy edges. After the
pointer flips, snapshot nodes beyond the newest three are removed; the
one the pointer names is always kept.
kgEdgeDensity counted every kg_edges row, including the ~16
semantic_neighbor edges per document that 8-NN embedding neighbours
write in both directions, plus file-lineage and entity-attachment edges.
Any embedded corpus therefore looked like a rich knowledge graph, so the
SCIENTIFIC-profile gate that should zero the KG leg on flat prose never
fired and graph richness scaled from embedding edges.
Add kgRelationalEdgeCount/Density, which exclude embedding kNN, file
lineage and layout, text segments, and entity-to-document attachments,
and use it for hasRichGraphTopology() and the tuner's richness scaling.
kgEdgeCount keeps its meaning as the reconciled-graph signal. The live
overlay carries the reconciled relational count forward.
The retired symbol stage never activates, so tuning.post_ingest.
symbol_concurrent and YAMS_POST_SYMBOL_CONCURRENT had no effect, and
status reported symbol_* queue metrics and post_symbol_limit that were
always zero. Remove the config key and environment override, the status
keys and CLI rows, the metrics snapshot fields, and the event-bus symbol
counters. The internal stage slot stays so budget and limiter indices
are unchanged; benchmark JSON keeps its symbol fields at zero.
SymbolEnricher boosted results by looking up code-symbol nodes that no
longer exist since v0.20, so it ran KG queries for the top results and
never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the
symbol_rank_* search stats, and the symbolRank option on the service,
retrieval, MCP and CLI request types. The IPC field stays in the wire
format but is ignored, and --symbol-rank/--no-symbol-rank remain
accepted as hidden no-ops so existing scripts keep working.
Graph explore looked up entry points in symbol_metadata, which no longer
exists. Resolve them from node labels, aliases, and ingest path nodes
instead, gather a wider candidate pool, and rank it by query-term
coverage per file before applying the symbol budget so broad early
terms cannot crowd out the file that answers the query.
Impact and affected-test analysis walked code-symbol edges that are no
longer produced; they now return a warning saying the symbol graph was
removed in v0.20 instead of silently empty results.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
Report topology_route_bq_used / topology_route_bq_candidates in search
debug stats so benchmarks can tell BQ-shortlisted routing apart from
ANN and exhaustive routing. Let retrieval_quality_bench write
bq_candidate_limit, bq_prefix_dim, sgc_hops and
simeon_rerank_outer_maxsim into its per-run config from YAMS_BENCH_*
harness settings, read through getenv_optional.
sgcHops, sgcNormalize, bqCandidateLimit and bqPrefixDimension existed
only as struct fields, so no production path could enable them. Resolve
topology.sgc_hops / topology.sgc_normalize into the TopologyManager
build config (hops capped at 4 to bound over-smoothing) and
search.topology.bq_candidate_limit / bq_prefix_dim into the routing
options, request, and fingerprint. The snapshot cache rebuilds only the
route index when the BQ prefix changes. All default to off.
Drop the snapshot interner, which was allocated but never used.
storeBatch wrote per-document keys before the snapshot node, so a
failure in between left documents pointing at a snapshot that was never
persisted. Write the snapshot node first and the latest pointer last so
a partial store leaves the previous snapshot authoritative.
Remove only the two per-document keys still written, in one transaction
via a new removeMetadataBatch, and stop the membership slow path from
reading structural keys that migration 41 prunes.
The metadata store cached the latest batch in an unguarded mutable
optional that search threads read and wrote, and handed out a full copy
on every loadLatest, so search kept the snapshot resident twice.
Publish the latest batch and its membership index as one immutable
shared snapshot under a mutex, add loadLatestShared to hand it out
without copying, and feed the routing snapshot cache from it. Historical
loads no longer displace the resident latest snapshot, and a reader that
raced a write never installs the older batch.
Implement three research-backed retrieval optimizations connecting
paperbridge literature findings with YAMS subsystems:
1. SGC Feature Smoothing (Proposal A):
- Add sgcHops and sgcNormalize to TopologyBuildConfig.
- Diffuse document embeddings across reciprocal nearest-neighbor graph
edges prior to clustering, formalizing Lean SGC.lean.
- Re-normalize diffused embeddings to preserve unit hypersphere metric geometry.
- Wire into ConnectedComponent, Louvain, and KMeans topology engines.
2. Matryoshka Prefix Binary Quantization (Proposal C):
- Add maxDimensions parameter to BinaryQuantizer::quantize and
maxPrefixDimension to BinaryQuantizedIndex::build.
- Support fast 64-bit word-aligned prefix Hamming shortlisting in
SparseGuidedClusterRouter via bqPrefixDimension.
3. Fragment Outer-MaxSim Reranker (Proposal B):
- Implement SimeonScoringMode { SingleVectorCosine, FragmentOuterMaxSim }
in SimeonModelProvider.
- Add extractDocumentFragments and late-interaction outer-MaxSim scoring
inspired by ColBERTv2/PLAID/SPLATE to prevent relevance dilution in long docs.
- Expose simeonRerankOuterMaxSim on SearchEngineConfig and ConfigResolver.
- Add Catch2 test coverage and verify BEIR scifact benchmark metrics.
Signed-off-by: trvon <me@trevon.dev>
- Add BinaryQuantizer and BinaryQuantizedIndex in include/yams/vector/binary_quantization.h
and src/vector/binary_quantization.cpp implementing 1-bit sign quantization,
packed 64-bit word storage, hardware popcount Hamming distance, and cosine
similarity estimation.
- Integrate centroidBqIndex into SparseRouteIndex, constructing compact BQ
centroids alongside dense ANN indices during SparseGuidedClusterRouter::buildRouteIndex.
- Add bqCandidateLimit to TopologyRouteRequest and tracking metrics (bqUsed,
bqCandidates, bqDistanceEvaluations) to SparseRouteWork.
- Enable fast candidate pre-filtering in SparseGuidedClusterRouter::route with
1-bit Hamming distance scans, providing sub-millisecond coarse candidate
shortlisting and automatic fallback when the dense ANN index is omitted.
- Add Catch2 unit test suites in tests/unit/vector/binary_quantization_catch2_test.cpp
and tests/unit/topology/topology_baseline_catch2_test.cpp.
Signed-off-by: trvon <me@trevon.dev>
