* test(graph): cover --topology-clusters --scope-cwd outside src/include
Follow-up to #280. --topology-clusters --scope-cwd still built its
scope from <cwd>/src and <cwd>/include, so a corpus laid out any other
way had no scoped members:
- The scope path set must hold every stored path under the cwd, also
when the cwd is reached through a symlink, and not a sibling that
shares the prefix.
- GraphTopologySupport must build the set from its own metadata repo.
The existing cluster-scope case built its path set by hand from the
old src/include contract. It now builds the set from /repo, so
/repo/tests/ is in scope and a document outside /repo is not.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): scope --topology-clusters --scope-cwd to the whole cwd
Follow-up to #280. Topology views built their --scope-cwd path set from
<cwd>/src and <cwd>/include only, a leftover from the code-symbol graph
removed in v0.20. Corpora laid out any other way reported no scoped
members.
Build the set from the same roots as the --list-type ranges: the cwd,
lexical and symlink-resolved. Keep only paths really under a root, since
the prefix query matches with LIKE. The topology view no longer prunes
build/, packages and IDE paths, so it matches --list-type.
GraphTopologySupport now builds the set from its own metadata repo
instead of the CLI app context. Remove buildGraphCodeScopePathSet and
buildGraphCurrentScopePathSet, which have no other callers, and use one
"Scoped to paths under the current directory" note for both views.
Signed-off-by: trvon <me@trevon.dev>
* test(cli): cover graph --name on a document with a path:file: node
Follow-up to #280. yams graph --name <path> tried path:file:<path>
node keys before the document lookup. v0.20 ingestion writes
path:logical: and path:<snapshot>: keys, so the probe misses on a
fresh corpus; after `yams doctor repair --graph` (or NL entity
extraction) the key exists, and --name printed a raw traversal of the
file node instead of the document's related documents.
Smoke cases through the in-process CLI:
- --name on a file with a path:file: node returns its hash and its
semantic neighbours (red).
- A relative --name resolves under the cwd even though a newer copy
with the same relative path exists in another checkout.
- A bare file name still resolves by name.
The portability allowlist entry for this file moves with its getenv
line.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): resolve graph --name through the document lookup only
Follow-up to #280. yams graph --name first traversed path:file:<path>
node keys. v0.20 ingestion writes path:logical:<path> and
path:<snapshot>:<path>, which only chain to blob versions; semantic
neighbours hang off doc:<hash>. On a fresh corpus the probe always
missed. After `yams doctor repair --graph` or NL entity extraction it
hit and printed a raw file-node traversal instead of the related
documents.
Drop the probe and go straight to the document lookup, which starts at
doc:<hash> since #280. Resolving to path:logical: instead would show
the version chain, not related documents; --node-key still traverses
any node.
Send --name as the path ingestion stores: relative to the client cwd,
lexically normalized, symlinks resolved. The daemon resolved relative
names against its own cwd and fell back to a suffix match, which picks
the newest copy across checkouts. Fall back to the name as given when
it does not resolve, so bare file names and suffixes still work.
build_graph_file_node_candidates had no other callers; its tests are
replaced by tests for buildGraphDocumentNameCandidates.
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover --limit on the document graph lookup
Follow-up to #280. yams graph <hash> and --name ignored --limit: the
daemon capped the graph query at 100 nodes and the same-directory
fallback at 20 documents.
Smoke cases through the in-process CLI:
- --limit 1 returns one related document by name and by hash, and the
first of the uncapped list.
- --limit 1 caps the same-directory fallback for a document without
graph edges.
- Without --limit the related lists are unchanged.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): pass --limit through the document graph lookup
Follow-up to #280. yams graph <hash> and --name sent --depth but not
--limit; DocumentService fixed the graph query at 100 nodes and the
same-directory fallback at 20 documents.
Add GetRequest.graph_limit (proto field 27) and
RetrieveDocumentRequest::graphLimit, and pass them from the CLI through
the dispatcher. A limit caps the related list, graph neighbours, path
history and the same-directory fallback alike. The graph query never
asks for fewer than 100 nodes, since path and blob nodes count toward
that limit.
The CLI sends a limit only when --limit is given; 0 keeps the old caps,
so default output is unchanged. The binary GetRequest layout appends
the field and still reads payloads without it.
Unit tests cover the proto round trip, the dispatcher mapping and the
DocumentService caps.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* test(graph): cover document lookup and --scope-cwd listing (#280)
Red tests for the two graph CLI bugs in #280:
- GetResponse graph fields and file name must survive the proto wire.
- DocumentService graph lookup by hash and by name must return the
same documents as traversing the doc:<hash> node.
- The document graph JSON must name the document when only name is set.
- KG path-range scoping must match document nodes by label and
snapshot path nodes by $.path.
- listByType --scope-cwd must keep document and path nodes under the
client cwd (including via a symlinked cwd) outside src/ and include/.
The existing code-roots dispatcher case now places its out-of-scope
file outside the scope root, since the scope is the whole cwd.
Signed-off-by: trvon <me@trevon.dev>
* fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280)
yams graph <hash> and --name returned graphEnabled=false with no
related documents while --node-key doc:<hash> listed neighbours:
- The GetResponse proto binding dropped graphEnabled, related and
fileName, so socket clients never saw graph data.
- DocumentService started traversal at blob:<hash>, which has no
outgoing edges, and filtered out semantic_neighbor. Start at the
doc:<hash> node and allow semantic_neighbor.
- GraphQueryService did not hydrate documentHash for doc: nodes, so
reached documents were skipped.
- The presenter read only fileName for "document"; fall back to name.
--list-type <t> --scope-cwd returned nothing:
- The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the
whole cwd, lexical and symlink-resolved.
- The KG range predicate ignored document labels and $.path on
snapshot path nodes. Match both.
Signed-off-by: trvon <me@trevon.dev>
* docs(skills): correct graph document lookup examples
--format dot is only rendered for node traversal, so show it with
--node-key doc:<hash>. Describe --scope-cwd as cwd-scoped.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.
Shutdown now tells in-flight work to stop before joining:
- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
on every leased connection under the pool mutex, which keeps each
lease open while it is interrupted. DatabaseManager exposes it for
both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
join wait slice through a new joinWithTimeout hook, because SQLite
drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
before the interrupt, and publishes it as AppContext::cancellationSignal.
DocumentService::list and GrepService::grep check it between query
and hydration steps and between document batches, and return
OperationCancelled; SearchService checks it on entry and forwards it
as SearchExecutionContext::hostCancellationSignal, which the engine's
existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
componentTimeout is 0, and a canceled search stops waiting for the
query embedding / concept tasks. Those tasks now own copies of their
inputs instead of capturing the caller's stack, so abandoning them is
safe.
Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.
Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep replication responsive
Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.
Reconstructed from ff62d3f8, 8e232721, and e60f513d.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated direct transport
Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.
Reconstructed from 96f9db88 and c46de77b.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): exchange causal deltas
Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.
Reconstructed from 1b8be146 and 41883338.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): integrate direct sync services
Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.
Reconstructed from a19aaa44.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): apply replicated document state
Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.
Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.
Signed-off-by: trvon <git@trevon.dev>
* test(p2p): register direct sync gates
Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.
Reconstructed from 360b9115 and 89416ccb.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): require bounded peer enrollment
Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.
Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.
Signed-off-by: trvon <git@trevon.dev>
* fix(p2p): persist quarantine and deletes
Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.
Reconstructed from 74ce01e0 and 19888604.
Signed-off-by: trvon <git@trevon.dev>
* feat(p2p): add authenticated cold bootstrap
Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.
Reconstructed from 8a9f5acf.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
* fix(graph): handle qualified C++ symbols
Signed-off-by: trvon <me@trevon.dev>
* refactor(vector)!: remove dead transition APIs
Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.
BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): harden PQ snapshot lifecycle
Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): expose route coordinates
Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.
Signed-off-by: trvon <me@trevon.dev>
* feat(xplan): add vector decision gates
Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): formalize coordinate routing
Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): serialize SymSpell readers
SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make mutations atomic
Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.
* fix(search)!: surface SymSpell write errors
Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.
BREAKING CHANGE: SymSpellSearch mutations now return Result values.
* feat(topology): formalize candidate expansion
* feat(vector): add coordinate-aware embeddings
* feat(topology): preserve protected relation cover
* feat(search): add graph candidate rescue
* feat(daemon): expose retrieval hydration telemetry
* feat(xplan): add rescue and hydration gates
* refactor(content): dedupe media handlers
Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.
* feat(lifecycle): prune stale corpus data
Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.
* feat(search): add topology candidate rescue
Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.
* feat(search): expose evidence paths
* docs: define mobile host contract
* formal: model bounded candidate rescue
* fix(prune): inspect extensionless signatures
* fix(prune): tolerate stale extensions
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): harden model state ownership
Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.
* refactor(daemon): consolidate ingest pollers
Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.
* refactor(daemon): unify repair execution
Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.
* refactor(daemon): unify runtime tuning
Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.
* refactor(daemon)!: remove dead IPC surfaces
Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.
BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.
* refactor(daemon): hide test hooks
Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.
* fix(daemon): avoid replaying successful adds
A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.
* bench(ingest): enforce lossless pipeline runs
Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.
* bench(search): expose cold seed ingest timings
Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.
* perf(ingest): batch deferred content indexing
Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.
* feat(metadata): expose insert writer metrics
Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.
* fix(daemon): retry transient refused connects
The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.
* bench(ingest): attribute submission path costs
Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.
* perf(ingest): batch durable document storage
Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.
* perf(ingest): coalesce bulk post-processing
Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.
* bench(ingest): record post-ingest batch size
Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.
* perf(ingest): batch queued storage waves
Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.
* perf(ingest): route bulk enrichment batches
Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.
* test(daemon): initialize task fixtures
* perf(ingest): batch post-ingest dispatch
* perf(ingest): widen directory storage waves
Borrow from a full wave when one document would otherwise fall through to single-document storage.
* perf(storage): bound batch writer fanout
Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.
* perf(ingest): align storage batch drains
Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.
* fix(ingest): commit metadata state atomically
Fold first-version, extracted-title, and embedding-completion state
into existing transactions.
Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.
* bench(ingest): measure production submission
Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.
* fix(xplan): preserve native failures
Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.
* fix(daemon): preserve embedding success hashes
* fix(daemon): serialize pooled socket shutdown
* fix(daemon): harden batch publication shutdown
* fix(storage): use compat jthread shim
* fix(daemon): avoid poller template shadowing
* fix(search): keep vector pipeline test enabled
* fix: harden shutdown and fallback paths
Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
* perf(daemon): coalesce metadata writes
* refactor(metadata): split repository ops
* fix(metadata): harden repository invariants
* feat: test coverage and continued refactor work in metadata code
* fix(graph): handle path-like explore queries
- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH
* fix(daemon): quiesce shutdown background work
- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain
* fix(daemon): cancel repair work on shutdown
Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.
* perf(metadata): reduce sqlite lock contention
* perf(metadata): flatten batch content retries
* fix(daemon): align shutdown wait budgets
A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.
* perf(metadata): bound feedback write waits
* perf(metadata): prep batch content before tx
* fix(daemon): honor no-embeddings tasks
Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.
* perf(daemon): batch fallback metadata writes
* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements
* perf(metadata): flatten insert retries
* fix(metadata): clear CI portability blockers
Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.
* perf(metadata): batch writes and steady bench
* fix(daemon): skip inactive post-ingest pollers
* perf(metadata): tighten tag and path caches
* perf(metadata): cache small batch upserts
* fix(daemon): harden embedded host teardown
* feat(docs): updating docs policies and language
* perf(metadata): tune doc update retries
* perf(metadata): bound download retries
* perf(metadata): bound download metadata waits
* bench(metadata): scope post-ingest content path
* fix(daemon): export embedded test hooks
* fix(daemon): harden test helper linkage
* fix(build): add missing standard includes
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter
Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close
* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler
* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly
ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.
* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict
setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.
* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build
* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs
* fix(build): restore hooks and shrink srht
Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.
Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.
Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.
* refactor(storage): drop dead code, add coverage
* refactor(cli): polish grep/search graph UX
* fix(daemon): stabilize shutdown and CI paths
- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work
* test(daemon): add db vacuum coverage
* test(daemon): add WAL startup recovery
- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown
* refactor(grep): consolidate indexed content scan
* fix(grep): remove pre-scan candidate truncation
* refactor(runtime): split startup, harden storage
* feat(bench): add manifest retrieval corpus
* fix(bench): run fusion experiment on simeon
* fix(cli): sharpen search graph hints
* fix(build): refresh version metadata
* fix(ci): stabilize sync search and srht packaging
* fix(ci): unblock windows and shrink artifacts
* fix(ci): stabilize linux runtime checks
* fix(build): shrink srht manifest
