Commit graph

1,853 commits

Author SHA1 Message Date
trvon
79232e5471
refactor(graph): drop --impact and --affected-tests
v0.20 stopped producing code-symbol nodes and calls/references/
inherits/implements edges, and migration 40 deleted the existing ones.
GraphContextService::impact() and affectedTests() had already been
reduced to returning a "code-symbol graph was removed" warning, so
`yams graph --impact`, `--affected-tests` and `--test-pattern` (and the
MCP graph actions `impact` and `affected_tests`) could only ever print
that warning.

Remove the dead surface:
- CLI: the three flags, their execute/render helpers, and the --impact
  mention in the --global help text.
- MCP: the two actions, their schema entries and the changed_files /
  test_pattern request fields. Unknown graph actions now fail with
  InvalidArgument instead of silently running a default query.
- Service: impact()/affectedTests() and their request/response structs.

The GraphImpact/GraphAffectedTests IPC messages stay for wire
compatibility. The dispatcher now answers them directly with the same
removal warning, without touching the metadata repository.

Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph
node labels and aliases and walk edges of any relation, so they still
work on document, path, version and entity graphs.

Tests: the service-level impact/affected-tests case moves to the
dispatcher graph suite; the sandbox smoke test now expects --impact to
be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and
blast-radius questions at `yams grep`.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:46 -06:00
trvon
7f91141014
refactor(cli): handle SessionCommand Mode::None explicitly
Mode::None is the initial value of mode_ and means "no subcommand
parsed". execute() only reached it through a catch-all default, which
also hid any future Mode enumerator that someone forgot to dispatch.

Give None its own case (same InvalidArgument error as before) and drop
the default so -Wswitch flags an unhandled mode at compile time. The
enumerator stays: removing it would make an unparsed command silently
default to Init.

Add a CLI test that `yams session` with no subcommand is rejected; the
None path stays a defensive guard because require_subcommand() stops
that case at parse time.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:46 -06:00
trvon
4be71457d0
refactor(ipc): route Connected->ReadingHeader through state impl
The ConnectionFsm state classes produce transitions through Next, and
to_public() maps every ImplState back to the public enum. No state class
ever returned ImplState::ReadingHeader, so that mapping was unreachable:
on_readable() bypassed the state classes and called transition() with a
hard-coded switch instead.

ReadingHeader itself is live (the request handler drives on_readable()
and checks for the state), so wire it rather than remove it: add an
on_readable() hook to StateBase, have ConnectedState return
ReadingHeader, and let ConnectionFsm::on_readable() dispatch through the
current state like every other event.

ConnectedState::on_header_parsed() returned ReadingPayload or
WritingHeader, neither of which the legality matrix allows from
Connected, so every such transition was rejected. Drop the override; the
observable behavior (header events are ignored until on_readable()) is
unchanged.

Pin the transition table with a Catch2 case covering on_readable in each
reachable state, header-only and payload frames, streaming keep-alive,
close, error, and accept, and give the previously assertion-free
"Body parsing" section real expectations.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:46 -06:00
trvon
17591fc3fd
refactor(daemon): remove unused ClientConfig::acceptCompressed
ClientConfig::acceptCompressed was written by RetrievalService's config
mapping and compared in the CLI client-pool equality operator, but no
DaemonClient or transport code ever read it, so it could not change
behavior. Compression is negotiated per request through
GetRequest::acceptCompressed, which RetrievalOptions::acceptCompressed
still feeds via makeGetRequest.

Drop the field, the mapping, the equality term (configs that differed
only in this flag now share a pooled client, which is what their
behavior already was), and the mapping-test assertion. The per-request
flag and its tests are unchanged.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:46 -06:00
trvon
0d4e649c3d
refactor(services): drop unused RetrievalService::cat
RetrievalService::cat had no callers anywhere in the tree (CLI, MCP,
services, tests, benchmarks). Content reads go through get/getByName*
instead. Remove the method and its declaration.

The daemon CatRequest/CatResponse protocol messages stay: DaemonClient::cat
is still exercised by the daemon socket integration test and the
multi-client ingestion benchmark, the dispatcher still handles it, and
removing the message would break wire compatibility with older clients.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:46 -06:00
trvon
f8fce1db18
fix(daemon): destroy the SocketServer only after its handlers are gone
YamsDaemon::stop() stopped the SocketServer and destroyed it while the
IOCoordinator threads were still running, and only then stopped and
destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for
connections to drain, and the handle_connection coroutine frames keep a
raw pointer to the server in their cleanup guard. A connection that
outlived the drain window either resumed on an I/O thread after the
server was freed, or had its frame destroyed together with the
io_context, and both paths decremented the connection counters of the
freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The
ServiceManager-init failure path stopped the server the same way and
left the I/O threads running.

Tear the IPC layer down in one place, teardownSocketServer(): stop the
server (which still releases its acceptors and accept strand while the
io_context runs), stop and join the I/O threads so no handler can
resume, destroy the IOCoordinator so the io_context destroys the pending
handler frames while the server is valid, and only then destroy the
server. All daemon stop and startup-failure paths use it.

The unit test stops the io_context under the connection handlers so
they are deterministically pending when the teardown runs; the daemon
integration test parks the I/O threads past the drain window.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:43 -06:00
trvon
b421c126ac
fix(daemon): break the connection lifetime timer reference cycle
The lifetime timer callback on main-socket connections captured the
shared_ptr that owns it so it could re-arm itself. That self-reference is
never released: every main connection leaked the callback and what it
holds (socket, timer, tracked socket, completion handler). When the
io_context is torn down with the connection still pending, the leaked
completion handler also keeps the handle_connection frame alive, so its
cleanup guards never run and the connection stays counted as active.

Re-arm through a weak_ptr and let the connection coroutine, which lives
exactly as long as the timer is needed, own the callback.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:43 -06:00
trvon
037a33be80
fix(daemon): let connection handlers co-own the slot semaphore
SocketServer::stop() waits at most ~2s for connection handlers to drain
and then frees the connection slot semaphores. A proxy connection that is
still draining releases its slot from a guard in the handler coroutine
frame, so when that frame later resumes, or is destroyed with the
io_context, it writes to the freed semaphore (ASAN heap-use-after-free in
SemaphoreGuard::~SemaphoreGuard).

Hold the semaphores through shared_ptr and give each handler frame its
own reference. The handler is now a captureless coroutine lambda whose
arguments are copied into the frame, so it no longer reads captures from
a closure that lives only for one accept-loop iteration.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:42 -06:00
Trevon
9c54cb02fa
Merge pull request #224 from trvon/fix/allowlist-request-dispatcher
fix(lint): update RequestDispatcher allowlist lines after the p2p and shutdown merges
2026-09-27 15:57:20 -06:00
trvon
f21f7b24f8
chore(release): pin the next release to 0.20.3
The vectors.db vacuum and session expiry are feat commits, which would
bump the pre-1.0 minor version; this is a stability release of the 0.20 line.

Release-As: 0.20.3
Signed-off-by: trvon <me@trevon.dev>
2026-09-27 15:55:25 -06:00
trvon
d32c9b669b
fix(lint): update RequestDispatcher allowlist lines after the p2p and shutdown merges
#216 and #220 each shifted the two reviewed getenv reads in
RequestDispatcher.cpp and each updated the allowlist on its own; merged
together the entries point at the old lines and the portability check fails.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 15:54:25 -06:00
Trevon
aa36c2d069
Merge pull request #222 from trvon/chore/data-dir-cleanup
chore(daemon): data-dir cleanup via migration and repair maintenance
2026-09-27 15:53:07 -06:00
Trevon
66fc421702
Merge pull request #219 from trvon/fix/snapshot-aggregates-membership
fix(metadata): summarize snapshots from membership
2026-09-27 15:52:49 -06:00
Trevon
057933b99a
Merge pull request #218 from trvon/test/reenable-post-ingest-persist
test(repair): re-enable the post-ingest persistence regression test
2026-09-27 15:52:38 -06:00
Trevon
11c024489d
Merge pull request #217 from trvon/chore/remove-tree-command
chore(cli): remove the unregistered tree command
2026-09-27 15:52:31 -06:00
Trevon
7f5db9daf1
Merge pull request #216 from trvon/fix/p2p-inbound-status
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
fix(p2p): record and report inbound session failures
2026-09-27 15:52:17 -06:00
Trevon
51e1ef24d5
Merge pull request #215 from trvon/fix/clients-decode-compressed
fix(client): decode compressed transfers for every client
2026-09-27 15:52:09 -06:00
trvon
bbec4093cd
fix(daemon): rate-limit repeating warnings that flood the daemon log
With 5 x 10 MB rotation the daemon log covered about three hours on the
audited machine, mostly one of a few lines repeated:

- "Failed to submit extraction ...: entity_graph_jobs channel full"
  (11k lines) from GraphComponent, echoed per batch by PostIngestQueue's
  "KG batch failed", plus "Entity channel full, dropping job";
- the per-document "Title+NL extraction for <hash> in N ms" info line;
- session-watcher scan/indexing warnings retried every pass.

Add yams::common::LogRateLimiter, a small lock-free helper that admits
one message per interval and reports how many it held back ("(N similar
messages suppressed)"), and use it at those sites (60 s for the queue
paths, 10 min for the session watcher, including its first-failure
warning per path). The Title+NL timing line is per-document progress
and moves to debug. The drop counters are unchanged, so status still
shows the full numbers.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 14:00:25 -06:00
trvon
86699d80f5
fix(cli): show vector index as pending until vectors initialize
`yams daemon status` printed "Vector Index: disabled by configuration"
whenever the daemon had not attempted vector init yet. That includes
every start where vectors are enabled but the metadata integrity check
is still running, so a slow start looked like a misconfiguration.

The daemon now publishes readiness key vector_db_disabled (only when
true, so clients that list every non-ready key do not show it) when
vectors are disabled by configuration. The renderer says "disabled / by
configuration" only in that case; otherwise it shows "pending" with what
it is waiting on (e.g. "waiting for database (Checking Integrity)"). The
detailed view's Vector DB row gets the same distinction instead of "Not
initialized".

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 14:00:25 -06:00
trvon
f87b64eb40
feat(daemon): expire idle session files from repair maintenance
~/.local/state/yams/sessions grows without bound: tool integrations
create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files
out of 1,764 on one machine) and nothing ever removes them.

Add expireIdleSessions() next to the session store and run it from
RepairService's idle maintenance once a day, only while no client is
connected. A session file is deleted only when nobody has created,
opened, closed or modified it for [daemon.maintenance]
session_expiry_days (typed config, default 30, 0 disables), and it is
kept when it is:

- the current session,
- watching a directory that still exists,
- still tagged on documents (they stay reachable through `session
  merge` / `discard`; without a metadata repository ownership is
  unknown and nothing expires),
- not valid JSON (not ours to judge).

Tests build a store shaped like the audited one and check each rule,
plus the RepairService wiring against a real metadata repository and
that a connected client defers the scan.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 14:00:25 -06:00
trvon
8eb7d96c2c
chore(migration): drop orphaned code-symbol artifacts from the data dir
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:

- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
  the audited machine) that `yams init` / `yams config grammar` built
  and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
  every start with two warnings (the ABI host's skip plus the plugin
  manager's "load failed"), and the refusal flipped the plugin host FSM
  through Failed.

Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.

The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 14:00:11 -06:00
trvon
6a71e79a47
feat(daemon): vacuum vectors.db from idle repair maintenance
vectors.db had no reclaim path at all. On a real install it had grown to
39.7 GB with 48% of its pages on the freelist (about 19 GB of dead
space), while yams.db already had a startup auto-VACUUM with a size,
free-page-ratio and free-disk check.

Move that decision into a shared sqlite_vacuum helper so both databases
follow the same rules, and let RepairService's background loop check
vectors.db once a day (15 minutes after start, typed config
[daemon.maintenance] vector_vacuum_interval_hours, 0 disables).

VACUUM holds the vectors.db write lock for the whole rewrite, so it only
starts while the daemon is idle: no client connections, resource
pressure below Warning, no queued or in-flight embedding writes, no
VectorIndexCoordinator rebuild or bulk load, and no on-demand repair.
It runs on its own connection with file-backed temp storage and a short
busy timeout, so a live writer makes it back off instead of waiting, and
a progress handler rolls it back as soon as any of those signals change
or shutdown starts. Readers on the owner's connection are unaffected in
WAL mode. Besides the data-dir free-space rule it also requires the temp
directory to hold a copy of the live pages.

Tests cover the thresholds (including the audited 39.7 GB / 48% case),
a real vacuum that shrinks a file, abort and live-writer back-off, the
admission truth table, and RepairService vacuuming a live VectorDatabase
file only when idle, which stays writable afterwards.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 14:00:11 -06:00
trvon
344f3c588f
fix(daemon): sweep stale integrity-stamp temp and claim files
The clean-shutdown stamp is written through yams.db.integrity.tmp and
consumed by renaming it to yams.db.integrity.claim.<pid>.<n>. Both are
removed on the normal path, but a crash between the rename and the unlink
leaves them behind, and nothing ever cleaned them up, so they accumulate in
the data directory across crashes.

sweepStaleDbIntegrityStampArtifacts() removes them conservatively:
- only the exact .tmp name and .claim.<pid>.<n> names are considered; the
  live stamp and any other file are never touched;
- a file must be older than a minimum age, so an in-flight publish or
  consume is never raced;
- a claim is removed only when its pid is provably not running (kill(pid, 0)
  reports ESRCH / OpenProcess reports no such process); an unparsable owner
  keeps the file.

It runs from RepairService::runRecoveryArtifactMaintenance() alongside the
corrupt-DB retention cleanup, so it only happens in the background when
maintenance is allowed, with stampArtifactMinAge (1h) in
RepairService::Config.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:57:35 -06:00
trvon
8da4df3568
fix(daemon): delete corrupt DB copies only after confirmed salvage
Corrupt-DB handling could destroy the only copy of documents:

- removeCorruptDbFiles() deleted every readable yams.db.corrupt-* after a
  startup or `repair --orphans --remove-corrupt` salvage, including DBs whose
  salvageFromCorruptDb() had failed or only partly copied, and DBs that were
  never salvaged because their document count was <= the live DB's.
- A corrupt DB whose count was <= the live DB's was otherwise kept forever
  with only a "skipping" log, with no path that ever decided it was safe to
  drop.
- quarantineAndRecreate() renamed onto yams.db.corrupt-<second>, so two
  quarantines in the same second overwrote the first copy.

Deletion is now decided by evidence, not by which code path ran:
corruptDbSalvageConfirmed() opens the corrupt DB read-only, attaches the
live DB and checks that every document hash in it is present in the live
DB (salvage copies documents keyed by hash). removeSalvagedCorruptDbs()
removes an artifact only when that check succeeds and the artifact is older
than a retention window (quarantine timestamp from its name, else mtime);
unreadable, unconfirmed or young artifacts are kept and reported with the
reason.

Startup salvage no longer deletes anything. RepairService owns cleanup:
runRecoveryArtifactMaintenance() runs from the background loop when
maintenanceAllowed(), every recoveryArtifactSweepInterval (6h), with
corruptDbRetention (7 days) in RepairService::Config. An explicit
`repair --orphans --remove-corrupt` waives the retention window but still
requires confirmation. yams doctor explains the retention policy next to
the artifacts it lists.

The "yams.db" / ".corrupt-" naming is centralized in db_recovery.h
(kMetadataDbFileName, kCorruptDbMarker, corruptDbPrefix(),
isCorruptDbFileName(), listCorruptDbs()) and used by salvage, recovery,
RepairService and the doctor checks instead of four hand-rolled scans.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:57:29 -06:00
trvon
fb3ed2877d
fix(daemon): quarantine malformed WAL/SHM instead of deleting them
When a stale-WAL checkpoint at startup (ServiceManager::recoverStaleWalIfPresent)
or the shutdown checkpoint (DatabaseManager::shutdown) failed with a
"malformed"/"corrupt" error, both paths deleted yams.db-wal and yams.db-shm.
A WAL that SQLite cannot checkpoint may still contain committed transactions
that never reached the main file, so deleting it discarded them with no copy
left for manual recovery.

Both paths now call quarantineSqliteSidecars(), which renames the sidecars
next to the database as yams.db-wal.quarantine-<UTC ts> (and -shm), never
over an earlier quarantine, and logs where each file went. SQLite no longer
treats them as sidecars, so the database still continues from its last
checkpoint, but the bytes are preserved. The WAL is moved first; if it cannot
be moved the SHM is left in place so SQLite sees a consistent pair.

yams doctor now lists quarantined sidecars alongside corrupt DB artifacts so
they are not forgotten.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:57:19 -06:00
trvon
8aae9245ef
fix(search): stop the simeon-lexical build when shutdown begins
During a search storm on a large corpus, daemon shutdown blocked in
Phase 6.7 (SearchEngineManager::clearEngine) until the 120 s watchdog
forced an exit. Dropping the engine destroys SimeonLexicalBackend, whose
destructor joins the BM25 build thread, and the build only looked at its
stop token once every 1024 documents, each needing a metadata fetch.
Stop was also requested only from that destructor, so the build had no
head start while the earlier shutdown phases ran.

SimeonLexicalBackend::requestStop() now flags the build without joining
it and makes later buildAsync() calls no-ops. The build checks the flag
(alongside the jthread token) before every document fetch in the corpus
and fragment passes and before fragment geometry. SearchEngine and
SearchEngineManager expose requestShutdown(), and ServiceManager calls
it in Phase 4, so the build thread has normally exited by the time
clearEngine() destroys the engine.

simeon's own finalize, concept-mining and PMI-learning steps take no
stop token, so a build already inside one of them still runs it to
completion; making those interruptible needs a change in the simeon
submodule.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:20:55 -06:00
trvon
6aacda7913
fix(daemon): cancel in-flight request work when shutdown begins
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.

Shutdown now tells in-flight work to stop before joining:

- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
  on every leased connection under the pool mutex, which keeps each
  lease open while it is interrupted. DatabaseManager exposes it for
  both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
  join wait slice through a new joinWithTimeout hook, because SQLite
  drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
  before the interrupt, and publishes it as AppContext::cancellationSignal.
  DocumentService::list and GrepService::grep check it between query
  and hydration steps and between document batches, and return
  OperationCancelled; SearchService checks it on entry and forwards it
  as SearchExecutionContext::hostCancellationSignal, which the engine's
  existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
  componentTimeout is 0, and a canceled search stops waiting for the
  query embedding / concept tasks. Those tasks now own copies of their
  inputs instead of capturing the caller's stack, so abandoning them is
  safe.

Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.

Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 13:05:07 -06:00
trvon
df8097abbb
fix(daemon): name in-flight worker jobs in shutdown join diagnostics
When the Phase 5 WorkCoordinator join times out, the log only printed a
tid_hash per stuck worker, so the handler that kept the daemon from
quiescing (and from publishing the clean-shutdown integrity stamp) could
not be identified without attaching a debugger.

Give each worker a "current job" slot. WorkCoordinator::JobScope records
a static job name and start time on the calling worker, restoring the
enclosing job on exit; offload_to_worker and WorkCoordinator::post now
require a job name and run the callable inside a scope. The join timeout
detail line reports each stuck worker's job name and age, and
activeJobs() exposes the same snapshot for tests and diagnostics.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 12:45:50 -06:00
trvon
f72abc7cc7
fix(p2p): count sessions the transport refuses as inbound failures
The TLS listener refuses a connection before the session handler runs
when the handshake fails, the peer pin is not allowed, or capacity is
exhausted; it logged that only at debug level, so the most common inbound
failure (an unenrolled peer) was still invisible. P2pListener::Options
gains an onRejected observer; P2pManager records those refusals with
stage 'transport', through the same path as later-stage failures.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:50:37 -06:00
trvon
a57889e783
fix(metadata): summarize snapshots from membership
getSnapshots, getSnapshotInfo and batchGetSnapshotInfo read only each
document's latest snapshot_id key, so a re-stored document disappeared
from every earlier snapshot's file count, and a snapshot whose documents
were all re-stored vanished from the list.

- getSnapshots lists tree_snapshots, document_snapshots and latest keys.
  It deliberately skips legacy snapshot_id:<id> keys, which would be a
  full-table scan on large corpora until the backfill finishes.
- batchGetSnapshotInfo counts members from document_snapshots, legacy
  keys not yet moved, and latest keys, each looked up by id through an
  index; label and git commit come from tree_snapshots first.
- getSnapshotInfo delegates to the batch query, removing three
  near-duplicate queries.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:21:50 -06:00
trvon
bda974f261
fix(p2p): record and report inbound session failures
P2pManager::handleInbound discarded runInboundSession's outcome, so a
rejected or failed inbound peer left no log line and no counter; the
InboundSessionStage it computed was never read.

- Count inbound sessions and failures, keep the last failure's stage,
  error and time, and log a warning naming the stage.
- Expose them through P2pManager::inboundStats, the memory-sync status
  (MemorySyncResponse, protobuf fields 17-21) and 'yams p2p status'
  (text and --json).

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:21:31 -06:00
trvon
946cd59b11
test(repair): re-enable the post-ingest persistence regression test
'post-ingest success should persist extracted content' had been skipped
unconditionally since c051423e for lack of a deterministic completion
signal. It already waits on the observable result (the document's
extraction status), and post-ingest completion is fenced since #157, so
run it again.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:18:33 -06:00
trvon
55c51299a6
chore(cli): remove the unregistered tree command
961a1281 dropped 'tree' from the command registry while removing CLI
read paths that opened the metadata database locally; tree_command.cpp
still built but nothing could reach it. The path-tree statistics it
showed are not exposed through the daemon, so reconnecting it would mean
a new daemon request; remove the file until a path-tree view is added
under 'yams graph'.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:17:50 -06:00
trvon
fb38c7c091
fix(client): decode compressed transfers for every client
Compressed GetResponse payloads were decoded only by 'yams get' (since
#214) and the MCP server, each with its own copy of the decoder. Any other
client of the daemon (library users, future commands) got the framed
bytes. In-process callers of DocumentService::retrieve defaulted to
acceptCompressed=true, so the daemon's CatRequest handler and the mobile
bindings would also have returned a frame instead of the document.

- DaemonClient::get decodes a compressed response once it is complete and
  returns the document; the compression fields stay for display, and a
  payload that does not decode is an error. The transfer stays compressed
  on the wire.
- RetrieveDocumentRequest::acceptCompressed defaults to false; only
  callers whose receiver decodes should opt in.
- Remove the now-redundant decoders from 'yams get' and the MCP get
  handler.
- The integration test now requires a compressed request to deliver the
  stored document, identical to a plain one.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 08:11:17 -06:00
Trevon
7150be549e
Merge pull request #214 from trvon/fix/get-raw-manifest
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
fix(cli): decode compressed content in yams get
2026-09-27 07:30:15 -06:00
trvon
73e5d3831c
fix(cli): decode compressed content in yams get
Since v0.7.3 'yams get' has requested compressed transfers
(acceptCompressed), and the daemon has answered with a framed payload: a
CompressionHeader followed by a Zstandard body. Only the MCP server
decoded it; 'yams get' wrote the frame itself to stdout or -o, so users
saw the 'CNRK' header and a zstd frame instead of the document. 'yams cat'
was unaffected because it does not request compression.

- Move the frame format into compression/framed_payload
  (encodeFramedPayload/decodeFramedPayload). DocumentService encodes with
  it; 'yams get' and MCP decode with it.
- The decoder checks the header, body size and CRC, and the decoded size
  and CRC. A payload that fails is an error; MCP used to fall back to
  returning the framed bytes as content.

Verified end to end: 'yams get' output (stdout and -o) is byte-identical
to the added file while the transfer stays compressed (51.75x).

Signed-off-by: trvon <me@trevon.dev>
2026-09-26 21:10:00 -06:00
Trevon
841aeb7b02
Merge pull request #213 from trvon/fix/document-snapshot-membership
fix(metadata): record snapshot membership in its own table
2026-09-26 19:15:26 -06:00
Trevon
6342531af7
Merge pull request #212 from trvon/fix/pool-slow-hold-label
fix(metadata): name the pool in slow connection-hold warnings
2026-09-26 19:15:19 -06:00
Trevon
c4a609dcac
Merge pull request #211 from trvon/fix/session-watcher-progress
fix(daemon): keep the session watcher progressing past a failing path
2026-09-26 19:15:14 -06:00
trvon
3637567512
fix(metadata): record snapshot membership in its own table
Every store added two metadata rows per snapshot to the document,
snapshot_id:<id> and snapshot_time:<id>, and nothing removed them.
Content-addressed documents are shared across paths and passes, so they
piled up: on one 93 GB database they were 43.1M of 44.0M metadata rows,
one file carried 52,367 snapshots, and 'yams list' timed out hydrating
2.69M rows for its 100 documents. Meanwhile findDocumentsBySnapshot read
only the latest snapshot_id, so a re-stored document dropped out of every
earlier snapshot.

- Migration 42 creates document_snapshots(document_id, snapshot_id,
  snapshot_time), keyed on (document_id, snapshot_id). It only creates the
  table, so startup stays fast.
- The insert transaction records membership; stores keep only the latest
  snapshot_id/snapshot_time keys.
- findDocumentsBySnapshot and file history read membership, falling back
  to legacy keys through one shared parser (legacySnapshotEntries).
- getMetadataForDocuments skips legacy keys via idx_metadata_doc_key: on
  the 93 GB database, 0.21 s instead of 18.5 s for list's 100 documents.
- RepairService moves legacy keys in the background, 1000 snapshots per
  batch, 200 ms apart, only below Warning pressure.

Signed-off-by: trvon <me@trevon.dev>
2026-09-26 13:20:18 -06:00
trvon
2ecdaefa90
fix(metadata): name the pool in slow connection-hold warnings
Read and write pools share the slow-holder threshold, but the warning
always said 'slow write-connection hold'. A slow read-only 'yams list'
(tag client_list, 42 s) was reported as a write hold, which pointed the
investigation at a nonexistent write. Say read or write from the pool's
readOnly config.

Signed-off-by: trvon <me@trevon.dev>
2026-09-26 13:10:46 -06:00
trvon
4f5d77e31b
fix(daemon): keep the session watcher progressing past a failing path
A single removal that failed returned before the watcher advanced its
snapshot, so every later pass saw the same files as changed, re-ran
addDirectory on all of them, and retried the removal forever. Removal
used force=false, and a path indexed more than once fails with 'Multiple
documents match', so one such path wedged the watcher permanently. A
live daemon re-indexed 1,793 files every 2 s this way and logged a
warning on each pass.

- Hold back only the paths that fail: a changed file whose indexing
  failed stays changed, a removal that failed stays pending, and the
  snapshot advances for everything else.
- Remove a deleted path with force: the file is gone, so every document
  stored at it is stale.
- Warn once per failing path and retry quietly after that.
- Honor the watched directory's .gitignore (the unused gitignorePatterns
  and gitignoreMtime state), and skip .git, so ignored build output is
  not tracked, restatted, or removed.

Signed-off-by: trvon <me@trevon.dev>
2026-09-26 11:54:14 -06:00
Trevon
edc17df83c
Merge pull request #210 from trvon/fix/status-integrity-phase
fix(daemon): report the startup integrity check as its own phase
2026-09-26 10:53:44 -06:00
trvon
0b1b162e2a
fix(daemon): report the startup integrity check as its own phase
Without a trusted clean-shutdown stamp, startup runs a full PRAGMA
quick_check before the database is ready. On a 93 GB database that took
over 11 minutes, and status showed 'Database: Opening' while the log
repeated 'still opening database', so a healthy scan looked like a hang.

Add the checking_integrity database phase, set it around the blocking
check, render it as 'Checking integrity ... full scan after unclean stop'
in 'yams daemon status', and name it in the startup watchdog log. The
check itself is unchanged: it still blocks readiness, and still recovers
confirmed corruption.

Signed-off-by: trvon <me@trevon.dev>
2026-09-26 08:56:51 -06:00
Trevon
4aa3ca15b7
Merge pull request #208 from trvon/docs/agents-merge-policy
docs(agents): state merge methods and back-merge rule for promotion
2026-09-25 10:58:50 -06:00
trvon
365329a1b3
docs(agents): state merge methods and back-merge rule for promotion
Squash-merged promotion (#203) and back-merge (#205) PRs dropped main's
commits from experimental, so Release Candidate rejected the next
candidate. Record which PR kinds need a merge commit, the back-merge
check, the BEGIN_COMMIT_OVERRIDE hazard behind the 9,908-line notes, and
Release-As for pinning a version.

Signed-off-by: trvon <me@trevon.dev>
2026-09-25 09:01:57 -06:00
Trevon
2265030e2a
chore: merge main into experimental (restore release history) (#207)
* chore: release 0.20.2

Release-As: 0.20.2

* chore(main): release 0.20.2 (#204)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-25 08:54:22 -06:00
trvon
43dbc7d13a
Merge branch 'main' into experimental (restore 0.20.2 history) 2026-09-25 08:50:57 -06:00
Trevon
468e1f9982
Revert "ci: make immutable uploads atomic" (#206)
This reverts commit 296e7320.

296e7320 moved R2 publication from wrangler (authenticated by
CLOUDFLARE_API_TOKEN) to boto3 over R2's S3 API, which needs
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (and EXPERIMENTAL_* pairs). Those
secrets were never configured, so the first stable release built after
the change (v0.20.2, run 36084509789) failed at 'Publish channel package
repositories to R2' with 'Missing channel-specific Cloudflare R2 S3
authority', and the APT/RPM/Arch repositories were not updated.

Restore the wrangler path so releases publish with the existing token.
2026-09-25 08:44:43 -06:00
Trevon
85819a2bb7
chore: back-merge main (0.20.2 release) into experimental (#205)
* chore: release 0.20.2

Release-As: 0.20.2

* chore(main): release 0.20.2 (#204)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-25 07:33:09 -06:00