Picks up the Worker fix 525a0a4 (preserve plugin cache policy) merged
in yams-repo PR #1; the pointer had stayed at dca5bd6.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the wrangler upload path, which passes
--content-type, and removed the S3 "ContentType" field this step grepped
for. Act Sanity has failed on experimental since then.
Signed-off-by: trvon <me@trevon.dev>
The snippet wrote /etc/pacman.d/yams.conf but nothing includes that file,
so pacman never saw the repository. Append the [yams] section to
/etc/pacman.conf instead.
Signed-off-by: trvon <me@trevon.dev>
${{ cond && '' || 'experimental/' }} always yields 'experimental/' because
'' is falsy in Actions expressions, so the generated USAGE.txt for stable
releases pointed YUM and Arch users at experimental paths that return 404.
Invert the expression. The YUM instructions also claimed an unsigned
repository although repomd.xml is signed whenever a channel key is set;
enable repo_gpgcheck with the channel key in that case.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the pre-September upload code, which passed
--force, while the workflow still pins wrangler 4.69.0. That version's
'r2 object put' has no --force option (it overwrites without prompting)
and exits with 'Unknown argument: force', so the v0.20.3 release failed
its first R2 upload and no apt, yum or Arch repository was published.
Immutable objects are still checked with 'get' before any write.
Signed-off-by: trvon <me@trevon.dev>
Feature PRs merged with merge commits carry the PR title in the merge
commit body, which Release Please parsed as a second conventional commit.
Keep the entry that links the original commit.
Signed-off-by: trvon <me@trevon.dev>
WranglerObjectStore passes str(Path("/wrangler")) to the command, which
is \\wrangler on Windows, so the Windows lane failed the literal
"/wrangler" comparison.
Signed-off-by: trvon <me@trevon.dev>
CI runs the unit lanes with YAMS_SQLITE_VEC_SKIP_INIT=1, so the test's
VectorDatabase came up without vector tables and the post-VACUUM insert
failed on every platform. The case already clears the other vector
toggles; clear this one too.
Signed-off-by: trvon <me@trevon.dev>
When the daemon holds background work back for a busy host, a user
looking at a stale topology or an unfinished repair scan needs to see
why. Status had no host signal at all.
Carry the governor's host sample through the metrics snapshot and
StatusResponse: source (psi, loadavg), whether the host counts as busy,
the PSI cpu/io/memory "some avg10" values or the per-CPU load average,
and which deferrable work is currently held back. The proto transport
piggybacks the fields on request_counts like the other string status
fields; the native serializer appends them as optional tail fields.
`yams daemon status` (brief and detailed) shows a Host Load row, e.g.
"Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and
`yams status --json` gains a host_pressure object. Document the new
[tuning.resource] keys next to the other typed tuning sections.
Signed-off-by: trvon <me@trevon.dev>
requestTopologyRebuild() re-posted itself immediately whenever ingest,
post-ingest, embedding or KG work was still in flight. It released its
in-progress flag before re-posting, so every request that arrived while
the daemon was busy (one per embedding batch) could win the flag and
start its own re-post loop. Those loops never slept: they occupied the
worker pool until the ingest drained, and a slower drain on a loaded
host meant more of them for longer.
Per-thread sampling during a 1899-file ingest on a 2x oversubscribed
48-core host showed ~36 worker threads at 100% for the whole drain. For
the same ingest (120s window, background work only):
daemon CPU seconds before after
idle host 41, 41 23, 20, 19
loaded host 96, 95 28, 30, 30 (deferral off)
Arm a single timer instead: a busy daemon re-checks after 500ms, a
rebuild deferred for host pressure after 5s, and only one retry is
armed at a time. post_ingest_drain still requests a rebuild directly
when the pipeline empties, so topology freshness does not wait on the
timer in the common case (idle-host topology completion is unchanged at
~7s after ingest).
Add a ServiceManager test for the deferral path: with the host marked
busy the rebuild is held and the governor reports it deferred; once the
host calms down nothing is held back and, when a vector database is
available, the armed retry builds the artifacts without a new request.
Signed-off-by: trvon <me@trevon.dev>
Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
The governor only measured the daemon itself (RSS, own CPU), so it could
not tell that the machine was busy. On a loaded host the daemon still
started background work at full width, and that work ran far less
efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU
seconds on an idle 48-core host and 96 CPU seconds with the host 2x
oversubscribed (stress-ng), and took 4x longer to finish.
Sample a cheap host-wide signal on the existing governor tick, at most
once per second: Linux PSI "some avg10" for cpu, io and memory, falling
back to the one-minute load average per online CPU; on macOS the load
average plus the kernel VM pressure level. A busy host is a flag with a
release band (signals must drop to 75% of the threshold to clear) so
work does not flap around a threshold.
Host load stays a separate dimension from the governor's pressure
level. Folding it into the level would reuse the Warning response,
which blocks model loads and trims foreground concurrency; a busy build
on the same machine should not stop a search from loading its model.
Background scheduling consumes the flag in a follow-up change.
Thresholds are typed config under [tuning.resource]: host_pressure,
host_cpu_pressure_pct (40), host_io_pressure_pct (30),
host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in
the runtime tuning status with their provenance.
Signed-off-by: trvon <me@trevon.dev>
libyams_mobile links YAMS, SQLite, Boost and the rest statically and
exported all of it (about 127k dynamic symbols). In a process that also
carries YAMS, such as the mobile test binary running a DaemonHarness, the
dynamic linker bound some of the daemon's metadata calls (for example
makeSqliteKnowledgeGraphStore) into the library's copy while the rest of
the daemon used its own. Two SQLite copies then opened the same yams.db.
POSIX advisory locks are per process, so neither copy saw the other's
WAL-index locks; one truncated the -shm file and the other faulted with
SIGBUS in walFindFrame while initialising the knowledge-graph store.
A linker version script (Linux) and exported-symbols list (macOS) keep
everything but yams_mobile_* local. The library now exports exactly the
51 API functions, and scripts/ci/check_mobile_abi.sh still passes.
This is a host-build fix; it does not claim device support.
Signed-off-by: trvon <me@trevon.dev>
libyams_mobile stopped compiling without anyone noticing because no CI
lane built it: every lane forced YAMS_ENABLE_MOBILE_BINDINGS=false, and
the "Check mobile ABI surface" step was commented out and pointed at
scripts/ci/check_mobile_abi.sh, which was never committed.
Add a linux-x86_64-mobile matrix entry that configures with
enable-mobile-bindings and enable-mobile-tests, compiles only the
yams_mobile and catch2_mobile_abi targets, runs the ABI check, and runs
the mobile meson suite. Other lanes keep mobile bindings off.
Add scripts/ci/check_mobile_abi.sh. It compiles the public header as C11
and C++20 and takes the address of every YAMS_MOBILE_API function, so a
declaration hidden from C (the extern "C" bug fixed in the previous
commit) fails. It then checks that the library exports exactly the
declared yams_mobile_* functions with unmangled names.
This lane proves the Linux x86_64 host build only. It does not show that
an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
With -Denable-mobile-bindings=true, libyams_mobile no longer compiled:
- diff/cat called IDocumentService::retrieveDocument, which was replaced
by retrieve(); they now fetch stored bytes through retrieve() with
acceptCompressed left false, so content comes back uncompressed, and
map service errors through map_error_code instead of NOT_FOUND.
- repair awaited DaemonClient::call() as a future and read response
fields that no longer exist; it now goes through daemon_call() and
folds RepairResponse::operationResults into the C result.
- YAMS_PROFILE_ZONE is not a defined macro; use YAMS_ZONE_SCOPED_N.
- doctor built its version with std::string(char); emit JSON instead.
- yams_mobile_string_view_destroy, _cat_result_destroy and
_restore_result_destroy were declared but never defined.
The repair/diff/cat/restore/model/doctor declarations sat outside the
extern "C" block and inside #ifdef __cplusplus, so C and FFI hosts could
not see them and C++ builds exported mangled names. Move them into the
C block.
Daemon-mode contexts handed their own worker_pool executor to the
process-wide connection pool keyed by socket path. The pool outlived
yams_mobile_context_destroy, so a second context reused sockets bound to
a freed thread pool (ASAN heap-use-after-free in AsioConnection) and the
daemon was left with connections that never drained, which then tripped
a use-after-free in daemon shutdown. Contexts now use single-use
connections and disconnect the client before joining the worker pool.
Tests: add an embedded cat/diff test (exact bytes, line diff, identical
docs, not-found mapping) and a daemon test that creates and destroys two
contexts and checks the server returns to its baseline connection count.
This proves the Linux x86_64 host build and host-side behavior only. It
does not show that an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
v0.20 stopped producing code-symbol nodes and calls/references/
inherits/implements edges, and migration 40 deleted the existing ones.
GraphContextService::impact() and affectedTests() had already been
reduced to returning a "code-symbol graph was removed" warning, so
`yams graph --impact`, `--affected-tests` and `--test-pattern` (and the
MCP graph actions `impact` and `affected_tests`) could only ever print
that warning.
Remove the dead surface:
- CLI: the three flags, their execute/render helpers, and the --impact
mention in the --global help text.
- MCP: the two actions, their schema entries and the changed_files /
test_pattern request fields. Unknown graph actions now fail with
InvalidArgument instead of silently running a default query.
- Service: impact()/affectedTests() and their request/response structs.
The GraphImpact/GraphAffectedTests IPC messages stay for wire
compatibility. The dispatcher now answers them directly with the same
removal warning, without touching the metadata repository.
Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph
node labels and aliases and walk edges of any relation, so they still
work on document, path, version and entity graphs.
Tests: the service-level impact/affected-tests case moves to the
dispatcher graph suite; the sandbox smoke test now expects --impact to
be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and
blast-radius questions at `yams grep`.
Signed-off-by: trvon <me@trevon.dev>
Mode::None is the initial value of mode_ and means "no subcommand
parsed". execute() only reached it through a catch-all default, which
also hid any future Mode enumerator that someone forgot to dispatch.
Give None its own case (same InvalidArgument error as before) and drop
the default so -Wswitch flags an unhandled mode at compile time. The
enumerator stays: removing it would make an unparsed command silently
default to Init.
Add a CLI test that `yams session` with no subcommand is rejected; the
None path stays a defensive guard because require_subcommand() stops
that case at parse time.
Signed-off-by: trvon <me@trevon.dev>
The ConnectionFsm state classes produce transitions through Next, and
to_public() maps every ImplState back to the public enum. No state class
ever returned ImplState::ReadingHeader, so that mapping was unreachable:
on_readable() bypassed the state classes and called transition() with a
hard-coded switch instead.
ReadingHeader itself is live (the request handler drives on_readable()
and checks for the state), so wire it rather than remove it: add an
on_readable() hook to StateBase, have ConnectedState return
ReadingHeader, and let ConnectionFsm::on_readable() dispatch through the
current state like every other event.
ConnectedState::on_header_parsed() returned ReadingPayload or
WritingHeader, neither of which the legality matrix allows from
Connected, so every such transition was rejected. Drop the override; the
observable behavior (header events are ignored until on_readable()) is
unchanged.
Pin the transition table with a Catch2 case covering on_readable in each
reachable state, header-only and payload frames, streaming keep-alive,
close, error, and accept, and give the previously assertion-free
"Body parsing" section real expectations.
Signed-off-by: trvon <me@trevon.dev>
ClientConfig::acceptCompressed was written by RetrievalService's config
mapping and compared in the CLI client-pool equality operator, but no
DaemonClient or transport code ever read it, so it could not change
behavior. Compression is negotiated per request through
GetRequest::acceptCompressed, which RetrievalOptions::acceptCompressed
still feeds via makeGetRequest.
Drop the field, the mapping, the equality term (configs that differed
only in this flag now share a pooled client, which is what their
behavior already was), and the mapping-test assertion. The per-request
flag and its tests are unchanged.
Signed-off-by: trvon <me@trevon.dev>
RetrievalService::cat had no callers anywhere in the tree (CLI, MCP,
services, tests, benchmarks). Content reads go through get/getByName*
instead. Remove the method and its declaration.
The daemon CatRequest/CatResponse protocol messages stay: DaemonClient::cat
is still exercised by the daemon socket integration test and the
multi-client ingestion benchmark, the dispatcher still handles it, and
removing the message would break wire compatibility with older clients.
Signed-off-by: trvon <me@trevon.dev>
YamsDaemon::stop() stopped the SocketServer and destroyed it while the
IOCoordinator threads were still running, and only then stopped and
destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for
connections to drain, and the handle_connection coroutine frames keep a
raw pointer to the server in their cleanup guard. A connection that
outlived the drain window either resumed on an I/O thread after the
server was freed, or had its frame destroyed together with the
io_context, and both paths decremented the connection counters of the
freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The
ServiceManager-init failure path stopped the server the same way and
left the I/O threads running.
Tear the IPC layer down in one place, teardownSocketServer(): stop the
server (which still releases its acceptors and accept strand while the
io_context runs), stop and join the I/O threads so no handler can
resume, destroy the IOCoordinator so the io_context destroys the pending
handler frames while the server is valid, and only then destroy the
server. All daemon stop and startup-failure paths use it.
The unit test stops the io_context under the connection handlers so
they are deterministically pending when the teardown runs; the daemon
integration test parks the I/O threads past the drain window.
Signed-off-by: trvon <me@trevon.dev>
The lifetime timer callback on main-socket connections captured the
shared_ptr that owns it so it could re-arm itself. That self-reference is
never released: every main connection leaked the callback and what it
holds (socket, timer, tracked socket, completion handler). When the
io_context is torn down with the connection still pending, the leaked
completion handler also keeps the handle_connection frame alive, so its
cleanup guards never run and the connection stays counted as active.
Re-arm through a weak_ptr and let the connection coroutine, which lives
exactly as long as the timer is needed, own the callback.
Signed-off-by: trvon <me@trevon.dev>
SocketServer::stop() waits at most ~2s for connection handlers to drain
and then frees the connection slot semaphores. A proxy connection that is
still draining releases its slot from a guard in the handler coroutine
frame, so when that frame later resumes, or is destroyed with the
io_context, it writes to the freed semaphore (ASAN heap-use-after-free in
SemaphoreGuard::~SemaphoreGuard).
Hold the semaphores through shared_ptr and give each handler frame its
own reference. The handler is now a captureless coroutine lambda whose
arguments are copied into the frame, so it no longer reads captures from
a closure that lives only for one accept-loop iteration.
Signed-off-by: trvon <me@trevon.dev>
The vectors.db vacuum and session expiry are feat commits, which would
bump the pre-1.0 minor version; this is a stability release of the 0.20 line.
Release-As: 0.20.3
Signed-off-by: trvon <me@trevon.dev>
#216 and #220 each shifted the two reviewed getenv reads in
RequestDispatcher.cpp and each updated the allowlist on its own; merged
together the entries point at the old lines and the portability check fails.
Signed-off-by: trvon <me@trevon.dev>
With 5 x 10 MB rotation the daemon log covered about three hours on the
audited machine, mostly one of a few lines repeated:
- "Failed to submit extraction ...: entity_graph_jobs channel full"
(11k lines) from GraphComponent, echoed per batch by PostIngestQueue's
"KG batch failed", plus "Entity channel full, dropping job";
- the per-document "Title+NL extraction for <hash> in N ms" info line;
- session-watcher scan/indexing warnings retried every pass.
Add yams::common::LogRateLimiter, a small lock-free helper that admits
one message per interval and reports how many it held back ("(N similar
messages suppressed)"), and use it at those sites (60 s for the queue
paths, 10 min for the session watcher, including its first-failure
warning per path). The Title+NL timing line is per-document progress
and moves to debug. The drop counters are unchanged, so status still
shows the full numbers.
Signed-off-by: trvon <me@trevon.dev>
`yams daemon status` printed "Vector Index: disabled by configuration"
whenever the daemon had not attempted vector init yet. That includes
every start where vectors are enabled but the metadata integrity check
is still running, so a slow start looked like a misconfiguration.
The daemon now publishes readiness key vector_db_disabled (only when
true, so clients that list every non-ready key do not show it) when
vectors are disabled by configuration. The renderer says "disabled / by
configuration" only in that case; otherwise it shows "pending" with what
it is waiting on (e.g. "waiting for database (Checking Integrity)"). The
detailed view's Vector DB row gets the same distinction instead of "Not
initialized".
Signed-off-by: trvon <me@trevon.dev>
~/.local/state/yams/sessions grows without bound: tool integrations
create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files
out of 1,764 on one machine) and nothing ever removes them.
Add expireIdleSessions() next to the session store and run it from
RepairService's idle maintenance once a day, only while no client is
connected. A session file is deleted only when nobody has created,
opened, closed or modified it for [daemon.maintenance]
session_expiry_days (typed config, default 30, 0 disables), and it is
kept when it is:
- the current session,
- watching a directory that still exists,
- still tagged on documents (they stay reachable through `session
merge` / `discard`; without a metadata repository ownership is
unknown and nothing expires),
- not valid JSON (not ours to judge).
Tests build a store shaped like the audited one and check each rule,
plus the RepairService wiring against a real metadata repository and
that a connected client defers the scan.
Signed-off-by: trvon <me@trevon.dev>
v0.20 removed the code-symbol graph, but installs still carry its
leftovers:
- <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on
the audited machine) that `yams init` / `yams config grammar` built
and nothing loads any more.
- A yams_symbol_extractor plugin under the install prefix is refused on
every start with two warnings (the ABI host's skip plus the plugin
manager's "load failed"), and the refusal flipped the plugin host FSM
through Failed.
Remove the grammars once through metadata migration 43, so the cleanup
is recorded in schema_migrations and never repeats. It only acts on a
data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-*
files directly in grammars/, removes the directory only when nothing
else is left in it, and leaves a symlinked grammars/ (and its target)
alone, so nothing outside the data directory can be touched.
The install prefix is not ours to clean, so the stale plugin is only
reported: one info line per process naming the file that can be
deleted. PluginManager recognises that refusal and skips it quietly
instead of warning and dispatching a load failure.
Signed-off-by: trvon <me@trevon.dev>
vectors.db had no reclaim path at all. On a real install it had grown to
39.7 GB with 48% of its pages on the freelist (about 19 GB of dead
space), while yams.db already had a startup auto-VACUUM with a size,
free-page-ratio and free-disk check.
Move that decision into a shared sqlite_vacuum helper so both databases
follow the same rules, and let RepairService's background loop check
vectors.db once a day (15 minutes after start, typed config
[daemon.maintenance] vector_vacuum_interval_hours, 0 disables).
VACUUM holds the vectors.db write lock for the whole rewrite, so it only
starts while the daemon is idle: no client connections, resource
pressure below Warning, no queued or in-flight embedding writes, no
VectorIndexCoordinator rebuild or bulk load, and no on-demand repair.
It runs on its own connection with file-backed temp storage and a short
busy timeout, so a live writer makes it back off instead of waiting, and
a progress handler rolls it back as soon as any of those signals change
or shutdown starts. Readers on the owner's connection are unaffected in
WAL mode. Besides the data-dir free-space rule it also requires the temp
directory to hold a copy of the live pages.
Tests cover the thresholds (including the audited 39.7 GB / 48% case),
a real vacuum that shrinks a file, abort and live-writer back-off, the
admission truth table, and RepairService vacuuming a live VectorDatabase
file only when idle, which stays writable afterwards.
Signed-off-by: trvon <me@trevon.dev>
The clean-shutdown stamp is written through yams.db.integrity.tmp and
consumed by renaming it to yams.db.integrity.claim.<pid>.<n>. Both are
removed on the normal path, but a crash between the rename and the unlink
leaves them behind, and nothing ever cleaned them up, so they accumulate in
the data directory across crashes.
sweepStaleDbIntegrityStampArtifacts() removes them conservatively:
- only the exact .tmp name and .claim.<pid>.<n> names are considered; the
live stamp and any other file are never touched;
- a file must be older than a minimum age, so an in-flight publish or
consume is never raced;
- a claim is removed only when its pid is provably not running (kill(pid, 0)
reports ESRCH / OpenProcess reports no such process); an unparsable owner
keeps the file.
It runs from RepairService::runRecoveryArtifactMaintenance() alongside the
corrupt-DB retention cleanup, so it only happens in the background when
maintenance is allowed, with stampArtifactMinAge (1h) in
RepairService::Config.
Signed-off-by: trvon <me@trevon.dev>
Corrupt-DB handling could destroy the only copy of documents:
- removeCorruptDbFiles() deleted every readable yams.db.corrupt-* after a
startup or `repair --orphans --remove-corrupt` salvage, including DBs whose
salvageFromCorruptDb() had failed or only partly copied, and DBs that were
never salvaged because their document count was <= the live DB's.
- A corrupt DB whose count was <= the live DB's was otherwise kept forever
with only a "skipping" log, with no path that ever decided it was safe to
drop.
- quarantineAndRecreate() renamed onto yams.db.corrupt-<second>, so two
quarantines in the same second overwrote the first copy.
Deletion is now decided by evidence, not by which code path ran:
corruptDbSalvageConfirmed() opens the corrupt DB read-only, attaches the
live DB and checks that every document hash in it is present in the live
DB (salvage copies documents keyed by hash). removeSalvagedCorruptDbs()
removes an artifact only when that check succeeds and the artifact is older
than a retention window (quarantine timestamp from its name, else mtime);
unreadable, unconfirmed or young artifacts are kept and reported with the
reason.
Startup salvage no longer deletes anything. RepairService owns cleanup:
runRecoveryArtifactMaintenance() runs from the background loop when
maintenanceAllowed(), every recoveryArtifactSweepInterval (6h), with
corruptDbRetention (7 days) in RepairService::Config. An explicit
`repair --orphans --remove-corrupt` waives the retention window but still
requires confirmation. yams doctor explains the retention policy next to
the artifacts it lists.
The "yams.db" / ".corrupt-" naming is centralized in db_recovery.h
(kMetadataDbFileName, kCorruptDbMarker, corruptDbPrefix(),
isCorruptDbFileName(), listCorruptDbs()) and used by salvage, recovery,
RepairService and the doctor checks instead of four hand-rolled scans.
Signed-off-by: trvon <me@trevon.dev>
When a stale-WAL checkpoint at startup (ServiceManager::recoverStaleWalIfPresent)
or the shutdown checkpoint (DatabaseManager::shutdown) failed with a
"malformed"/"corrupt" error, both paths deleted yams.db-wal and yams.db-shm.
A WAL that SQLite cannot checkpoint may still contain committed transactions
that never reached the main file, so deleting it discarded them with no copy
left for manual recovery.
Both paths now call quarantineSqliteSidecars(), which renames the sidecars
next to the database as yams.db-wal.quarantine-<UTC ts> (and -shm), never
over an earlier quarantine, and logs where each file went. SQLite no longer
treats them as sidecars, so the database still continues from its last
checkpoint, but the bytes are preserved. The WAL is moved first; if it cannot
be moved the SHM is left in place so SQLite sees a consistent pair.
yams doctor now lists quarantined sidecars alongside corrupt DB artifacts so
they are not forgotten.
Signed-off-by: trvon <me@trevon.dev>
During a search storm on a large corpus, daemon shutdown blocked in
Phase 6.7 (SearchEngineManager::clearEngine) until the 120 s watchdog
forced an exit. Dropping the engine destroys SimeonLexicalBackend, whose
destructor joins the BM25 build thread, and the build only looked at its
stop token once every 1024 documents, each needing a metadata fetch.
Stop was also requested only from that destructor, so the build had no
head start while the earlier shutdown phases ran.
SimeonLexicalBackend::requestStop() now flags the build without joining
it and makes later buildAsync() calls no-ops. The build checks the flag
(alongside the jthread token) before every document fetch in the corpus
and fragment passes and before fragment geometry. SearchEngine and
SearchEngineManager expose requestShutdown(), and ServiceManager calls
it in Phase 4, so the build thread has normally exited by the time
clearEngine() destroys the engine.
simeon's own finalize, concept-mining and PMI-learning steps take no
stop token, so a build already inside one of them still runs it to
completion; making those interruptible needs a change in the simeon
submodule.
Signed-off-by: trvon <me@trevon.dev>
A clean `yams daemon restart` never published the clean-shutdown
integrity stamp, so every start paid for a full PRAGMA quick_check
(13-27 min on a 93 GB database). Request handlers offloaded to
WorkCoordinator threads (list, grep, get, search) kept running through
Phase 5: nothing interrupted a long SQLite statement or a long result
loop, both 5 s and 30 s joins expired, the workers were abandoned, and
the stamp was disabled.
Shutdown now tells in-flight work to stop before joining:
- ConnectionPool::interruptActiveConnections() calls sqlite3_interrupt
on every leased connection under the pool mutex, which keeps each
lease open while it is interrupted. DatabaseManager exposes it for
both pools. Phase 4 interrupts once, and Phase 5 re-interrupts on each
join wait slice through a new joinWithTimeout hook, because SQLite
drops an interrupt that lands between two statements.
- ServiceManager owns a shutdown cancellation flag, set at Phase 4
before the interrupt, and publishes it as AppContext::cancellationSignal.
DocumentService::list and GrepService::grep check it between query
and hydration steps and between document batches, and return
OperationCancelled; SearchService checks it on entry and forwards it
as SearchExecutionContext::hostCancellationSignal, which the engine's
existing cancellation checks and the fanout collector now honor.
- The concept-extraction wait no longer blocks without bound when
componentTimeout is 0, and a canceled search stops waiting for the
query embedding / concept tasks. Those tasks now own copies of their
inputs instead of capturing the caller's stack, so abandoning them is
safe.
Interruption stays scoped to shutdown: the flag is only set in Phase 4
and cleared on initialize, and interrupting an idle lease does not fail
its next statement.
Also erase a connection from the leased set when its rollback throws;
it was left behind as a dangling pointer, which the interrupt walk
would have dereferenced.
Signed-off-by: trvon <me@trevon.dev>
When the Phase 5 WorkCoordinator join times out, the log only printed a
tid_hash per stuck worker, so the handler that kept the daemon from
quiescing (and from publishing the clean-shutdown integrity stamp) could
not be identified without attaching a debugger.
Give each worker a "current job" slot. WorkCoordinator::JobScope records
a static job name and start time on the calling worker, restoring the
enclosing job on exit; offload_to_worker and WorkCoordinator::post now
require a job name and run the callable inside a scope. The join timeout
detail line reports each stuck worker's job name and age, and
activeJobs() exposes the same snapshot for tests and diagnostics.
Signed-off-by: trvon <me@trevon.dev>
The TLS listener refuses a connection before the session handler runs
when the handshake fails, the peer pin is not allowed, or capacity is
exhausted; it logged that only at debug level, so the most common inbound
failure (an unenrolled peer) was still invisible. P2pListener::Options
gains an onRejected observer; P2pManager records those refusals with
stage 'transport', through the same path as later-stage failures.
Signed-off-by: trvon <me@trevon.dev>
