InternalEventBus::get_or_create_channel returns an existing channel for a name and silently
discards the requested capacity, so the first requester sizes a channel for the whole process.
Several call sites depend on that silence, and five of them survived an earlier fix precisely
because nothing reported it.
Record each distinct (name, actual, requested) disagreement and warn once. The warning is emitted
outside the lock so a sink cannot re-enter the bus while mu_ is held. remove_channel now drops a
name's records too, so the list describes live disagreements and an assertion cannot become
order-dependent. The spdlog include is guarded with __has_include, matching
pressure_limited_poller.h, since this is an installed public header.
The recorded capacity is stored beside the type-erased channel pointer rather than read back through
static_pointer_cast<SpscQueue<T>>. A name is NOT guaranteed to map to one T: "model.events" is
registered as SpscQueue<ModelReadyEvent> at plugins/onnx/model_provider.cpp:1076 and as
SpscQueue<ModelLoadFailedEvent> at :1089/:1266/:1503, so reading capacity() through the cast would be
a wrong-type member access on a reachable production path. The cast itself is still unvalidated, but
that risk is unchanged from what callers already accepted for a doubly-registered name.
On the test side the drain helpers no longer pin a capacity: they drain and remove, so each case's
own declared capacity applies. The two embed_jobs sites that want a small ring drain first, so they
actually receive 2048 instead of silently inheriting a larger registered ring, and a
ScopedChannelRemoval guard drops that ring at scope exit. The throughput case asserts that no
post_ingest disagreement was recorded, guarding the drain-then-remove behaviour its ring depends on.
Evidence: with the pre-fix pinning helper restored, the diagnostic reports "channel 'post_ingest'
already exists with capacity 32 but 64 was requested" and the new assertion fails (10 assertions, 1
failed); with the fix the case passes 10/10 and the binary is 307/307. The 4-batch outcome itself is
load-dependent and did not reproduce on this machine, so the guard's value is that it detects the
precondition deterministically rather than waiting for a loaded runner.
Two defects in an earlier revision of this commit were found by the entry-review workflow (Gemini
3.8 Flash + Claude Opus, adjudicated) before the push, and are fixed here: one embed_jobs site never
established the 2048 its comment claimed, so the comment and this message were false; and the
capacity read went through the cast while claiming a one-T-per-name invariant that is already false
in production.
This does not reduce the mismatch count in this binary (9 -> 10 -> 9 distinct after these fixes, with
the false direction gone). The
remaining records are deliberate test-local rings plus production disagreements that need separate
fixes: embed_jobs (8192 vs 1000) and post_ingest (1000 vs 4096 in RepairService). The diagnostic also
surfaced a family that was not previously known: PostIngestQueue derives symbol_extraction,
entity_extraction, title_extraction and kg_jobs from its per-instance capacity_ via
boundedStageChannelCapacity, while those channels are process-wide singletons, so the
first-constructed queue sizes them for every later queue (32 vs 512 and 32 vs 64 both observed).
Signed-off-by: trvon <me@trevon.dev>
The case "keeps multi-doc batches when extraction concurrency is low"
asserts an exact batch count (extractionBatches == 3) and produced 4 on a
loaded macOS runner (16+16+15+1).
The binding constraint was not the coalescing window: the drain helper
created the shared post_ingest channel with capacity 32 before any queue
was constructed, and get_or_create_channel ignores the requested capacity
on a name hit, so the queue declared capacity 64 but got a 31-slot ring.
With 48 tasks the producer had to block on the channel until the poller
drained, which made batch boundaries a function of runner load.
Drain and then drop the channel so each case capacity applies, and add the
bus remove_channel() that makes that possible. The exact-count assertions
are kept, not relaxed.
Signed-off-by: trvon <me@trevon.dev>
PostIngestQueue::processed() counts prepared tasks before their content is
committed, so waiting on it and then reading repository state is racy.
Wait on the committed observable in the three affected cases and
characterize the counter ordering with a gated-insert regression.
Signed-off-by: trvon <me@trevon.dev>
Stress-tagged cases ran in the default unit lane because only ~[slow]
was filtered. Tag them [slow] and add ~[slow]/[slow] splits to the five
binaries (mcp_submodule, work_coordinator, daemon_background,
plugin_infrastructure, hybrid_search) so they run in the slow suite.
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
* fix(daemon): harden model state ownership
Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.
* refactor(daemon): consolidate ingest pollers
Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.
* refactor(daemon): unify repair execution
Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.
* refactor(daemon): unify runtime tuning
Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.
* refactor(daemon)!: remove dead IPC surfaces
Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.
BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.
* refactor(daemon): hide test hooks
Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.
* fix(daemon): avoid replaying successful adds
A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.
* bench(ingest): enforce lossless pipeline runs
Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.
* bench(search): expose cold seed ingest timings
Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.
* perf(ingest): batch deferred content indexing
Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.
* feat(metadata): expose insert writer metrics
Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.
* fix(daemon): retry transient refused connects
The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.
* bench(ingest): attribute submission path costs
Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.
* perf(ingest): batch durable document storage
Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.
* perf(ingest): coalesce bulk post-processing
Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.
* bench(ingest): record post-ingest batch size
Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.
* perf(ingest): batch queued storage waves
Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.
* perf(ingest): route bulk enrichment batches
Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.
* test(daemon): initialize task fixtures
* perf(ingest): batch post-ingest dispatch
* perf(ingest): widen directory storage waves
Borrow from a full wave when one document would otherwise fall through to single-document storage.
* perf(storage): bound batch writer fanout
Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.
* perf(ingest): align storage batch drains
Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.
* fix(ingest): commit metadata state atomically
Fold first-version, extracted-title, and embedding-completion state
into existing transactions.
Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.
* bench(ingest): measure production submission
Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.
* fix(xplan): preserve native failures
Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.
* fix(daemon): preserve embedding success hashes
* fix(daemon): serialize pooled socket shutdown
* fix(daemon): harden batch publication shutdown
* fix(storage): use compat jthread shim
* fix(daemon): avoid poller template shadowing
* fix(search): keep vector pipeline test enabled
* fix: harden shutdown and fallback paths
Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
* perf(daemon): coalesce metadata writes
* refactor(metadata): split repository ops
* fix(metadata): harden repository invariants
* feat: test coverage and continued refactor work in metadata code
* fix(graph): handle path-like explore queries
- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH
* fix(daemon): quiesce shutdown background work
- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain
* fix(daemon): cancel repair work on shutdown
Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.
* perf(metadata): reduce sqlite lock contention
* perf(metadata): flatten batch content retries
* fix(daemon): align shutdown wait budgets
A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.
* perf(metadata): bound feedback write waits
* perf(metadata): prep batch content before tx
* fix(daemon): honor no-embeddings tasks
Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.
* perf(daemon): batch fallback metadata writes
* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements
* perf(metadata): flatten insert retries
* fix(metadata): clear CI portability blockers
Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.
* perf(metadata): batch writes and steady bench
* fix(daemon): skip inactive post-ingest pollers
* perf(metadata): tighten tag and path caches
* perf(metadata): cache small batch upserts
* fix(daemon): harden embedded host teardown
* feat(docs): updating docs policies and language
* perf(metadata): tune doc update retries
* perf(metadata): bound download retries
* perf(metadata): bound download metadata waits
* bench(metadata): scope post-ingest content path
* fix(daemon): export embedded test hooks
* fix(daemon): harden test helper linkage
* fix(build): add missing standard includes
* fix(core): add assertions and harden daemon lifecycle
* fix(daemon): tighten shutdown lifecycle invariants
* fix(test): add PressureLimitedPoller capability-sleep shutdown test
* refactor(vector): harden backend contracts
Split the vector backend surface into narrower contracts and
move entity read paths to Result<> so callers can distinguish
backend errors from empty results.
Also fixes the TurboQuant lazy-init race, makes VectorDatabase
error state thread-safe, wires eager index preparation, and
adds regression plus memory-scaling benchmark coverage.
* fix(vector): address audit findings plus hardening
Fix the five confirmed bugs: coroutine lifetime hazard in
embedding_generator (await_with_timeout outliving captured state),
VectorSystemManager init retry latch poisoning, incremental HNSW
persistence race, vec0 registry key collisions, and simeon PMI
output_dim/config() drift.
Add invariant hardening: PostIngestQueue stage validation +
numChunks assertion, VectorSystemManager success postcondition,
VectorDatabase TurboQuant non-null assertions on all quantize/
dequantize paths, and sqlite_vec_backend Simeon PQ range and
divisibility preconditions.
Also adds regression coverage for init-retry and lock-busy paths
and a locked node snapshot helper for safe incremental HNSW
persistence.
* fix(test): add PressureLimitedPoller wake-timer shutdown tests
* chore(vector): resolve clang-tidy hygiene across queue, vdb, and sqlite-vec
Replace unprotected std::getenv calls with locked getenvCopy helpers,
add debug-level logging to previously-silent catch blocks in
PostIngestQueue, VectorDatabase, and SqliteVecBackend, wrap
PostIngestQueue timer-detach allocations in try-catch, and fix
unsigned-widening / optional-deref / unused-include hygiene.
Verified with LSP diagnostics and rebuilds of catch2_vector_submodule
and catch2_daemon_component_submodule.
* fix(daemon): harden PressureLimitedPoller inFlightCounter null check
* fix(daemon): harden PostIngestQueue wait_until deadline checks
* fix(daemon): clean up recovery sentinels regardless of salvage count
Previously the .recovered-* sentinel and .corrupt-* file cleanup
was gated on documentsSalvaged > 0, leaving quarantine
artifacts to accumulate on repeated crash/recovery cycles where
the corrupt DB had 0 salvageable docs.
Move cleanup outside the salvage-count guard so stale
quarantine artifacts are always removed after examination.
* feat(vec0): restore persisted ANN index on daemon restart
After vec0 rebuild, probe for vectors_<dim>_hnsw_meta shadow tables
and load a previously-persisted HNSW index. When found, the ANN
index is set directly on the vec0 table, skipping the O(n log n)
rebuild that was previously paid on every daemon restart.
Removes the stub restoreVec0AnnIndexIfPresentUnlocked() in favor
of inline restoration in rebuildVec0DimUnlocked(), which has
the vec0 table already created and ready for ann_index assignment.
* fix: enable DCHECK by default and assert metadata counter consistency
* fix(services): assert document deletion ordering content-before-metadata
* fix(test): add fork-based assertion validation tests
* fix(test): expand assertion tests for POSTCONDITION, DCHECK, UNREACHABLE
* fix(ci): enable -Denable-dcheck=true in all test CI lanes
* fix(test): address assertion review findings
* refactor(daemon): isolate poller config assertion checks
* refactor(services): reduce deletion and poller complexity
* style(ci): normalize test workflow spacing
* refactor(services): extract list query helpers
* refactor(services): extract list hydration helpers
* refactor(daemon): extract metrics snapshot helpers
* refactor(daemon): extract metrics counter snapshot
* refactor(daemon): extract metrics resource snapshot
* feat(graph): add daemon-backed explore
Wire graph explore through daemon IPC, keep CLI rendering stable,
and fall back locally when daemon transport or protocol skew fails.
Also add graph context service/tests, improve graph follow-up hints,
and keep release builds working with the current JSON library.
* fix(services): prevent prune delete metadata orphans
* fix(cli): honor data-dir in graph explore
- bypass daemon explore when the CLI pins a data dir so\n temp fixtures do not resolve against a long-lived global DB\n- guard snippet truncation and ratio math to remove Fold-\n reported underflow and div-by-zero paths\n- clamp zero-based symbol lines before rendering graph\n context
* fix(daemon): make lifecycle stop completion safe
* fix(daemon): gate db-open startup against shutdown
* fix(metadata): serialize FTS indexed counter updates
* test(daemon): simulate vector lock contention cross-process
* test(daemon): isolate post-ingest identity batch
* style(daemon): normalize test include order
* refactor(graph): split cli graph helpers
Keep graph command paths smaller and easier to evolve.
Lock graph explore/topology contracts into focused presenter,
dispatcher, and support tests so follow-up refactors keep the
current IPC and hot-path behavior intact.
* fix(config): harden expand_tilde and fix Windows XDG state path doubling
- expand_tilde: handle bare ~ and ~/ explicitly to avoid std::out_of_range
on substr(2) for single-char input
- getXDGStateHome: on Windows, go up two levels from %LOCALAPPDATA%\yams\state
to %LOCALAPPDATA% so callers' xdg/yams append doesn't double the yams segment
* fix(static-analysis): add logging to empty catch blocks and fix nosemgrep syntax
* fix(static-analysis): strip trailing text from nosemgrep comments
* fix(doctor): add logging to daemon_check empty catch blocks
* fix(db_repair): add logging to empty catch blocks
* fix(chunking): add precondition checks for rabin polynomial and windowSize
* fix(safety): add null precondition checks for Fold proved null-deref sites
* fix(daemon): widen init-failure dispatch, add null/dbz preconditions
InitializationFailedEvent races with shutdown — async init
fails after teardown begins. Widen canDispatch to accept
Ready/ShuttingDown. Add static_assert on SnapshotRing N>0.
Add YAMS_PRECONDITION null guards on hex/stages/data/bytes.
* ci(ubsan): use restricted build targets to avoid runner timeout
* fix(daemon): detach wake timers on Windows stop path to prevent SIGSEGV
* fix(config): cast parseTomlU32 to size_t for batch fields
* fix(ci): use static helper for Windows timer leak, restrict ubsan tests
