Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
The governor only measured the daemon itself (RSS, own CPU), so it could
not tell that the machine was busy. On a loaded host the daemon still
started background work at full width, and that work ran far less
efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU
seconds on an idle 48-core host and 96 CPU seconds with the host 2x
oversubscribed (stress-ng), and took 4x longer to finish.
Sample a cheap host-wide signal on the existing governor tick, at most
once per second: Linux PSI "some avg10" for cpu, io and memory, falling
back to the one-minute load average per online CPU; on macOS the load
average plus the kernel VM pressure level. A busy host is a flag with a
release band (signals must drop to 75% of the threshold to clear) so
work does not flap around a threshold.
Host load stays a separate dimension from the governor's pressure
level. Folding it into the level would reuse the Warning response,
which blocks model loads and trims foreground concurrency; a busy build
on the same machine should not stop a search from loading its model.
Background scheduling consumes the flag in a follow-up change.
Thresholds are typed config under [tuning.resource]: host_pressure,
host_cpu_pressure_pct (40), host_io_pressure_pct (30),
host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in
the runtime tuning status with their provenance.
Signed-off-by: trvon <me@trevon.dev>
