Commit graph

7 commits

Author SHA1 Message Date
trvon
433615044e
governor: release a memory episode on the memory hold, not the current RSS
compute_level picked the release hold from the RSS *after* the drop, so an
episode raised by the daemon's own memory released in 150 ms as soon as RSS
fell below the warning ratio — ignoring the documented 1000 ms memory hold.
A warning-level memory episode could therefore clear almost immediately.

The episode is now classified when it is raised: memory_driven_episode_ is set
when memory drove the level, and that decides the release hold. Critically this
does not clear when the level steps down (critical -> warning is the same
episode); the next escalation re-classifies it.

Tests: a memory warning -> calm transition is still warning at 200 ms and
999 ms and clears at 1000 ms, while a CPU-driven episode still clears in 150 ms.
2026-11-17 13:52:37 -07:00
trvon
a1d77dda9a
governor: fail-safe host pressure, diagnostic diskstats, capped scan buffers
Host pressure stays fail-safe: any of CPU/I/O/memory PSI above its profile
threshold, or low MemAvailable, still defers background work and can degrade
the exec guard. That is deliberate — global I/O PSI also counts waits with no
block-device counter (FUSE/network-backed I/O, display page flips), so idle
disks never prove pressure is benign. An audit of an earlier version that
vetoed I/O PSI below 5% block utilization rejected it on exactly that basis.

/proc/diskstats is therefore diagnostic telemetry only, exposed as
host.block_io_util_pct. It is keyed by major:minor so a reused device name
cannot compare unrelated counters, and reports unknown (-1) for the first
sample, a new/replaced/reset device, or an oversized read, instead of a false
0%.

scan_buffer_budget() now derives from the same effective budget that pressure
uses — the configured/profile budget capped by cgroup memory.max — with the
existing 32 MiB floor, so a cgroup-capped daemon cannot be granted scan
buffers larger than its own limit. Lock order stays mutex_ -> buffer_mutex_.

Tests: deterministic fail-safe cases across every diskstat value, the
new-device regression, scanner budget under a 100 MiB cgroup cap, and a seeded
property asserting severe I/O PSI always holds work whatever diskstats says.
2026-11-17 13:45:07 -07:00
trvon
5685600e57
Daemon memory: budget scan buffers, stream large images on the launch path, report throttling
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.

- Exec guard (assess_exec) no longer copies an image before deciding: a cached
  scan answers with no read; an image over guard_full_scan_max_bytes, or when
  the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
  new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
  (degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
  ScanBufferLease): in-memory image copies bounded to memory budget / 4
  (balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
  an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
  checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
  memory_full_pct / throttled, and resources.scan_buffers; doctor check
  resources.memory (problem when throttled at MemoryHigh, with the fix).

Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:36:14 -07:00
trvon
728b2462e7
Governor: background work no longer starves under sustained pressure
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
On a host that sits at critical/emergency for a long time (its own apps holding
most of RAM, or the daemon slightly over its soft budget), the starvation guard
was disabled (`!hard`), so correlation, retro-hunt and inventory deferred
forever — observed deferred 153 min with forced=0, which also starves the
retro-hunt delayed labels that Stage 4 training needs.

- Tiered starvation guard: it now fires under critical/emergency too, but after a
  level-scaled window (1x warning, 4x critical, 12x emergency) so background work
  is rare under load, not zero. A true host-memory crunch (host_severe_,
  MemAvailable below the severe floor) still holds everything, since forcing work
  then could tip the host toward OOM. exec guard/scan paths unchanged.
- Raise the balanced own-memory soft budget 512 -> 768 MiB: the daemon's steady
  footprint grew (telemetry buffers, the ingest server, the shadow model), so 512
  left it permanently over budget -> pinned critical/emergency. 768 fits the real
  baseline, still well under the 1.5 GiB cgroup cap.

Tests updated for the 768 budget and the tiered guard (critical forces after the
4x window; severe holds it). Suite 51/51; analyzer + clang-tidy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-28 22:33:25 -06:00
trvon
713122f845
Read process facts on macOS without procfs
include/strawberry/platform_process.hpp answers what /proc answers on
Linux: running PIDs (proc_listallpids), executable path
(proc_pidpath), parent and start time (sysctl KERN_PROC_PID, readable
for every process), argv (KERN_PROCARGS2) and a boot id
(kern.bootsessionuuid). A process is identified by its start time in
microseconds since the epoch plus the boot id, so node ids survive PID
reuse as on Linux. It is header-only, so the per-test source lists stay
as they are.

Users on macOS: resolve_process_ref (default root), the pipeline's
image hashing and exec-decision actor, and YaraScanner::scan_process
(scans the image by path). A synthetic procfs root still takes the
procfs code, so the existing procfs tests run on macOS too.

The resource governor samples Mach/sysctl on macOS: own CPU from
getrusage, RSS from task_info, load from getloadavg, available memory
from host_statistics64 (free + inactive + purgeable + speculative).

The behavior engine knows macOS debuggers (debugserver, dtrace) and
kext loaders (kernelmanagerd, kmutil), and macOS temp and Downloads
paths count as drop locations. The Linux lists are unchanged.

The KERN_PROCARGS2 parser is portable: a property test (exact argv
recovery, truncation never invents data, total on junk) and the
fuzz_procargs target run on every platform. test_platform_process
checks the live helpers against this process; the daemon test reads
the stopped state through sysctl on macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:55:09 -06:00
trvon
f1b948df24
Keep exec scanning alive under memory pressure; summarize repeats
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The KPI detect phase still dropped events after the correlation fix.
Cause: every detection kept ~12 KB of evidence in memory, 20k detections
pushed RSS past the 512 MB budget, the governor went to emergency, and in
emergency every live exec scan waited 5 s, so the queue overflowed and
stayed full (memory that is evidence never comes back by waiting).

- The governor holds live exec scans for at most 5 s per emergency
  episode, then lets them run (counted as forced); background work still
  waits.
- Repeated detections of the same image and rules: the first 20 per
  10-minute window are stored in full, then one summary indicator per
  key keeps the count and first/last pid and time (rewritten at most
  every 30 s). All detections are counted; alerts repeat at most once a
  minute per key. Status reports pipeline.repeats_aggregated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 07:32:30 -06:00
trvon
2d5e05e8db
Add resource governor, scan profiles and KPI harness
PSI/cgroup-aware governor with efficient/balanced/aggressive profiles,
background pacing and thread priorities; timing histograms and optional
Tracy zones; a capped Docker KPI harness with report/compare tooling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:16 -06:00