2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.
- Exec guard (assess_exec) no longer copies an image before deciding: a cached
scan answers with no read; an image over guard_full_scan_max_bytes, or when
the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
(degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
ScanBufferLease): in-memory image copies bounded to memory budget / 4
(balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
memory_full_pct / throttled, and resources.scan_buffers; doctor check
resources.memory (problem when throttled at MemoryHigh, with the fix).
Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allowlisting a hash suppresses new detections but does not delete findings
already stored for it (e.g. the /usr/bin/ss false-positive storm). That cleanup
now lives in doctor instead of a one-off compaction.
- DetectionPipeline::retire_allowlisted_findings(dry_run): a finding is stale
when it references at least one hash and every hash it references (from
metadata "hashes" and file_<sha256> nodes) is allowlisted, so a finding that
also implicates a non-allowlisted binary is never dropped. Mirrors
retire_superseded_findings (retire_finding + transaction).
- node_ids.hpp gains content_hash_of_file_node_id() (the inverse of
file_node_id for the content case) so the pipeline parses file ids through the
canonical helper, not a "file_" literal (source invariant R8).
- Doctor check findings.allowlisted (warn) with maintenance action
retire-allowlisted-findings; DoctorInputs.allowlisted_findings gathered by the
daemon; run_maintenance handles the action; CLI `doctor fix` help + completion
and API_CONTRACT updated.
Tests: test_retire_allowlisted_findings (two execs of a flagged binary -> a
finding; allowlist the hash -> dry-run lists it, apply retires it, idempotent);
doctor check ok/warn cases; CLI completion string. Suite 50/50; clang static
analyzer, clang-tidy and source invariants clean on the changed TUs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.
A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.
Also fixes status model.warmup_windows, which was never filled in.
Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Replay builds each window's WindowSummary as it goes (exec depth and
locations, connections, writes, ptrace, module loads, scan outcomes,
surprisal) and can collect training examples: features, a label (a
detection or late label touched the window), time, root program and the
rarity score. Sampled to at most 50k: positives up to a fifth,
negatives by reservoir with at most 1% per root program.
--inject-train traces become positive examples and are not evaluated.
- learn/train: the value head (and hidden layers) learn P(teacher
positive) with a class-weighted log loss and Adam; positive-unlabeled
data, so unlabeled is "not known bad", documented. Split by time
(newest 20% tested); reports test loss, top-k hits and ROC AUC for the
model and for the rarity baseline. Refuses with fewer than 5 positives.
Deterministic for a seed. The policy head waits for shadow-mode
inspection outcomes.
- CLI: `strawberry model train [store_dir] [--inject-train FILE]
[--epochs N] [--seed N] [--out FILE] [--json]` writes the shadow model
(generation after the newest existing one, a note saying what it was
trained on); `strawberry model show [file]`; `strawberry replay --model
FILE` ranks windows (and evaluates injections) by the model's value.
Config model_dir (default /var/lib/strawberry/model).
- doctor: the model check loads promoted.sbm and shadow.sbm; an invalid
file is a problem (it is never used).
Tests: test_train (gradients against finite differences, skipping steps
that cross a ReLU kink; a planted pattern learned to high AUC; determinism;
refusal without labels; roc_auc); test_replay R10 (examples, labels,
per-program cap, training traces, model ranking); CLI and doctor cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- doctor (control verb, read-only): store health and disk floor,
telemetry growth per day and days until the floor when retention is
unlimited, leftover store files, superseded findings, retro-hunt state,
thread/write errors, logs (journald rotates them), model (none yet).
Each check carries a maintenance action or advice. Pure checks over
gathered inputs (src/doctor.cpp), so they test without a daemon.
- maintain (control payload verb, privileged, rate limited, audited):
retire-superseded-findings and clean-store, each with dry_run.
- finding_retired evidence record: removes a finding from view; it stays
in the stream's history. CorrelationEngine::superseded_findings finds
old hash/rule findings by recomputing the stable id, and old lineage
findings only when the current one's parent process is among their
nodes (a reused pid on another boot is not a match). Lineage findings
now record parent_process_node_id.
- store::inspect_store_files / clean_store_leftovers: orphan/ files older
than N days and *.compact / MANIFEST.tmp temp files; never a listed
segment.
Tests: test_doctor (check statuses, superseded detection including pid
reuse and no-replacement cases, retirement surviving a reopen, cleanup
keeping recent orphans and listed segments); daemon test exercises both
verbs over the socket. fuzz_evidence_records covers finding_retired.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>