Written before any threat trace is collected or evaluated. Train on families first reported before 2025-07-01, evaluate on families first reported after it (new-family bucket primary), from public write-ups and Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training. Adds gate criterion 5 (model recall >= rarity-greedy on new families). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.7 KiB
Stage 4 inspection model: pre-registered promotion gate
Written 2026-10-27, before any promotion. The criteria below decide whether a
shadow model may be promoted (strawberry model promote, which checks them).
A changed method gets a new dated document; these numbers do not move after
results are seen. (Also in YAMS as source=decision,
task=strawberry-data-engine.)
What a promoted model may do
Request extra inspections only (memory scan, deeper rescan, closer watch of a window's descendants, rescan on the next rules update), within an hourly CPU budget (10/30/120 CPU-seconds by scan profile). It never skips, delays or replaces a rule-triggered scan, never blocks, contains or kills, and its output is never a finding by itself (Stage 5 decides that separately).
Measurement
strawberry replay --model <shadow.sbm> --policy-eval [--inject FILE] over
the host's telemetry:
- Candidates: every ranked window after the warm-up (default 12 h).
- Positives ("extra"): windows that rules did not flag at exec time and that
later got a late label (retro-hunt after a rules update, or an operator
quarantine/kill), plus injected evaluation-trace windows
(
config/traces/standard.trace; never the traces a model was trained on). Amended 2026-10-28 by the threat-trace pre-registration: plusthreats-eval.trace, and a criterion 5 on new threat families. - Policies, each with its own hourly budget and the same costs and reach
(
ShadowOptions): the model (its value through the shared chooser) and rarity-greedy (min(1, rarity score / 48) through the same chooser). - Efficiency = hits / inspection CPU-seconds. Recall = hits / positives.
Gate (all must hold)
- At least 10 extra positives in the evaluated history.
- Model efficiency >= 1.5 x rarity-greedy efficiency.
- Model recall >= rarity-greedy recall (efficiency is not bought by inspecting less of what matters).
- Model inspections stay within budget by construction; refusals are reported, not counted against it.
model promote checks 1-3 on the current history and refuses otherwise.
Before relying on a promoted model, the plan's fuller check also applies:
the gate holds on 3 non-overlapping replay periods and for 3 training seeds.
The tool reports the single run; the operator records the others (YAMS
source=evidence).
Known limits
- Payoffs assume an inspection of a positive window surfaces
reach(action)of it. Until the model has acted, real inspection outcomes are not observed; the policy head stays untrained until they are. - With few real late labels, the gate leans on injected traces, which measure targeting of known attack shapes, not of novel ones.