The rarity-greedy comparator acted on every evaluated window, so it spent each hour's budget on the first ~15 windows and scored 0 hits in every cell. Criteria 2 and 3 of the Stage 4 gate were void against it. rarity_calibrated keeps the same rarity value, chooser, costs and budget, and adds a per-node threshold frozen before the evaluated span: the m-th largest value over the trailing 24 h of label-free history between the warm-up and the --eval-after bound, with m set so expected spend matches the hourly budget. Without that history the comparator is uncalibrated and the gate fails by name; it never falls back to another rule. rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour ordering, not causal) are reported as diagnostics and never judged. fleet gate pools counts, keeps thresholds per node, and fails a pooled cell when any node is uncalibrated or lacks the comparator. Contract change: in `fleet gate --json`, rarity_hits and rarity_cpu_seconds now describe the calibrated comparator; the old numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds. Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md, sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95, indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is dated after the registration time in that document (2026-11-21T00:00Z); the YAMS record is what predates it. Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the strict analyzer on the changed TUs. Not run: fold on the changed TUs, make fuzz, the live Docker tests and the perf KPI. The comparator has not been run on validation-range data. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| range_attack.py | ||
| range_common.sh | ||
| range_down.sh | ||
| range_status.sh | ||
| range_up.sh | ||
| range_workload.sh | ||
| README.md | ||
| user-data.tmpl | ||
Real-sensor validation range
A self-contained local lab that produces the real inputs the Strawberry promotion gate consumes -
raw history, delayed labels and gate models. A few
Arch VMs run the real daemon (eBPF + fanotify exec guard) with realistic background activity, MITRE
Caldera runs emulated adversary activity against them on a private network, and the detector's
retro-hunt turns those incidents into the delayed labels the Stage 4 fleet gate counts. It extends the
single-VM de-risk in ../vm (validated 2026-11-08) to a multi-sensor fleet that runs over days.
Nothing on the host is published off-box: Caldera listens on 127.0.0.1:8888 only (rootless Docker,
capped), and the sensor VMs reach it over the qemu slirp gateway 10.0.2.2. The range is not
network-isolated, though: slirp is user-mode NAT, so guest traffic is NATed through the host, and
cloud-init runs package_update: true and pulls packages from the Arch mirrors on first boot. Read
"local" as loopback-published with no inbound exposure, not air-gapped. All payloads are inert
markers (a copy of /bin/sleep with a marker string appended); there is no real malware. See
../../../.claude-tracked memory caldera-range for the Caldera auth details.
Build for travel
VM disks and scheduler state live under $SB_RANGE_DIR (default /var/tmp/sb-range), not cleared on
reboot. The attack scheduler paces by its own uptime, so suspending the laptop or rebooting simply
pauses the range; range_up.sh resumes it and the sensors' stores keep growing across the gaps.
Use
deploy/validation/range/range_up.sh # bring up / resume (idempotent); build builddir first
deploy/validation/range/range_status.sh # health: caldera, agent liveness, sensors, incidents, labels
deploy/validation/range/range_down.sh # pause (keep disks + state); --destroy wipes them
Then, after at least the pre-registered 12 h warm-up plus enough history for 3 periods, run the real
gate from the Salt side (orch.strawberry_cycle then orch.strawberry_gate) — or, for a host-only
check, strawberry fleet gate over each sensor's replay --policy-eval --eval-model ... --periods 3
report. See docs/evaluation/stage4-fleet-gate.md.
Those reports are the gate's input, not a passing result. Replaying a sensor's whole store scores
windows inside the range its gate models recorded as their training range, so every cell publishes
in_sample: true and fails with FAIL: out-of-sample: ...: the gate refuses such a set instead of
promoting. A passing verdict requires the evaluation to be bounded after the models' training range
(replay --policy-eval --eval-after NS|auto: absolute ns, or auto = the latest training-range end
among the --eval-model files), so the reports' eval_first_ns falls after training_last_ns. The
bound also separates the label-free history the gate comparator (rarity_calibrated) freezes its
threshold from; without a bound the comparator is uncalibrated and the cell fails by name.
On this range the bound is fixed by the rarity-baseline pre-registration: the span up to 2026-11-20 was seen before the comparator was defined, so gate evidence comes only from windows after its registration time. On each sensor:
strawberry replay /var/lib/strawberry/evidence --policy-eval \
--eval-model seed1.sbm --eval-model seed2.sbm --eval-model seed3.sbm \
--periods 3 --eval-after 1795219200000000000 --json
then strawberry fleet gate <reports>. A run with an earlier bound is a diagnostic, not evidence.
What it exercises that the Docker fleet test cannot
- a real kernel: eBPF process monitoring and the fanotify exec guard (containers here cannot);
- a real baseline: days of varied benign activity the attacks must stand out from;
- the real label path: incidents are unknown to the rules at exec time, then a later rule update + retro-hunt labels the earlier windows — the delayed "extra positives" the gate measures, produced the way they occur in production, not injected.
Files
| file | role |
|---|---|
range_common.sh |
shared paths, host-safety floors, ssh/scp, verified VM/scheduler lookups |
range_up.sh |
boot/resume Caldera + sensor VMs, provision, start the scheduler (writes its pid file) |
range_workload.sh |
benign background activity inside each sensor (systemd unit) |
range_attack.py |
scheduler: one distinct Caldera incident per interval, then delayed labels |
range_status.sh / range_down.sh |
health; pause or destroy |
user-data.tmpl |
cloud-init for a sensor VM |
Requires: a built builddir, the ../vm base image + ssh key, the caldera:slim image, qemu,
cloud-localds. The $SB_RANGE_DIR runtime tree is disposable; nothing here writes to the repo.
Reserve about 10 GiB of available host memory for the three 2-GiB VMs, Caldera and headroom; do not
run compiler/build jobs concurrently with the range (a 2026-11-17 global OOM was caused by concurrent
compilers, not the range alone). range_up.sh checks MemAvailable and free disk once, before it starts
anything: below 10 GiB (RANGE_MIN_AVAIL_MB) or 20 GiB free (RANGE_MIN_DISK_GB) it refuses. That gate
is start-time only — nothing in the range pauses itself, so a long-running monitor must call
range_down.sh (pause, disks and state kept) when host headroom falls below its own floor (8 GiB is the
working one); range_status.sh prints the current headroom.
range_down.sh stops every sensor the range has on disk — every sensor-N id found in
RANGE_DIR/vms (pid file, disk or seed), not just the first SB_RANGE_SENSORS — ACPI-powering each
verified VM off, then TERM/KILL for any QEMU whose command line still names this range's disks. A
lowered sensor count, a stale pid file or a recycled PID therefore cannot leave a VM running. --destroy
refuses to delete while any such QEMU (or this range's scheduler) is still alive, and deletes only the
sensor-N.qcow2 / sensor-N-seed.iso names it created: a foreign *.qcow2 in vms/ is reported and
kept, never unlinked. vm_pid() only accepts a PID whose command line still names that sensor's QEMU
and disk, so a recycled PID is never counted as a live VM or signalled.
Each sensor VM gets SB_RANGE_VM_MEM MiB (default 2048). Lower it (e.g. 1536) on a host that
cannot keep ~8 GiB free with three VMs up: the daemon needs ~300 MiB, and letting a monitor
pause the range every few hours costs more evidence than a smaller footprint.
The attack scheduler is tracked per range: range_up.sh writes RANGE_DIR/state/scheduler.pid and
range_down.sh / range_status.sh act only on a PID whose command line runs this repo's
range_attack.py with SB_RANGE_DIR equal to this RANGE_DIR (falling back to a /proc scan with the
same test when the pid file is missing or stale) instead of a host-wide pkill -f range_attack.py, so
two ranges on one host never stop each other's scheduler. range_status.sh groups Caldera
registrations per sensor group and reports live vs stale (no beacon for more than
SB_RANGE_AGENT_STALE_S, default 300 s) or says that liveness is unknown; the registration total is
never presented as a VM count, because Caldera keeps a registration for every agent restart.
Sensor units use RuntimeDirectory=strawberry so control sockets and delayed-label reloads also work
after a guest reboot.