strawberry/deploy/validation/range/README.md
trvon a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00

6.8 KiB

Real-sensor validation range

A self-contained local lab that produces the real inputs the Strawberry promotion gate consumes - raw history, delayed labels and gate models. A few Arch VMs run the real daemon (eBPF + fanotify exec guard) with realistic background activity, MITRE Caldera runs emulated adversary activity against them on a private network, and the detector's retro-hunt turns those incidents into the delayed labels the Stage 4 fleet gate counts. It extends the single-VM de-risk in ../vm (validated 2026-11-08) to a multi-sensor fleet that runs over days.

Nothing on the host is published off-box: Caldera listens on 127.0.0.1:8888 only (rootless Docker, capped), and the sensor VMs reach it over the qemu slirp gateway 10.0.2.2. The range is not network-isolated, though: slirp is user-mode NAT, so guest traffic is NATed through the host, and cloud-init runs package_update: true and pulls packages from the Arch mirrors on first boot. Read "local" as loopback-published with no inbound exposure, not air-gapped. All payloads are inert markers (a copy of /bin/sleep with a marker string appended); there is no real malware. See ../../../.claude-tracked memory caldera-range for the Caldera auth details.

Build for travel

VM disks and scheduler state live under $SB_RANGE_DIR (default /var/tmp/sb-range), not cleared on reboot. The attack scheduler paces by its own uptime, so suspending the laptop or rebooting simply pauses the range; range_up.sh resumes it and the sensors' stores keep growing across the gaps.

Use

deploy/validation/range/range_up.sh      # bring up / resume (idempotent); build builddir first
deploy/validation/range/range_status.sh  # health: caldera, agent liveness, sensors, incidents, labels
deploy/validation/range/range_down.sh    # pause (keep disks + state); --destroy wipes them

Then, after at least the pre-registered 12 h warm-up plus enough history for 3 periods, run the real gate from the Salt side (orch.strawberry_cycle then orch.strawberry_gate) — or, for a host-only check, strawberry fleet gate over each sensor's replay --policy-eval --eval-model ... --periods 3 report. See docs/evaluation/stage4-fleet-gate.md.

Those reports are the gate's input, not a passing result. Replaying a sensor's whole store scores windows inside the range its gate models recorded as their training range, so every cell publishes in_sample: true and fails with FAIL: out-of-sample: ...: the gate refuses such a set instead of promoting. A passing verdict requires the evaluation to be bounded after the models' training range (replay --policy-eval --eval-after NS|auto: absolute ns, or auto = the latest training-range end among the --eval-model files), so the reports' eval_first_ns falls after training_last_ns. The replay-side bound and its FAIL text are in the tree; the --eval-after CLI flag is landing in parallel (API_CONTRACT.md). Until it is wired up this range provides real history, real delayed labels and real model provenance - and a documented refusal, not promotion evidence.

What it exercises that the Docker fleet test cannot

  • a real kernel: eBPF process monitoring and the fanotify exec guard (containers here cannot);
  • a real baseline: days of varied benign activity the attacks must stand out from;
  • the real label path: incidents are unknown to the rules at exec time, then a later rule update + retro-hunt labels the earlier windows — the delayed "extra positives" the gate measures, produced the way they occur in production, not injected.

Files

file role
range_common.sh shared paths, host-safety floors, ssh/scp, verified VM/scheduler lookups
range_up.sh boot/resume Caldera + sensor VMs, provision, start the scheduler (writes its pid file)
range_workload.sh benign background activity inside each sensor (systemd unit)
range_attack.py scheduler: one distinct Caldera incident per interval, then delayed labels
range_status.sh / range_down.sh health; pause or destroy
user-data.tmpl cloud-init for a sensor VM

Requires: a built builddir, the ../vm base image + ssh key, the caldera:slim image, qemu, cloud-localds. The $SB_RANGE_DIR runtime tree is disposable; nothing here writes to the repo. Reserve about 10 GiB of available host memory for the three 2-GiB VMs, Caldera and headroom; do not run compiler/build jobs concurrently with the range (a 2026-11-17 global OOM was caused by concurrent compilers, not the range alone). range_up.sh checks MemAvailable and free disk once, before it starts anything: below 10 GiB (RANGE_MIN_AVAIL_MB) or 20 GiB free (RANGE_MIN_DISK_GB) it refuses. That gate is start-time only — nothing in the range pauses itself, so a long-running monitor must call range_down.sh (pause, disks and state kept) when host headroom falls below its own floor (8 GiB is the working one); range_status.sh prints the current headroom.

range_down.sh stops every sensor the range has on disk — every sensor-N id found in RANGE_DIR/vms (pid file, disk or seed), not just the first SB_RANGE_SENSORS — ACPI-powering each verified VM off, then TERM/KILL for any QEMU whose command line still names this range's disks. A lowered sensor count, a stale pid file or a recycled PID therefore cannot leave a VM running. --destroy refuses to delete while any such QEMU (or this range's scheduler) is still alive, and deletes only the sensor-N.qcow2 / sensor-N-seed.iso names it created: a foreign *.qcow2 in vms/ is reported and kept, never unlinked. vm_pid() only accepts a PID whose command line still names that sensor's QEMU and disk, so a recycled PID is never counted as a live VM or signalled.

Each sensor VM gets SB_RANGE_VM_MEM MiB (default 2048). Lower it (e.g. 1536) on a host that cannot keep ~8 GiB free with three VMs up: the daemon needs ~300 MiB, and letting a monitor pause the range every few hours costs more evidence than a smaller footprint.

The attack scheduler is tracked per range: range_up.sh writes RANGE_DIR/state/scheduler.pid and range_down.sh / range_status.sh act only on a PID whose command line runs this repo's range_attack.py with SB_RANGE_DIR equal to this RANGE_DIR (falling back to a /proc scan with the same test when the pid file is missing or stale) instead of a host-wide pkill -f range_attack.py, so two ranges on one host never stop each other's scheduler. range_status.sh groups Caldera registrations per sensor group and reports live vs stale (no beacon for more than SB_RANGE_AGENT_STALE_S, default 300 s) or says that liveness is unknown; the registration total is never presented as a VM count, because Caldera keeps a registration for every agent restart. Sensor units use RuntimeDirectory=strawberry so control sockets and delayed-label reloads also work after a guest reboot.