The gate's own documentation claimed more than the tool did, in ways that mattered once the integrity checks landed. - "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it is now enforced (previous commit) and the text says which criterion the tool applies: complete evidence, refused false, every pooled cell passing. - The model digest proves agreement ACROSS reports. It cannot prove that the file on disk now is the one that was evaluated. Corrected everywhere. - The remedy was described inconsistently: cutting the training bundles earlier is NOT enough (evaluation still replays the whole store), a bounded evaluation range is required. Written consistently, with --eval-after NS|auto. - scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It now asserts >= 5, requires the provenance fields, and adds the honest-evidence invariant: today every cell is in-sample, so at least one cell must report in_sample true, none may pass, and each must carry the out-of-sample failure. A pass over in-sample cells is itself a failure. - The Salt/range runbooks advertised the flow as producing promotion evidence; it collects gate INPUT, and a passing verdict requires out-of-sample windows.
96 lines
6.8 KiB
Markdown
96 lines
6.8 KiB
Markdown
# Real-sensor validation range
|
|
|
|
A self-contained local lab that produces the real **inputs** the Strawberry promotion gate consumes -
|
|
raw history, delayed labels and gate models. A few
|
|
Arch VMs run the real daemon (eBPF + fanotify exec guard) with realistic background activity, MITRE
|
|
Caldera runs emulated adversary activity against them on a private network, and the detector's
|
|
retro-hunt turns those incidents into the delayed labels the Stage 4 fleet gate counts. It extends the
|
|
single-VM de-risk in `../vm` (validated 2026-11-08) to a multi-sensor fleet that runs over days.
|
|
|
|
Nothing on the host is published off-box: Caldera listens on `127.0.0.1:8888` only (rootless Docker,
|
|
capped), and the sensor VMs reach it over the qemu slirp gateway `10.0.2.2`. The range is **not
|
|
network-isolated**, though: slirp is user-mode NAT, so guest traffic is NATed through the host, and
|
|
cloud-init runs `package_update: true` and pulls packages from the Arch mirrors on first boot. Read
|
|
"local" as loopback-published with no inbound exposure, not air-gapped. All payloads are **inert
|
|
markers** (a copy of `/bin/sleep` with a marker string appended); there is no real malware. See
|
|
`../../../.claude`-tracked memory `caldera-range` for the Caldera auth details.
|
|
|
|
## Build for travel
|
|
|
|
VM disks and scheduler state live under `$SB_RANGE_DIR` (default `/var/tmp/sb-range`), **not** cleared on
|
|
reboot. The attack scheduler paces by its own uptime, so suspending the laptop or rebooting simply
|
|
pauses the range; `range_up.sh` resumes it and the sensors' stores keep growing across the gaps.
|
|
|
|
## Use
|
|
|
|
deploy/validation/range/range_up.sh # bring up / resume (idempotent); build builddir first
|
|
deploy/validation/range/range_status.sh # health: caldera, agent liveness, sensors, incidents, labels
|
|
deploy/validation/range/range_down.sh # pause (keep disks + state); --destroy wipes them
|
|
|
|
Then, after at least the pre-registered 12 h warm-up plus enough history for 3 periods, run the real
|
|
gate from the Salt side (`orch.strawberry_cycle` then `orch.strawberry_gate`) — or, for a host-only
|
|
check, `strawberry fleet gate` over each sensor's `replay --policy-eval --eval-model ... --periods 3`
|
|
report. See `docs/evaluation/stage4-fleet-gate.md`.
|
|
|
|
Those reports are the gate's *input*, not a passing result. Replaying a sensor's whole store scores
|
|
windows inside the range its gate models recorded as their training range, so every cell publishes
|
|
`in_sample: true` and fails with `FAIL: out-of-sample: ...`: the gate refuses such a set instead of
|
|
promoting. A passing verdict requires the evaluation to be bounded after the models' training range
|
|
(`replay --policy-eval --eval-after NS|auto`: absolute ns, or `auto` = the latest training-range end
|
|
among the `--eval-model` files), so the reports' `eval_first_ns` falls after `training_last_ns`. The
|
|
replay-side bound and its `FAIL` text are in the tree; the `--eval-after` CLI flag is landing in
|
|
parallel (`API_CONTRACT.md`). Until it is wired up this range provides real history, real delayed
|
|
labels and real model provenance - and a documented refusal, not promotion evidence.
|
|
|
|
## What it exercises that the Docker fleet test cannot
|
|
|
|
- a real kernel: eBPF process monitoring and the fanotify exec guard (containers here cannot);
|
|
- a real baseline: days of varied benign activity the attacks must stand out from;
|
|
- the real label path: incidents are **unknown to the rules at exec time**, then a later rule update +
|
|
retro-hunt labels the earlier windows — the delayed "extra positives" the gate measures, produced the
|
|
way they occur in production, not injected.
|
|
|
|
## Files
|
|
|
|
| file | role |
|
|
|---|---|
|
|
| `range_common.sh` | shared paths, host-safety floors, ssh/scp, verified VM/scheduler lookups |
|
|
| `range_up.sh` | boot/resume Caldera + sensor VMs, provision, start the scheduler (writes its pid file) |
|
|
| `range_workload.sh` | benign background activity inside each sensor (systemd unit) |
|
|
| `range_attack.py` | scheduler: one distinct Caldera incident per interval, then delayed labels |
|
|
| `range_status.sh` / `range_down.sh` | health; pause or destroy |
|
|
| `user-data.tmpl` | cloud-init for a sensor VM |
|
|
|
|
Requires: a built `builddir`, the `../vm` base image + ssh key, the `caldera:slim` image, `qemu`,
|
|
`cloud-localds`. The `$SB_RANGE_DIR` runtime tree is disposable; nothing here writes to the repo.
|
|
Reserve about 10 GiB of available host memory for the three 2-GiB VMs, Caldera and headroom; do not
|
|
run compiler/build jobs concurrently with the range (a 2026-11-17 global OOM was caused by concurrent
|
|
compilers, not the range alone). `range_up.sh` checks MemAvailable and free disk once, before it starts
|
|
anything: below 10 GiB (`RANGE_MIN_AVAIL_MB`) or 20 GiB free (`RANGE_MIN_DISK_GB`) it refuses. That gate
|
|
is start-time only — nothing in the range pauses itself, so a long-running monitor must call
|
|
`range_down.sh` (pause, disks and state kept) when host headroom falls below its own floor (8 GiB is the
|
|
working one); `range_status.sh` prints the current headroom.
|
|
|
|
`range_down.sh` stops every sensor the range has on disk — every `sensor-N` id found in
|
|
`RANGE_DIR/vms` (pid file, disk or seed), not just the first `SB_RANGE_SENSORS` — ACPI-powering each
|
|
verified VM off, then TERM/KILL for any QEMU whose command line still names this range's disks. A
|
|
lowered sensor count, a stale pid file or a recycled PID therefore cannot leave a VM running. `--destroy`
|
|
refuses to delete while any such QEMU (or this range's scheduler) is still alive, and deletes only the
|
|
`sensor-N.qcow2` / `sensor-N-seed.iso` names it created: a foreign `*.qcow2` in `vms/` is reported and
|
|
kept, never unlinked. `vm_pid()` only accepts a PID whose command line still names that sensor's QEMU
|
|
and disk, so a recycled PID is never counted as a live VM or signalled.
|
|
|
|
Each sensor VM gets `SB_RANGE_VM_MEM` MiB (default 2048). Lower it (e.g. 1536) on a host that
|
|
cannot keep ~8 GiB free with three VMs up: the daemon needs ~300 MiB, and letting a monitor
|
|
pause the range every few hours costs more evidence than a smaller footprint.
|
|
|
|
The attack scheduler is tracked per range: `range_up.sh` writes `RANGE_DIR/state/scheduler.pid` and
|
|
`range_down.sh` / `range_status.sh` act only on a PID whose command line runs this repo's
|
|
`range_attack.py` with `SB_RANGE_DIR` equal to this `RANGE_DIR` (falling back to a `/proc` scan with the
|
|
same test when the pid file is missing or stale) instead of a host-wide `pkill -f range_attack.py`, so
|
|
two ranges on one host never stop each other's scheduler. `range_status.sh` groups Caldera
|
|
registrations per sensor group and reports live vs stale (no beacon for more than
|
|
`SB_RANGE_AGENT_STALE_S`, default 300 s) or says that liveness is unknown; the registration total is
|
|
never presented as a VM count, because Caldera keeps a registration for every agent restart.
|
|
Sensor units use `RuntimeDirectory=strawberry` so control sockets and delayed-label reloads also work
|
|
after a guest reboot.
|