Commit graph

5 commits

Author SHA1 Message Date
trvon
f4adc12a74
gate: judge the model against a calibrated rarity comparator
The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.

rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.

rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.

Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.

Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.

Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 19:20:55 -07:00
trvon
a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
trvon
0e3d89d918
range: export the shared constants and keep shellcheck quiet
range_common.sh is a sourced library: range_up.sh, range_down.sh and
range_status.sh consume its constants, which shellcheck cannot see when it
analyses the file alone, so it reported every one as unused. Exporting them
states that interface, and the two $(cd ...) assignments are split from their
export so a failing cd is not masked.
2026-11-19 11:49:41 -07:00
trvon
39c5bc4efe
range: host-safety preflight, verified teardown, per-range scheduler identity
Host safety: range_up refuses to start below 10 GiB MemAvailable or 20 GiB free
disk, range_status prints the headroom, and the README documents both floors
and the 8 GiB pause a monitor should enforce. The 2026-11-17 global OOM was
caused by fifteen concurrent compilers plus the range, not the range alone, but
starting blind is what made it easy.

Teardown: sensor ids come from the vms/ contents rather than SB_RANGE_SENSORS,
so a lowered count or missing pid file cannot leave a VM behind; vm_pid accepts
a PID only when its command line still names that sensor's QEMU and disk, so a
recycled PID is never counted as live or signalled; a /proc sweep covers QEMUs
whose pid file was lost; --destroy refuses while any of them survives and
deletes only the sensor-N disks and seeds this range created.

Scheduler: range_up writes a per-range pid file and up/down/status act only on
a PID whose command line runs this repo's range_attack.py with SB_RANGE_DIR
resolving here, replacing the host-wide pkill that could kill another range.

Status: agents are reported live/stale/unknown per sensor group instead of a
raw registration total (six registrations for three VMs after restarts), and
the scheduler line now separates verified incidents from those without
execution proof, reporting "rule-pushed" rather than implying observed labels.
2026-11-17 13:45:22 -07:00
trvon
af5bfa78e5
validation: multi-sensor range for real promotion-gate evidence
An isolated local lab that extends the single-VM de-risk (deploy/validation/vm,
2026-11-08) to a multi-sensor fleet run over days, to produce the delayed-label
history the Stage 4 fleet gate needs. All local/offline: Caldera on loopback
(rootless Docker, capped), sensor VMs reach it only via the qemu slirp gateway,
no traffic leaves the host; payloads are inert markers (a copy of /bin/sleep
with a marker string), never real malware.

- range_up.sh: boot/resume Caldera + N Arch VMs running the real daemon
  (eBPF + fanotify), benign workload and a Caldera agent as systemd units;
  idempotent and travel-safe (VM disks + scheduler state persist under
  $SB_RANGE_DIR, the scheduler paces by its own uptime, so suspend/reboot just
  pauses it and range_up.sh resumes).
- range_workload.sh: varied benign activity so the detector learns a real
  baseline the emulated incidents must stand out from.
- range_attack.py: one distinct Caldera incident per 30-90 min per sensor
  (fresh ability + operation so each lands in its own replay window); incidents
  are unknown to the rules at exec time, then a later rule-file update + rules
  reload lets the retro-hunt label the earlier windows -- the delayed extra
  positives the gate measures, produced the production way, not injected.
- range_status.sh / range_down.sh (pause or --destroy), README, cloud-init.

Nothing here writes to the repo; the $SB_RANGE_DIR runtime tree is disposable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-15 09:43:22 -07:00