Commit graph

7 commits

Author SHA1 Message Date
trvon
f4adc12a74
gate: judge the model against a calibrated rarity comparator
The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.

rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.

rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.

Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.

Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.

Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 19:20:55 -07:00
trvon
a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
trvon
0e3d89d918
range: export the shared constants and keep shellcheck quiet
range_common.sh is a sourced library: range_up.sh, range_down.sh and
range_status.sh consume its constants, which shellcheck cannot see when it
analyses the file alone, so it reported every one as unused. Exporting them
states that interface, and the two $(cd ...) assignments are split from their
export so a failing cd is not masked.
2026-11-19 11:49:41 -07:00
trvon
aa7143e005
range: read Caldera link states correctly, treat a vanished operation as failed
The verifier treated every negative link status as "Caldera never ran it", but
Caldera's states are not signed that way: HIGH_VIZ=-5, UNTRUSTED=-4,
EXECUTE=-3, DISCARD=-2, PAUSE=-1, SUCCESS=0, ERROR=1, TIMEOUT=124. EXECUTE is
the normal in-flight state, so a healthy incident was failed on its first poll
and the range reported zero verified incidents while the marker really ran.

Links are now classified explicitly: EXECUTE and PAUSE keep polling, SUCCESS
still requires a completion timestamp, and DISCARD/UNTRUSTED/HIGH_VIZ/TIMEOUT
or a nonzero exit are terminal failures.

Reconcile also treated a missing operation as pending forever, so an incident
whose operation disappeared (Caldera recreated after the OOM) was retried
indefinitely. A 404 is now terminal ("operation no longer exists"), while any
other read error still stays pending and never becomes "executed". The Caldera
client turns a malformed JSON body into a RuntimeError instead of leaking a
decode error from deep inside the client.

Verified against the live range: a queued link (-3) polled to SUCCESS with a
finish timestamp and was recorded verified with the executing agent; synthetic
cases cover every state code, SUCCESS without a finish, 404 and 500.
2026-11-17 16:16:23 -07:00
trvon
39c5bc4efe
range: host-safety preflight, verified teardown, per-range scheduler identity
Host safety: range_up refuses to start below 10 GiB MemAvailable or 20 GiB free
disk, range_status prints the headroom, and the README documents both floors
and the 8 GiB pause a monitor should enforce. The 2026-11-17 global OOM was
caused by fifteen concurrent compilers plus the range, not the range alone, but
starting blind is what made it easy.

Teardown: sensor ids come from the vms/ contents rather than SB_RANGE_SENSORS,
so a lowered count or missing pid file cannot leave a VM behind; vm_pid accepts
a PID only when its command line still names that sensor's QEMU and disk, so a
recycled PID is never counted as live or signalled; a /proc sweep covers QEMUs
whose pid file was lost; --destroy refuses while any of them survives and
deletes only the sensor-N disks and seeds this range created.

Scheduler: range_up writes a per-range pid file and up/down/status act only on
a PID whose command line runs this repo's range_attack.py with SB_RANGE_DIR
resolving here, replacing the host-wide pkill that could kill another range.

Status: agents are reported live/stale/unknown per sensor group instead of a
raw registration total (six registrations for three VMs after restarts), and
the scheduler line now separates verified incidents from those without
execution proof, reporting "rule-pushed" rather than implying observed labels.
2026-11-17 13:45:22 -07:00
trvon
3cf5461d72
range: verify incidents actually ran, and keep pacing across restarts
The scheduler recorded an incident as complete the moment Caldera accepted the
operation, timed labels from that creation moment, and advanced its label
counter once a rule reload was acknowledged. Restarts also reset its clock, so
incidents could land minutes apart instead of one interval, and a malformed
state file was silently replaced — reusing marker ids that the range's
"unknown at exec time" contract depends on.

Now an incident is verified only from Caldera's own record: a link for this
ability, a non-negative exit, a completion timestamp and the agent that ran it,
polled with bounded retries. Unverified outcomes keep no execution time and are
excluded from label aging, so rule pushes are no longer presented as observed
labels. The next due time is persisted (a restart waits out the interval), the
pending id and marker are saved before the Caldera call, and load() fails
closed — exit 2, file untouched — for malformed, unreadable or wrong-schema
state, defaulting only when the file is absent. The Caldera client is pinned to
the range's loopback origin and /api/v2 paths.
2026-11-17 13:45:18 -07:00
trvon
af5bfa78e5
validation: multi-sensor range for real promotion-gate evidence
An isolated local lab that extends the single-VM de-risk (deploy/validation/vm,
2026-11-08) to a multi-sensor fleet run over days, to produce the delayed-label
history the Stage 4 fleet gate needs. All local/offline: Caldera on loopback
(rootless Docker, capped), sensor VMs reach it only via the qemu slirp gateway,
no traffic leaves the host; payloads are inert markers (a copy of /bin/sleep
with a marker string), never real malware.

- range_up.sh: boot/resume Caldera + N Arch VMs running the real daemon
  (eBPF + fanotify), benign workload and a Caldera agent as systemd units;
  idempotent and travel-safe (VM disks + scheduler state persist under
  $SB_RANGE_DIR, the scheduler paces by its own uptime, so suspend/reboot just
  pauses it and range_up.sh resumes).
- range_workload.sh: varied benign activity so the detector learns a real
  baseline the emulated incidents must stand out from.
- range_attack.py: one distinct Caldera incident per 30-90 min per sensor
  (fresh ability + operation so each lands in its own replay window); incidents
  are unknown to the rules at exec time, then a later rule-file update + rules
  reload lets the retro-hunt label the earlier windows -- the delayed extra
  positives the gate measures, produced the production way, not injected.
- range_status.sh / range_down.sh (pause or --destroy), README, cloud-init.

Nothing here writes to the repo; the $SB_RANGE_DIR runtime tree is disposable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-15 09:43:22 -07:00