The verifier treated every negative link status as "Caldera never ran it", but
Caldera's states are not signed that way: HIGH_VIZ=-5, UNTRUSTED=-4,
EXECUTE=-3, DISCARD=-2, PAUSE=-1, SUCCESS=0, ERROR=1, TIMEOUT=124. EXECUTE is
the normal in-flight state, so a healthy incident was failed on its first poll
and the range reported zero verified incidents while the marker really ran.
Links are now classified explicitly: EXECUTE and PAUSE keep polling, SUCCESS
still requires a completion timestamp, and DISCARD/UNTRUSTED/HIGH_VIZ/TIMEOUT
or a nonzero exit are terminal failures.
Reconcile also treated a missing operation as pending forever, so an incident
whose operation disappeared (Caldera recreated after the OOM) was retried
indefinitely. A 404 is now terminal ("operation no longer exists"), while any
other read error still stays pending and never becomes "executed". The Caldera
client turns a malformed JSON body into a RuntimeError instead of leaking a
decode error from deep inside the client.
Verified against the live range: a queued link (-3) polled to SUCCESS with a
finish timestamp and was recorded verified with the executing agent; synthetic
cases cover every state code, SUCCESS without a finish, 404 and 500.
The scheduler recorded an incident as complete the moment Caldera accepted the
operation, timed labels from that creation moment, and advanced its label
counter once a rule reload was acknowledged. Restarts also reset its clock, so
incidents could land minutes apart instead of one interval, and a malformed
state file was silently replaced — reusing marker ids that the range's
"unknown at exec time" contract depends on.
Now an incident is verified only from Caldera's own record: a link for this
ability, a non-negative exit, a completion timestamp and the agent that ran it,
polled with bounded retries. Unverified outcomes keep no execution time and are
excluded from label aging, so rule pushes are no longer presented as observed
labels. The next due time is persisted (a restart waits out the interval), the
pending id and marker are saved before the Caldera call, and load() fails
closed — exit 2, file untouched — for malformed, unreadable or wrong-schema
state, defaulting only when the file is absent. The Caldera client is pinned to
the range's loopback origin and /api/v2 paths.
An isolated local lab that extends the single-VM de-risk (deploy/validation/vm,
2026-11-08) to a multi-sensor fleet run over days, to produce the delayed-label
history the Stage 4 fleet gate needs. All local/offline: Caldera on loopback
(rootless Docker, capped), sensor VMs reach it only via the qemu slirp gateway,
no traffic leaves the host; payloads are inert markers (a copy of /bin/sleep
with a marker string), never real malware.
- range_up.sh: boot/resume Caldera + N Arch VMs running the real daemon
(eBPF + fanotify), benign workload and a Caldera agent as systemd units;
idempotent and travel-safe (VM disks + scheduler state persist under
$SB_RANGE_DIR, the scheduler paces by its own uptime, so suspend/reboot just
pauses it and range_up.sh resumes).
- range_workload.sh: varied benign activity so the detector learns a real
baseline the emulated incidents must stand out from.
- range_attack.py: one distinct Caldera incident per 30-90 min per sensor
(fresh ability + operation so each lands in its own replay window); incidents
are unknown to the rules at exec time, then a later rule-file update + rules
reload lets the retro-hunt label the earlier windows -- the delayed extra
positives the gate measures, produced the production way, not injected.
- range_status.sh / range_down.sh (pause or --destroy), README, cloud-init.
Nothing here writes to the repo; the $SB_RANGE_DIR runtime tree is disposable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>