Commit graph

18 commits

Author SHA1 Message Date
trvon
933a05c3bf
release: add a Debian forky (testing) build leg
Some checks failed
release / draft (push) Has been skipped
release / build (debian:forky, forky) (push) Failing after 4m13s
release / build (debian:trixie, trixie) (push) Successful in 1h6m14s
release / build (ubuntu:24.04, noble) (push) Successful in 1h1m20s
release / publish (push) Has been skipped
The Security Hub server runs Debian forky; build-release.sh accepts forky and the
release matrix gains a debian:forky leg (~1 h more per release, legs run one at a
time). Testing moves: rebuild after a library transition.
2026-11-20 07:44:12 -07:00
trvon
a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
trvon
d81ccbe7c8
gate: a bounded evaluation range, so out-of-sample evidence is possible
The in-sample check landed without any way to bound an evaluation, so every cell
was in-sample and every promotion path refused. Cutting the training bundles
earlier does not help: training spans the bundles, evaluation replays the whole
store with no lower bound, and the store keeps everything (retention 0), so
eval_first <= training_last always held.

replay --policy-eval gains --eval-after NS|auto (0 = unbounded). The bound is
applied where windows are ranked and where injected traces are planned, so the
periods partition only the out-of-sample span; auto derives it as the latest
training-range end over the loaded --eval-model files. Comparisons are written
without first + warm-up arithmetic so INT64_MAX timestamps cannot overflow. A
bound past the end of the store does not produce a silent empty cell: the cell
carries eval_error and promotion_gate refuses naming it.

Provenance also gets honest about width: training_last_ns is the last training
window's END (a window spans window_ns, so using its start left up to 60 s of
trained-on data looking disjoint), and the single-model path (model promote) now
uses the ranked span instead of the whole store, which had guaranteed a refusal.

Verified: R11c passes with 128 out-of-sample windows and a derived bound, and
unbounded returns to in-sample. Refuses an explicit bound later than all history.
2026-11-19 17:27:59 -07:00
trvon
0e3d89d918
range: export the shared constants and keep shellcheck quiet
range_common.sh is a sourced library: range_up.sh, range_down.sh and
range_status.sh consume its constants, which shellcheck cannot see when it
analyses the file alone, so it reported every one as unused. Exporting them
states that interface, and the two $(cd ...) assignments are split from their
export so a failing cd is not masked.
2026-11-19 11:49:41 -07:00
trvon
aa7143e005
range: read Caldera link states correctly, treat a vanished operation as failed
The verifier treated every negative link status as "Caldera never ran it", but
Caldera's states are not signed that way: HIGH_VIZ=-5, UNTRUSTED=-4,
EXECUTE=-3, DISCARD=-2, PAUSE=-1, SUCCESS=0, ERROR=1, TIMEOUT=124. EXECUTE is
the normal in-flight state, so a healthy incident was failed on its first poll
and the range reported zero verified incidents while the marker really ran.

Links are now classified explicitly: EXECUTE and PAUSE keep polling, SUCCESS
still requires a completion timestamp, and DISCARD/UNTRUSTED/HIGH_VIZ/TIMEOUT
or a nonzero exit are terminal failures.

Reconcile also treated a missing operation as pending forever, so an incident
whose operation disappeared (Caldera recreated after the OOM) was retried
indefinitely. A 404 is now terminal ("operation no longer exists"), while any
other read error still stays pending and never becomes "executed". The Caldera
client turns a malformed JSON body into a RuntimeError instead of leaking a
decode error from deep inside the client.

Verified against the live range: a queued link (-3) polled to SUCCESS with a
finish timestamp and was recorded verified with the executing agent; synthetic
cases cover every state code, SUCCESS without a finish, 404 and 500.
2026-11-17 16:16:23 -07:00
trvon
39c5bc4efe
range: host-safety preflight, verified teardown, per-range scheduler identity
Host safety: range_up refuses to start below 10 GiB MemAvailable or 20 GiB free
disk, range_status prints the headroom, and the README documents both floors
and the 8 GiB pause a monitor should enforce. The 2026-11-17 global OOM was
caused by fifteen concurrent compilers plus the range, not the range alone, but
starting blind is what made it easy.

Teardown: sensor ids come from the vms/ contents rather than SB_RANGE_SENSORS,
so a lowered count or missing pid file cannot leave a VM behind; vm_pid accepts
a PID only when its command line still names that sensor's QEMU and disk, so a
recycled PID is never counted as live or signalled; a /proc sweep covers QEMUs
whose pid file was lost; --destroy refuses while any of them survives and
deletes only the sensor-N disks and seeds this range created.

Scheduler: range_up writes a per-range pid file and up/down/status act only on
a PID whose command line runs this repo's range_attack.py with SB_RANGE_DIR
resolving here, replacing the host-wide pkill that could kill another range.

Status: agents are reported live/stale/unknown per sensor group instead of a
raw registration total (six registrations for three VMs after restarts), and
the scheduler line now separates verified incidents from those without
execution proof, reporting "rule-pushed" rather than implying observed labels.
2026-11-17 13:45:22 -07:00
trvon
3cf5461d72
range: verify incidents actually ran, and keep pacing across restarts
The scheduler recorded an incident as complete the moment Caldera accepted the
operation, timed labels from that creation moment, and advanced its label
counter once a rule reload was acknowledged. Restarts also reset its clock, so
incidents could land minutes apart instead of one interval, and a malformed
state file was silently replaced — reusing marker ids that the range's
"unknown at exec time" contract depends on.

Now an incident is verified only from Caldera's own record: a link for this
ability, a non-negative exit, a completion timestamp and the agent that ran it,
polled with bounded retries. Unverified outcomes keep no execution time and are
excluded from label aging, so rule pushes are no longer presented as observed
labels. The next due time is persisted (a restart waits out the interval), the
pending id and marker are saved before the Caldera call, and load() fails
closed — exit 2, file untouched — for malformed, unreadable or wrong-schema
state, defaulting only when the file is absent. The Caldera client is pinned to
the range's loopback origin and /api/v2 paths.
2026-11-17 13:45:18 -07:00
trvon
af5bfa78e5
validation: multi-sensor range for real promotion-gate evidence
An isolated local lab that extends the single-VM de-risk (deploy/validation/vm,
2026-11-08) to a multi-sensor fleet run over days, to produce the delayed-label
history the Stage 4 fleet gate needs. All local/offline: Caldera on loopback
(rootless Docker, capped), sensor VMs reach it only via the qemu slirp gateway,
no traffic leaves the host; payloads are inert markers (a copy of /bin/sleep
with a marker string), never real malware.

- range_up.sh: boot/resume Caldera + N Arch VMs running the real daemon
  (eBPF + fanotify), benign workload and a Caldera agent as systemd units;
  idempotent and travel-safe (VM disks + scheduler state persist under
  $SB_RANGE_DIR, the scheduler paces by its own uptime, so suspend/reboot just
  pauses it and range_up.sh resumes).
- range_workload.sh: varied benign activity so the detector learns a real
  baseline the emulated incidents must stand out from.
- range_attack.py: one distinct Caldera incident per 30-90 min per sensor
  (fresh ability + operation so each lands in its own replay window); incidents
  are unknown to the rules at exec time, then a later rule-file update + rules
  reload lets the retro-hunt label the earlier windows -- the delayed extra
  positives the gate measures, produced the production way, not injected.
- range_status.sh / range_down.sh (pause or --destroy), README, cloud-init.

Nothing here writes to the repo; the $SB_RANGE_DIR runtime tree is disposable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-15 09:43:22 -07:00
trvon
3fd5d23b77
threat traces checker + fleet gate in the dockerized Salt test
- scripts/threat_traces.py: enforces the mechanical rules of the 2026-10-28
  threat-trace pre-registration: required # meta keys; eval families first
  reported 2025-07-01..2026-10-28 with a new-family/new-variant bucket and
  write-up sources only; training families before the cutoff with every
  fifth family (first-report order) in dev; no family in both files;
  RFC 5737 addresses and .example domains only; size caps; and that every
  file compiles with replay's parser. --sha256 prints the freeze digest.
- tests/trace_check.cpp: strawberry-trace-check dev tool (replay's parser).
- strawberry.gate: optional gate.warmup_hours (unset = the pre-registered
  12 h; TEST pillars only).
- deploy/salt/test/fleet_run.sh: gate phase - tops sensor history past the
  10 min injection floor, runs orch.strawberry_gate, asserts 3 gate models
  were published, 9 pooled cells over 3 sensors with positives, sensors keep
  their reports, and still no promoted model anywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:36:24 -07:00
trvon
6f5c62b31c
Stage 4 over fleet data: per-node 3x3 policy evaluation, pooled fleet promotion gate
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:

- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
  evaluates every model on every equal time slice of the ranked history; each
  (model, period) cell simulates the model and rarity-greedy with their own
  hourly budgets exactly as policy_eval does (policy_eval.cells). With one
  model equal to --model and one period the cell equals the single-model
  evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
  positives, inspections, CPU-s, hits, refusals) per (model, period) and
  applies learn::promotion_gate unchanged to each; passes only if all pass.
  Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
  cycle now also publishes all three seed models to dist/gate/; strawberry.gate
  (sensors) evaluates them with the pre-registered evaluation traces; the
  operator-run orch.strawberry_gate pushes the reports back and pools them;
  evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
  criteria and numbers unchanged).

Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:18:04 -07:00
trvon
80070a6497
fleet test: judge --since auto against a same-moment full export
Round-1 bundles are not a fair baseline: the 120 s cursor overlap is a large
share of a few-minute test history. Verified on the live fleet: incremental
192/206/194 windows vs full 230/244/232.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:32 -07:00
trvon
39ffa02ae6
Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes
deploy/salt/test/: builds a release tarball from HEAD in ubuntu:24.04 (CI path),
boots systemd containers (rootless, no --privileged), master==central + 3
sensors with the host's Salt onedir mounted read-only, applies the real states,
runs a workload with planted YARA detections, then two orchestrated cycles.
43 checks (install/config/idempotence, real daemons + telemetry, R2 cache
miss/hit, cp.push collect, 3-seed cycle, canary hold then fleet rollout, no
promoted model anywhere, fleet bits in round 2, --since cursor, dedup).

State fixes found by running them for real:
- init.sls no longer includes strawberry.distribute: a plain state.apply on
  sensors shipped models fleet-wide and bypassed the canary; only the
  orchestration distributes now.
- install.sls: CI tarballs carry a +g<sha> suffix that 'strawberry version'
  does not print, so install.sh re-ran on every apply; record the installed
  release and compare base version + marker.
- orch collect: mv -f (re-collected bundles must not be skipped).
- central cycle: builds the overlay first; a training refusal for too few
  teacher labels (normal on a young fleet) publishes the overlay only and
  keeps the sensors' shadow, any other error still fails.
- distribute.sls: serves only what the cycle published (cp.hash_file; the
  cp.list_master file-list cache can miss a just-published file).
Render check gains overlay-only and full publish cases (20 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:31 -07:00
trvon
b1c7cc41dd
fleet: e2e harness + fleet train/overlay JSON, *.sbx-only bundle listing
fleet train advertised --json but printed text, so the central cycle's
train-seedN.json evidence was not JSON; it now emits one object incl.
duplicate_windows. fleet overlay gains --json (bundles_superseded,
overlays_skipped). Both commands read only *.sbx (no dotfiles) from the
bundle dir, so a cursor or stray file is never fed to the merge.

scripts/fleet_e2e.sh drives the real binary + central cycle script over N
synthetic sensors: --since auto export chain, Salt-pull simulation, 3-seed
train + overlay, sha256 publish, determinism, overlay distribution and a
round-2 export carrying fleet rarity features. 6/6 checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:00:57 -07:00
trvon
265915ac3b
salt: README diagram uses the fleet overlay CLI
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:58:16 -07:00
trvon
4436bee457
Fleet F1: merge dedup per node+window, overlay newest-per-node, export --since
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:03 -07:00
trvon
25d0da1121
Salt: install/config/export/central/distribute states + central cycle orchestration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:43:42 -07:00
trvon
120e296acb
Stage 6: local multi-node fleet-loop validation (gap #4)
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
run-fleet-loop.sh + README: proves the distributed loop end to end with real binaries
(seed N nodes -> round-1 export -> central fleet train + fleet overlay -> distribute ->
round-2 export), asserting the fleet signal propagates (round-1 fleet_feat=0 ->
round-2 fleet_feat>0, N distinct nodes, non-empty overlay). Verified: 2 nodes PASS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 09:32:32 -07:00
trvon
621d00b86e
Stage 6: Salt pillar + deployment contract for distributed mode
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Pillar data (deploy/salt/pillar/) + README for the deploying agent: hub-and-spoke
(sensor export -> central fleet train + overlay -> distribute), role assignment,
CLI surface, security invariants, and the honest gaps (no `fleet overlay` CLI yet,
node_id not wired, no multi-node/docker-swarm validation).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 08:21:44 -07:00