strawberry/deploy/salt/STATES.md
trvon a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00

6.5 KiB

Strawberry Salt states

States consuming the pillar in pillar/strawberry/init.sls (see README.md for the architecture: hub-and-spoke, Salt-pull, no node-to-node transport).

deploy/salt/
  pillar/top.sls, pillar/strawberry/{init,central}.sls
  salt/strawberry/{init,install,config,export,central,distribute}.sls
  salt/orch/strawberry_cycle.sls
  files/strawberry-central-cycle.sh, strawberry-{export,central-cycle}.{service,timer}
scripts/test_salt_states.sh        # offline render check

Wiring (master)

# /etc/salt/master.d/strawberry.conf
file_recv: True                    # lets minions cp.push their bundles
file_roots:
  base:
    - /srv/salt                    # holds strawberry/dist/ (written by the cycle)
    - <repo>/deploy/salt/salt      # salt://strawberry/*, salt://orch/*
    - <repo>/deploy/salt           # salt://files/*
pillar_roots:
  base:
    - <repo>/deploy/salt/pillar

central.dist_dir (default /srv/salt/strawberry/dist) must be a directory inside a file_roots entry so it resolves as salt://strawberry/dist/. Pillar top matches by minion id ('*' = sensor defaults, 'strawberry-central' adds strawberry.central); pillar top cannot match on pillar. Set strawberry:release:version and artifact_sha256 (string, or a map per distro trixie/noble) in your own pillar.

States

  • install: downloads strawberry-<ver>-<distro>-<arch>.tgz (arch is Debian style, amd64) from the forgejo release with source_hash: sha256=<pin>, extracts to /opt/strawberry/<ver>, runs its scripts/install.sh only when strawberry version differs. Empty version/pin renders a failing state (fail closed). Nodes never build.
  • config: edits only the keys it owns in /etc/strawberry/strawberry.conf (capture_redact_cmdline=true, rule_auto_fetch=false, model_shadow, scan_profile, model_dir, db_path, optional node_id), restarts strawberry on change. Pillar capture_redact_cmdline: false fails the run.
  • export (sensor): export_dir, export_retention_hours (= retain_local_days x 24), export_max_mb; strawberry-export.timer runs the export as root, Nice=19, idle I/O, with --since auto when export.incremental is true (requires the cursor feature in the installed release).
  • central: 0700 bundle/work/dist dirs, /usr/local/sbin/strawberry-central-cycle.sh, nightly strawberry-central-cycle.timer.
  • distribute (sensor; NOT part of init, applied only by the orchestration so it stays canary-gated and never runs before the first cycle has published): shadow.sbm and fleet-overlay.sbo from salt://strawberry/dist/ into model_dir (0600 root), hashes from the .sha256 sidecars. Nothing else is placed in model_dir.

The cycle

salt-run state.orchestrate orch.strawberry_cycle (master must also be the central node):

  1. sensors: systemctl start strawberry-export.service
  2. sensors: salt-call cp.push each <export_dir>/export-*.sbx (lands under /var/cache/salt/master/minions/<id>/files/...); central moves them into bundle_dir. If central is a different host from the master, replace the move with an rsync or file.recurse of that cache tree to the central node.
  3. central: strawberry-central-cycle.sh trains seeds 1, 2, 3 (kept with a summary.json under work_dir/run-<stamp>/ as evidence), builds the overlay, publishes seed 1 as shadow.sbm plus fleet-overlay.sbo and sidecars to dist_dir.
  4. canary-first distribution: distribute.canary_minions get strawberry.distribute; the rest of the sensors only when distribute.canary_ok: true (the operator flips it after checking the canaries). Orchestration reads the master's pillar; override with pillar='{"strawberry":{"distribute":{"canary_ok":true}}}' on the command line.

Operator-only: promotion

No state, orchestration or script runs the model-gate step. After reviewing the three seeds in summary.json and the canaries, the operator runs it by hand on a node, with strawberry model promote and the pre-registered gate evidence (docs/evaluation/stage4-promotion-gate.md). Distribution only ever delivers a shadow model.

Security notes

  • No inbound listener on sensors: the master pulls via cp.push over the Salt channel.
  • Release binaries are pinned by sha256 (source_hash); a missing pin stops the run.
  • capture_redact_cmdline=true is enforced by config; bundles are redacted at export.
  • Model and overlay files are 0600 root; the cycle script publishes atomically.
  • Rules come from the release's pinned snapshot (rule_auto_fetch=false) on fleet nodes.

Check

scripts/test_salt_states.sh renders sensor, central and orchestration SLS offline (scratch config dir, no root) and asserts the invariants above.

Fleet promotion gate (orch.strawberry_gate, operator-run)

After a cycle has trained models (enough teacher labels), the cycle also publishes the three seed models to salt://strawberry/dist/gate/. To collect gate evidence (a passing verdict additionally requires out-of-sample windows - see step 3):

salt-run state.orchestrate orch.strawberry_gate
  1. strawberry.gate on every sensor: fetches the three gate models (sha256 from the sidecars) and runs one replay pass over the node's own history: 3 models x 3 periods, with the pre-registered evaluation traces (pillar strawberry:gate). Raw telemetry stays on the node.
  2. The reports are cp.pushed to the master (== central) and pooled by strawberry fleet gate.
  3. Evidence lands in central.work_dir/gate-<UTC stamp>/ (reports/<minion>.json, verdict.json).

A failing gate is a recorded verdict, not an orchestration failure. The gate also refuses in-sample evidence: every cell publishes in_sample, and a cell whose evaluated windows overlap the model's recorded training range fails with FAIL: out-of-sample: ..., so a full-store replay of the history the models trained on yields a refused verdict. A passing verdict needs a bounded evaluation range (replay --policy-eval --eval-after NS|auto: absolute ns, or auto = the latest training-range end among the gate models). The replay side is in place (ReplayOptions::eval_after_ns, eval_after_from_models, provenance eval_error); the CLI flag is landing in parallel, so a build without it cannot produce a passing gate - see API_CONTRACT.md. Procedure and criteria: docs/evaluation/stage4-fleet-gate.md (criteria unchanged from stage4-promotion-gate.md). Promotion stays an operator step on the nodes after reviewing the evidence; no state runs it.