strawberry/deploy/salt/STATES.md
trvon 933a05c3bf
Some checks failed
release / draft (push) Has been skipped
release / build (debian:forky, forky) (push) Failing after 4m13s
release / build (debian:trixie, trixie) (push) Successful in 1h6m14s
release / build (ubuntu:24.04, noble) (push) Successful in 1h1m20s
release / publish (push) Has been skipped
release: add a Debian forky (testing) build leg
The Security Hub server runs Debian forky; build-release.sh accepts forky and the
release matrix gains a debian:forky leg (~1 h more per release, legs run one at a
time). Testing moves: rebuild after a library transition.
2026-11-20 07:44:12 -07:00

117 lines
6.5 KiB
Markdown

# Strawberry Salt states
States consuming the pillar in `pillar/strawberry/init.sls` (see README.md for the
architecture: hub-and-spoke, Salt-pull, no node-to-node transport).
```
deploy/salt/
pillar/top.sls, pillar/strawberry/{init,central}.sls
salt/strawberry/{init,install,config,export,central,distribute}.sls
salt/orch/strawberry_cycle.sls
files/strawberry-central-cycle.sh, strawberry-{export,central-cycle}.{service,timer}
scripts/test_salt_states.sh # offline render check
```
## Wiring (master)
```yaml
# /etc/salt/master.d/strawberry.conf
file_recv: True # lets minions cp.push their bundles
file_roots:
base:
- /srv/salt # holds strawberry/dist/ (written by the cycle)
- <repo>/deploy/salt/salt # salt://strawberry/*, salt://orch/*
- <repo>/deploy/salt # salt://files/*
pillar_roots:
base:
- <repo>/deploy/salt/pillar
```
`central.dist_dir` (default `/srv/salt/strawberry/dist`) must be a directory inside a
`file_roots` entry so it resolves as `salt://strawberry/dist/`. Pillar top matches by minion
id (`'*'` = sensor defaults, `'strawberry-central'` adds `strawberry.central`); pillar top
cannot match on pillar. Set `strawberry:release:version` and `artifact_sha256` (string, or a
map per distro `trixie`/`noble`/`forky`) in your own pillar.
## States
- `install`: downloads `strawberry-<ver>-<distro>-<arch>.tgz` (arch is Debian style,
`amd64`) from the forgejo release with `source_hash: sha256=<pin>`, extracts to
`/opt/strawberry/<ver>`, runs its `scripts/install.sh` only when `strawberry version` differs.
Empty version/pin renders a failing state (fail closed). Nodes never build.
- `config`: edits only the keys it owns in `/etc/strawberry/strawberry.conf`
(`capture_redact_cmdline=true`, `rule_auto_fetch=false`, `model_shadow`, `scan_profile`,
`model_dir`, `db_path`, optional `node_id`), restarts `strawberry` on change. Pillar
`capture_redact_cmdline: false` fails the run.
- `export` (sensor): `export_dir`, `export_retention_hours` (= `retain_local_days` x 24),
`export_max_mb`; `strawberry-export.timer` runs the export as root, `Nice=19`, idle I/O,
with `--since auto` when `export.incremental` is true (requires the cursor feature in
the installed release).
- `central`: 0700 bundle/work/dist dirs, `/usr/local/sbin/strawberry-central-cycle.sh`,
nightly `strawberry-central-cycle.timer`.
- `distribute` (sensor; NOT part of `init`, applied only by the orchestration so it stays canary-gated and
never runs before the first cycle has published): `shadow.sbm` and `fleet-overlay.sbo` from `salt://strawberry/dist/`
into `model_dir` (0600 root), hashes from the `.sha256` sidecars. Nothing else is placed in
`model_dir`.
## The cycle
`salt-run state.orchestrate orch.strawberry_cycle` (master must also be the central node):
1. sensors: `systemctl start strawberry-export.service`
2. sensors: `salt-call cp.push` each `<export_dir>/export-*.sbx` (lands under
`/var/cache/salt/master/minions/<id>/files/...`); central moves them into `bundle_dir`.
If central is a different host from the master, replace the move with an rsync or
`file.recurse` of that cache tree to the central node.
3. central: `strawberry-central-cycle.sh` trains seeds 1, 2, 3 (kept with a `summary.json`
under `work_dir/run-<stamp>/` as evidence), builds the overlay, publishes seed 1 as
`shadow.sbm` plus `fleet-overlay.sbo` and sidecars to `dist_dir`.
4. canary-first distribution: `distribute.canary_minions` get `strawberry.distribute`; the
rest of the sensors only when `distribute.canary_ok: true` (the operator flips it after
checking the canaries). Orchestration reads the master's pillar; override with
`pillar='{"strawberry":{"distribute":{"canary_ok":true}}}'` on the command line.
## Operator-only: promotion
No state, orchestration or script runs the model-gate step. After reviewing the three seeds
in `summary.json` and the canaries, the operator runs it by hand on a node, with
`strawberry model promote` and the pre-registered gate evidence
(`docs/evaluation/stage4-promotion-gate.md`). Distribution only ever delivers a shadow model.
## Security notes
- No inbound listener on sensors: the master pulls via `cp.push` over the Salt channel.
- Release binaries are pinned by sha256 (`source_hash`); a missing pin stops the run.
- `capture_redact_cmdline=true` is enforced by `config`; bundles are redacted at export.
- Model and overlay files are 0600 root; the cycle script publishes atomically.
- Rules come from the release's pinned snapshot (`rule_auto_fetch=false`) on fleet nodes.
## Check
`scripts/test_salt_states.sh` renders sensor, central and orchestration SLS offline
(scratch config dir, no root) and asserts the invariants above.
## Fleet promotion gate (`orch.strawberry_gate`, operator-run)
After a cycle has trained models (enough teacher labels), the cycle also publishes the three seed
models to `salt://strawberry/dist/gate/`. To collect gate evidence (a *passing* verdict additionally
requires out-of-sample windows - see step 3):
salt-run state.orchestrate orch.strawberry_gate
1. `strawberry.gate` on every sensor: fetches the three gate models (sha256 from the sidecars) and runs
one replay pass over the node's own history: 3 models x 3 periods, with the pre-registered
evaluation traces (`pillar strawberry:gate`). Raw telemetry stays on the node.
2. The reports are `cp.push`ed to the master (== central) and pooled by `strawberry fleet gate`.
3. Evidence lands in `central.work_dir/gate-<UTC stamp>/` (`reports/<minion>.json`, `verdict.json`).
A failing gate is a recorded verdict, not an orchestration failure. The gate also refuses
in-sample evidence: every cell publishes `in_sample`, and a cell whose evaluated windows overlap the
model's recorded training range fails with `FAIL: out-of-sample: ...`, so a full-store replay of the
history the models trained on yields a refused verdict. A passing verdict needs a bounded evaluation
range (`replay --policy-eval --eval-after NS|auto`: absolute ns, or `auto` = the latest training-range
end among the gate models). The replay side is in place (`ReplayOptions::eval_after_ns`,
`eval_after_from_models`, provenance `eval_error`); the CLI flag is landing in parallel, so a build
without it cannot produce a passing gate - see `API_CONTRACT.md`. Procedure and criteria:
`docs/evaluation/stage4-fleet-gate.md` (criteria unchanged from `stage4-promotion-gate.md`).
Promotion stays an operator step on the nodes after reviewing the evidence; no state runs it.