Some checks failed
release / draft (push) Has been skipped
release / build (debian:forky, forky) (push) Failing after 4m13s
release / build (debian:trixie, trixie) (push) Successful in 1h6m14s
release / build (ubuntu:24.04, noble) (push) Successful in 1h1m20s
release / publish (push) Has been skipped
The Security Hub server runs Debian forky; build-release.sh accepts forky and the release matrix gains a debian:forky leg (~1 h more per release, legs run one at a time). Testing moves: rebuild after a library transition.
117 lines
6.5 KiB
Markdown
117 lines
6.5 KiB
Markdown
# Strawberry Salt states
|
|
|
|
States consuming the pillar in `pillar/strawberry/init.sls` (see README.md for the
|
|
architecture: hub-and-spoke, Salt-pull, no node-to-node transport).
|
|
|
|
```
|
|
deploy/salt/
|
|
pillar/top.sls, pillar/strawberry/{init,central}.sls
|
|
salt/strawberry/{init,install,config,export,central,distribute}.sls
|
|
salt/orch/strawberry_cycle.sls
|
|
files/strawberry-central-cycle.sh, strawberry-{export,central-cycle}.{service,timer}
|
|
scripts/test_salt_states.sh # offline render check
|
|
```
|
|
|
|
## Wiring (master)
|
|
|
|
```yaml
|
|
# /etc/salt/master.d/strawberry.conf
|
|
file_recv: True # lets minions cp.push their bundles
|
|
file_roots:
|
|
base:
|
|
- /srv/salt # holds strawberry/dist/ (written by the cycle)
|
|
- <repo>/deploy/salt/salt # salt://strawberry/*, salt://orch/*
|
|
- <repo>/deploy/salt # salt://files/*
|
|
pillar_roots:
|
|
base:
|
|
- <repo>/deploy/salt/pillar
|
|
```
|
|
|
|
`central.dist_dir` (default `/srv/salt/strawberry/dist`) must be a directory inside a
|
|
`file_roots` entry so it resolves as `salt://strawberry/dist/`. Pillar top matches by minion
|
|
id (`'*'` = sensor defaults, `'strawberry-central'` adds `strawberry.central`); pillar top
|
|
cannot match on pillar. Set `strawberry:release:version` and `artifact_sha256` (string, or a
|
|
map per distro `trixie`/`noble`/`forky`) in your own pillar.
|
|
|
|
## States
|
|
|
|
- `install`: downloads `strawberry-<ver>-<distro>-<arch>.tgz` (arch is Debian style,
|
|
`amd64`) from the forgejo release with `source_hash: sha256=<pin>`, extracts to
|
|
`/opt/strawberry/<ver>`, runs its `scripts/install.sh` only when `strawberry version` differs.
|
|
Empty version/pin renders a failing state (fail closed). Nodes never build.
|
|
- `config`: edits only the keys it owns in `/etc/strawberry/strawberry.conf`
|
|
(`capture_redact_cmdline=true`, `rule_auto_fetch=false`, `model_shadow`, `scan_profile`,
|
|
`model_dir`, `db_path`, optional `node_id`), restarts `strawberry` on change. Pillar
|
|
`capture_redact_cmdline: false` fails the run.
|
|
- `export` (sensor): `export_dir`, `export_retention_hours` (= `retain_local_days` x 24),
|
|
`export_max_mb`; `strawberry-export.timer` runs the export as root, `Nice=19`, idle I/O,
|
|
with `--since auto` when `export.incremental` is true (requires the cursor feature in
|
|
the installed release).
|
|
- `central`: 0700 bundle/work/dist dirs, `/usr/local/sbin/strawberry-central-cycle.sh`,
|
|
nightly `strawberry-central-cycle.timer`.
|
|
- `distribute` (sensor; NOT part of `init`, applied only by the orchestration so it stays canary-gated and
|
|
never runs before the first cycle has published): `shadow.sbm` and `fleet-overlay.sbo` from `salt://strawberry/dist/`
|
|
into `model_dir` (0600 root), hashes from the `.sha256` sidecars. Nothing else is placed in
|
|
`model_dir`.
|
|
|
|
## The cycle
|
|
|
|
`salt-run state.orchestrate orch.strawberry_cycle` (master must also be the central node):
|
|
|
|
1. sensors: `systemctl start strawberry-export.service`
|
|
2. sensors: `salt-call cp.push` each `<export_dir>/export-*.sbx` (lands under
|
|
`/var/cache/salt/master/minions/<id>/files/...`); central moves them into `bundle_dir`.
|
|
If central is a different host from the master, replace the move with an rsync or
|
|
`file.recurse` of that cache tree to the central node.
|
|
3. central: `strawberry-central-cycle.sh` trains seeds 1, 2, 3 (kept with a `summary.json`
|
|
under `work_dir/run-<stamp>/` as evidence), builds the overlay, publishes seed 1 as
|
|
`shadow.sbm` plus `fleet-overlay.sbo` and sidecars to `dist_dir`.
|
|
4. canary-first distribution: `distribute.canary_minions` get `strawberry.distribute`; the
|
|
rest of the sensors only when `distribute.canary_ok: true` (the operator flips it after
|
|
checking the canaries). Orchestration reads the master's pillar; override with
|
|
`pillar='{"strawberry":{"distribute":{"canary_ok":true}}}'` on the command line.
|
|
|
|
## Operator-only: promotion
|
|
|
|
No state, orchestration or script runs the model-gate step. After reviewing the three seeds
|
|
in `summary.json` and the canaries, the operator runs it by hand on a node, with
|
|
`strawberry model promote` and the pre-registered gate evidence
|
|
(`docs/evaluation/stage4-promotion-gate.md`). Distribution only ever delivers a shadow model.
|
|
|
|
## Security notes
|
|
|
|
- No inbound listener on sensors: the master pulls via `cp.push` over the Salt channel.
|
|
- Release binaries are pinned by sha256 (`source_hash`); a missing pin stops the run.
|
|
- `capture_redact_cmdline=true` is enforced by `config`; bundles are redacted at export.
|
|
- Model and overlay files are 0600 root; the cycle script publishes atomically.
|
|
- Rules come from the release's pinned snapshot (`rule_auto_fetch=false`) on fleet nodes.
|
|
|
|
## Check
|
|
|
|
`scripts/test_salt_states.sh` renders sensor, central and orchestration SLS offline
|
|
(scratch config dir, no root) and asserts the invariants above.
|
|
|
|
## Fleet promotion gate (`orch.strawberry_gate`, operator-run)
|
|
|
|
After a cycle has trained models (enough teacher labels), the cycle also publishes the three seed
|
|
models to `salt://strawberry/dist/gate/`. To collect gate evidence (a *passing* verdict additionally
|
|
requires out-of-sample windows - see step 3):
|
|
|
|
salt-run state.orchestrate orch.strawberry_gate
|
|
|
|
1. `strawberry.gate` on every sensor: fetches the three gate models (sha256 from the sidecars) and runs
|
|
one replay pass over the node's own history: 3 models x 3 periods, with the pre-registered
|
|
evaluation traces (`pillar strawberry:gate`). Raw telemetry stays on the node.
|
|
2. The reports are `cp.push`ed to the master (== central) and pooled by `strawberry fleet gate`.
|
|
3. Evidence lands in `central.work_dir/gate-<UTC stamp>/` (`reports/<minion>.json`, `verdict.json`).
|
|
|
|
A failing gate is a recorded verdict, not an orchestration failure. The gate also refuses
|
|
in-sample evidence: every cell publishes `in_sample`, and a cell whose evaluated windows overlap the
|
|
model's recorded training range fails with `FAIL: out-of-sample: ...`, so a full-store replay of the
|
|
history the models trained on yields a refused verdict. A passing verdict needs a bounded evaluation
|
|
range (`replay --policy-eval --eval-after NS|auto`: absolute ns, or `auto` = the latest training-range
|
|
end among the gate models). The replay side is in place (`ReplayOptions::eval_after_ns`,
|
|
`eval_after_from_models`, provenance `eval_error`); the CLI flag is landing in parallel, so a build
|
|
without it cannot produce a passing gate - see `API_CONTRACT.md`. Procedure and criteria:
|
|
`docs/evaluation/stage4-fleet-gate.md` (criteria unchanged from `stage4-promotion-gate.md`).
|
|
Promotion stays an operator step on the nodes after reviewing the evidence; no state runs it.
|