80 lines
4.4 KiB
Markdown
80 lines
4.4 KiB
Markdown
# Strawberry distributed deployment (Salt) — contract for the deploying agent
|
|
|
|
This directory holds the **Salt pillar** (data contract) for deploying Strawberry's
|
|
distributed learning. The SLS **states** (install/config/egress/distribute) are to be
|
|
authored separately (by the deploying agent) and consume this pillar.
|
|
|
|
- `pillar/strawberry.sls` — the full pillar schema with defaults + required fields.
|
|
- `pillar/top.sls` — example role assignment (sensor vs central).
|
|
|
|
## Architecture (what this release actually is)
|
|
|
|
**Hub-and-spoke, NOT peer-to-peer.**
|
|
|
|
```
|
|
sensor nodes central node
|
|
------------ ------------
|
|
daemon -> store (also a sensor, optionally)
|
|
strawberry export --out bundle.sbx --(Salt pull)--> bundle_dir/
|
|
strawberry fleet train bundle_dir --out shadow.sbm
|
|
strawberry fleet overlay bundle_dir --out fleet-overlay.sbo
|
|
model_dir/ <--(Salt push)---------- distribute: shadow.sbm + fleet-overlay.sbo
|
|
(round 2+ exports carry fleet_bits)
|
|
```
|
|
|
|
- There is **no node-to-node transport**, no gossip/DHT, no decentralized aggregation.
|
|
- Egress is **central-pull** (Salt orchestration / `cp.push` of a sealed, redacted bundle).
|
|
The bundle artifact contract is stable, so an object store / mTLS collector can replace
|
|
the pull later without daemon changes.
|
|
- **Promotion is operator-only, never automatic.** States distribute `shadow.sbm`; a model
|
|
only acts after `strawberry model promote` passes the pre-registered gate
|
|
(`docs/evaluation/stage4-promotion-gate.md`). Keep promotion a human step.
|
|
|
|
## CLI surface the states invoke (exists today on `main`)
|
|
|
|
- `strawberry export <store_dir> --out <file>` — sensor egress. Auto-includes fleet rarity
|
|
when `<model_dir>/fleet-overlay.sbo` is present.
|
|
- `strawberry fleet train <bundle_dir> --out <model.sbm>` — central merge + train shadow model.
|
|
|
|
## Current state and remaining gaps (honest status)
|
|
|
|
Closed on `main`:
|
|
|
|
- `strawberry fleet overlay <bundle_dir> --out fleet-overlay.sbo` exists (sums per-node rarity
|
|
sketches; the pillar's `central.overlay_out` can run).
|
|
- `node_id` is a daemon config key: export stamps `node_uid` via `read_node_uid(config.node_id)`.
|
|
- Incremental export: `strawberry export --export-dir DIR --since auto` (or `--since <unix_ns>`).
|
|
`auto` reads `DIR/.export-cursor` (one decimal int64 = the previous export's `time_high_ns`);
|
|
a missing or unparsable cursor means a full export (with a note). After a successful export the
|
|
cursor is rewritten atomically (tmp + rename, 0640). The cursor does not match the prune glob
|
|
(`export-*.sbx`). The next export starts 120 s before the cursor so a window that was still open
|
|
is re-shipped. Labeled windows always re-ship (the replay exposes no label timestamp, so a
|
|
delayed label cannot be dated). Rarity records are always included (the sketch is cumulative).
|
|
|
|
Merge semantics on the central node (repeated exports pulled into ONE bundle dir are safe):
|
|
|
|
- `fleet train` dedups per (node_uid, window_id) across bundles: first-seen order kept, label =
|
|
positive in ANY bundle, features/start/rarity from the newest bundle (manifest
|
|
`created_unix_ns`, tie: later path). Windows with the same id on different nodes are separate.
|
|
- `fleet overlay` uses only the NEWEST bundle per node (its rarity sketch is cumulative, so summing
|
|
every bundle would count a node k times). A bundle stamped `fleet-overlay` in the dir is skipped.
|
|
|
|
Remaining gap:
|
|
|
|
1. **No multi-node validation.** The pipeline is unit/component + single-binary tested; it has
|
|
**not** been run as a real multi-node fleet (N daemons -> central pull -> merge across hosts
|
|
-> distribute -> re-export). A docker-swarm end-to-end is the recommended acceptance test
|
|
before fleet rollout (the plan's deferred `docker-live` milestone).
|
|
|
|
## Paths (binary defaults)
|
|
|
|
- store / `db_path`: `/var/lib/strawberry/evidence`
|
|
- `model_dir`: `/var/lib/strawberry/model` (holds `shadow.sbm`, `promoted.sbm`, `rarity.state`,
|
|
`fleet-overlay.sbo`)
|
|
|
|
## Security invariants the states must preserve
|
|
|
|
- `capture_redact_cmdline: true` (argv redaction before storage) — do not disable on a fleet.
|
|
- No new inbound listener on sensor nodes — export is an outbound read-only artifact pulled
|
|
over the trusted channel; the daemon sandbox stays intact.
|
|
- Reviewed, sha256-pinned binaries only; verify the pin before install.
|