strawberry/docs/store-reduction.md
trvon ebade1ba80
Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:09 -07:00

82 lines
4.3 KiB
Markdown

# Scheduled telemetry reduction
`strawberry store reduce` rewrites sealed telemetry segments and drops dead,
benign process subtrees (stage 7a). Evidence (findings, detections and their
provenance chains) is untouched. On the measured corpus it cut execs by about
91% and windows by about 92% (replay and correlation roughly 11x cheaper) with
all detections preserved.
`store reduce` needs exclusive access to the store, so it refuses while the
daemon is running. The scheduled path automates **stop, reduce, start**:
- `strawberry-reduce.timer` fires weekly (randomized by up to 1 hour).
- `strawberry-reduce.service` runs `/usr/local/lib/strawberry/strawberry-reduce.sh`.
- The wrapper checks `store_reduce_enabled` in `/etc/strawberry/strawberry.conf`.
If it is not `true`/`1`, it prints `strawberry-reduce: disabled
(store_reduce_enabled)` and exits 0 without touching the daemon.
- Otherwise it stops `strawberry.service`, runs `store reduce --yes --json`
under `nice -n 19`, and starts the daemon again. An EXIT trap restarts the
daemon on every exit path (reduce failure, signal, error). Only `kill -9` of
the wrapper itself can bypass it. The wrapper's exit code is the reduce exit
code, so a failed run shows as a failed unit in `systemctl`/`journalctl -u
strawberry-reduce`.
The scheduled path is **7a only**: no `--skeleton` or `--merge`. A manual
`strawberry store reduce` is never gated and always available.
## Before you enable it
- **Monitoring gap.** The daemon is stopped for the length of each run, and no
live telemetry or detections are recorded during it.
- **Retention change.** Reduction permanently drops benign telemetry. Run the
recall gate against a copy of your store first to confirm detections survive.
- The unit is intentionally less sandboxed than `strawberry-update.service`:
it must reach systemd to stop and start the daemon and rewrite the store.
## Enable
The installer places the units and wrapper but enables nothing.
```sh
# 1. in /etc/strawberry/strawberry.conf
store_reduce_enabled=true
# 2. start the timer
sudo systemctl enable --now strawberry-reduce.timer
# one-off run: sudo systemctl start strawberry-reduce.service
```
To turn it off, `systemctl disable --now strawberry-reduce.timer` (or set
`store_reduce_enabled=false`, which makes any fired run a no-op).
## Merge horizon and CPR mode (manual `store reduce --merge`)
`--merge` (stage 7c) thins repeated `(node, type, target)` connect/file_write
accesses of old records. Two knobs, both valid only together with `--merge`
(otherwise a usage error):
- `--merge-horizon H` (whole hours, >= 1): merge only records older than `H`
hours before the newest sealed telemetry record. Default (flag absent) is the
2-day GC margin. Values below 1 hour are never used: the library clamps to
1 h (`kMinMergeHorizonNs`) so records still being written near the seal
boundary are not merged. GC (7a) and skeletonization (7b) always keep the
2-day margin; the active segment is never touched.
- `--merge-mode sample|cpr` (default `sample`). `sample` is the 7c sampler
(groups > 8 keep head 4, tail 1 and a 1-in-32 sample); with no horizon flag
it is byte-identical to before. `cpr` is the Causality-Preserving Reduction
of Xu et al. (CCS'16): a repeated access is dropped only when the node's
immediately preceding record was the same access, so every causal ordering
survives (connect X, exec child, connect X keeps both connects). Each run of
identical accesses keeps its first record, and the last occurrence of every
group is always kept.
Safety argument: merge never removes a node, a process_exec, a flagged record
or anything in a flagged node's keep closure; it only thins repeats of the same
access by non-flagged, live-or-dead benign nodes, so a shorter horizon cannot
cause a delayed retro-hunt label to lose a subtree. CPR is causality-preserving
per node: the run-length-collapsed access sequence of each node is unchanged.
Limits: interleaving is judged from same-node records and from an exec whose
parent is that node; each sealed segment start is a run boundary (the plan pass
is per segment), so a run spanning two segments keeps its first record in each.
`--json` adds `merge_mode` and `merge_horizon_hours` (48 = default margin).
The scheduled wrapper has no `--merge` knob, so it is unchanged.