strawberry/docs/store-reduction.md
trvon ebade1ba80
Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:09 -07:00

4.3 KiB

Scheduled telemetry reduction

strawberry store reduce rewrites sealed telemetry segments and drops dead, benign process subtrees (stage 7a). Evidence (findings, detections and their provenance chains) is untouched. On the measured corpus it cut execs by about 91% and windows by about 92% (replay and correlation roughly 11x cheaper) with all detections preserved.

store reduce needs exclusive access to the store, so it refuses while the daemon is running. The scheduled path automates stop, reduce, start:

  • strawberry-reduce.timer fires weekly (randomized by up to 1 hour).
  • strawberry-reduce.service runs /usr/local/lib/strawberry/strawberry-reduce.sh.
  • The wrapper checks store_reduce_enabled in /etc/strawberry/strawberry.conf. If it is not true/1, it prints strawberry-reduce: disabled (store_reduce_enabled) and exits 0 without touching the daemon.
  • Otherwise it stops strawberry.service, runs store reduce --yes --json under nice -n 19, and starts the daemon again. An EXIT trap restarts the daemon on every exit path (reduce failure, signal, error). Only kill -9 of the wrapper itself can bypass it. The wrapper's exit code is the reduce exit code, so a failed run shows as a failed unit in systemctl/journalctl -u strawberry-reduce.

The scheduled path is 7a only: no --skeleton or --merge. A manual strawberry store reduce is never gated and always available.

Before you enable it

  • Monitoring gap. The daemon is stopped for the length of each run, and no live telemetry or detections are recorded during it.
  • Retention change. Reduction permanently drops benign telemetry. Run the recall gate against a copy of your store first to confirm detections survive.
  • The unit is intentionally less sandboxed than strawberry-update.service: it must reach systemd to stop and start the daemon and rewrite the store.

Enable

The installer places the units and wrapper but enables nothing.

# 1. in /etc/strawberry/strawberry.conf
store_reduce_enabled=true
# 2. start the timer
sudo systemctl enable --now strawberry-reduce.timer
# one-off run: sudo systemctl start strawberry-reduce.service

To turn it off, systemctl disable --now strawberry-reduce.timer (or set store_reduce_enabled=false, which makes any fired run a no-op).

Merge horizon and CPR mode (manual store reduce --merge)

--merge (stage 7c) thins repeated (node, type, target) connect/file_write accesses of old records. Two knobs, both valid only together with --merge (otherwise a usage error):

  • --merge-horizon H (whole hours, >= 1): merge only records older than H hours before the newest sealed telemetry record. Default (flag absent) is the 2-day GC margin. Values below 1 hour are never used: the library clamps to 1 h (kMinMergeHorizonNs) so records still being written near the seal boundary are not merged. GC (7a) and skeletonization (7b) always keep the 2-day margin; the active segment is never touched.
  • --merge-mode sample|cpr (default sample). sample is the 7c sampler (groups > 8 keep head 4, tail 1 and a 1-in-32 sample); with no horizon flag it is byte-identical to before. cpr is the Causality-Preserving Reduction of Xu et al. (CCS'16): a repeated access is dropped only when the node's immediately preceding record was the same access, so every causal ordering survives (connect X, exec child, connect X keeps both connects). Each run of identical accesses keeps its first record, and the last occurrence of every group is always kept.

Safety argument: merge never removes a node, a process_exec, a flagged record or anything in a flagged node's keep closure; it only thins repeats of the same access by non-flagged, live-or-dead benign nodes, so a shorter horizon cannot cause a delayed retro-hunt label to lose a subtree. CPR is causality-preserving per node: the run-length-collapsed access sequence of each node is unchanged. Limits: interleaving is judged from same-node records and from an exec whose parent is that node; each sealed segment start is a run boundary (the plan pass is per segment), so a run spanning two segments keeps its first record in each.

--json adds merge_mode and merge_horizon_hours (48 = default margin). The scheduled wrapper has no --merge knob, so it is unchanged.