strawberry/docs/evaluation/stage4-execution-gate.md
trvon bbfda149a5
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 4: execute a promoted model's inspections, under guardrails
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:

- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
  recorded as indicators of kind model_inspection (findings titled "YARA
  match found by a deeper rescan the inspection model requested") and
  positive labels.

watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.

Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
  never blocks); WorkClass::model_inspect lets the governor defer it, and
  a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
  decision's generation (demoting or replacing it revokes queued work),
  that the root's node id and raw image path still match (decisions carry
  root_image, not the feature template), and skips protected processes.
  A deep rescan discards its result if the process, image or rules changed
  during the scan.
- An acting decision is durably written before it is dispatched; every
  attempt becomes an inspection_outcome record (queued, scanned, skipped,
  deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
  exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
  wiring points, are reviewed by hand). Models never block, contain, kill
  or create findings directly.
- Config model_act and model_lambda; status model block reports acting
  stats; timings.model_inspection.

docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.

Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.

Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 21:03:57 -06:00

5.9 KiB
Raw Permalink Blame History

Stage 4 execution-policy gate (2026-10-27)

This is a method amendment to the original promotion gate, written before evaluating the new execution path. It does not change that gate's numeric thresholds. Previous promotion results must be rerun with this method.

Deployed action set

The daemon can execute memory_scan and deep_rescan, or do nothing. It does not implement watch_subtree; rule-change rescanning remains the independent retro-hunt service. Both model and rarity-greedy replay policies are restricted to the executable action set through executable_policy(). Promoted decisions are greedy, as in replay; epsilon exploration remains shadow-only.

model promote and replay --policy-eval use the configured scan profile's 10 / 30 / 120 estimated inspection seconds per hour (efficient / balanced / aggressive). Expected costs and reach are still the pre-registered constants. These are allocation estimates, not measured CPU consumption or a hard CPU-time quota. Real durations appear in inspection_outcome and timings.model_inspection; ResourceGovernor admission, bounded queues, paced reads, byte limits and scanner timeouts remain the runtime resource controls. Calibrating cost estimates and proving an actual inspection CPU budget remain required before claiming the full Stage 4 resource milestone.

Acceptance and compatibility

The original thresholds still apply: at least 10 extra positives, model hits per estimated inspection second at least 1.5 times rarity-greedy, and model recall no lower. The operator must still record three disjoint replay periods and three training seeds before relying on a promoted model. One CLI run does not establish that evidence, nor does this implementation claim it.

After a passing gate, model promote appends an integrity-covered inspection-execution-v1 budget=<seconds> suffix to the promoted model's note. The daemon refuses old promoted files and files evaluated for a different scan profile, reporting model.promoted_error. This prevents an upgrade from silently activating a model evaluated over unsupported actions. Re-promote explicitly; do not edit the note to bypass evaluation. No model is promoted by a daemon, MCP tool, or by this implementation work.

Runtime contract

  • model_shadow=false or telemetry=false disables both shadow and acting model processing. A shadow file alone never authorizes execution.
  • At startup and approximately once a minute, the daemon loads shadow.sbm and promoted.sbm from model_dir. Missing, corrupt, symlinked, wrong-owner, group-writable or world-writable model files are not used. Removal/replacement revokes queued work when observed; an already running scanner is not forcibly interrupted. The directory must remain operator-controlled.
  • A decision is synced to telemetry before dispatch. Failure to record it means no execution. Queues are bounded, and refusals/drop counters are visible.
  • Acting jobs must identify a stable process node (PID, boot prefix, start time) and the same executable. Protected processes are refused. Memory jobs retain this identity through the queue and recheck it around the scan. Rule/operator jobs have priority over queued model jobs; a scan already in progress can still occupy the shared scanner until its timeout.
  • Deep rescans open the current image, read a bounded paced private copy, drop page cache and recheck image/rules identity before storing any new YARA hits. On Linux this uses /proc/<pid>/exe. macOS has no equivalent descriptor; its path-based image check is weaker and requires separate live validation.
  • Model output is never itself a finding or a response instruction. A real YARA match can enter the existing evidence/correlation path and remains subject to allowlisting and the existing independent response policy.
  • Outcomes are root-only telemetry, not an MCP mutation surface. Shadow and acting records remain distinct; replay's shadow_logged tally excludes acting decisions.

This increment does not implement online retraining, learned inspection costs, subtree watching, or Stage 5 advisory detections. Live model-triggered inspection and constrained-lane evidence must be reported separately from unit coverage.

Validation status (2026-10-27)

This implementation passed 48 local tests, 46 tests in a clean-object Linux Docker build, and the privileged exec-guard, behavior, file-watch, YARA and daemon-runtime checks. Three seeded sweeps cover inspection/replay/I/O properties. Source invariants pass; selected Fold/clang-tidy checks were reviewed with their bounded coverage retained. The final sanitizer refresh ran all 20 fuzz targets for 10 seconds each with fail-fast UBSan, plus a 60-second health parser follow-up. An earlier recovered UBSan error was fixed with a regression; that earlier run is not counted as clean.

Three quiet, alternating baseline/changed KPI pairs used the same rule snapshot, 2 CPUs and 2 GiB per container. All six runs passed the existing thresholds:

Pair Baseline idle CPU Changed idle CPU Exec-decision p95, both Dropped events, both
1 1.23% 0.22% 0.031 ms 0
2 0.13% 0.23% 0.031 ms 0
3 0.12% 0.20% 0.031 ms 0

These are default-lane measurements: every run had no active model, and the changed runs submitted zero model inspections. They do not satisfy an acting-model resource or efficacy gate. Peak container memory varied (baseline 305.5–342.8 MB, changed 316.9–378.8 MB), so passing these thresholds is not a blanket claim of resource improvement. Earlier noisy runs, including an idle-CPU failure, are retained in the evidence rather than discarded. No model was promoted and no host service was changed during this validation. Evidence summaries, artifact references and triage are recorded in YAMS under task strawberry-stage4-execution; local run artifacts use batch 20261027-173147 under build/perf/ and build/audit/stage4/.