strawberry/AGENTS.md
trvon 50254dc8cc
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 31m52s
release / build (ubuntu:24.04, noble) (push) Successful in 29m30s
release / publish (push) Has been skipped
Docs: codify the system + lean AGENTS/CLAUDE for open-source navigation
- README: front door states the full system (core + experimental shadow-only
  inspection model + distributed fleet learning); new "Fleet learning" usage
  section; learn/ + fleet_export/merge + governor in Architecture; a proper
  "Documentation" map; license corrected MIT -> GPLv3 (matches LICENSE).
- AGENTS.md: crisp YAMS work-loop (retrieve/inspect/persist/report), replaced
  the stale ~64-line dated "Current Project Reality" with an accurate concise
  state + pointers (PLAN/DEVELOPER/git/YAMS), trimmed stale work-order/milestone,
  added "Keep This File Lean".
- CLAUDE.md: work-loop + evidence rule up top.
- gitignore the local validation/attack-range harness (deploy/validation/{docker,vm}).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 10:41:05 -07:00

279 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# AGENTS.md
## Documentation Map
- [RESEARCH.md](RESEARCH.md) — literature-backed rationale for Strawberry's architecture and scope
- [PLAN.md](PLAN.md) — execution order, status, and next milestones
- [README.md](README.md) — public project overview and build/run instructions
- [DEVELOPER.md](DEVELOPER.md) — implementation notes and code structure
- [OFFLINE_INSTALL.md](OFFLINE_INSTALL.md) — hybrid Conan/submodule air-gapped install workflow
- [UX.md](UX.md) — analyst workflow notes and the desktop app (Qt6 + tray) direction
- [API_CONTRACT.md](API_CONTRACT.md) — control-socket protocol, access model, and CLI/evidence JSON shapes
- [docs/prompts/PROMPT-eng-codex.md](docs/prompts/PROMPT-eng-codex.md) — generic YAMS-first operating loop: retrieval contract, metadata, handoff, ask-first list
- [docs/prompts/PROMPT-strawberry-audit.md](docs/prompts/PROMPT-strawberry-audit.md) — audit flow with brick (structure), fold (symbolic SAST) and pipe (state flow)
## Operating Model
Work loop: **retrieve → inspect → act → verify → persist → report**. Run it after a
new session, model switch or compaction; reuse evidence already in context instead of
repeating work. The full generic loop (claim, checkpoints, handoff, ask-first list) is
`docs/prompts/PROMPT-eng-codex.md`, shared with `../yams`/`../brick`. YAMS (`yams` CLI)
is the durable cross-session memory and code index — not the transcript. In brief:
1. **Retrieve.** `yams search "strawberry <task>" --limit 5 --paths-only`, hydrate hits
with `yams get --name <name> --max-bytes 4000` (≤3 first). Notes are leads, not proof;
revalidate mutable facts against current source/runtime.
2. **Inspect.** `yams grep -F "<symbol>" --paths-only` then read the source (`rg` as
fallback). `git status --short` to separate this task's edits from existing work.
3. **Persist.** After meaningful work, store verified findings, decisions and handoffs
in YAMS (`task=strawberry-<topic>`, `source=note|decision|evidence`, `owner=opencode`),
not as new local Markdown; checkpoint long work, re-index changed files (below). Confirm
by read-back — a queued/`--sync` receipt is not saved. Product docs (README, API_CONTRACT,
PLAN, DEVELOPER) stay in the repo and must keep matching the code.
4. **Report.** State changes, the checks actually run and their results, and remaining
uncertainty — with evidence (file:line, check/result). A passing test covers only what
it exercises; an empty search is not proof of absence.
Re-index this task's changed files at checkpoints:
```bash
yams add . --recursive --include "*.cpp,*.hpp,*.h,*.c,*.md,*.build,*.yar,*.service,*.timer,*.path,*.sh,*.conf" \
--exclude "builddir/*,third_party/*,build/*,.ccache/*" \
--metadata "mode=engineering,task=<task>,phase=checkpoint,owner=opencode,source=code"
```
Audits use `docs/prompts/PROMPT-strawberry-audit.md` (brick = structure, fold = bounded
symbolic findings, pipe = `enum class` state flow): verify every finding in source before
fixing or reporting, land a fix with a test, store triage in YAMS (`source=evidence`).
## Background Work & Resource Governor
Strawberry runs on desktops: a scanner that freezes the machine gets
uninstalled. `ResourceGovernor` (`include/strawberry/resource_governor.hpp`,
ported from `../yams`) samples the daemon's CPU/RSS, host PSI (loadavg
fallback), `MemAvailable` and cgroup v2 limits, and the scan profile
(`scan_profile = efficient | balanced | aggressive`) bounds every heavy path.
- Code that starts a thread, loop or job asks the governor first:
`admit(WorkClass::…)` / `wait_admitted(…)`. Background classes (inventory,
memory scans, correlation, rule reloads, blocklist sync) wait while the host
is busy; the starvation guard (`background_max_deferral_s`) still runs them.
- The launch path is never deferred: the exec guard may only *degrade* to
hash/IOC/quarantine (`guard_should_degrade`) and must answer before
`prevention_timeout_ms`; its YARA timeout is capped at that deadline.
- Background reads draw from `pace_read` and drop page cache afterwards;
background threads run at the profile's nice/ioprio
(`src/thread_priority.hpp`). Scans happen outside `DetectionPipeline::mutex_`.
- No free-running loops, no unbounded queues, no clear-all caches on hot
paths, no full-file hashing on the eBPF event thread.
- A new or changed background path needs a constrained-lane KPI before and
after (`make perf-kpi`, `scripts/perf_kpi_report.py --compare`), 3 repeats
for any claim. Code reading is not evidence.
- Timings: record into `PerfStats` (`include/strawberry/perf_stats.hpp`) and
add Tracy zones (`STRAWBERRY_ZONE`, `-Denable_profiling=true`) for any new
heavy step; surface them in status `timings`.
## Invariants and Fuzzing
**Per-change checks.** Every change that touches I/O, callbacks, threads,
parsers or persistence runs, before it is called done:
| Check | Command | Catches |
|---|---|---|
| Unit + property + source invariants | `nice -n 19 meson test -C builddir --num-processes 2` | logic, `tests/test_invariants.cpp`, `tests/source_invariants.py` (iostream/`std::sto*`/unguarded threads/non-`noexcept` C callbacks/empty `catch`) |
| Fault injection | `test_fault_injection` (part of meson test; sweep seeds with `STRAWBERRY_PROPERTY_SEED`) | exceptions, hangs or wrong results when reads/opens fail (ESRCH, EIO, EINTR storms, short reads) |
| Fuzzing | `make fuzz` (clang, capped, niced; `fuzz_io_faults` + parser targets) | crashes, escapes and timeouts on hostile input and I/O |
| fold | `scripts/fold_changed.sh [base]` (fold's LLVM-19 image, `--cpus=2 --memory=6g`, cached) | bounded symbolic findings in changed TUs, including `exception-escape` (exceptions reaching noexcept functions, destructors, C callbacks, thread entries) |
| clang-tidy | `nice -n 19 clang-tidy -p builddir <changed files>` (`.clang-tidy`; enforced checks are errors) | `bugprone-empty-catch` today; more checks are added as the tree is cleaned |
| clang static analyzer (strict) | `make analyze` (`scripts/analyze_changed.sh [base]`; clang-check `--analyze` on changed TUs via `builddir/compile_commands.json`, `nice -n 19`, 2 at a time; any diagnostic fails the run, `ANALYZE_STRICT=0` reports only) | null deref, use-after-free, leaks, uninitialized reads and path-sensitive logic bugs the type system and clang-tidy miss |
| brick / pipe | see `docs/prompts/PROMPT-strawberry-audit.md` | structure, `enum class` state flow |
| Live + KPI | `make docker-live-tests`, `make perf-kpi` (3× `--compare`) | root-only paths, resource regressions |
Verify every tool finding in source before fixing or reporting it, and store
the triage in YAMS (`source=evidence`). Fold's `exception-escape` detector,
`source_invariants.py` R4/R5 and the fault-injection properties together cover
exceptions escaping into C frames, destructors and threads. Fold sees one TU at
a time and the invariants are syntactic, so keep all three. The clang static
analyzer is path-sensitive within a TU (null deref, use-after-free, leaks,
uninitialized reads); it complements fold's bounded symbolic cross-call analysis
and the syntactic invariants rather than replacing either — run all of them, and
index the change in YAMS at each checkpoint before and after.
- Decision logic and parsers carry property tests in `tests/test_invariants.cpp`
(seeded; failures print the seed). A new rule that must always hold gets a
property, not just an example.
- Every parser of untrusted or attacker-influenced input has a libFuzzer
target in `fuzz/` (`make fuzz`, clang, capped and niced). New parsers get a
target and seeds in `fuzz/seeds/<target>`.
- Every file or syscall I/O on a runtime path goes through `include/strawberry/sys_io.hpp`
(`io::read`, `io::pread`, …) so the fault-injection property tests and
`fuzz_io_faults` can fail it. Read loops bound EINTR retries (`io::kMaxEintrRetries`);
a short file at EOF is an error, never a silent prefix.
- **No empty `catch` blocks** (R7, clang-tidy `bugprone-empty-catch`). A
comment is not handling. Handle the exception (log, count, return an error),
or, where nothing more is possible (destructors, `noexcept` C callbacks,
error-reporting fallbacks), record it with `STRAWBERRY_NOTE_SWALLOWED()`
from `include/strawberry/swallowed.hpp`: it counts the exception (status
`resources.swallowed_exceptions`) and logs file:line without allocating or
throwing.
- New static rules go in `tests/source_invariants.py` with a `bad_*.cpp`
fixture in `tests/fixtures/invariants/` (the self-test proves the rule fires).
## Purpose
Strawberry is an early-stage Linux host security scanner prototype in C++23. The intended design remains:
- eBPF/libbpf for process/runtime telemetry
- YARA for file/process scanning
- an embedded local store for indicators + relationship graphs
- a minimal rule-based correlation layer over persisted runtime evidence
- a small daemon/state-machine shell around those components
## Current Project Reality
A working C++23 host security scanner, well past prototype but not finished.
**Status and next steps: `PLAN.md`. Implementation detail: `DEVELOPER.md`. History:
git log / YAMS task notes.** The load-bearing facts:
- **Build/test spine green** — meson/ninja (`-Denable_tests=true`), thin `make` wrapper,
hybrid Conan offline deps. Runtime validation for eBPF/`/proc`/systemd/memory-scan is
Linux-only (`make docker-live-tests`); static review + most unit tests run anywhere.
- **Store** — `Database` over the compressed event store (`src/store/`: zstd segments,
sealed footers, crash-atomic MANIFEST, LogStore recovery); persists indicators, nodes,
edges, findings, labels, telemetry; legacy unqlite import on first open. Evidence 0640,
telemetry root-only 0600.
- **Monitoring** — `ProcessMonitor`: procfs polling + Linux libbpf exec/behavior events
(eBPF falls back to procfs). Process ids via `make_process_node_id()`
(`process_<pid>_<boot8>_<starttime>`) — never hand-build them.
- **Detection** — `DetectionPipeline` + `YaraScanner` (file / `/proc/<pid>/exe` / trigger-only
memory scans, bounded in-memory copy), IOC feeds with per-feed trust, behavior rules;
narrow `CorrelationEngine` (a handful of YARA/hash/lineage/behavior rules — NOT a general
incident engine). Rules auto-update via `PolicyStore` + sandboxed `strawberry-update`.
- **Response** — fanotify exec guard (`audit` default, `enforce` opt-in, fails open),
OpenSnitch containment (hash-pinned), quarantine/suspend/kill by node id. See Guardrails.
- **Learning (Stage 4, shadow-only)** — `learn/`: offline MLP training, live shadow scoring,
replay-gated operator promotion. It *steers inspection budget*, it is not a detector; Stage 5
advisory detection is deferred.
- **Distributed fleet (Stage 6)** — `strawberry export` (redacted bundles: windows, features,
labels, rarity sketches) → central `fleet train` + `fleet overlay` → per-node + fleet-wide
rarity as a model feature. See `fleet_export.*` / `fleet_merge.*` and the
[[distributed-fleet-salt]] note.
- **Interfaces** — Qt6 desktop app + tray (`app/`, `enable_app`); `strawberry-mcp`
(`mcp/`, `enable_mcp`) read-only agent interface over stdio (`scan-fd` fd-passing, never
path-opening); yams-style CLI (`src/cli_commands.cpp`, `noun verb` table, `--json`). All
mutation goes through `ControlServer` (`/run/strawberry/control.sock`, group `strawberry`,
peer-checked, rate-limited, audited); the app reads the DB read-only.
- **macOS** — builds + `meson test` pass (GitHub CI `macos`); Endpoint Security backends
(exec/`AUTH_EXEC`/file-watch) compile with libproc fallback but need root + the ES
entitlement to run live. Linux paths unchanged.
- Still process-centric: no thread/TID telemetry (thread search is future UX, not current).
## Storage Guidance
- keep the `Database` public interface narrow and stable; storage specifics live in `src/store/`
- every decoder takes untrusted bytes and never throws; new record fields are tagged (old readers skip them), record type ids are append-only
- all store I/O goes through `strawberry::io` (fault injection); store changes need the S/M/L properties, the differential test and the store fuzzers green
- evidence is group-readable (0640), telemetry root-only (0600)
## Build System Guidance
Meson/Ninja remains the source of truth.
- Prefer fixing Meson rather than replacing it.
- The repository now also exposes a **thin GNU Make wrapper** for convenience.
- A root `conanfile.py` supports a hybrid offline dependency flow for OpenSSL, zstd and optional extras; nothing is vendored.
- The root Conan recipe forces a C++23 Meson native-file setting so `conan profile detect` defaults do not break the build.
- The repo now also seeds a local `yara/4.5.5` Conan recipe under `conan/local_recipes_index/` for air-gapped or pinned builds.
- The root `Dockerfile` is now the Linux build/test container definition and should stay aligned with the real dependency set.
- The Docker image defaults to `gcc`/`g++` for C++23 inside the Ubuntu 24.04 container.
- `make app` runs the desktop app from the build tree; `make demo` seeds demo evidence and opens it in the app. Qt6 (Core, Gui, Widgets, DBus, Network) is optional; `-Denable_app=disabled` skips the app.
- If someone asks for `make` support, keep it as a wrapper around Meson/Ninja unless explicitly told otherwise.
- If “Make” refers to **Make.com**, treat that as a separate integration effort and do not assume it means GNU Make.
## Recommended Work Order
Current priorities and sequencing live in `PLAN.md`. Standing principles: keep build/test
green; grow the correlation engine only when the evidence model supports a rule cleanly;
keep control-socket/JSON contract changes small and documented in `API_CONTRACT.md`; reuse
the read-only evidence + fixed control verbs for CLI/MCP; defer thread-level UX until
thread/TID telemetry exists; expand packaging/hardening only once the core is stable.
## Testing Priorities
Prioritize tests in this order:
1. **Build configuration tests**
- clean configure
- clean compile
- test toggle works
2. **Pure unit tests**
- hash utilities
- config parsing/serialization
- database persistence semantics
- graph traversal behavior
3. **Integration tests**
- YARA rule load from a temp directory
- file-scan match persistence
- multi-hit indicator persistence for the same hash
- synthetic process-event ingestion pipeline
- synthetic procfs event → `DetectionPipeline` → persisted indicator/graph flow
4. **Linux runtime smoke tests**
- daemon starts with test config
- telemetry path initializes cleanly
- privileged libbpf smoke harness observes a live exec event
- test YARA hit is stored and queryable
- live daemon-owned telemetry path stores a test hit through the real daemon wrapper
5. **Correlation and operator tests**
- multiple related raw events collapse into one explainable finding when intended
- process-first query surfaces return lineage plus related indicators/files
- findings-first query surfaces can pivot back to underlying evidence
## Repo Conventions
- Public headers live in `include/strawberry/`
- Implementations live in `src/`
- Keep public interfaces small and dependency-light
- Prefer RAII and `std::unique_ptr`
- Keep platform-specific code isolated behind implementation details
- Preserve XDG-style default data placement for local storage
## Agent Guardrails
When working in this repo:
- do not describe the current YARA path as fully complete: file/executable and process-memory scans exist, but memory scans run only on triggers (memfd/deleted/ptrace target/operator request), not continuously, and need real-host tuning
- do not describe the current runtime pipeline as fully complete: synthetic procfs-driven scan/store coverage and one live libbpf-to-store smoke pass now exist, but richer live-event enrichment and daemon-grade runtime handling are still the next major steps
- do not describe the daemon runtime as fully complete: the reusable wrapper, daemon-owned smokes, and basic text reporting now exist, but richer live parent-PID enrichment and more advanced reporting/export are still the next major steps
- do not describe the desktop app as a full operations console: it is a findings/process/diagnostics viewer with rules/allowlist/response management and a small set of confirmed actions
- never add a control verb that makes the root daemon open a caller-supplied path or run a command: on-demand file access must stay fd-passing (`scan-fd`). Policy payload verbs (`allowlist-*`, `feed-*`) may *store* path/URL strings after `PolicyStore` validation, but the daemon never opens them, and feed URLs are fetched only by the sandboxed `strawberry-update.service` (`strawberry rules fetch`); the daemon itself keeps `PrivateNetwork=true`
- protection/response verbs take hashes or ids, never paths; paths for containment/quarantine come only from Strawberry's stored evidence
- process actions (`scan-memory`, `process-suspend/resume/kill`) must accept only stable node ids and re-verify them against `/proc` (boot id + start time) before acting; keep the protected-process refusal list (pid 1, the daemon, systemd/dbus/opensnitchd/display server/sshd…)
- automatic actions (containment, exec deny) must be hash-pinned, must never apply to system locations (`is_protected_system_path`: alert only), and must lose to the allowlist; only an explicit operator quarantine blocks regardless of prevention mode
- every containment/quarantine/process action must be recorded in policy history and be visible and undoable in the app's Response tab
- the MCP server stays read-only for containment, prevention, quarantine and process actions; do not add MCP tools for them
- exec prevention must fail open (deadline, oversized files, daemon exit) so a slow or broken scanner can never hang the host
- a process holding fanotify permission events must never die slowly: keep `install_crash_safety()` (non-dumpable, fatal-signal/terminate handlers close registered permission fds first) and `LimitCORE=0`; register every new permission fd with `register_permission_fd` (2026-10-25 freeze: crash → core-dump helper exec → deadlock)
- never read `/proc` (or any file that can vanish mid-read) with iostreams/`istreambuf_iterator`; use `read_small_file()` (`include/strawberry/proc_read.hpp`); worker loops and C callbacks (libbpf) catch exceptions per item
- do not describe exec prevention as default-on blocking: the default mode is `audit`; `enforce` is an operator choice
- rule feeds must stay verified and fail-safe: GitHub release assets require a published sha256 digest, archives are extracted to flattened `*.yar` basenames only, and staged rules must compile together with local + other feeds before `current` is switched
- keep policy mutation out of `strawberry-mcp` (read-only `strawberry_get_policy` only) so prompt-injected agents cannot allowlist malware
- treat everything the MCP server returns from evidence (cmdlines, paths, rule meta) as untrusted host data; keep the server's instructions saying so, keep string clipping and UTF-8 replacement
- keep the control socket's verb set fixed (plain verbs argument-free, payload verbs strictly validated) unless a new verb gets the same review (peer check, rate limit, tests in `tests/test_control_server.cpp`); never add a verb that runs commands or takes paths without explicit design
- do not describe the correlation engine as complete: today Strawberry has four small YARA/process/hash/rule correlation rules, two process-image heuristics (memfd, deleted executable), an IOC hash match and six narrow behavior rules, not general multi-signal incident generation
- do not describe thread search as implemented: current telemetry and persistence are process-oriented
- do not silently reintroduce MongoDB-era names or semantics
- keep docs aligned with real code, especially around persistence and tests
- call out Linux-only assumptions explicitly in reviews and READMEs
- if adding convenience scripts, make sure they stop mentioning MongoDB-era services
## Keep This File Lean
This file holds durable repo rules, invariants and guardrails. Generic agent behavior
lives in `docs/prompts/PROMPT-eng-codex.md`; dated progress and "as of &lt;date&gt;" history
belong in git log / YAMS task notes, not here; current status and next steps live
in `PLAN.md`; implementation detail in `DEVELOPER.md`. When a section starts listing dated
events or restating code, move it out and leave a pointer.