strawberry/AGENTS.md
trvon caf52de559
Add a macOS CI job and document the port
The macos job (macos-15, Homebrew deps, Qt app skipped) builds and
runs meson test. It only runs on GitHub: Forgejo also reads
.github/workflows and has no macOS runner.

README: macOS build, the Linux to macOS backend map, ES requirements
and installed files. API_CONTRACT: macOS socket path and peer
credentials, the endpoint_security and libproc backend values, and
the macOS status fields. DEVELOPER: platform map and the GitHub +
Forgejo remotes. AGENTS: current state of the port, including that
the ES backends have not run live yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 20:53:27 -06:00

27 KiB
Raw Permalink Blame History

AGENTS.md

Documentation Map

  • RESEARCH.md — literature-backed rationale for Strawberry's architecture and scope
  • PLAN.md — execution order, status, and next milestones
  • README.md — public project overview and build/run instructions
  • DEVELOPER.md — implementation notes and code structure
  • OFFLINE_INSTALL.md — hybrid Conan/submodule air-gapped install workflow
  • UX.md — analyst workflow notes and the desktop app (Qt6 + tray) direction
  • API_CONTRACT.md — control-socket protocol, access model, and CLI/evidence JSON shapes
  • docs/prompts/PROMPT-eng-codex.md — generic YAMS-first operating loop: retrieval contract, metadata, handoff, ask-first list
  • docs/prompts/PROMPT-strawberry-audit.md — audit flow with brick (structure), fold (symbolic SAST) and pipe (state flow)

Operating Model

Follow docs/prompts/PROMPT-eng-codex.md (shared with ../yams, ../brick): YAMS-first retrieval (yams grep / search / graph --explore, then hydrate with yams cat --hash), a start claim, checkpoints and a completion handoff. Use task slugs like strawberry-<topic> and agent id opencode-<task>.

Development memory: store audits, triage notes, decisions and handoffs in YAMS (source=note|decision|evidence, plus task, phase, owner=opencode), not as new local Markdown. Product docs (README, API_CONTRACT, PLAN, DEVELOPER) stay in the repo and must keep matching the code. Re-index changed files at checkpoints:

yams add . --recursive --include "*.cpp,*.hpp,*.h,*.c,*.md,*.build,*.yar,*.service,*.timer,*.path,*.sh,*.conf" \
  --exclude "builddir/*,third_party/*,build/*,.ccache/*" \
  --metadata "mode=engineering,task=<task>,phase=checkpoint,owner=opencode,source=code"

Audits use the flow in docs/prompts/PROMPT-strawberry-audit.md: brick for structure, fold for bounded symbolic findings, pipe for enum class state flow. Every finding is verified in source before it is fixed or reported, a fix lands with a test, and the triage is stored in YAMS (source=evidence).

Background Work & Resource Governor

Strawberry runs on desktops: a scanner that freezes the machine gets uninstalled. ResourceGovernor (include/strawberry/resource_governor.hpp, ported from ../yams) samples the daemon's CPU/RSS, host PSI (loadavg fallback), MemAvailable and cgroup v2 limits, and the scan profile (scan_profile = efficient | balanced | aggressive) bounds every heavy path.

  • Code that starts a thread, loop or job asks the governor first: admit(WorkClass::…) / wait_admitted(…). Background classes (inventory, memory scans, correlation, rule reloads, blocklist sync) wait while the host is busy; the starvation guard (background_max_deferral_s) still runs them.
  • The launch path is never deferred: the exec guard may only degrade to hash/IOC/quarantine (guard_should_degrade) and must answer before prevention_timeout_ms; its YARA timeout is capped at that deadline.
  • Background reads draw from pace_read and drop page cache afterwards; background threads run at the profile's nice/ioprio (src/thread_priority.hpp). Scans happen outside DetectionPipeline::mutex_.
  • No free-running loops, no unbounded queues, no clear-all caches on hot paths, no full-file hashing on the eBPF event thread.
  • A new or changed background path needs a constrained-lane KPI before and after (make perf-kpi, scripts/perf_kpi_report.py --compare), 3 repeats for any claim. Code reading is not evidence.
  • Timings: record into PerfStats (include/strawberry/perf_stats.hpp) and add Tracy zones (STRAWBERRY_ZONE, -Denable_profiling=true) for any new heavy step; surface them in status timings.

Invariants and Fuzzing

Per-change checks. Every change that touches I/O, callbacks, threads, parsers or persistence runs, before it is called done:

Check Command Catches
Unit + property + source invariants nice -n 19 meson test -C builddir --num-processes 2 logic, tests/test_invariants.cpp, tests/source_invariants.py (iostream/std::sto*/unguarded threads/non-noexcept C callbacks/empty catch)
Fault injection test_fault_injection (part of meson test; sweep seeds with STRAWBERRY_PROPERTY_SEED) exceptions, hangs or wrong results when reads/opens fail (ESRCH, EIO, EINTR storms, short reads)
Fuzzing make fuzz (clang, capped, niced; fuzz_io_faults + parser targets) crashes, escapes and timeouts on hostile input and I/O
fold scripts/fold_changed.sh [base] (fold's LLVM-19 image, --cpus=2 --memory=6g, cached) bounded symbolic findings in changed TUs, including exception-escape (exceptions reaching noexcept functions, destructors, C callbacks, thread entries)
clang-tidy nice -n 19 clang-tidy -p builddir <changed files> (.clang-tidy; enforced checks are errors) bugprone-empty-catch today; more checks are added as the tree is cleaned
brick / pipe see docs/prompts/PROMPT-strawberry-audit.md structure, enum class state flow
Live + KPI make docker-live-tests, make perf-kpi (3× --compare) root-only paths, resource regressions

Verify every tool finding in source before fixing or reporting it, and store the triage in YAMS (source=evidence). Fold's exception-escape detector, source_invariants.py R4/R5 and the fault-injection properties together cover exceptions escaping into C frames, destructors and threads. Fold sees one TU at a time and the invariants are syntactic, so keep all three.

  • Decision logic and parsers carry property tests in tests/test_invariants.cpp (seeded; failures print the seed). A new rule that must always hold gets a property, not just an example.
  • Every parser of untrusted or attacker-influenced input has a libFuzzer target in fuzz/ (make fuzz, clang, capped and niced). New parsers get a target and seeds in fuzz/seeds/<target>.
  • Every file or syscall I/O on a runtime path goes through include/strawberry/sys_io.hpp (io::read, io::pread, …) so the fault-injection property tests and fuzz_io_faults can fail it. Read loops bound EINTR retries (io::kMaxEintrRetries); a short file at EOF is an error, never a silent prefix.
  • No empty catch blocks (R7, clang-tidy bugprone-empty-catch). A comment is not handling. Handle the exception (log, count, return an error), or, where nothing more is possible (destructors, noexcept C callbacks, error-reporting fallbacks), record it with STRAWBERRY_NOTE_SWALLOWED() from include/strawberry/swallowed.hpp: it counts the exception (status resources.swallowed_exceptions) and logs file:line without allocating or throwing.
  • New static rules go in tests/source_invariants.py with a bad_*.cpp fixture in tests/fixtures/invariants/ (the self-test proves the rule fires).

Purpose

Strawberry is an early-stage Linux host security scanner prototype in C++23. The intended design remains:

  • eBPF/libbpf for process/runtime telemetry
  • YARA for file/process scanning
  • an embedded local store for indicators + relationship graphs
  • a minimal rule-based correlation layer over persisted runtime evidence
  • a small daemon/state-machine shell around those components

Current Project Reality

Treat this repository as a prototype with a working build/test spine, not a finished scanner.

As of April 27, 2026, the most important facts are:

  1. Build and test baseline is green

    • meson_options.txt replaced the broken meson.options flow.
    • meson setup builddir -Denable_tests=true, meson compile -C builddir, and meson test -C builddir --verbose are expected to work.
    • A thin Makefile wrapper is available for make setup, make build, make test, and make run.
  2. The embedded database path is real in this checkout

    • Database is backed by the event store (src/store/: zstd segments, sealed footers, crash-atomic MANIFEST, LogStore recovery).
    • Strawberry persists IndicatorEntry, GraphNode, GraphEdge and findings across reopen; a differential test checks every query against the old store's semantics.
    • A legacy unqlite-format database.db is imported on the first writer open (verified, old file kept as .migrated-<ts>).
    • The unqlite-cpp submodule is gone; src/store/legacy_kv.* reads the old format for the importer.
  3. The detection pipeline is still incomplete

    • ProcessMonitor now has a minimal procfs-polling runtime path plus a Linux libbpf exec-event path, and a privileged Docker smoke run on April 26, 2026 validated live tracepoint attach plus one observed /bin/echo exec event.
    • A reusable DetectionPipeline now owns the scan/store path, and a Docker-validated synthetic runtime test on April 26, 2026 proved procfs event ingestion can trigger YARA matching and persisted process→indicator graph storage.
    • A second privileged Docker smoke run on April 26, 2026 proved one real libbpf-delivered exec event can also flow through DetectionPipeline into persisted indicator + graph storage.
    • A reusable StrawberryDaemon wrapper now owns config loading, TinyFSM transitions, process-monitor startup, and pipeline ownership for the production binary.
    • A daemon-level synthetic test on April 26, 2026 now proves procfs event ingestion can also flow through StrawberryDaemon into persisted indicator + graph storage.
    • A third privileged Docker smoke run on April 26, 2026 now proves one real libbpf-delivered exec event can also flow through StrawberryDaemon itself into persisted indicator + graph storage.
    • A new IncidentReporter now renders concise human-readable summaries over persisted graph state, including first/last seen timing and relative span formatting, and the main binary can print them with --trace-process or --trace-node.
    • A synthetic demo-trace seed path exists (--seed-demo-trace, config/strawberry.demo.conf, make demo) for evaluating the app/reporting flow without a running daemon.
    • The prototype direction is system improvements first: collect a fuller Linux process corpus and refine correlation before expanding UI.
    • The operator UI is the Qt6 desktop app + tray indicator strawberry-tray (app/, meson option enable_app). The temporary Python web evaluator (scripts/strawberry_ui_server.py, ui/, WEB_DEPLOYMENT.md) was removed on October 24, 2026; do not reintroduce an HTTP server for local use.
    • strawberry-mcp (mcp/, meson option enable_mcp, needs nlohmann_json) is the agent interface: MCP over stdio with evidence tools, strawberry_scan_paths, reload and rescan (--read-only hides the last three). On-demand scans use the scan-fd verb: the client opens the file as the user and passes only the descriptor (SCM_RIGHTS); the daemon reads it into memory (≤ scan_max_bytes), never stores results, and audit-logs uid/pid.
    • YARA rules auto-update: PolicyStore (policy.json next to the DB) holds rule feeds (default: YARA Forge core GitHub release) and the allowlist (sha256 / path / rule); strawberry-update.timer + .path run the sandboxed updater daily or on update-rules; the daemon loads local rules plus each enabled feeds/<id>/current in its own YARA namespace. Allowlisted detections are suppressed before storage and counted. The app has Rules and Allowlist tabs; changes go through payload verbs on the control socket (group strawberry, rate-limited, audited).
    • Process images are scanned from a bounded in-memory copy (process_scan_max_bytes), never mmap, and exec events go through a bounded worker queue (events_dropped counts overflow and eBPF ring-buffer drops).
    • The app runs as the desktop user, reads the DB via Database::open_read_only(), and controls the daemon only through ControlServer (/run/strawberry/control.sock, group strawberry, peer-credential checked, rate-limited, audit-logged). Verbs: plain status / policy / reload-rules / rescan / update-rules, fd-passing scan-fd, and one-line JSON payload verbs for allowlist/feeds/protection/containment/quarantine/memory scan/process actions (see API_CONTRACT.md). strawberry --status/--policy-set/... use the same socket.
    • October 25, 2026 protection pass (M1–M5): IOC list feeds, OpenSnitch blocklists + automatic containment, fanotify exec prevention (audit default), eBPF/fanotify behavior telemetry, process-memory YARA scans (yr_rules_scan_proc) and response actions. The service now also needs CAP_SYS_ADMIN and CAP_KILL (systemd-analyze security ≈ 3.2). Root-only tests run via make docker-live-tests.
    • October 25, 2026 macOS port: builds and passes meson test on macOS (Homebrew deps, CI job macos on GitHub). Endpoint Security backends for exec events, AUTH_EXEC prevention and file watching, libproc polling fallback, Mach resource sampler, launchd packaging (packaging/macos/, scripts/macos_install.sh). The ES backends compile and fall back correctly but have not run live: that needs root + the ES entitlement (SIP-disabled VM for ad hoc signing); use make macos-live-tests. Platform map: DEVELOPER.md "Platforms". Linux code paths are unchanged; Linux changes must still pass make docker-live-tests.
    • A first minimal CorrelationEngine now stores explainable findings for process-executable YARA matches grouped by process/hash.
    • A second rule now stores lineage findings when one parent has multiple matched descendants.
    • A third rule now stores findings when the same matched executable hash appears across multiple processes.
    • A fourth rule now stores findings when the same YARA rule appears across multiple processes.
    • Finding list/show commands support --json for UI and automation clients.
    • A ProcessQuery surface supports process search by PID, path, hash, YARA rule, and finding evidence, with JSON output; the desktop app's Processes tab uses it directly.
    • Process search results include first/last seen evidence timestamps and immediate parent/child lineage summaries.
    • The correlation engine is intentionally narrow: it is not yet a general subgraph matcher, anomaly scorer, or full incident engine.
    • The current model is still process-centric. There is no first-class thread/TID telemetry, so “find threads” should be treated as a planned future UX capability, not a current feature.
    • The procfs backend can now emit an initial startup inventory of already-running processes when scan_existing_processes_on_start=true is passed through daemon config / monitor options. This is the current preferred path for building a host-wide process corpus for correlation refinement.
    • Procfs-driven events now also persist parent_pid and event_timestamp_ns, and create a minimal spawned lineage edge when parent information is available.
    • Matched runtime events now also persist an executable-image file node and provenance edges: process → file (executed) and file → indicator (content_matched).
    • YaraScanner now has optional libyara-backed rule compilation and file scanning.
    • scan_process currently performs a best-effort /proc/<pid>/exe scan, not full process-memory scanning.
    • YARA hit metadata now persists through Database, and the daemon path links process nodes to indicator nodes with matched edges.
    • A Conan-backed libyara build and unit-test pass succeeded on April 25, 2026 on the current macOS host, so real rule compilation and file-scan assertions are now locally validated.
    • Linux-specific /proc/<pid>/exe behavior is now covered by a Docker-validated Linux unit test on April 26, 2026.
    • Procfs-based process-event ingestion is covered by a synthetic procfs unit test on April 26, 2026.
    • The Linux Docker build now also validates that the minimal libbpf exec-event object compiles on April 26, 2026.
    • The live libbpf smoke harnesses currently match the observed exec by executable path; callback PIDs may differ from the child PID seen inside a privileged container because of namespace/kernel PID differences.
    • The libbpf path now includes a best-effort fixed-width argv preview (argv0/argv1) for cases where /proc/<pid>/cmdline cannot be read.
    • A privileged /bin/echo smoke on April 26, 2026 validated that live argv preview can surface echo strawberry_process_monitor_live_smoke, and the staged live-pipeline smoke also surfaced live_echo strawberry_live_bpf_pipeline_smoke.
    • The libbpf path now opportunistically attempts sched_process_fork for parent-PID enrichment, but that extra tracepoint is not guaranteed to attach in every privileged container/runtime.
    • In the privileged container smoke path, parent_pid may still be unavailable for live libbpf events because /proc/<host-pid> enrichment can fail across PID namespaces.
    • The daemon is still best understood as a shell around partially implemented subsystems.
    • October 24, 2026 host-readiness pass: the BPF object builds on Arch (UAPI types), execveat is hooked, events resolve and scan through /proc/<pid>/exe (deleted/memfd flagged as heuristic indicators), process node IDs are process_<pid>_<boot8>_<starttime> (legacy process_<pid> when start time is unknown; use make_process_node_id() / indicator process_node_id metadata rather than building IDs by hand), the daemon fails closed without rules or a writable DB, the DB has a single-writer flock plus read-only query opens and transactions, correlation is batched, rule severity comes from meta.severity.
  4. Runtime target is Linux

    • Static review and most unit tests can run anywhere.
    • Real validation for eBPF, /proc, systemd, and process-memory scanning should be treated as Linux-only work.

Storage Guidance

  • keep the Database public interface narrow and stable; storage specifics live in src/store/
  • every decoder takes untrusted bytes and never throws; new record fields are tagged (old readers skip them), record type ids are append-only
  • all store I/O goes through strawberry::io (fault injection); store changes need the S/M/L properties, the differential test and the store fuzzers green
  • evidence is group-readable (0640), telemetry root-only (0600)

Build System Guidance

Meson/Ninja remains the source of truth.

  • Prefer fixing Meson rather than replacing it.
  • The repository now also exposes a thin GNU Make wrapper for convenience.
  • A root conanfile.py supports a hybrid offline dependency flow for OpenSSL, zstd and optional extras; nothing is vendored.
  • The root Conan recipe forces a C++23 Meson native-file setting so conan profile detect defaults do not break the build.
  • The repo now also seeds a local yara/4.5.5 Conan recipe under conan/local_recipes_index/ for air-gapped or pinned builds.
  • The root Dockerfile is now the Linux build/test container definition and should stay aligned with the real dependency set.
  • The Docker image defaults to gcc/g++ for C++23 inside the Ubuntu 24.04 container.
  • make app runs the desktop app from the build tree; make demo seeds demo evidence and opens it in the app. Qt6 (Core, Gui, Widgets, DBus, Network) is optional; -Denable_app=disabled skips the app.
  • If someone asks for make support, keep it as a wrapper around Meson/Ninja unless explicitly told otherwise.
  • If “Make” refers to Make.com, treat that as a separate integration effort and do not assume it means GNU Make.

The next meaningful work should happen in roughly this order:

  1. Keep build/test green while landing follow-on changes.
  2. Keep the daemon-level synthetic and privileged Linux smoke paths aligned with the real binary startup path.
  3. Validate and harden startup process inventory plus ongoing process-event collection on Linux.
  4. Improve runtime enrichment that feeds correlation: executable identity, command line, timestamps, parent/child lineage, and short-lived process handling.
  5. Extend the minimal rule/heuristic correlation engine only when the evidence model supports the rule cleanly.
  6. Add process-first and findings-first query surfaces on top of that correlation layer.
  7. Keep the desktop app focused (findings, processes, diagnostics, reload/rescan); grow it only as evidence quality justifies.
  8. Keep control-socket and JSON contract changes limited and documented in API_CONTRACT.md.
  9. Agent-facing access (CLI/MCP) should reuse the same read-only evidence and fixed control verbs.
  10. Defer thread-level UX until Strawberry actually captures thread/TID telemetry.
  11. Expand packaging, service hardening, and external automation surfaces only after the above are stable.

Testing Priorities

Prioritize tests in this order:

  1. Build configuration tests

    • clean configure
    • clean compile
    • test toggle works
  2. Pure unit tests

    • hash utilities
    • config parsing/serialization
    • database persistence semantics
    • graph traversal behavior
  3. Integration tests

    • YARA rule load from a temp directory
    • file-scan match persistence
    • multi-hit indicator persistence for the same hash
    • synthetic process-event ingestion pipeline
    • synthetic procfs event → DetectionPipeline → persisted indicator/graph flow
  4. Linux runtime smoke tests

    • daemon starts with test config
    • telemetry path initializes cleanly
    • privileged libbpf smoke harness observes a live exec event
    • test YARA hit is stored and queryable
    • live daemon-owned telemetry path stores a test hit through the real daemon wrapper
  5. Correlation and operator tests

    • multiple related raw events collapse into one explainable finding when intended
    • process-first query surfaces return lineage plus related indicators/files
    • findings-first query surfaces can pivot back to underlying evidence

Repo Conventions

  • Public headers live in include/strawberry/
  • Implementations live in src/
  • Keep public interfaces small and dependency-light
  • Prefer RAII and std::unique_ptr
  • Keep platform-specific code isolated behind implementation details
  • Preserve XDG-style default data placement for local storage

Agent Guardrails

When working in this repo:

  • do not describe the current YARA path as fully complete: file/executable and process-memory scans exist, but memory scans run only on triggers (memfd/deleted/ptrace target/operator request), not continuously, and need real-host tuning
  • do not describe the current runtime pipeline as fully complete: synthetic procfs-driven scan/store coverage and one live libbpf-to-store smoke pass now exist, but richer live-event enrichment and daemon-grade runtime handling are still the next major steps
  • do not describe the daemon runtime as fully complete: the reusable wrapper, daemon-owned smokes, and basic text reporting now exist, but richer live parent-PID enrichment and more advanced reporting/export are still the next major steps
  • do not describe the desktop app as a full operations console: it is a findings/process/diagnostics viewer with rules/allowlist/response management and a small set of confirmed actions
  • never add a control verb that makes the root daemon open a caller-supplied path or run a command: on-demand file access must stay fd-passing (scan-fd). Policy payload verbs (allowlist-*, feed-*) may store path/URL strings after PolicyStore validation, but the daemon never opens them, and feed URLs are fetched only by the sandboxed strawberry-update.service (strawberry --update-rules); the daemon itself keeps PrivateNetwork=true
  • protection/response verbs take hashes or ids, never paths; paths for containment/quarantine come only from Strawberry's stored evidence
  • process actions (scan-memory, process-suspend/resume/kill) must accept only stable node ids and re-verify them against /proc (boot id + start time) before acting; keep the protected-process refusal list (pid 1, the daemon, systemd/dbus/opensnitchd/display server/sshd…)
  • automatic actions (containment, exec deny) must be hash-pinned, must never apply to system locations (is_protected_system_path: alert only), and must lose to the allowlist; only an explicit operator quarantine blocks regardless of prevention mode
  • every containment/quarantine/process action must be recorded in policy history and be visible and undoable in the app's Response tab
  • the MCP server stays read-only for containment, prevention, quarantine and process actions; do not add MCP tools for them
  • exec prevention must fail open (deadline, oversized files, daemon exit) so a slow or broken scanner can never hang the host
  • a process holding fanotify permission events must never die slowly: keep install_crash_safety() (non-dumpable, fatal-signal/terminate handlers close registered permission fds first) and LimitCORE=0; register every new permission fd with register_permission_fd (2026-10-25 freeze: crash → core-dump helper exec → deadlock)
  • never read /proc (or any file that can vanish mid-read) with iostreams/istreambuf_iterator; use read_small_file() (include/strawberry/proc_read.hpp); worker loops and C callbacks (libbpf) catch exceptions per item
  • do not describe exec prevention as default-on blocking: the default mode is audit; enforce is an operator choice
  • rule feeds must stay verified and fail-safe: GitHub release assets require a published sha256 digest, archives are extracted to flattened *.yar basenames only, and staged rules must compile together with local + other feeds before current is switched
  • keep policy mutation out of strawberry-mcp (read-only strawberry_get_policy only) so prompt-injected agents cannot allowlist malware
  • treat everything the MCP server returns from evidence (cmdlines, paths, rule meta) as untrusted host data; keep the server's instructions saying so, keep string clipping and UTF-8 replacement
  • keep the control socket's verb set fixed (plain verbs argument-free, payload verbs strictly validated) unless a new verb gets the same review (peer check, rate limit, tests in tests/test_control_server.cpp); never add a verb that runs commands or takes paths without explicit design
  • do not describe the correlation engine as complete: today Strawberry has four small YARA/process/hash/rule correlation rules, two process-image heuristics (memfd, deleted executable), an IOC hash match and six narrow behavior rules, not general multi-signal incident generation
  • do not describe thread search as implemented: current telemetry and persistence are process-oriented
  • do not silently reintroduce MongoDB-era names or semantics
  • keep docs aligned with real code, especially around persistence and tests
  • call out Linux-only assumptions explicitly in reviews and READMEs
  • if adding convenience scripts, make sure they stop mentioning MongoDB-era services

Current Useful Milestone

The current baseline milestone is:

  • repo configures successfully
  • unit tests compile and pass
  • embedded DB persists indicators and graph data to disk

The next milestone should add:

  • persisted YARA hit metadata tied to the database/graph path
  • one libyara-enabled Linux validation pass
  • one privileged Docker-validated live libbpf exec-event pass
  • one synthetic end-to-end scan stored in the database
  • one privileged Docker-validated live daemon-owned scan stored in the database
  • one concise human-readable incident trace report over stored graph data
  • one desktop app + tray indicator for viewing stored evidence and scanner health on the current system ✅
  • one real correlation pass that emits explainable findings over related evidence ✅
  • one process-first investigation surface suitable for eventual Qt desktop work