strawberry/PLAN.md
trvon dc50f06f8d
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Drop unqlite-cpp and import legacy databases
Nothing shipped used the library after the event store landed. A small
reader/writer for its file format (src/store/legacy_kv) replaces it for
the legacy code, and the submodule, subproject and library test are
removed.

A writer open now imports a legacy database.db file into the event
store: indicators are replayed in an order consistent with the old
hash/rule indexes, the result is verified against the old file, the
old file is kept as database.db.migrated-<ts>, and an import
interrupted between its final renames is finished on the next open.
Read-only handles read a legacy file directly and switch to the store
once it is imported. Ids re-stored under another hash are reported
(their old index entries cannot be carried over).

Tests: legacy import properties I1-I6 (locked and corrupt files are
left untouched), the differential test now shares its comparisons, and
fuzz_legacy_kv covers the file parser. A copy of a real 137 KB legacy
database imported cleanly into 7 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:46:21 -06:00

25 KiB
Raw Permalink Blame History

Implementation Plan

Execution plan for turning Strawberry from scaffold into a minimally real, research-backed prototype.

Linked Documents

Planning Principle

This plan follows the literature in RESEARCH.md:

  • reliable runtime telemetry before advanced analytics
  • signatures as enrichment, not the only detector
  • graph context as a force multiplier
  • compact, inspectable local workflows before bigger platform ambitions

Current Status Snapshot

As of April 27, 2026:

  • Phase 0 is complete: the repo configures and compiles again
  • Phase 1 is complete: the current unit suite passes
  • Phase 2 is complete in the current checkout: Strawberry persists indicators and graph data across reopen through vendored third_party/unqlite-cpp (replaced by the compressed event store on 2026-10-25; the submodule was removed)
  • direct persistence tests now cover reopen, commit, rollback, and read-only behavior
  • Phase 3 has started: optional libyara-backed rule loading and file scanning are implemented
  • YARA hit metadata now persists through the database path, including multi-hit support for the same hash and process→indicator graph linkage
  • scan_process now performs a best-effort /proc/<pid>/exe scan when libyara is available
  • a Conan-backed libyara build and test pass succeeded on April 25, 2026 on the current macOS host, including real rule compilation and file-scan assertions
  • Linux-specific /proc/<pid>/exe behavior is now covered by a Docker-validated unit test on April 26, 2026
  • a minimal procfs-polling process-event path now exists on April 26, 2026
  • a reusable DetectionPipeline now exists, and a Docker-validated synthetic runtime test proves procfs event ingestion can drive executable scanning plus persisted indicator/graph storage on April 26, 2026
  • a minimal libbpf exec-event backend now compiles in the Linux Docker build on April 26, 2026
  • a privileged Linux Docker smoke run now validates live libbpf tracepoint attach and exec-event delivery on April 26, 2026
  • a second privileged Linux Docker smoke run now validates live libbpf exec-event persistence through DetectionPipeline on April 26, 2026
  • a reusable StrawberryDaemon wrapper now owns the daemon shell around config loading, process-monitor startup, and DetectionPipeline
  • a daemon-level synthetic test now validates procfs event ingestion through StrawberryDaemon into persisted indicator/graph storage on April 26, 2026
  • a third privileged Linux Docker smoke run now validates live libbpf exec-event persistence through StrawberryDaemon itself on April 26, 2026
  • a new IncidentReporter now renders concise human-readable summaries over persisted graph state with first/last seen timing, and the main binary can print them with --trace-process / --trace-node
  • the operator UI is now the Qt6 desktop app + tray indicator strawberry-tray (October 24, 2026); the temporary Python web evaluator was removed
  • a synthetic demo-trace seed path exists (make demo), so the app/reporting flow can be evaluated without a running daemon
  • the active direction remains system improvements first so real Linux process collection and correlation evidence quality lead the work
  • a first minimal correlation engine now exists: Strawberry can group process-executable YARA indicators by process/hash and persist an explainable finding
  • a second correlation rule now flags multiple matched descendants under one parent process lineage
  • a third correlation rule now flags the same matched executable hash across multiple processes
  • a fourth correlation rule now flags the same YARA rule across multiple processes
  • findings now have JSON output through the CLI for UI/automation consumers
  • a process search query surface now exists for PID, path, hash, YARA rule, and finding pivots, with JSON output for UI/automation consumers
  • the correlation engine is intentionally small and rule-based; it is not yet a general subgraph matcher, anomaly detector, or full incident engine
  • the current data model is still process-centric, not thread-centric; there is no first-class TID/thread telemetry or thread search UX yet
  • procfs polling now has an optional startup inventory mode that emits already-running processes before continuing to watch for newly observed PIDs; this is the current path for collecting a fuller Linux process corpus for correlation refinement
  • the libbpf path now has a best-effort fixed-width live argv preview (argv0/argv1); both the /bin/echo smoke and the staged pipeline smoke validated it on April 26, 2026
  • the libbpf path now opportunistically attempts sched_process_fork for parent-PID enrichment, but the privileged container smoke confirms that this additional tracepoint may be denied while the main exec-event path still succeeds
  • procfs-driven runtime events now also persist parent_pid and event_timestamp_ns, and create minimal spawned lineage edges on April 26, 2026
  • matched runtime events now also persist executable-image file nodes with executed and content_matched provenance edges on April 26, 2026
  • a thin GNU Make wrapper now exists for local make setup/build/test/run
  • a root Conan 2 recipe and air-gap workflow doc now exist for offline dependency installs
  • a seeded local yara/4.5.5 Conan recipe now exists under conan/local_recipes_index/
  • a Linux-oriented root Dockerfile now exists for build/test validation with libyara enabled
  • the Docker image now defaults to gcc/g++ because the Ubuntu 24.04 container path was the cleanest working route for C++23 + vendored unqlite-cpp

Host-readiness hardening (October 24, 2026):

  • the BPF object now builds on Arch (UAPI __u32/__u64 types instead of glibc <stdint.h>); previously it silently failed there and the daemon fell back to procfs polling
  • execveat tracepoints are attached, so fexecve and memfd (fileless) executions are observed
  • events resolve through /proc/<pid>/exe and scan/hash that magic link, so relative paths, replaced files, deleted binaries, memfd images and other mount namespaces are handled
  • MemfdExecution / DeletedExecutable heuristic indicators feed per-process correlation
  • process nodes use a stable identity (process_<pid>_<boot8>_<starttime>), so PID reuse no longer merges processes
  • the BPF backend now also emits the startup inventory, and its timestamps are converted to wall clock
  • the daemon fails closed when no rules compile (require_rules=true) or the DB cannot be opened for writing
  • database: single-writer flock, read-only CLI/UI queries (no lost updates), transactional per-event writes, batched correlation, and a scan-result cache
  • rule severity comes from meta.severity (default medium); starter rules were replaced with a small, high-signal set
  • systemd unit: capability-bounded, NoNewPrivileges, read-only /home and shared /tmp (so binaries there can still be scanned), private state dir
  • (the web UI server hardened in this pass was later replaced by the desktop app; see Phase 8)

Next for host use: worker-pool scanning off the ring-buffer thread with drop counters, fuller argv capture, on-demand memory scans for flagged processes, file-write/network/module telemetry, structured alert output, and an allowlist.

Phase 0 — Make the repo buildable ✅

Goals

  • repair build metadata
  • eliminate obvious compile blockers
  • get the project configuring cleanly

Delivered

  1. Replaced invalid meson.options usage with valid Meson option handling.
  2. Fixed the DaemonStateMachine type definition problem in include/strawberry/daemon_states.hpp.
  3. Removed compile-time migration fallout and stale MongoDB-era references from the active build path.
  4. Re-ran configure and compile until the build graph was stable.

Exit Criteria

  • meson setup builddir -Denable_tests=true succeeds
  • meson compile -C builddir succeeds

Phase 1 — Repair and modernize tests ✅

Goals

  • align tests with current public interfaces
  • establish a trustworthy regression baseline

Delivered

  1. Updated tests/test_config.cpp to use db_path and current config defaults.
  2. Kept HashUtils tests compiling and passing.
  3. Added database-focused tests for:
    • indicator round-trip
    • graph round-trip
    • persistence across reopen

Exit Criteria

  • meson test -C builddir --verbose passes for the current unit suite

Phase 2 — Real embedded persistence ✅

Goals

  • turn the previous in-memory Database stub into real local persistence
  • keep the public interface stable

Delivered

  1. Integrated git@github.com:trvon/unqlite-cpp.git behind src/database.cpp.
    • vendored location: third_party/unqlite-cpp
  2. Persisted:
    • IndicatorEntry
    • GraphNode
    • GraphEdge
  3. Preserved XDG-friendly defaults.
  4. Added direct persistence tests under /tmp for:
    • reopen
    • commit
    • rollback
    • read-only enforcement

Exit Criteria

  • indicators survive process restart
  • graph nodes and edges survive reopen
  • query helpers work against persisted data

Phase 3 — Real YARA integration 🟡

Goals

  • replace placeholder load/scan logic with actual libyara-backed behavior

Delivered so far

  1. Implemented rule compilation/loading from a file or directory when Strawberry is built with libyara.
  2. Implemented file scanning with real match extraction when libyara is available.
  3. Implemented a best-effort scan_process bridge that scans /proc/<pid>/exe rather than process memory.
  4. Added YARA-focused tests that:
    • exercise graceful fallback when libyara is absent
    • contain real scan assertions for libyara-enabled builds
    • now include a Linux-only /proc/<pid>/exe smoke test for scan_process
  5. Persisted YARA hit metadata through the database path and added graph linkage tests for process→indicator relationships.
  6. Validated the real libyara file-scan path through the Conan-backed dependency flow and fixed a Conan/Meson C++ standard mismatch by forcing a C++23 consumer toolchain.
  7. Validated the Linux scan_process path in the Docker Ubuntu 24.04 test image on April 26, 2026.
  8. Added a reusable DetectionPipeline and a synthetic runtime integration test that proves procfs event ingestion can trigger YARA matching and persisted indicator/graph storage on April 26, 2026.

Remaining Tasks

  1. Thread the persisted YARA hit path through more realistic end-to-end daemon/runtime flows.
  2. If feasible, add true process-memory scanning as a guarded Linux-only feature.
  3. Record at least one libyara-enabled synthetic end-to-end scan/store run on Linux.
  4. Start the minimal libbpf/eBPF exec-event ingestion path.

Exit Criteria

  • a temp rule set can be loaded on a libyara-enabled host
  • a synthetic malicious sample triggers a stored match
  • YARA hit metadata is persisted alongside related process/graph context
  • at least one libyara-enabled automated test pass is recorded

Phase 4 — Real eBPF process telemetry

Goals

  • replace placeholder monitoring logic with actual runtime event capture

Delivered so far

  1. Replaced the empty ProcessMonitor loop with a minimal procfs-polling backend that emits new-process callbacks.
  2. Captured per-process basics from procfs:
    • pid
    • executable path
    • SHA-256 hash of the executable
    • command line
  3. Added a synthetic procfs unit test for new-process detection and callback delivery.
  4. Added a minimal libbpf-backed exec-event build path:
    • BPF object source for sys_enter_execve / sys_exit_execve
    • user-space libbpf loader and ring-buffer handling
    • automatic fallback to procfs polling when libbpf is unavailable or attach/load fails
  5. Validated in the Linux Docker build that the eBPF object compiles cleanly on Ubuntu 24.04.
  6. Added a privileged Docker live smoke harness and validated one real /bin/echo exec event reaches user space.

This is now a useful Linux-oriented bridge with both a real privileged Docker smoke pass and a real privileged Docker libbpf-to-store smoke pass, but it is not yet a richer daemon-grade runtime ingestion pipeline.

Tasks

  1. Fold the live libbpf-to-store harness behavior into a daemon/runtime harness, not just standalone smoke executables. ✅
  2. Capture at least:
    • pid
    • parent pid if available
    • executable path
    • timestamp
  3. Improve event context beyond the current executable path + comm bridge in the live libbpf path (for example, parent PID and richer argv handling if feasible).
  4. Keep a Linux-only smoke path in Docker/CI for the live telemetry path.

Exit Criteria

  • a real Linux exec event reaches user space
  • Strawberry creates at least one persisted process node and relationship edge from live telemetry

Phase 5 — Basic provenance graph usefulness

Goals

  • make the graph layer operationally useful, not just present

Tasks

  1. Record process lineage edges.
  2. Optionally record executable file edges.
  3. Add simple queries:
    • related nodes from a process
    • all matches connected to a lineage
  4. Add a concise CLI or log output for incident trace summaries. ✅

Exit Criteria

  • one suspicious event can be traced through stored relationships in a human-readable way

Phase 6 — Packaging and automation surfaces

Goals

  • add convenience surfaces only after the core path is real

Tasks

  1. Keep the GNU Make wrapper aligned with Meson.
  2. Keep the Conan/offline workflow aligned with real dependency usage, including the local YARA recipe.
  3. Keep the Linux Docker image aligned with the real dependency and test path.
  4. Clean up install/service scripts as runtime features become real.
  5. If the target is Make.com, define a small stable CLI contract first, then automate it.
  6. Document Linux-only runtime expectations clearly.

Exit Criteria

  • operators can build, test, and run the scanner through one stable interface
  • external automation does not depend on undocumented internal behavior

Phase 7 — Detection correlation engine

Goals

  • move from stored evidence to ranked, explainable findings
  • correlate multiple related events into one analyst-facing detection object

Delivered so far

  1. Added FindingRecord persistence/query support to Database.
  2. Added CorrelationEngine with a deterministic process_executable_yara_match rule.
  3. Added lineage_multiple_yara_matches for parent processes with multiple matched descendants.
  4. Added same_hash_multiple_process_matches for repeated matched executable hashes across processes.
  5. Added same_yara_rule_multiple_processes for repeated YARA rule hits across processes.
  6. Added CLI surfaces:
    • --correlate
    • --list-findings
    • --show-finding <id>
    • --json for structured finding output
  7. Updated the demo seed path to emit a correlated finding.
  8. Extended the minimal local web GUI with finding list/show endpoints and controls.
  9. Added unit coverage proving multiple related process YARA hits, lineage hits, and same-hash process reuse collapse into deterministic findings.

Tasks

  1. Extend the minimal correlation input model over existing persisted data:
    • process nodes
    • file nodes
    • indicator nodes
    • spawned, executed, matched, and content_matched edges
    • timestamps, hashes, paths, command lines, and parent PID where available
  2. Add more small rule/heuristic detections before ML:
    • suspicious executable matched by YARA and executed by a short-lived process
    • repeated matched descendants in the same lineage window
    • one file image tied to multiple suspicious process executions
    • process lineage containing multiple matched artifacts in a bounded time range
  3. Enrich durable finding records with:
    • finding ID
    • severity or score
    • reason/explanation
    • related node IDs
    • related indicator IDs
    • first/last seen timestamps
  4. Expand CLI and programmatic queries for:
    • list findings
    • show one finding
    • pivot from finding → process lineage → indicators
  5. Add tests proving that additional raw event families collapse into one correlated finding when appropriate.

Exit Criteria

  • Strawberry can generate at least one explainable finding from multiple persisted graph artifacts ✅
  • the finding can be queried independently of the raw node/edge dump ✅
  • at least two distinct correlation rules exist ✅

Goals

  • move from a debug-style trace viewer to a real operator workflow
  • make process-centric investigation fast and explainable
  • a native desktop app + tray indicator as the operator surface, with no network listener
  • agent-facing access (CLI today, MCP next) over the same read-only evidence and fixed control verbs

Delivered so far

  1. Added ProcessQuery for process-centric pivots over graph nodes, indicators, and findings.
  2. Added CLI support:
    • --search-process <query>
    • --search-process <query> --json
  3. Added local web UI process search backed by structured JSON.
  4. Added parent/child lineage summaries to process search results.
  5. Added first/last seen evidence timestamps to process search results.
  6. Added local web UI deep links from process search results to trace and finding views.
  7. Added local web UI finding cards that pivot from findings back to process traces and process search.
  8. Added related file/indicator summaries to process search cards, making the local evaluator a lightweight process detail panel.
  9. Added client-side sort/filter controls for process search and finding cards.
  10. Added operator-friendly ns/us/ms/s duration formatting for trace, finding, and process-search timestamps.
  11. Added local-web copy/download affordances for rendered traces, process-search results, and findings.
  12. Split process-search cards into structured identity, timing, lineage, related-file, related-indicator, and finding sections.
  13. Added explicit UI server host/port controls, with safe loopback-only default and --allow-remote required for non-loopback binds.
  14. Documented the Python UI server as a temporary evaluator and the target C++ Boost.Beast/Asio API service direction.
  15. Added API_CONTRACT.md to pin the current web JSON contract before porting endpoints into C++.
  16. Added /api/process for exact process-detail JSON by PID or process node ID.
  17. Wired the local web UI to /api/process with a focused process-detail panel and detail pivots from search/finding results.
  18. Added a loopback web API contract test for health/defaults/trace/findings/process search/process detail against seeded demo data.
  19. Added WEB_DEPLOYMENT.md covering local, SSH-tunnel, container/lab-bind, and reverse-proxy prototype modes.
  20. Added unit coverage for search by PID, path, hash, YARA rule, finding metadata, lineage summaries, and process seen timestamps.
  21. October 24, 2026: replaced the web evaluator (items 3, 6–9, 11–19) with a Qt6 desktop app + tray indicator (app/, strawberry-tray):
    • Findings, Processes and Diagnostics tabs; tray colour by scanner health and unseen findings; desktop notifications for new high/critical findings.
    • Daemon ControlServer Unix socket (status, reload-rules, rescan; group strawberry; peer-credential checked; rescan rate-limited), plus strawberry --status/--reload-rules/--rescan.
    • The service runs with Group=strawberry, so evidence is 0640 group-readable; the app reads it via Database::open_read_only().
    • Tests: tests/test_control_server.cpp, app/tests/test_tray_state.cpp, pipeline stats/rescan coverage.
    • Removed scripts/strawberry_ui_server.py, ui/, scripts/test_web_api_contract.py, WEB_DEPLOYMENT.md; API_CONTRACT.md now documents the socket protocol and CLI JSON.

Tasks

  1. ✅ Agent-facing interface (October 24, 2026): strawberry-mcp (MCP over stdio) with status, findings, process search/detail, fd-passing scan_paths, reload and rescan; --read-only mode; strawberry --scan-file. Next: optionally record on-demand scan hits as evidence, and an MCP resource/notification for new findings.
  2. ✅ (October 24, 2026) Auto-updating YARA rule feeds (YARA Forge core by default, digest-verified, compile-checked, atomic switch, sandboxed updater with daily timer + "Update now"), allowlist (hash/path/rule) with suppression in the pipeline, Rules/Allowlist tabs in the app, read-only strawberry_get_policy MCP tool; process scans from bounded memory copies; exec events via a bounded worker queue with drop counters.
  3. Expand the process-first views as evidence grows (argv, file writes, network).
  4. Defer thread-level pivots until Strawberry actually captures TID/thread telemetry.
  5. Package the app (desktop entry/autostart) for more desktops; keep it read-only plus fixed verbs.

Exit Criteria

  • an operator can start from a process or finding and pivot across related evidence without using raw DB internals ✅
  • scanner health and new high-severity findings are visible without opening a terminal ✅
  • operator controls never require root or a network listener ✅

Phase 9 — Protection and response (October 25, 2026)

Delivered (all unit/integration tested; root-only paths validated in a privileged container):

  • M1 threat-intel feeds: list feed kinds (hashes, domains, ips, nets) with validation, caps and a shrink guard; default CERT-PL, Spamhaus DROP, Feodo C2 and malware-hash feeds; IOC_KnownMalwareHash; YARA meta.score → severity
  • M2 OpenSnitch: blocklist rules from list feeds; automatic per-binary containment at ≥ contain_min_severity (hash-pinned with EnableChecksums); Response tab with undo
  • M3 exec prevention: fanotify FAN_OPEN_EXEC_PERM guard, off/audit/enforce, fail-open deadline, verdict cache, system paths never denied
  • M4 behavior telemetry: eBPF connect/ptrace/module events, fanotify persistence/drop watcher, six behavior rules
  • M5 memory scan + response: yr_rules_scan_proc memory scans (automatic for memfd/deleted/ptrace targets, or on request); quarantine (blocks execution in every mode + network); process-suspend/resume/kill with PID/start-time re-verification and protected-process refusals

Open items:

  • host validation with OpenSnitch hash pinning on, prevention in audit, then enforce
  • behavior rules need real-host tuning (false-positive review of persistence writes and connects)
  • network events are kept in memory only and persisted when relevant; no long-term network history
  • no thread/TID telemetry yet

Phase 10 — Resource governor and profiling (October 25, 2026)

Delivered: always-on timing histograms (status timings), optional Tracy build, constrained-lane KPI (make perf-kpi), ResourceGovernor (PSI / loadavg / MemAvailable / cgroup v2, levels with hysteresis, deferral with a starvation guard, read pacing), scan profiles, shared guard/pipeline scan cache, scans outside the pipeline lock, no hashing on the eBPF thread, paced inventory queue, governed memory scans, rule-reload backoff, unchanged OpenSnitch lists not rewritten, per-thread nice/ioprio, systemd limits.

Open: database whole-file rewrite per commit and O(F²) finding index (measure with timings.db_commit / correlation on a long-running host first); fuzz and invariant tests for parsers and decision logic.

Immediate Next Actions

These are the best next steps now:

  1. validate startup process inventory on Linux with STRAWBERRY_PROCESS_MONITOR_BACKEND=procfs and scan_existing_processes_on_start=true
  2. feed that fuller process corpus through DetectionPipeline so correlation work is based on real host-wide evidence rather than GUI demo data
  3. improve runtime enrichment that matters for correlation: stable executable identity, parent/child lineage, timestamps, command-line capture, and exit/short-lived-process handling
  4. keep improving libbpf enrichment in parallel, especially parent-PID coverage and a hybrid BPF-events-plus-procfs-inventory mode when feasible
  5. add or refine correlation rules only when the collected evidence model supports them cleanly
  6. keep desktop-app work limited to what new evidence needs; next interface work is agent access (MCP/CLI)

Note: event timestamps are wall-clock nanoseconds (eBPF boot-time stamps are converted at ingestion); synthetic demo data may still use small raw values.

Suggested Next Deliverable

The next meaningful deliverable should be:

  • buildable repository
  • passing unit tests
  • persisted local indicators and graph data
  • real YARA file scanning on a libyara-enabled host
  • privileged Docker-validated live libbpf exec-event delivery
  • one synthetic end-to-end daemon-owned scan stored in the database
  • one privileged Docker-validated live daemon-owned scan stored in the database
  • one Linux procfs startup inventory pass that collects already-running processes into the same pipeline
  • one minimal explainable correlated finding generated from the stored graph/evidence path
  • one process-first CLI/query surface that can pivot from a finding into lineage and related artifacts

That gives Strawberry a real detection and investigation spine before deeper eBPF, provenance, and desktop UX work.