Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
546 lines
33 KiB
Markdown
546 lines
33 KiB
Markdown
# Implementation Plan
|
||
|
||
Execution plan for turning Strawberry from scaffold into a minimally real, research-backed prototype.
|
||
|
||
## Linked Documents
|
||
|
||
- [RESEARCH.md](RESEARCH.md)
|
||
- [AGENTS.md](AGENTS.md)
|
||
- [DEVELOPER.md](DEVELOPER.md)
|
||
- [UX.md](UX.md)
|
||
- [API_CONTRACT.md](API_CONTRACT.md)
|
||
|
||
## Planning Principle
|
||
|
||
This plan follows the literature in [RESEARCH.md](RESEARCH.md):
|
||
|
||
- reliable runtime telemetry before advanced analytics
|
||
- signatures as enrichment, not the only detector
|
||
- graph context as a force multiplier
|
||
- compact, inspectable local workflows before bigger platform ambitions
|
||
|
||
## Active work — data engine and staged learning (October 27, 2026)
|
||
|
||
The newer data-engine **stages** are separate from the original scaffold
|
||
**phases** retained below. Safety/event-store work and richer telemetry have
|
||
landed, followed by incremental correlation, rarity/replay, injected trace
|
||
evaluation and retro-hunt. Stage 4 has offline model training, live shadow
|
||
scoring and operator promotion with replay gates.
|
||
|
||
The current Stage 4 increment wires promoted decisions to bounded additional
|
||
memory scans and uncached image rescans, with governor admission, stable process
|
||
identity checks, outcome telemetry and a shared executable-action replay policy.
|
||
See [the execution-policy gate](docs/evaluation/stage4-execution-gate.md).
|
||
Old promotion files are not automatically activated; they require re-evaluation.
|
||
The increment has passed local and clean-object Docker tests, privileged Linux
|
||
smokes, sanitizer/property checks and three paired default-lane KPI comparisons.
|
||
Those KPI runs had no acting model; they do not validate active inspection cost
|
||
or efficacy. Detailed coverage and limitations are in the execution-policy gate.
|
||
The live rarity model is kept across daemon restarts (`model_dir/rarity.state`,
|
||
checksummed, saved hourly and at stop), so a restart no longer repeats the
|
||
one-hour shadow warm-up.
|
||
This does **not** complete Stage 4: measured-cost budget enforcement, online
|
||
training, full multi-period/multi-seed promotion evidence and live model-triggered
|
||
inspection validation remain. Stage 5 advisory detections have not been enabled.
|
||
|
||
Offline training-data follow-up: replay now retains a seeded reservoir of teacher
|
||
positives even after negative examples fill the sample, instead of retaining only
|
||
first-arriving positives. Sampling bookkeeping is bounded by retained examples.
|
||
Restored rarity clocks also handle full-range elapsed times and complete decay
|
||
without signed overflow. The per-program cap now bounds the combined positive +
|
||
negative count per entity, and replay label joins resolve operator retractions
|
||
in stored order (allowlist-remove by entry id, quarantine-remove by hash),
|
||
preserving independent positives and other entries. Parent audit repairs cover
|
||
multi-hash entry removal, delayed suppression observations, quarantine provenance,
|
||
malformed-label failures and capped-entity refresh with bounded slot back-links
|
||
and allocated capacity (geometric compaction, not just logical entry limits).
|
||
Fresh host training readiness is **unavailable**: this session cannot read the
|
||
root-only telemetry stream; historical scarcity is not current evidence. The
|
||
readiness proposal requires total vs retained labels, temporal split, provenance,
|
||
concentration, age and full replay/training RSS measurement. Measured-cost
|
||
calibration and the unchanged three-period × three-seed promotion gate remain;
|
||
these fixes do not train, promote or activate an operational model.
|
||
|
||
Reuse-first diagnostics follow-up: existing replay now has an opt-in
|
||
`--training-summary` mode, not a parallel readiness engine. It reuses the label
|
||
resolver/sampler, reports actual retained classes and anonymous concentration,
|
||
keeps ordinary collection off, and says `assessment: not_assessed`. The reader
|
||
exposes skipped/invalid input observations; replay rejects incomplete evidence
|
||
or telemetry without exporting a usable partial report (including to trainer
|
||
loader callers). Decode availability is observed per segment: a block that needs
|
||
a zstd dictionary this build does not have is *unavailable*, and a complete block
|
||
header that fails validation is *invalid*; only a short tail an active writer
|
||
still owns stays tolerated. Snapshot summaries additionally require a stable current
|
||
manifest and no torn tail; ordinary active-tail/fragment/fallback replay remains
|
||
supported. Verifier JSON config chatter is captured, but verifier `ok` with
|
||
partial-coverage notes is intentionally not redefined as a full-read verdict.
|
||
CLI-local wall histograms/Tracy zones are not RSS or actual CPU evidence. Local
|
||
synthetic regressions cover these contracts; independent full validation remains
|
||
required before acceptance. No operational model is trained/promoted/activated,
|
||
no daemon/control/persistence format changes, and no host-readiness or promotion
|
||
claim follows from these descriptive counts.
|
||
|
||
Findings-engine repair follow-up (in progress): regressions reproduce scoped
|
||
allowlist cleanup hiding independent evidence, stale incremental membership after
|
||
policy changes, namespace/source rule-name conflation, network attribution across
|
||
PID reuse, and misleading instance counts. The repair keeps real indicator
|
||
support and immutable policy views, explicitly rebuilds/withdraws managed findings
|
||
on policy changes, preserves typed rule identities and history, and joins network
|
||
history only to an observed stable process lifetime without per-connect procfs
|
||
reads. Review found and reproduced an offline full-rebuild gap; daemon and
|
||
offline persistence now share managed-set reconciliation, and the legacy CLI
|
||
reads policy without seeding or migrating it. Fold then exposed an unguarded
|
||
shutdown-flush exception path. A one-shot allocation-failure RED regression now
|
||
passes with best-effort flush handling and a destructor guard, including raw-data
|
||
recovery and writer release. Post-rework local functional validation passes 52/52 Meson tests, 16 seeded
|
||
checks, 24 ASan/UBSan runs and 22 fuzz targets (6,390,642 executions, no crash
|
||
artifacts or sanitizer diagnostics). Native/LSP checks retain baseline warnings;
|
||
Fold's six selected daemon TUs retain 49 source-triaged candidates but no longer
|
||
report the shutdown escape. This is not diagnostic-clean or production sign-off.
|
||
The runner recorded 21m02s monotonic elapsed time, but UTC timestamps span 34m14s.
|
||
Subsequent journal review found a suspend during the batch (02:18:20–02:31:34 UTC);
|
||
`CLOCK_MONOTONIC` excludes suspended time. The historical wall cap is not
|
||
retroactively certified. An audit-only replacement uses `CLOCK_BOOTTIME` timerfd /
|
||
pidfd waits and records all three clocks; 15 focused regression/integration tests
|
||
pass. At that follow-up its future heavy-run copy remained unarmed; no controlled
|
||
physical-suspend test or new heavy validation was run. All 10 deferred YAMS artifacts
|
||
were later recovered byte-exactly without a service restart.
|
||
|
||
The separately authorized November 2 privileged/KPI follow-up hardened the harness
|
||
and built immutable actual-dirty-baseline/candidate images (14 harness tests passed
|
||
inside each). Five live fixtures passed; three dedicated BPF smokes failed and
|
||
remain unresolved, not attributed to the FE changes. Three alternating constrained
|
||
KPI pairs loaded 5,140 rules from the same three files. All six workloads completed
|
||
within 300 BOOTTIME seconds, but **all six failed the 2% idle-CPU gate**: baseline
|
||
3.16/5.42/2.12%, candidate 2.54/2.99/4.15%. Every run recorded zero dropped events
|
||
and a 0.007 ms exec-decision p95; maxima were approximately 1.01–1.04 s with
|
||
1–3 guard timeouts, so p95 is not a universal launch-latency bound. Maximum sampled
|
||
daemon RSS was 444.27–462.20 MiB baseline and 449.49–463.28 MiB candidate; these
|
||
are whole-daemon measurements without an acting model, not model inspection cost.
|
||
Sampler errors and the failed BPF predicates are retained. No thresholds were
|
||
relaxed or failed BPF tests rerun. The full container unit matrix was not rebuilt;
|
||
the earlier native functional/sanitizer/fuzz evidence remains separate. These
|
||
validation runs did not deploy the changes, establish the host-crash cause, or clear
|
||
the privileged/resource/real-data/model-promotion gates.
|
||
|
||
The subsequent CPU-cache repair distinguishes an oversized-image refusal from a
|
||
completed scan. Repeated cache hits no longer hash an unknown oversized image to
|
||
EOF, and a refusal cannot satisfy the exec guard as a clean YARA result. Local
|
||
regressions cover known/unknown hashes, inventory/live events and size boundaries;
|
||
52/52 native tests, six ASan/UBSan suite runs and 22 fuzz targets passed (10,253,990
|
||
executions, no crash artifacts or sanitizer diagnostics; leak detection disabled).
|
||
This does not clear Fold, live/KPI acceptance or post-rollout CPU measurements.
|
||
OpenSnitch's separately observed MD5 cost remains a distinct follow-up; disabling
|
||
checksums or weakening firewall rules is not part of this repair.
|
||
|
||
## Historical Status Snapshot
|
||
|
||
As of **April 27, 2026** (historical; later sections and the active-work summary
|
||
above supersede remaining-task lists from this snapshot):
|
||
|
||
- Phase 0 is complete: the repo configures and compiles again
|
||
- Phase 1 is complete: the current unit suite passes
|
||
- Phase 2 is complete in the current checkout: Strawberry persists indicators and graph data across reopen through vendored `third_party/unqlite-cpp` (replaced by the compressed event store on 2026-10-25; the submodule was removed)
|
||
- direct persistence tests now cover reopen, commit, rollback, and read-only behavior
|
||
- Phase 3 has started: optional libyara-backed rule loading and file scanning are implemented
|
||
- YARA hit metadata now persists through the database path, including multi-hit support for the same hash and process→indicator graph linkage
|
||
- `scan_process` now performs a best-effort `/proc/<pid>/exe` scan when libyara is available
|
||
- a Conan-backed libyara build and test pass succeeded on **April 25, 2026** on the current macOS host, including real rule compilation and file-scan assertions
|
||
- Linux-specific `/proc/<pid>/exe` behavior is now covered by a Docker-validated unit test on **April 26, 2026**
|
||
- a minimal procfs-polling process-event path now exists on **April 26, 2026**
|
||
- a reusable `DetectionPipeline` now exists, and a Docker-validated synthetic runtime test proves procfs event ingestion can drive executable scanning plus persisted indicator/graph storage on **April 26, 2026**
|
||
- a minimal libbpf exec-event backend now compiles in the Linux Docker build on **April 26, 2026**
|
||
- a privileged Linux Docker smoke run now validates live libbpf tracepoint attach and exec-event delivery on **April 26, 2026**
|
||
- a second privileged Linux Docker smoke run now validates live libbpf exec-event persistence through `DetectionPipeline` on **April 26, 2026**
|
||
- a reusable `StrawberryDaemon` wrapper now owns the daemon shell around config loading, process-monitor startup, and `DetectionPipeline`
|
||
- a daemon-level synthetic test now validates procfs event ingestion through `StrawberryDaemon` into persisted indicator/graph storage on **April 26, 2026**
|
||
- a third privileged Linux Docker smoke run now validates live libbpf exec-event persistence through `StrawberryDaemon` itself on **April 26, 2026**
|
||
- a new `IncidentReporter` now renders concise human-readable summaries over persisted graph state with first/last seen timing, and the main binary can print them with `--trace-process` / `--trace-node`
|
||
- the operator UI is now the Qt6 desktop app + tray indicator `strawberry-tray` (October 24, 2026); the temporary Python web evaluator was removed
|
||
- a synthetic demo-trace seed path exists (`make demo`), so the app/reporting flow can be evaluated without a running daemon
|
||
- the active direction remains **system improvements first** so real Linux process collection and correlation evidence quality lead the work
|
||
- a first minimal correlation engine now exists: Strawberry can group process-executable YARA indicators by process/hash and persist an explainable finding
|
||
- a second correlation rule now flags multiple matched descendants under one parent process lineage
|
||
- a third correlation rule now flags the same matched executable hash across multiple processes
|
||
- a fourth correlation rule now flags the same YARA rule across multiple processes
|
||
- findings now have JSON output through the CLI for UI/automation consumers
|
||
- a process search query surface now exists for PID, path, hash, YARA rule, and finding pivots, with JSON output for UI/automation consumers
|
||
- the correlation engine is intentionally small and rule-based; it is not yet a general subgraph matcher, anomaly detector, or full incident engine
|
||
- the current data model is still **process-centric**, not **thread-centric**; there is no first-class TID/thread telemetry or thread search UX yet
|
||
- procfs polling now has an optional startup inventory mode that emits already-running processes before continuing to watch for newly observed PIDs; this is the current path for collecting a fuller Linux process corpus for correlation refinement
|
||
- the libbpf path now has a best-effort fixed-width live argv preview (`argv0`/`argv1`); both the `/bin/echo` smoke and the staged pipeline smoke validated it on **April 26, 2026**
|
||
- the libbpf path now opportunistically attempts `sched_process_fork` for parent-PID enrichment, but the privileged container smoke confirms that this additional tracepoint may be denied while the main exec-event path still succeeds
|
||
- procfs-driven runtime events now also persist `parent_pid` and `event_timestamp_ns`, and create minimal `spawned` lineage edges on **April 26, 2026**
|
||
- matched runtime events now also persist executable-image file nodes with `executed` and `content_matched` provenance edges on **April 26, 2026**
|
||
- a thin GNU Make wrapper now exists for local `make setup/build/test/run`
|
||
- a root Conan 2 recipe and air-gap workflow doc now exist for offline dependency installs
|
||
- a seeded local `yara/4.5.5` Conan recipe now exists under `conan/local_recipes_index/`
|
||
- a Linux-oriented root `Dockerfile` now exists for build/test validation with libyara enabled
|
||
- the Docker image now defaults to `gcc`/`g++` because the Ubuntu 24.04 container path was the cleanest working route for C++23 + vendored `unqlite-cpp`
|
||
|
||
Host-readiness hardening (**October 24, 2026**):
|
||
|
||
- the BPF object now builds on Arch (UAPI `__u32/__u64` types instead of glibc `<stdint.h>`); previously it silently failed there and the daemon fell back to procfs polling
|
||
- `execveat` tracepoints are attached, so `fexecve` and memfd (fileless) executions are observed
|
||
- events resolve through `/proc/<pid>/exe` and scan/hash that magic link, so relative paths, replaced files, deleted binaries, memfd images and other mount namespaces are handled
|
||
- `MemfdExecution` / `DeletedExecutable` heuristic indicators feed per-process correlation
|
||
- process nodes use a stable identity (`process_<pid>_<boot8>_<starttime>`), so PID reuse no longer merges processes
|
||
- the BPF backend now also emits the startup inventory, and its timestamps are converted to wall clock
|
||
- the daemon fails closed when no rules compile (`require_rules=true`) or the DB cannot be opened for writing
|
||
- database: single-writer `flock`, read-only CLI/UI queries (no lost updates), transactional per-event writes, batched correlation, and a scan-result cache
|
||
- rule severity comes from `meta.severity` (default `medium`); starter rules were replaced with a small, high-signal set
|
||
- systemd unit: capability-bounded, `NoNewPrivileges`, read-only `/home` and shared `/tmp` (so binaries there can still be scanned), private state dir
|
||
- (the web UI server hardened in this pass was later replaced by the desktop app; see Phase 8)
|
||
|
||
Next for host use: worker-pool scanning off the ring-buffer thread with drop counters, fuller argv capture, on-demand memory scans for flagged processes, file-write/network/module telemetry, structured alert output, and an allowlist.
|
||
|
||
## Phase 0 — Make the repo buildable ✅
|
||
|
||
### Goals
|
||
|
||
- repair build metadata
|
||
- eliminate obvious compile blockers
|
||
- get the project configuring cleanly
|
||
|
||
### Delivered
|
||
|
||
1. Replaced invalid `meson.options` usage with valid Meson option handling.
|
||
2. Fixed the `DaemonStateMachine` type definition problem in `include/strawberry/daemon_states.hpp`.
|
||
3. Removed compile-time migration fallout and stale MongoDB-era references from the active build path.
|
||
4. Re-ran configure and compile until the build graph was stable.
|
||
|
||
### Exit Criteria
|
||
|
||
- `meson setup builddir -Denable_tests=true` succeeds
|
||
- `meson compile -C builddir` succeeds
|
||
|
||
## Phase 1 — Repair and modernize tests ✅
|
||
|
||
### Goals
|
||
|
||
- align tests with current public interfaces
|
||
- establish a trustworthy regression baseline
|
||
|
||
### Delivered
|
||
|
||
1. Updated `tests/test_config.cpp` to use `db_path` and current config defaults.
|
||
2. Kept `HashUtils` tests compiling and passing.
|
||
3. Added database-focused tests for:
|
||
- indicator round-trip
|
||
- graph round-trip
|
||
- persistence across reopen
|
||
|
||
### Exit Criteria
|
||
|
||
- `meson test -C builddir --verbose` passes for the current unit suite
|
||
|
||
## Phase 2 — Real embedded persistence ✅
|
||
|
||
### Goals
|
||
|
||
- turn the previous in-memory `Database` stub into real local persistence
|
||
- keep the public interface stable
|
||
|
||
### Delivered
|
||
|
||
1. Integrated `git@github.com:trvon/unqlite-cpp.git` behind `src/database.cpp`.
|
||
- vendored location: `third_party/unqlite-cpp`
|
||
2. Persisted:
|
||
- `IndicatorEntry`
|
||
- `GraphNode`
|
||
- `GraphEdge`
|
||
3. Preserved XDG-friendly defaults.
|
||
4. Added direct persistence tests under `/tmp` for:
|
||
- reopen
|
||
- commit
|
||
- rollback
|
||
- read-only enforcement
|
||
|
||
### Exit Criteria
|
||
|
||
- indicators survive process restart
|
||
- graph nodes and edges survive reopen
|
||
- query helpers work against persisted data
|
||
|
||
## Phase 3 — Real YARA integration 🟡
|
||
|
||
### Goals
|
||
|
||
- replace placeholder load/scan logic with actual libyara-backed behavior
|
||
|
||
### Delivered so far
|
||
|
||
1. Implemented rule compilation/loading from a file or directory when Strawberry is built with libyara.
|
||
2. Implemented file scanning with real match extraction when libyara is available.
|
||
3. Implemented a best-effort `scan_process` bridge that scans `/proc/<pid>/exe` rather than process memory.
|
||
4. Added YARA-focused tests that:
|
||
- exercise graceful fallback when libyara is absent
|
||
- contain real scan assertions for libyara-enabled builds
|
||
- now include a Linux-only `/proc/<pid>/exe` smoke test for `scan_process`
|
||
5. Persisted YARA hit metadata through the database path and added graph linkage tests for process→indicator relationships.
|
||
6. Validated the real libyara file-scan path through the Conan-backed dependency flow and fixed a Conan/Meson C++ standard mismatch by forcing a C++23 consumer toolchain.
|
||
7. Validated the Linux `scan_process` path in the Docker Ubuntu 24.04 test image on **April 26, 2026**.
|
||
8. Added a reusable `DetectionPipeline` and a synthetic runtime integration test that proves procfs event ingestion can trigger YARA matching and persisted indicator/graph storage on **April 26, 2026**.
|
||
|
||
### Remaining Tasks
|
||
|
||
1. Thread the persisted YARA hit path through more realistic end-to-end daemon/runtime flows.
|
||
2. If feasible, add true process-memory scanning as a guarded Linux-only feature.
|
||
3. Record at least one libyara-enabled synthetic end-to-end scan/store run on Linux.
|
||
4. Start the minimal `libbpf`/eBPF exec-event ingestion path.
|
||
|
||
### Exit Criteria
|
||
|
||
- a temp rule set can be loaded on a libyara-enabled host
|
||
- a synthetic malicious sample triggers a stored match
|
||
- YARA hit metadata is persisted alongside related process/graph context
|
||
- at least one libyara-enabled automated test pass is recorded
|
||
|
||
## Phase 4 — Real eBPF process telemetry
|
||
|
||
### Goals
|
||
|
||
- replace placeholder monitoring logic with actual runtime event capture
|
||
|
||
### Delivered so far
|
||
|
||
1. Replaced the empty `ProcessMonitor` loop with a minimal procfs-polling backend that emits new-process callbacks.
|
||
2. Captured per-process basics from procfs:
|
||
- pid
|
||
- executable path
|
||
- SHA-256 hash of the executable
|
||
- command line
|
||
3. Added a synthetic procfs unit test for new-process detection and callback delivery.
|
||
4. Added a minimal libbpf-backed exec-event build path:
|
||
- BPF object source for `sys_enter_execve` / `sys_exit_execve`
|
||
- user-space libbpf loader and ring-buffer handling
|
||
- automatic fallback to procfs polling when libbpf is unavailable or attach/load fails
|
||
5. Validated in the Linux Docker build that the eBPF object compiles cleanly on Ubuntu 24.04.
|
||
6. Added a privileged Docker live smoke harness and validated one real `/bin/echo` exec event reaches user space.
|
||
|
||
This is now a useful Linux-oriented bridge with both a real privileged Docker smoke pass and a real privileged Docker libbpf-to-store smoke pass, but it is **not yet a richer daemon-grade runtime ingestion pipeline**.
|
||
|
||
### Tasks
|
||
|
||
1. Fold the live libbpf-to-store harness behavior into a daemon/runtime harness, not just standalone smoke executables. ✅
|
||
2. Capture at least:
|
||
- pid
|
||
- parent pid if available
|
||
- executable path
|
||
- timestamp
|
||
3. Improve event context beyond the current executable path + `comm` bridge in the live libbpf path (for example, parent PID and richer argv handling if feasible).
|
||
4. Keep a Linux-only smoke path in Docker/CI for the live telemetry path.
|
||
|
||
### Exit Criteria
|
||
|
||
- a real Linux exec event reaches user space
|
||
- Strawberry creates at least one persisted process node and relationship edge from live telemetry
|
||
|
||
## Phase 5 — Basic provenance graph usefulness
|
||
|
||
### Goals
|
||
|
||
- make the graph layer operationally useful, not just present
|
||
|
||
### Tasks
|
||
|
||
1. Record process lineage edges.
|
||
2. Optionally record executable file edges.
|
||
3. Add simple queries:
|
||
- related nodes from a process
|
||
- all matches connected to a lineage
|
||
4. Add a concise CLI or log output for incident trace summaries. ✅
|
||
|
||
### Exit Criteria
|
||
|
||
- one suspicious event can be traced through stored relationships in a human-readable way
|
||
|
||
## Phase 6 — Packaging and automation surfaces
|
||
|
||
### Goals
|
||
|
||
- add convenience surfaces only after the core path is real
|
||
|
||
### Tasks
|
||
|
||
1. Keep the GNU Make wrapper aligned with Meson.
|
||
2. Keep the Conan/offline workflow aligned with real dependency usage, including the local YARA recipe.
|
||
3. Keep the Linux Docker image aligned with the real dependency and test path.
|
||
4. Clean up install/service scripts as runtime features become real.
|
||
5. If the target is **Make.com**, define a small stable CLI contract first, then automate it.
|
||
6. Document Linux-only runtime expectations clearly.
|
||
|
||
### Exit Criteria
|
||
|
||
- operators can build, test, and run the scanner through one stable interface
|
||
- external automation does not depend on undocumented internal behavior
|
||
|
||
## Phase 7 — Detection correlation engine
|
||
|
||
### Goals
|
||
|
||
- move from stored evidence to ranked, explainable findings
|
||
- correlate multiple related events into one analyst-facing detection object
|
||
|
||
### Delivered so far
|
||
|
||
1. Added `FindingRecord` persistence/query support to `Database`.
|
||
2. Added `CorrelationEngine` with a deterministic `process_executable_yara_match` rule.
|
||
3. Added `lineage_multiple_yara_matches` for parent processes with multiple matched descendants.
|
||
4. Added `same_hash_multiple_process_matches` for repeated matched executable hashes across processes.
|
||
5. Added `same_yara_rule_multiple_processes` for repeated YARA rule hits across processes.
|
||
6. Added CLI surfaces:
|
||
- `--correlate`
|
||
- `--list-findings`
|
||
- `--show-finding <id>`
|
||
- `--json` for structured finding output
|
||
6. Updated the demo seed path to emit a correlated finding.
|
||
7. Extended the minimal local web GUI with finding list/show endpoints and controls.
|
||
8. Added unit coverage proving multiple related process YARA hits, lineage hits, and same-hash process reuse collapse into deterministic findings.
|
||
|
||
### Tasks
|
||
|
||
1. Extend the minimal correlation input model over existing persisted data:
|
||
- process nodes
|
||
- file nodes
|
||
- indicator nodes
|
||
- `spawned`, `executed`, `matched`, and `content_matched` edges
|
||
- timestamps, hashes, paths, command lines, and parent PID where available
|
||
2. Add more small rule/heuristic detections before ML:
|
||
- suspicious executable matched by YARA and executed by a short-lived process
|
||
- repeated matched descendants in the same lineage window
|
||
- one file image tied to multiple suspicious process executions
|
||
- process lineage containing multiple matched artifacts in a bounded time range
|
||
3. Enrich durable finding records with:
|
||
- finding ID
|
||
- severity or score
|
||
- reason/explanation
|
||
- related node IDs
|
||
- related indicator IDs
|
||
- first/last seen timestamps
|
||
4. Expand CLI and programmatic queries for:
|
||
- list findings
|
||
- show one finding
|
||
- pivot from finding → process lineage → indicators
|
||
5. Add tests proving that additional raw event families collapse into one correlated finding when appropriate.
|
||
|
||
### Exit Criteria
|
||
|
||
- Strawberry can generate at least one explainable finding from multiple persisted graph artifacts ✅
|
||
- the finding can be queried independently of the raw node/edge dump ✅
|
||
- at least two distinct correlation rules exist ✅
|
||
|
||
## Phase 8 — Analyst UX and search
|
||
|
||
### Goals
|
||
|
||
- move from a debug-style trace viewer to a real operator workflow
|
||
- make process-centric investigation fast and explainable
|
||
- a native desktop app + tray indicator as the operator surface, with no network listener
|
||
- agent-facing access (CLI today, MCP next) over the same read-only evidence and fixed control verbs
|
||
|
||
### Delivered so far
|
||
|
||
1. Added `ProcessQuery` for process-centric pivots over graph nodes, indicators, and findings.
|
||
2. Added CLI support:
|
||
- `--search-process <query>`
|
||
- `--search-process <query> --json`
|
||
3. Added local web UI process search backed by structured JSON.
|
||
4. Added parent/child lineage summaries to process search results.
|
||
5. Added first/last seen evidence timestamps to process search results.
|
||
6. Added local web UI deep links from process search results to trace and finding views.
|
||
7. Added local web UI finding cards that pivot from findings back to process traces and process search.
|
||
8. Added related file/indicator summaries to process search cards, making the local evaluator a lightweight process detail panel.
|
||
9. Added client-side sort/filter controls for process search and finding cards.
|
||
10. Added operator-friendly ns/us/ms/s duration formatting for trace, finding, and process-search timestamps.
|
||
11. Added local-web copy/download affordances for rendered traces, process-search results, and findings.
|
||
12. Split process-search cards into structured identity, timing, lineage, related-file, related-indicator, and finding sections.
|
||
13. Added explicit UI server host/port controls, with safe loopback-only default and `--allow-remote` required for non-loopback binds.
|
||
14. Documented the Python UI server as a temporary evaluator and the target C++ Boost.Beast/Asio API service direction.
|
||
15. Added `API_CONTRACT.md` to pin the current web JSON contract before porting endpoints into C++.
|
||
16. Added `/api/process` for exact process-detail JSON by PID or process node ID.
|
||
17. Wired the local web UI to `/api/process` with a focused process-detail panel and detail pivots from search/finding results.
|
||
18. Added a loopback web API contract test for health/defaults/trace/findings/process search/process detail against seeded demo data.
|
||
19. Added `WEB_DEPLOYMENT.md` covering local, SSH-tunnel, container/lab-bind, and reverse-proxy prototype modes.
|
||
20. Added unit coverage for search by PID, path, hash, YARA rule, finding metadata, lineage summaries, and process seen timestamps.
|
||
21. **October 24, 2026: replaced the web evaluator (items 3, 6–9, 11–19) with a Qt6 desktop app + tray indicator** (`app/`, `strawberry-tray`):
|
||
- Findings, Processes and Diagnostics tabs; tray colour by scanner health and unseen findings; desktop notifications for new high/critical findings.
|
||
- Daemon `ControlServer` Unix socket (`status`, `reload-rules`, `rescan`; group `strawberry`; peer-credential checked; rescan rate-limited), plus `strawberry --status/--reload-rules/--rescan`.
|
||
- The service runs with `Group=strawberry`, so evidence is `0640` group-readable; the app reads it via `Database::open_read_only()`.
|
||
- Tests: `tests/test_control_server.cpp`, `app/tests/test_tray_state.cpp`, pipeline stats/rescan coverage.
|
||
- Removed `scripts/strawberry_ui_server.py`, `ui/`, `scripts/test_web_api_contract.py`, `WEB_DEPLOYMENT.md`; `API_CONTRACT.md` now documents the socket protocol and CLI JSON.
|
||
|
||
### Tasks
|
||
|
||
1. ✅ Agent-facing interface (October 24, 2026): `strawberry-mcp` (MCP over stdio) with status, findings, process search/detail, fd-passing `scan_paths`, reload and rescan; `--read-only` mode; `strawberry --scan-file`. Next: optionally record on-demand scan hits as evidence, and an MCP resource/notification for new findings.
|
||
2. ✅ (October 24, 2026) Auto-updating YARA rule feeds (YARA Forge core by default, digest-verified, compile-checked, atomic switch, sandboxed updater with daily timer + "Update now"), allowlist (hash/path/rule) with suppression in the pipeline, Rules/Allowlist tabs in the app, read-only `strawberry_get_policy` MCP tool; process scans from bounded memory copies; exec events via a bounded worker queue with drop counters.
|
||
3. Expand the process-first views as evidence grows (argv, file writes, network).
|
||
3. Defer thread-level pivots until Strawberry actually captures TID/thread telemetry.
|
||
4. Package the app (desktop entry/autostart) for more desktops; keep it read-only plus fixed verbs.
|
||
|
||
### Exit Criteria
|
||
|
||
- an operator can start from a process or finding and pivot across related evidence without using raw DB internals ✅
|
||
- scanner health and new high-severity findings are visible without opening a terminal ✅
|
||
- operator controls never require root or a network listener ✅
|
||
|
||
## Phase 9 — Protection and response (October 25, 2026)
|
||
|
||
Delivered (all unit/integration tested; root-only paths validated in a privileged container):
|
||
|
||
- **M1 threat-intel feeds**: list feed kinds (`hashes`, `domains`, `ips`, `nets`) with validation, caps and a shrink guard; default CERT-PL, Spamhaus DROP, Feodo C2 and malware-hash feeds; `IOC_KnownMalwareHash`; YARA `meta.score` → severity
|
||
- **M2 OpenSnitch**: blocklist rules from list feeds; automatic per-binary containment at ≥ `contain_min_severity` (hash-pinned with `EnableChecksums`); Response tab with undo
|
||
- **M3 exec prevention**: fanotify `FAN_OPEN_EXEC_PERM` guard, `off`/`audit`/`enforce`, fail-open deadline, verdict cache, system paths never denied
|
||
- **M4 behavior telemetry**: eBPF connect/ptrace/module events, fanotify persistence/drop watcher, six behavior rules
|
||
- **M5 memory scan + response**: `yr_rules_scan_proc` memory scans (automatic for memfd/deleted/ptrace targets, or on request); `quarantine` (blocks execution in every mode + network); `process-suspend/resume/kill` with PID/start-time re-verification and protected-process refusals
|
||
|
||
Open items:
|
||
|
||
- host validation with OpenSnitch hash pinning on, prevention in `audit`, then `enforce`
|
||
- behavior rules need real-host tuning (false-positive review of persistence writes and connects)
|
||
- network events are kept in memory only and persisted when relevant; no long-term network history
|
||
- no thread/TID telemetry yet
|
||
|
||
## Phase 10 — Resource governor and profiling (October 25, 2026)
|
||
|
||
Delivered: always-on timing histograms (status `timings`), optional Tracy
|
||
build, constrained-lane KPI (`make perf-kpi`), `ResourceGovernor` (PSI /
|
||
loadavg / MemAvailable / cgroup v2, levels with hysteresis, deferral with a
|
||
starvation guard, read pacing), scan profiles, shared guard/pipeline scan
|
||
cache, scans outside the pipeline lock, no hashing on the eBPF thread, paced
|
||
inventory queue, governed memory scans, rule-reload backoff, unchanged
|
||
OpenSnitch lists not rewritten, per-thread nice/ioprio, systemd limits.
|
||
|
||
Open: database whole-file rewrite per commit and O(F²) finding index (measure
|
||
with `timings.db_commit` / `correlation` on a long-running host first); fuzz
|
||
and invariant tests for parsers and decision logic.
|
||
|
||
## Immediate Next Actions
|
||
|
||
These are the best next steps now:
|
||
|
||
1. validate startup process inventory on Linux with `STRAWBERRY_PROCESS_MONITOR_BACKEND=procfs` and `scan_existing_processes_on_start=true`
|
||
2. feed that fuller process corpus through `DetectionPipeline` so correlation work is based on real host-wide evidence rather than GUI demo data
|
||
3. improve runtime enrichment that matters for correlation: stable executable identity, parent/child lineage, timestamps, command-line capture, and exit/short-lived-process handling
|
||
4. keep improving libbpf enrichment in parallel, especially parent-PID coverage and a hybrid BPF-events-plus-procfs-inventory mode when feasible
|
||
5. add or refine **correlation rules** only when the collected evidence model supports them cleanly
|
||
6. keep desktop-app work limited to what new evidence needs; next interface work is agent access (MCP/CLI)
|
||
|
||
Note: event timestamps are wall-clock nanoseconds (eBPF boot-time stamps are converted at ingestion); synthetic demo data may still use small raw values.
|
||
|
||
## Suggested Next Deliverable
|
||
|
||
The next meaningful deliverable should be:
|
||
|
||
- buildable repository
|
||
- passing unit tests
|
||
- persisted local indicators and graph data
|
||
- real YARA file scanning on a libyara-enabled host
|
||
- privileged Docker-validated live libbpf exec-event delivery
|
||
- one synthetic end-to-end daemon-owned scan stored in the database
|
||
- one privileged Docker-validated live daemon-owned scan stored in the database
|
||
- one Linux procfs startup inventory pass that collects already-running processes into the same pipeline
|
||
- one minimal explainable correlated finding generated from the stored graph/evidence path
|
||
- one process-first CLI/query surface that can pivot from a finding into lineage and related artifacts
|
||
|
||
That gives Strawberry a real detection and investigation spine before deeper eBPF, provenance, and desktop UX work.
|