Commit graph

16 commits

Author SHA1 Message Date
trvon
1c9b005645
store: incremental read-only refresh (fixes the tray spinning a core)
Root cause, from the Tracy capture the CPU watcher took when the tray spun
(2026-11-15 08:15, 67% for 15 s): database.reload_read_only rebuilt the whole
evidence view from every segment — mean 1.4 s, max 1.7 s, 4x in 20 s — because
every 3 s poll that saw the daemon append evidence triggered a full reload.

- store::scan_store_read_only_from + ReadCursor (per segment: id, sealed,
  bytes, records delivered). When the previously seen segments are an
  unchanged prefix (same ids, sealed ones byte-identical), only records
  appended since are delivered: the rest of the previously last segment
  (skipping what was applied) and new segments. Any other change —
  compaction, retention, reduction, a rewritten or shrunk segment — delivers
  everything and the caller rebuilds.
- Database read-only refresh uses it; records apply in the same order a full
  rebuild uses, so the view is identical. read_only_reload_counts() reports
  full vs incremental refreshes.

Live, on the real evidence store: the change-driven reload went from ~1.4 s
to 2.8 ms; the tray did ~97 ms of work in a 60 s window.

Test (test_incremental_view, seeded property): a long-lived reader
refreshing across random appends, upserts, seals and a compaction always
equals a freshly opened reader; appends refresh incrementally (42-49 per run),
the compaction forces a rebuild, and no change reports no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:46:00 -07:00
trvon
ebade1ba80
Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:09 -07:00
trvon
cbcb9896cd
Stage 7c: NodeMerge collapse redundant repeated benign accesses on live nodes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
9aa9c9bb46
Stage 7b: minimal reachable skeleton (skeletonize dead-benign nodes far from labels)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
c679a03561
Stage 7a: LogGC dead benign subtree GC (causality-preserving telemetry reduction)
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Successful in 53m38s
release / publish (push) Has been skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-10 19:32:12 -07:00
trvon
29e490dcab
Replay: add strict read-only training summaries
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.
2026-11-03 06:54:17 -07:00
trvon
1acaf9f563
Store housekeeping: read mtimes on macOS (st_mtimespec)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 10:04:35 -06:00
trvon
8fda6206c0
Build on macOS: portable syscalls, sockets and tray
The tree now compiles on macOS. Each Linux-only call gets a Darwin
branch, and the Linux branch stays as it was:

- io::fdatasync uses fcntl(F_FULLFSYNC): fsync alone only reaches the
  drive cache on APFS, and the store's crash atomicity relies on it.
- Control socket: FD_CLOEXEC set after socket/accept/recvmsg (no
  SOCK_CLOEXEC, accept4 or MSG_CMSG_CLOEXEC), SO_NOSIGPIPE, peer
  credentials from LOCAL_PEERCRED + LOCAL_PEERPID, getgrouplist with
  int groups and a growing buffer. launchd has no RuntimeDirectory, so
  the socket directory is created (0750, access group) when missing.
  Default socket path is /var/run/strawberry/control.sock.
- CRC32C detects ARMv8 CRC with hw.optional.armv8_crc32 (no auxv).
- stat mtime via st_mtimespec; the BPF shared header uses stdint
  typedefs off Linux.
- Background threads join the PRIO_DARWIN_BG band (lower CPU and I/O);
  crash safety zeroes RLIMIT_CORE in place of PR_SET_DUMPABLE.
- The desktop app drops Qt DBus off Linux and shows notifications as
  tray messages (app/notifier_tray.cpp).
- make setup passes Homebrew's keg-only libarchive to meson.

Tests: behavior_live (Linux ptrace API) builds on Linux only, the
control-server fd count reads /dev/fd, the crash test checks the core
limit on macOS, and the runtime pipeline test no longer references a
YARA-only helper when YARA is absent (that also broke Linux builds
without libyara).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:54:54 -06:00
trvon
b55723547d
Add doctor checks and maintenance actions to the daemon
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
- doctor (control verb, read-only): store health and disk floor,
  telemetry growth per day and days until the floor when retention is
  unlimited, leftover store files, superseded findings, retro-hunt state,
  thread/write errors, logs (journald rotates them), model (none yet).
  Each check carries a maintenance action or advice. Pure checks over
  gathered inputs (src/doctor.cpp), so they test without a daemon.
- maintain (control payload verb, privileged, rate limited, audited):
  retire-superseded-findings and clean-store, each with dry_run.
- finding_retired evidence record: removes a finding from view; it stays
  in the stream's history. CorrelationEngine::superseded_findings finds
  old hash/rule findings by recomputing the stable id, and old lineage
  findings only when the current one's parent process is among their
  nodes (a reused pid on another boot is not a match). Lineage findings
  now record parent_process_node_id.
- store::inspect_store_files / clean_store_leftovers: orphan/ files older
  than N days and *.compact / MANIFEST.tmp temp files; never a listed
  segment.

Tests: test_doctor (check statuses, superseded detection including pid
reuse and no-replacement cases, retirement surviving a reopen, cleanup
keeping recent orphans and listed segments); daemon test exercises both
verbs over the socket. fuzz_evidence_records covers finding_retired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 09:27:23 -06:00
trvon
d54d987b3a
Retro-hunt: re-scan binaries seen before when the rules change
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The main source of teacher labels over time: rules improve daily, while
most malware runs once.

- The pipeline remembers every binary that ran (image_seen evidence
  record: sha256, path, first/last seen, size; at most one write per hash
  a day). Deleted and memfd images are skipped: nothing to find again.
- Each rules load computes a digest of the rule files and IOC hash lists
  (paths after resolving symlinks, sizes, mtimes).
- RetroHunter (src/retro_hunt.cpp, own thread, idle priority, paced at
  retro_hunt_pace_ms, deferred by the governor as WorkClass::retro_hunt)
  re-scans when the digest differs from the last completed pass, most
  recently seen first. Only paths that still hold the same content are
  scanned (O_NOFOLLOW, regular files, hash compared); others count as
  changed or missing. A completed pass is recorded (retro_hunt record) so
  a restart does not repeat it; an interrupted one starts over.
- New matches (rules not recorded before for that hash, not allowlisted)
  become indicators with scan_kind retro_hunt, a file/indicator graph
  link, a positive label and a notification. Correlation turns them into
  findings titled "YARA match on a binary that ran here (found after a
  rules update)". Nothing is blocked or contained from here.
- Replay reads positive labels from the evidence stream and marks windows
  that ran a binary labelled bad later (LATE in the CLI).
- Status: a retro_hunt block (API_CONTRACT.md); config: retro_hunt,
  retro_hunt_pace_ms.

Tests: test_retro_hunt (remembered once; passes only on a digest change
and survive a restart; a clean-at-exec binary matching a new rule becomes
an indicator, label, finding and notification; a replaced binary is
counted as changed, never scanned as the old one; a stop abandons a pass
unrecorded); test_replay R9 (late labels, from a real store).
fuzz_evidence_records covers the new records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 08:59:38 -06:00
trvon
e746eaa70d
Correlate incrementally: rebuild only the groups that changed
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Every finding is a function of one group of indicators (a process, a
parent, a hash or a rule). IncrementalCorrelator keeps group membership
(ids and keys, not indicator contents) and rebuilds only the groups that
new or changed indicators touch. Each pass used to copy every indicator
and rebuild every finding; it now costs about the size of what changed.

- correlation_engine.cpp: the four group loops become per-group
  functions (re-indented; logic unchanged), shared by the full pass and
  the incremental one.
- EvidenceView keeps a bounded change log of written indicator ids
  (resets after clear(), at first use, or past 65536 entries);
  Database::take_indicator_changes / find_indicators expose it, with
  rolled-back writes reported as removed.
- DetectionPipeline holds its lock only to read changes and members and
  to store findings, as before.

Fix found by the oracle test: an indicator naming a process node but no
pid formed a second group for the same process with the same finding id,
and the two findings overwrote each other. The group pid now comes from
the node id when the pid is missing (finding ids are unchanged).

tests/test_correlation_incremental.cpp: random batches of new and updated
indicators give exactly the findings the full pass after every batch
stored (I1, I2); a batch touching one process rebuilds only its groups
(I3); the store's change log, including a rolled-back write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 19:51:57 -06:00
trvon
7c1a7fc1c1
Bound finding size and say why a store write is refused
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Found by the KPI detect phase (2-3 persist failures per run):
- a finding for one rule across thousands of processes listed every
  related id and outgrew the 256 KiB block limit, so the correlation
  commit was refused; related lists are now capped at 200 (the most
  recent indicators) with the totals in metadata
- a single record up to the 1 MiB record limit may now form a block on
  its own (the decoder bound only loosens)
- a refused append records its reason ("rejected: ...") instead of
  logging an empty error
- the Database no longer logs every indicator, node, edge and finding it
  stores (175k lines in a 4-minute run); failures are still logged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 06:43:46 -06:00
trvon
833ae7c423
Add the store's disk floor, retention, verify and compact
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 1 increment 7.
- Disk floor: below max(store_min_free_mb, store_min_free_pct) free,
  telemetry is refused and counted (a gap record with the count and
  time range is written once space returns) or, with
  store_floor_action=evict_oldest_telemetry, the oldest sealed
  telemetry is deleted; evidence continues until store_hard_min_free_mb.
- Retention (telemetry only, off by default): store_retention_days and
  store_max_mb, applied when a segment is sealed. Evidence is never
  deleted by retention.
- Status gains a store block (floor, free space, per-stream sizes,
  compression ratio, drops, deletions); health reports a low disk,
  a failed evidence store and unsaved detections.
- strawberry --store-verify checks the manifest and every segment
  (read-only, safe while the daemon runs; root-only telemetry is noted,
  not flagged). strawberry --store-compact rewrites sealed segments at
  zstd level 19 with the daemon stopped, never touching a damaged one.
- The manifest records each sealed segment's uncompressed size.

Tests L7-L11 cover the floor, gaps, retention by age and size,
eviction, compaction and verify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 06:19:29 -06:00
trvon
648457802a
Forbid empty catch blocks
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Every catch now handles the exception or records it with
STRAWBERRY_NOTE_SWALLOWED() (strawberry/swallowed.hpp): an atomic count,
exported in status as resources.swallowed_exceptions, plus a file:line
log line written with write(2), so it is safe out of memory and in
destructors. The 16 empty catches (store error fallbacks, destructors,
YARA compiler callback, thread entry logging) now use it. Status also
reports thread_errors, which was counted but never exported.

Enforced by source invariant R7 (with a self-test fixture) and by
clang-tidy bugprone-empty-catch in a new .clang-tidy (errors; inherits
parent config). AGENTS.md and CLAUDE.md state the rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:52:45 -06:00
trvon
dc50f06f8d
Drop unqlite-cpp and import legacy databases
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Nothing shipped used the library after the event store landed. A small
reader/writer for its file format (src/store/legacy_kv) replaces it for
the legacy code, and the submodule, subproject and library test are
removed.

A writer open now imports a legacy database.db file into the event
store: indicators are replayed in an order consistent with the old
hash/rule indexes, the result is verified against the old file, the
old file is kept as database.db.migrated-<ts>, and an import
interrupted between its final renames is finished on the next open.
Read-only handles read a legacy file directly and switch to the store
once it is imported. Ids re-stored under another hash are reported
(their old index entries cannot be carried over).

Tests: legacy import properties I1-I6 (locked and corrupt files are
left untouched), the differential test now shares its comparisons, and
fuzz_legacy_kv covers the file parser. A copy of a real 137 KB legacy
database imported cleanly into 7 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:46:21 -06:00
trvon
adb40e5b62
Add compressed event store
zstd block codec (CRC32C, varint tagged records), segment files with
sealed footers and block index, crash-atomic MANIFEST, LogStore
(lock, recovery, rolls, orphaning, heal with backoff), the evidence
view, and the legacy unqlite store kept for import. Property tests
under injected faults and crashes, a differential test against the old
store, and fuzz targets for every decoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:17 -06:00