Commit graph

7 commits

Author SHA1 Message Date
trvon
8411c70ba8
macOS: build and test fixes found re-validating on the Mac (57/57 green)
- fleet.cpp: HOST_NAME_MAX is Linux-only; use a 256-byte buffer (the
  POSIX/RFC 1035 limit gethostname() truncates to).
- test_replay: two dangling pointers - victim(run(...)) returned a pointer
  into a temporary report destroyed at the end of the statement. Passed on
  Linux by allocator luck; failed deterministically on macOS for every seed.
- test_yara_scanner: a brace initializer with a comma inside assert() (the
  macro splits it on macOS), and ::close needs <unistd.h> outside the
  Linux-only block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:35:34 -07:00
trvon
db4c49155f
yara: rule partitions — scan each file only with the rules that can match it
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.

Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
  after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
  pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
  size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
  kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
  priority order (big non-PE first) under rule_partition_budget_pct (default
  100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
  would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
  and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
  differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
  partition, full_scans); config rule_partitions (default true) and
  rule_partition_budget_pct.

Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:58:39 -07:00
trvon
4586eed50d
yara engine: range/fast-mode scanning API, ELF load ranges, engine A/B bench, opt-in guard partial scan
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
  full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
  builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
  differences, but the corpus has no rule matches, so parity is not proven.

- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
  zero-copy blocks; file offsets and filesize keep file semantics) with
  EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
  bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
  loaded segments of ELF images >= 4 MiB at launch; such results are marked
  `partial`, never cached, and the pipeline still scans the whole file. Status
  pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
  not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).

Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:51:10 -07:00
trvon
8f4cb586b8
Rules R2: node-local compiled-rules cache keyed on libyara version + sources
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:54:28 -07:00
trvon
b46e367e6d
Record YARA string hits as rule_match telemetry
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Each rule match from a fresh scan now writes a rule_match record: the
rule, namespace, severity, rules generation and its matched strings
(identifier, offset, length, 64-bit FNV-1a digest of the captured bytes).
The matched bytes themselves are never stored. Cache hits write no
rule_match, since their scan_result already names the rules.

Hits are collected in the libyara callback (libyara 4+) and capped at
16 strings per rule, 4 hits per string and 256 per scan. The callback
also tolerates a null user_data now.

Tests: caps, offsets and digests against a real scan; record round-trip
with the encode cap; the telemetry record fuzzer covers rule_match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 11:08:58 -06:00
trvon
ad1676a594
Integrate features into the daemon, build and docs
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Wire the policy, feeds, containment, exec guard, behavior, governor,
crash safety and event store into the daemon, detection pipeline, CLI
and meson build (zstd now required); Dockerfile, conan, systemd unit
(LimitCORE=0, resource limits) and installer updates; existing tests
adapted (store paths are directories now). Detections are counted and
notified even when evidence cannot be persisted (persist_failures in
status). prevention_timeout_ms is clamped and behavior dedupe handles
out-of-order events. Docs: API contract, developer guide, plan, UX,
agent rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:30 -06:00
trvon
22a653f617 Implement runtime pipeline, reporting, and local UI 2026-04-26 16:31:01 -04:00