Commit graph

102 commits

Author SHA1 Message Date
trvon
ffc7917957
feat: symbolic IR, swift backend, binary-to-fuzz pipeline, SARIF fixes
- Rust IR encoder (encodeRustIr): walks HIR expr trees, fires SafetyObserver hooks
- Swift SIL frontend (fold-swift): swiftc -emit-sil parser, demangling, NDJSON output
- SwiftBackend: --mode swift CLI routing, shells out to fold-swift
- Binary-to-fuzz: Mach-O export discovery, ARM64 ABI param-count inference, dlopen harness generation
- Language-agnostic IR: json_keys canonical constants, Call/Member/ArraySubscript/Cast/HeapNew nodes
- Dogfood fixes: SARIF kind field, cross-detector dedup, harness_file in predictions, Swift stoi safety
- Corpuses: swift-sil-z3-corpus.md, binary-fuzz-corpus.md
- AGENTS.md: cross-language wire format rule
2026-05-27 15:29:14 -04:00
trvon
95aac8209c
feat: expand rust security scanner coverage 2026-05-27 11:32:05 -04:00
trvon
ce1c353847
feat: harden security scanner routing 2026-05-27 10:36:57 -04:00
trvon
b4c80b32b8
docs: capture library-expansion follow-ups in plan doc
Three outstanding workstreams after the matrix sweep:

1. scan_runner.cpp dispatch hook for uninit-local-read (C5).
   Uncommitted in local edits; entangled with parallel session's
   C2 clang::CallGraph work. Coordinated commit pending.

2. Native UAF detector. Carved out from C4 (ConsumedAnalysis was
   wrong-tool -- needs source attributes). New detector should
   be a CFG-walker tracking freed_ptrs per function.

3. Per-row precision baselines for Z5, Z10, C5, C7, C8, C9.
   Z4 has one (docs/perf-audit/runs/matrix/Z4-curl-2026-05-26.json);
   the rest are pending.

Also: cross-cutting cleanup (workspace junk to gitignore,
parallel-session rust-frontend edits, catch2 chained-comparison
lint).
2026-05-26 09:21:20 -04:00
trvon
1dc20ca920
docs(matrix): mark C4 unscheduled -- wrong-tool FP
clang::consumed::ConsumedAnalysis only activates on classes marked
[[clang::consumable]] + member functions marked
[[clang::callable_when(state)]] + [[clang::return_typestate(state)]].
Real-world C code (curl, yams, kernel-style) has none of these
attributes; without auto-injection the analysis is silent.

A native UAF detector remains a real opportunity, but it's a
CFG-walker shape (track malloc/free on tracked pointers + flag
derefs after free), not ConsumedAnalysis. Carved out as a future
detector row rather than wedged into C4's framing.
2026-05-26 09:16:03 -04:00
trvon
5acc896eef
docs(matrix): mark C3 unscheduled -- sized wrong, no forcing function
clang ships the dataflow framework but NO ready-made pointer-
nullability analyzer. Bundled UncheckedOptionalAccessModel is for
std::optional, not raw pointers. C3 would require a custom
DataflowAnalysis<NullPtrLattice> with transfer functions for every
pointer-touching expression -- L, not M as originally sized.

Original sizing column referenced functions_prechecked_safe but
that counter measures syntactic sink-absence, not nullability.

fold already has multiple condition-based non-null pre-passes
(enclosingThenBranchProvesNonNull, ternaryArmProvesNonNull,
priorFunctionTextProvesNonNull, staticParamContractProvesNonNull,
current_debugassert_nonnull_). Reconsider when a concrete null-
deref FP catalog says clang dataflow would catch what these miss.
2026-05-26 09:15:18 -04:00
trvon
55e1d53a9c
docs(matrix): mark C1 unscheduled -- no forcing function
Surveyed sites: dead_branch.cpp has 7 dyn_cast uses, all
single-line type-tag dispatch. security_walker.cpp has 48 of the
same shape. ASTMatchers would be MORE verbose for type-tag
dispatch (MatchFinder + bind + callback structure).

They win for pattern-structure queries (regex name + nth arg
shape + ...), which fold's existing detectors don't need.
"Apply selectively, not bulk" was correct; on inspection there's
no concrete bulk-replaceable site that benefits.
2026-05-26 09:13:48 -04:00
trvon
63e519fb8a
docs(matrix): mark Z9 unscheduled -- infrastructure absent
z3 4.13+ (fold's pinned version) does not export interpolant
APIs; they were a 4.5-era research feature that was removed.
Z3_get_interpolant / mk_interpolant aren't in the linked headers.

Even if available, existing buildCounterexample already emits
concrete model values like `(e.g., when ptr=0)` which are easier
to read than formula slices for the UX-upgrade claim.
2026-05-26 09:12:52 -04:00
trvon
644a6b6b2d
docs(matrix): link Z4 curl baseline JSON
Captures curl scan profile after Z4 (lambda memcpy) lands:
analyze_ms=565020, null-deref wall_ms=344675 / 2,533 queries.

Established convention: each shipped row's Status column links
to its baseline JSON under docs/perf-audit/runs/matrix/.
2026-05-26 09:12:10 -04:00
trvon
da7e072028
perf(symbolic): model memcpy/memmove via z3::lambda (Z4)
Replaces the per-byte `store(store(...))` chain in modelMemcpyLikeCall
with a single `z3::lambda(i, ite(in_range, src_byte, orig_byte))`.
Drops the `!arg_values[2].is_numeral()` bail so symbolic copy_bytes
now reaches the solver instead of being silently skipped.

Concrete cap (kMaxModeledCopyBytes=4096) is kept to prevent
pathological bit-blast explosion on giant memcpy()s. strcpy is
unchanged -- literal bytes are short + concrete and the lambda
form doesn't help there.

z3 simplifies the lambda away at concrete-index selects so QF_BV
stays viable; no logic-change cascade. Full unit suite stays
57/57 green.
2026-05-26 08:47:20 -04:00
trvon
0f462e5b73
docs(matrix): mark C10 unscheduled -- architectural-mismatch FP
clang::CrossTranslationUnitContext is a callee-AST importer via
ASTImporter; fold's CrossTuCache is a property-summary cache.
Different paradigms.

Clang CTU exists for analyses that walk callee statements (CSA-
style). fold deliberately avoids that -- summary_encoder serialises
bodies to SMT-LIB strings, detectors consume properties not AST.
Adopting clang CTU would mean redesigning the analysis paradigm,
not extending it.

The matrix's "selective adoption" framing implied a real selective
use case; on inspection there isn't one in fold's design.
2026-05-25 23:11:59 -04:00
trvon
a791230552
feat(report): C9 FixIts + unified-diff --fix-output
Adds the FixIt collector + per-file unified-diff patch writer
behind a new `fold scan --fix-output <path>` flag. Patch-only --
no in-place rewrites -- so a bad suggestion can't silently
overwrite source.

Consumers (only when the transform is provably sound):
- UninitLocalRead: scalar integer locals only.
  `int x;` -> `int x = 0;`. Non-scalar types deferred.
- DeprecatedApi: regex extracts `[Uu]se foo()` from the
  deprecation message; silent skip when unparseable rather
  than hallucinating a rewrite.

Patches include 3 lines of context above + below each fix so
`git apply --check` accepts them. End-to-end verified against a
synthetic fixture.
2026-05-25 16:40:43 -04:00
trvon
d6e76f45ab
docs(matrix): mark C6 unscheduled -- current-state FP
dead_statement.cpp:140 already filters macro-ID anchors
(battle-tested in 2026-05-15 curl audit: doh.c 9,060 -> 0
findings). Remaining FPs are CFG-construction artifacts
(range-for desugaring, EH stubs), not preprocessor-induced.

fold has no multi-config scan mode, so "variant -D" was a
speculative failure mode. If a real dead-branch macro FP surfaces,
the fix is a one-line isMacroID() filter on dead_branch.cpp, not
a Preprocessor-hook row.
2026-05-25 12:16:48 -04:00
trvon
48dac13d53
feat(detect): uninit-local-read detector (C5 wiring)
Graduates C5's UninitVariablesAnalysis wrapper from
measurement-only into a real detector. Emits a finding per read of
an uninitialised local in the TU's main-file functions.

Scope is calibrated (C5's wrong-hypothesis pivot): clang's analyzer
flags LOCAL read-before-write, NOT parameter nullability. Tests
pin this explicitly (`int p; return p; -> no finding`).

End-to-end on synthetic fixture: 3 expected findings (read of
uninit local, self-init `int x = x`, subsequent use of the
self-init). Parameter reads correctly skipped.

scan_runner.cpp dispatch is unstaged in this commit -- parallel
session has unrelated in-flight edits to that file. The detector
is registered and tested; scan-runner wiring lands in a follow-up.
2026-05-25 11:52:20 -04:00
trvon
95b1ae777b
docs(matrix): mark Z6 unscheduled -- infrastructure FP
`inferCountedLoopInvariant` already runs in security_walker, but
its result only feeds spdlog::debug -- `refineRange` and
`recordRangeInOctagon` run unconditionally regardless of invariant
proof.

Extending fixedpoint to mutual recursion before wiring the existing
single-loop invariant into analysis state would add capability
nothing reads. The actual fix is precision-bug-shaped (consume the
result), not library-expansion-shaped.
2026-05-25 11:25:41 -04:00
trvon
d410bb7ba7
feat(fuzz): refine SAT models for smaller integer witnesses (Z5)
When a witness recorder is active, run z3::optimize.minimize on the
SAT model before extracting bytes for fuzz seeds. Refines integer
params toward the smallest path-and-hazard-satisfying value, so
seeds emit `arg=1` instead of `arg=0xDEADBEEF`.

Measured on synthetic access_at_offset(p, offset) with the path
`0 < offset && offset < 100`: primary-witness median 33 -> 5
(85% smaller). divide_by_count(p, count) with `count > 0` was
already path-minimal so no change.

Only the primary witness is refined; recordAlternateWitnesses
still uses the original model so the corpus covers the
equivalence class.

Curl-scale validation deferred -- fuzz-prep on full curl crashes
pre-existing (verified Z5-OFF baseline exit 139).
2026-05-25 11:22:38 -04:00
trvon
071f469d1d
docs(matrix): batch audit -- Z3 FP, Z4 site correction
Z3 ("byte-unroll elimination") was wrong-tracked: encoder doesn't
unroll strcpy/strlen byte-by-byte. Concrete cases use
recordKnownStringDestinationLength; symbolic cases bail. The real
Z3 opportunity is a capability addition, not a perf win.

Z4's matrix insertion site (heap_model.cpp::mergeFrom) was wrong --
that path is pure C++ hashmap union with no z3 queries. Real site
is encoder.cpp:688/715 where memcpy/strcpy unroll per-byte
store-chains capped at kMaxModeledCopyBytes and bail on symbolic
copy_bytes. Array lambdas would replace the loop body with one
expression and handle symbolic length.
2026-05-25 10:13:55 -04:00
trvon
72bc61f2c4
Document library expansion audit results 2026-05-25 09:55:55 -04:00
trvon
43827fa780
docs(matrix): mark Z2 unscheduled -- design FP
HeapRegion is a C++ struct, not a serialised z3 value. Its only
z3-typed fields (base, size) are already z3 BV exprs; freed/in_loop
are plain bools. The struct never crosses TUs, so a z3 tuple sort
saves zero bytes in the cross-TU cache. mergeFrom is hashmap union
with no z3 queries -- "alias-query count" was an imaginary timer.

Real opportunity at this site is path-sensitive freed merge, but
that's a heap-model correctness fix, not a library-expansion row.
2026-05-25 09:55:37 -04:00
trvon
c29a290be2
Reduce Curl high-signal false positives 2026-05-25 09:49:07 -04:00
trvon
6b5cae57a3
docs(matrix): mark Z1 unscheduled -- measured FP on curl
Instrumented `instantiateSummaryExpr` + `instantiateSummaryPredicate`
+ `instantiateSummaryPredicateInContext` directly: 3,052 calls /
238 ms on a full curl scan = 0.042% of analyze_ms (570 s).

Original `summary_lookup_ms` hypothesis was wrong-tracked; that
timer measures the hash-map lookup, not the parse_string path Z1
would replace. Quantifier e-matching would also add semantic-drift
risk for no measurable win.
2026-05-25 09:47:48 -04:00
trvon
07a39e73a7
perf(symbolic): compact serialised summaries via z3 let-bindings
Drop pp.min-alias-size from 10 (default) to 2 so shared subterms
collapse under `let` in serialize(). Measured 39.9% byte reduction
on encoder-shaped n=100 corpus (94.9KB -> 57.0KB). Roundtrip
preserved -- deserialize() parses let-form via z3's SMT-LIB reader.
2026-05-25 09:19:13 -04:00
trvon
1a7b6c1302
feat(detect): deprecated-api detector via clang::Decl::isDeprecated
New AST-only detector that flags call sites whose target decl is
marked deprecated under any of the three idioms:

- `[[deprecated]]` / `[[deprecated("why")]]` (C++14 attribute)
- `__attribute__((deprecated))` (GCC/Clang extension)
- `__attribute__((availability(...,deprecated=N.M)))` (Apple)

Per-call rather than per-decl so a migration that updates call
sites one at a time shows monotone progress. Deprecation reason
(when present) carries through into finding.message. System-header
call sites are skipped so the report stays scoped to the user's
own code.

Curl baseline: 52 real findings, e.g.
  ldap.c:269  ldap_url_parse — first deprecated in macOS 10.11
  multi-formadd.c:50  curl_formadd — since 7.56.0. Use curl_mime_init()

Refs docs/library-expansion-matrix.md row C7.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 00:47:38 -04:00
trvon
634607e1ea
perf(security): add detector sink slicing hooks 2026-05-24 23:45:12 -04:00
trvon
3c02c123f6
feat(symbolic): /*@ requires p != null */ contract seeding
Adds fold::symbolic::parseFunctionContracts(fd, ctx) — reads
RawCommentList for a function decl and returns the param indices
proven non-null by `/*@ requires <param> != null */` clauses.
SecurityWalker calls applyInitialFunctionContracts before walkCFG,
so within-function derefs see the contract via the existing
isDefinitelyNonNull pre-filter.

Grammar v1: bare `<ident> != null|NULL|nullptr`. Multi-line and
multi-clause supported. Unrecognised shapes are silently rejected
so future grammar growth doesn't break existing files.

applyHostToolchain now passes -fparse-all-comments. Without it
clang's lexer drops ordinary block comments before they reach the
AST, so the contract syntax can't ride along on `/*` ... `*/`.

Scope: suppresses within-function derefs on contracted params.
Does NOT suppress call-site findings — that needs CrossTUCache
integration so callee CalleeProperties carry the precondition.
Filed as a follow-up on matrix row C8.

Refs docs/library-expansion-matrix.md row C8.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 23:31:32 -04:00
trvon
64247e9409
docs: library-expansion matrix for Z3 + Clang opportunities
Catalogs 10 Z3 + 10 Clang APIs fold doesn't yet use, each with a
file-cited insertion site, scored payoff, and complexity/risk
columns. Top-3 sequencing nominates Z8 (unsat-core caching) →
C5 (UninitializedValues measurement) → C8 (RawCommentList).

Z8 + C5 ship in this same round; matrix rows updated with
"shipped" status. Perf-audit doc gains a single line pointing at
the matrix from its Phase-2 backlog header — the matrix is the
canonical surface for library-expansion picks now.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 23:03:02 -04:00
trvon
58f392899c
Add precision-preserving profiling prechecks 2026-05-24 20:31:55 -04:00
trvon
57f13a7643
docs(build): canonical BUILD.md with per-platform deps
Top-level BUILD.md so users see the dependency list (libz3-dev,
libclang-dev, llvm-dev, etc.) without spelunking docs/. Lists apt
package names for Ubuntu/Debian, dnf names for Fedora/RHEL, and
brew names for macOS — kept in sync with docker/Dockerfile.linux-
validate, which is the executable spec.

- README points at BUILD.md instead of docs/BUILDING.md.
- docs/BUILDING.md trimmed to radare2 vendoring notes only (the
  topic that didn't belong in a top-level quick-start).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 18:11:31 -04:00
trvon
331a68467c
Optimize security detectors and add quality gates 2026-05-23 20:42:05 -04:00
trvon
159e71ec1a
bench: 5-run post-cache baseline for the curl workload
Captured via bench/perf_audit_run.py --label baseline-post-cache
--runs 5. State of the tree: writtenvars cache (52218478) +
dead-statement range-for filter (8895d50) + all phase-timing
instrumentation from this audit session.

5-run summary:
  observed_wall_s            436.6s  iqr=17.8s  range=427.6..452.1
  total_wall_ms              436,568 ± 17,837
  scan.serial_dispatch_ms    436,498 ± 17,836  (all of it; --profile forces serial)

  detector breakdown (medians, ms):
    null-deref   wall=270,437  walker=267,126  encode=6,109  hook=4,748  z3_queries=5,531
    dead         wall=8,526
    equiv        wall=2,173
    clone        wall=204

The 4% IQR is tight enough to attribute sub-5% optimizations
reliably; bigger changes need fewer samples. Future perf PRs
should `--diff docs/perf-audit/runs/baseline-post-cache.json`
against this snapshot.

Also archives the earlier 2-run smoke output as
docs/perf-audit/runs/smoke.json.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 15:17:16 -04:00
trvon
9d08cb73a9
docs(perf-audit): update fold-on-fold triage with dead-statement fix outcome
Document the 152 → 16 reduction from the range-for + structured-
binding filter (commit 8895d50). The remaining 16 findings are
all the same idiom: defensive `return "unknown"` after an
exhaustive enum switch. Lays out three resolution options and
recommends documenting it as a known FP idiom rather than
refactoring fold's own source.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 14:51:01 -04:00
trvon
28c5155bfa
docs(perf-audit): point at bench/perf_audit_run.py for future PRs
The audit's "measurement discipline" section now names the
concrete tool to use for any sub-15% perf claim — the multi-run
harness committed in the previous commit. Includes the
copy-paste invocation pattern for before/after diff.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 13:25:59 -04:00
trvon
21886984af
bench: multi-run perf measurement harness with median+IQR
Adds bench/perf_audit_run.py which runs fold N times against a
fixed compile DB and reports median + IQR + min/max for every
metric in --profile JSON (scan-level + per-detector). Output
snapshots land under docs/perf-audit/runs/<label>.json so
before/after diffs are git-tracked.

Motivation: the audit's writtenvars-cache commit (5221847)
showed sub-15% perf moves can't be reliably attributed from a
single run — cross-baseline variance was ~15% just from system
load. Future perf PRs need a statistical baseline. This harness
is the single-machine version of that discipline.

Usage:
  bench/perf_audit_run.py --label baseline --runs 5
  bench/perf_audit_run.py --label after-fix --runs 5 \\
      --diff docs/perf-audit/runs/baseline.json

The --diff mode prints a side-by-side with significance flagged
("yes"/"no") when the delta exceeds 1.5×IQR. That's a rough rule
of thumb; for high-confidence claims, run 10+ samples.

Smoke-tested with --runs 2 (15min wall) on the curl baseline:
2-run IQR = 13s on a ~440s scan (~3% band). 5 runs should
tighten that.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 13:25:21 -04:00
trvon
52218478f9
perf(walker): cache collectWrittenVars per Stmt (audit remediation #1)
WriteCollector is a RecursiveASTVisitor over a stmt subtree.
5 call sites in security_walker.cpp consulted it per-statement
during the main walk, the buildNonNullAtBlockEntry pre-pass, and
unrollLoop. Same stmt subtrees re-walked across these sites.

Add cachedCollectWrittenVars(stmt) member function with a
per-SecurityWalker std::unordered_map<const clang::Stmt*, set>
cache. WriteCollector's output is a pure function of the stmt
subtree (no SymbolicEnv state, no ctx mutation), so the cache
is always safe. Map lives one function's analysis (SecurityWalker
is per-function-scoped).

Converted all 5 call sites to the cached wrapper.

Perf: ran the curl baseline after the change (4 workers, 306 TUs,
2339 functions). Single-run wall = 451s, vs. 442–465s spread across
prior single-run baselines. Within run-to-run variance; the audit
doc adds a "measurement discipline" section noting that 5+ runs +
median are needed to validate sub-15% wins. Cache is architecturally
sound regardless — pure memoization with no behavior change.

44/44 tests still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 13:02:06 -04:00
trvon
8c154612b9
perf(profile): hook_ms — fourth pivot, walker bookkeeping IS the cost
Add a ScopedHookTimer RAII helper to SolverStats and instrument the
6 hot per-detector hook bodies (onPointerDeref, onDivision,
onBinaryArith, onTruncation, onSignedUnsignedConversion,
onArraySubscript). Each now records its own hook_ms which surfaces
per-detector in --profile JSON.

For null-deref on the curl baseline:

  wall_ms:                 344,644
    walker_ms:             340,329
      encode_ms:            24,156   (7.1% of walker)
      hook_ms:              11,860   (3.5% of walker)
      state_merge_ms:           75
      [unaccounted]:       304,238   (89.4% — walker bookkeeping)

Z3 total: ~33ms across the entire scan.

This is the fourth wrong-hypothesis pivot in this audit:
1. Hash-walk fusion: 0.04% of total          ← wrong
2. State-merge optimization: <0.1%           ← wrong
3. Encoder optimization: 7.1%                ← wrong
4. Hook body optimization: 3.5%              ← wrong

The 304s lives in the walker's per-stmt bookkeeping:
- M22-M28 non-null marking (6 separate accumulators)
- collectWrittenVars (per-stmt RecursiveASTVisitor)
- refineRanges / refineOctagon (per-branch)
- buildNonNullAtBlockEntry (per-function pre-pass)
- BranchLivenessBuilder / WriteCollector ancillary passes

Phase-level instrumentation can't subdivide further without
O(stmts) timer overhead. Updated docs/perf-audit.md with
flame-graph-based profiling as the next step + 4 specific
remediation candidates ranked by likely payoff.

44/44 tests still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:41:54 -04:00
trvon
5ebe38d502
perf(profile): encode_ms — second pivot, encoder is not the bottleneck
Subdivide walker_ms with a new encode_ms field that times the
outermost encodeExpr / encodeCondition / applyStmt /
encodeConditionWithShortCircuit calls from the SecurityWalker
(8 call sites, each wrapped with a ScopedEncodeTimer RAII helper).

For null-deref on the curl baseline:

  wall_ms:                 274,332
    walker_ms:             270,956
      encode_ms:             6,451  (2.4% of walker)
      state_merge_ms:           0
      walker-encode-merge: 264,505  (97.6% — bookkeeping + hooks)

This is the third pivot in two sessions of perf-audit work. The
encoder hypothesis is wrong: 264s of the 271s walker time is NOT
in encodeExpr family. The actual cost lives in walker per-stmt
bookkeeping (range/octagon refinement, M22/M23/M25/M26/M27/M28
non-null marking, CFG block lookup, branch liveness) and in the
hooks' SAT-prep work (assumption-vector construction, expression
hashing, multi-model sampling from commit 4c343d4).

Z3 itself remains tiny (~33ms across the entire scan).

Updated docs/perf-audit.md item #2 with the corrected numbers and
the next-round remediation hypothesis: add hook_ms next; expect
it to dominate.

44/44 tests still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:25:24 -04:00
trvon
78ef40d72f
perf(profile): per-detector walker_ms + state_merge_ms instrumentation
Subdivide the security-detector wall_ms bucket into walker time and
merge time (audit backlog item #3). For null-deref's 287s wall on
the curl baseline the new fields surface a surprise:

  null-deref wall_ms:       286,610
    walker_ms:              283,206   (98.8% — all in walkCFG/walk)
    state_merge_ms:             142   (0.05% — merge is NOT the target)
    walker - merge:         283,064   (encoder + hook dispatch)

The state-merge hypothesis in the original audit (env copy/diff at
if-joins as the cost driver) is empirically wrong; merge is
essentially free. The 283s lives in encoder calls + observer hook
dispatch + RecursiveASTVisitor traversal.

Updated docs/perf-audit.md item #2 with the corrected numbers and a
narrower remediation sketch: subdivide walker_ms further into
visit_ms / encode_ms / hook_ms in the next round.

Mechanics:
- SolverStats gains walker_ms + state_merge_ms (std::chrono::ms).
- SecurityWalker reuses its existing walk_start clock to populate
  stats_->walker_ms; new merge timer wraps the resolved-binding
  block in walkIf.
- applySolverStats propagates both into DetectorProfile.
- profile.cpp emits both fields in --profile JSON.

New baseline: docs/perf-audit/baseline-phases-2026-05-23.json.
44/44 tests still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:43:11 -04:00
trvon
ef7601b884
docs(positioning): triage heuristic — input-reachability filters most FPs
Add a "What fold reliably catches vs what's noise" section informed
by recent real-codebase usage. Two concrete claims now in the doc:

1. On tight, well-typed codebases most security findings are FPs
   because the bounded symbolic model can't carry domain invariants
   the codebase enforces elsewhere.

2. The reliable signal is the CLI-input → arithmetic-hole pattern.
   Where untrusted input flows into integer math or pointer
   arithmetic without a bounds check, fold catches it; the fix is
   usually a one-line defensive check at the boundary.

Plus a four-step triage heuristic users can apply when reading a
report: filter by input reachability; trust ConcreteWitness;
triage Proved against reachability; heavily filter Heuristic.

Recommends `--detectors security` (which includes taint-track) as
the mechanical version of the input-reachability check.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:29:11 -04:00
trvon
67f90ec6bc
docs: add fold performance audit (2026-05-23)
Audit doc citing the new ScanProfile instrumentation, the curl
baseline, the fold-on-fold static-analysis pass, and a Phase-2
backlog ranked by observed cost rather than guesses.

Headline findings:

- The plan's "highest-leverage" target (5-walk hash fusion in the
  clone phase) turned out to be 0.04% of total wall on curl.
  Pivoted to instrumentation; doc records the pivot honestly.
- New named bucket `serial_dispatch_ms - sum(detector.wall_ms)`
  exposes 146s (32%) of previously-unaccounted wall, almost
  entirely clang ASTUnit construction per TU.
- null-deref dominates detector wall at 65% of total. ~99% of
  that is encoder + state-merging, not Z3.
- Per-phase null-deref instrumentation is now item #3 in the
  backlog; PCH for project headers is item #1.

Cross-references the instrumentation commit (503799a) and the
fold-on-fold pass (previous commit) so consumers can reproduce
all numbers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:19:13 -04:00
trvon
57cf9fd439
docs(perf-audit): fold-on-fold static-analysis pass
Run fold against its own 134-TU compile DB with
--detectors clone,dead-statement,equiv --clone-type 3.

Results:
- 487 clone findings (Type-1/2/3, 80%+ similarity)
- 152 dead-statement findings
- 0 equiv findings (signature-bucket misses; scout-rule extractions
  moved colliding helpers out)

docs/perf-audit/fold-on-fold.md triages the top duplication clusters:
- `mixHash` 5+ copies across security-detector anonymous namespaces
- `supportsParam` / `collectConjuncts` repeated in many detectors
- `onFunctionEnd` boilerplate across all 5 security observers
- `ir::*::valid` / `operator bool` / `operator==` per-type boilerplate
- Test-fixture duplication (intentional; not a refactor target)

Plus a triage suggestion for the 152 dead-statement findings (cache
short-circuits usually safe; encoder bails usually intentional).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:17:14 -04:00
trvon
503799a64c
perf(profile): instrument compile-db load + fanout + serial dispatch
Add three new phase-level timings to ScanProfile:
- compile_db_load_ms: loadCompilationDatabase + buildSyntheticCompileDB
- fanout_orchestration_ms: SubprocessFanout::runSync wall (spawn + IPC + wait)
- serial_dispatch_ms: in-process runDetectorsOnShard wall (--profile forces
  serial today; this is the dominant phase when profiling)

Motivation: the baseline curl scan showed sum(detector.wall_ms) = 298s
out of 455s total wall, leaving 157s (34%) unaccounted. With this
instrumentation the user can see at a glance where the gap lives.

For the captured baseline (curl, 306 TUs, 2339 functions, --profile,
serial):
  total_wall_ms:           455_756
  compile_db_load_ms:           2  (negligible)
  fanout_orchestration_ms:      0  (not used in profile mode)
  serial_dispatch_ms:      455_682
  detector wall_ms sum:    298_361
  unaccounted in dispatch: 157_321  ← clang AST build + cross-TU priming
                                       + call-graph build + fingerprinting

Phase-2 work will subdivide the 157s further. This commit just
surfaces it as a named metric in the JSON.

44/44 tests still green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:05:30 -04:00
trvon
3d5b310c92
docs: add CLI design philosophy
Durable policy doc that every CLI change reviews against. Adapted
from sage's CLI methodology, fold-specific.

Sections: purpose; workflow stages + canonical verb table
(scan/show/confirm/fuzz prep+run/report/serve/worker); 13 design
principles (one canonical path per intent, lifecycle layering,
scan-vs-confirm separation, help that teaches the graph, errors
that recover, parse-time validation, non-interactive first-class,
machine-readable stability, metadata from the real graph,
verification on the built binary, deliberate back-compat, scriptable
streams, literature as constraint); required review checklist;
preferred flow patterns; anti-patterns; audit appendix.

The audit appendix names 13 anti-patterns observable in fold's CLI
today; #1-#6 are addressed by the Phase-1 refactor commits in this
series (C1: --quiet + hide worker + fuzz group, C2: detector parse
validation, C3: errorWithRecovery, C4: r2 noise filter). #7-#13
are tracked as Phase-2 follow-ups.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 22:38:42 -04:00
trvon
da4c848cbc
feat(rust): MVP Rust frontend — unsafe-deref detection + clone clustering
First non-clang frontend for fold. Scans Rust source for the
subset of issues that meaningfully exist there: raw-pointer
dereferences in `unsafe` blocks (Heuristic confidence) and
within-Rust Type-1/2/3 clones via the existing clone detector.
Reuses the M30 frontend-agnostic IR + AnalysisBackend
abstraction unchanged.

Architecture: a separate `tools/fold-rust` cargo binary that the
C++ `RustBackend` shells out to. Two subprocess invocation
modes:

- `extract --unit <file>`: standalone .rs file, no Cargo
  context (limited; external-crate imports won't resolve).
- `cargo-check --crate-dir <dir>`: invokes `cargo check` with
  RUSTC_WORKSPACE_WRAPPER set to ourselves so cargo supplies
  the full --extern/-L/--cfg per-TU. Mirrors the
  clippy/miri/kani pattern.

RustBackend::discoverUnits walks up from each `.rs` input to
find the nearest Cargo.toml and collapses sibling files to one
crate unit; orphan files fall back to single-file extract.
scan_router auto-detects --mode=rust when all paths are `.rs`;
mixed Rust + C/C++ warns and defaults to source mode.

Includes:

- tools/fold-rust/: cargo crate pinned to nightly-2026-05-22.
  HIR-driven DerefHunter (raw-pointer dereference detection
  via typeck `is_raw_ptr()`) and HashWalker (FNV-1a + mix64
  port of src/ir/normalize.cpp for FunctionSummary hashes).
- include/fold/ir/expr_json.{hpp,cpp}: RustExprNode +
  RustNodeArena + bidirectional JSON wire format for
  ir::Expr; round-trip tested across 8 Kinds.
- include/fold/frontend/rust_backend.{hpp,cpp}: AnalysisBackend
  impl. posix_spawnp + NDJSON parser. Caches summaries +
  findings across extractFunctions calls; polymorphic
  runDetectorsOnShard dispatch routes summaries to
  detectClones.
- src/cli/{main,scan_router,scan_runner}.cpp: --mode=rust
  branch + autoDetectRust(). Mode validation extended to
  accept rust.
- docs/rust-frontend.md: scope, wire format, rustc-private
  pin + bump procedure, verification snippets.

44/44 unit + integration tests still green. Verified
end-to-end on a 3-function rename-pair fixture: 1 clone
cluster + 1 raw-deref finding through both `extract` and
`cargo-check` paths.

Out of scope (tracked separately): equiv + class-dup
detectors (need encoder frontend-agnostification),
dead-statement (needs Rust CFG), cross-language clones
(needs unified kind-tag taxonomy), Proved/ConcreteWitness
promotion (needs encoder integration), `cargo fuzz`
harnesses.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 22:36:28 -04:00
trvon
0caf2b39a3
docs: positioning + M31 Phase 3 concolic scoping
Two long-pending docs:

- docs/positioning.md: sales-free comparison of fold against
  clang-tidy, commercial SAST, CodeQL/Joern, CBMC/ESBMC, and
  KLEE/SymCC/S2E. Tells engineers when to choose fold and
  when not to. Closes task #88.

- docs/m31-phase3-concolic.md: scoping for the closed-loop
  symbolic execution piece of M31 Phase 3. Compares
  vendoring SymCC's LLVM pass (Option A) vs the LibAFL +
  symcc-rt bridge (Option B). Recommends Option B for
  maintenance leverage; stages P3.1 (plumbing), P3.2
  (LibAFL harness), P3.3 (promotion path).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 22:35:15 -04:00
trvon
a51cdf5748
M19–M30 + scout-rule encoder refactor
Multi-phase work merged into one commit:

* M19: flow-sensitive guard tracking (stmtAlwaysExits, walkIf early-exit)
* M20–M27: non-null marking across deref, addr-of, callsite preconds,
  if/while/switch guards, AddrOf-of-subexpr, CFG-block-scoped maps
* M30 Phase A: dialect-agnostic SourceLoc + LocationMapper + frontend/
  boundary marker; ClangLocationMapper wired through SafetyObserver
* M30 Phase B v2: QueryTracer + 13 SafetyObserver hooks migrated to
  ir::Expr / ir::Call / ir::MemberRef / ir::Symbol / ir::SourceLoc
* M30 Phase C/D/D2/D3: ir::Symbol, ir::Expr (with Kind), ir::Call,
  ir::MemberRef opaque types; ir_expr_clang.cpp houses the clang
  adapters; encodeExpr migrated to switch(kind) dispatch

Scout-rule extractions from encoder.cpp (2800 → 2371 lines):
* encoder_callee_predicates.{hpp,cpp} — isMallocLikeCall, isFreeCall,
  isAssertCall as pure std::string_view predicates
* encoder_type_queries.{hpp,cpp} — typeSizeBytes, sourceWidth,
  isUnsignedLike, byteWidthExpr
* encoder_callee_names.{hpp,cpp} — calleeName, qualifiedFunctionName,
  qualifiedCalleeName (with internal-linkage @-suffix)
* encoder_coerce.{hpp,cpp} — coerceToBv, coerceToExactBv, coerceToBool,
  coerceToType (BV-64 storage model documented at the boundary)
* encoder_reachability.{hpp,cpp} — ReachabilityCounter impl, markEncoded,
  recordEncoded, recordBailedNode, public profiling glue
* encoder_summary_instantiate.cpp — SMT-LIB summary substitution
  (instantiateSummaryExpr / Predicate / PredicateInContext +
  applySummaryParamPost; kills the redundant *Impl pass-through layer)
2026-05-21 11:17:50 -04:00
trvon
c4e42cec0a
Add scan router and MCP job queue
Split source and binary scan routing with merge-only hybrid support, binary provenance, and router-backed MCP scan contexts.

Add async MCP scan jobs for long-running scans, detector precision fixes, benchmark scripts, docs, and integration coverage.
2026-05-19 00:24:12 -04:00
trvon
a56d5dac2b
Phase 1: Clang CFG traversal layer + test coverage 2026-05-15 19:17:01 -04:00
trvon
0452353d2e
Implement walkSwitch: analyze case bodies in switch statements
Results: +86 TP across all CWEs (476:+3, 369:+18, 190:+51, 121:+14)
2026-05-15 18:32:45 -04:00
trvon
7af5927729
Update benchmarks doc with final numbers 2026-05-15 15:13:58 -04:00
trvon
72ceaa7e4b
Update benchmarks doc with current numbers 2026-05-15 11:21:17 -04:00