8ac9ddb175
- skills/fold-sast: rename to skill-fold-sast and expand the scan/precision guidance for C++/Rust/Swift lanes - docs/architecture: record call-graph phase decomposition and the optimization-loop TP/FP/TN/FN evidence policy - docs/mcp-tools + README: update budget/per-query defaults (1800s/6000ms) and note the OpenGrep pre-push hook
377 lines
14 KiB
Markdown
377 lines
14 KiB
Markdown
# fold benchmark harness
|
||
|
||
Drives fold and comparator SAST tools across a corpus, writes raw findings
|
||
per tool, scores them against a gold standard, and produces a precision /
|
||
recall / F1 table you can paste into a PR or write up.
|
||
|
||
## Layout
|
||
|
||
```
|
||
bench/
|
||
├── run_bench.py # subprocess driver: corpus → per-tool JSON
|
||
├── run_summary_ab.py # fold summary off/on benchmark with profile deltas
|
||
├── run_curl_ab.py # curl source-vs-binary backend A/B harness
|
||
├── compare_literature.py # Fold runs → literature/ceiling report
|
||
├── literature_baselines.json # paperbridge-grounded external baselines
|
||
├── score.py # JSON + gold → summary.md
|
||
├── corpora/ # corpus fetchers + cached source trees
|
||
│ ├── fold-fixtures/ # symlink → ../tests/fixtures/clones (zero-download)
|
||
│ ├── poj-104/ # fetch.sh stub; see "Adding POJ-104" below
|
||
│ └── sard-cwe-561/ # fetch.sh stub; see "Adding SARD" below
|
||
├── gold/ # gold-standard answer files
|
||
│ └── fold-fixtures.json
|
||
└── results/ # one dated subdirectory per run
|
||
```
|
||
|
||
## Smoke run (no downloads)
|
||
|
||
```bash
|
||
cd bench
|
||
python3 run_bench.py \
|
||
--corpus corpora/fold-fixtures \
|
||
--name fold-fixtures \
|
||
--tools fold,clang-tidy \
|
||
--processes 1
|
||
python3 score.py \
|
||
--run-dir results/<latest> \
|
||
--gold gold/fold-fixtures.json
|
||
```
|
||
|
||
|
||
## Optimization micro-benchmarks
|
||
|
||
Use `run_micro_opt.py` for synthetic before/after validation of fold speed and
|
||
false-positive optimization patches. It generates focused C++ projects, runs
|
||
`fold scan --profile`, and writes `summary.json` / `summary.md` plus raw SARIF
|
||
and profile artifacts. See `docs/internal/perf-audit/micro-benchmarks.md` for the case
|
||
matrix.
|
||
|
||
For a complete real-project or precision iteration, follow
|
||
[`docs/architecture/optimization-loop.md`](../docs/architecture/optimization-loop.md).
|
||
It pins the workload through existing manifests, requires profile attribution
|
||
and repeated measurements, and keeps source/binary provenance and explicit
|
||
precision labels separate.
|
||
|
||
```bash
|
||
python3 bench/run_micro_opt.py --case all --repeats 3
|
||
python3 bench/run_micro_opt.py --case branch-merge --size 200 --repeats 5
|
||
```
|
||
|
||
|
||
## Quality gate
|
||
|
||
Use `run_quality_gate.py` to collect the focused micro-benchmarks plus correctness
|
||
smokes into one manifest. It wraps `run_micro_opt.py`, `run_summary_ab.py`,
|
||
`run_bench.py` + `score.py`, and optional curl/Juliet runs.
|
||
|
||
### Project manifests
|
||
|
||
`scripts/benchmarks/run_project_eval.py` is the generic real-project runner for
|
||
C++ source, Rust workspaces, and binaries. It uses versioned manifests under
|
||
`tests/benchmarks/projects/` and writes SARIF, profiles, immutable run artifacts,
|
||
finding-stability summaries, and optional disposition/baseline comparisons.
|
||
|
||
```sh
|
||
python3 scripts/benchmarks/run_project_eval.py \
|
||
--manifest tests/benchmarks/projects/yams_source.json \
|
||
--out-dir /tmp/fold-yams-eval
|
||
|
||
python3 scripts/benchmarks/run_project_eval.py \
|
||
--manifest tests/benchmarks/projects/sage_rust.json \
|
||
--out-dir /tmp/fold-sage-eval
|
||
```
|
||
|
||
Chromium is an opt-in production-scale canary. The committed manifest selects
|
||
eight deterministic translation units across `base`, `net`, `services/network`,
|
||
and `url`; it does not attempt an unbounded monorepo scan. From a sibling
|
||
`../chromium/src` checkout, generate the compilation database using Chromium's
|
||
official helper, then run the shared evaluator:
|
||
|
||
```sh
|
||
cd ../chromium/src
|
||
tools/clang/scripts/generate_compdb.py -p out/Default > compile_commands.json
|
||
cd ../../fold
|
||
python3 scripts/benchmarks/run_project_eval.py \
|
||
--manifest tests/benchmarks/projects/chromium_source.json \
|
||
--out-dir /tmp/fold-chromium-eval
|
||
```
|
||
|
||
Build the selected Chromium targets first when generated headers are absent.
|
||
The manifest skips cleanly when no checkout exists, so Chromium remains a
|
||
local/periodic benchmark rather than a required CI download.
|
||
|
||
Include a project run in the local quality gate only when its reviewed baseline
|
||
is stable:
|
||
|
||
```sh
|
||
python3 bench/run_quality_gate.py --skip-curl --skip-juliet \
|
||
--project-manifest tests/benchmarks/projects/yams_source.json
|
||
```
|
||
|
||
```bash
|
||
python3 bench/run_quality_gate.py --skip-curl --skip-juliet
|
||
python3 bench/run_quality_gate.py --juliet-cwes 476 369 --juliet-timeout 300
|
||
```
|
||
|
||
## Summary A/B run
|
||
|
||
Use `run_summary_ab.py` to compare `fold scan` with summaries disabled
|
||
against `--build-summaries` on the same compile database. By default it uses
|
||
`bench/corpora/interproc`, runs `null-deref`, `div-by-zero`, `oob`, and
|
||
`int-overflow`, enables profile output, and scores against
|
||
`<corpus>/gold.json` when present.
|
||
|
||
```bash
|
||
python3 bench/run_summary_ab.py
|
||
python3 bench/run_summary_ab.py --repeats 3
|
||
python3 bench/run_summary_ab.py \
|
||
--corpus bench/corpora/interproc \
|
||
--detectors null-deref,div-by-zero,oob,int-overflow \
|
||
--extra-arg=--trace-z3
|
||
```
|
||
|
||
Each run writes `summary.json`, `summary.md`, per-variant SARIF, profile JSON,
|
||
and stdout/stderr logs under `bench/results/<timestamp>_summary-ab/`. The
|
||
Markdown report includes finding-count deltas, Z3 query and detector wall-time
|
||
deltas, exact added/removed finding keys, optional precision/recall/F1, and an
|
||
`optimization_signals` payload that separates useful TP/recall gain from pure
|
||
implementation overhead.
|
||
Set `LLVM_BIN=/path/to/llvm/bin` if the default Homebrew LLVM path is not
|
||
available.
|
||
|
||
## curl Source-vs-Binary A/B
|
||
|
||
Use `run_curl_ab.py` to exercise the source backend and the radare2 binary
|
||
backend on curl in one run. The harness writes `source.sarif`, `binary.sarif`,
|
||
logs, `summary.json`, and `summary.md` under `bench/results/<timestamp>_curl-ab/`.
|
||
|
||
By default it looks for curl source at `../curl-source/curl`, configures a
|
||
temporary CMake build to obtain `compile_commands.json`, scans the source build,
|
||
and compares it against `/usr/bin/curl` with the binary backend.
|
||
|
||
Fast smoke run against one source file and only a few binary functions:
|
||
|
||
```bash
|
||
python3 bench/run_curl_ab.py \
|
||
--source-detectors clone \
|
||
--source-max-files 8 \
|
||
--source-path-regex '/src/tool_main\.c$' \
|
||
--binary-max-functions 8 \
|
||
--processes 1
|
||
```
|
||
|
||
Source security detectors are intentionally more expensive on production C code.
|
||
For source-security A/Bs, start with a small file cap and explicit budgets:
|
||
|
||
```bash
|
||
python3 bench/run_curl_ab.py \
|
||
--source-detectors null-deref,div-by-zero,int-overflow,oob \
|
||
--source-max-files 2 \
|
||
--source-path-regex '/(lib|src)/' \
|
||
--source-budget-seconds 10 \
|
||
--source-per-query-ms 250 \
|
||
--source-per-tu-timeout 15 \
|
||
--binary-max-functions 16
|
||
```
|
||
|
||
If curl is already configured, point directly at its compile database:
|
||
|
||
```bash
|
||
python3 bench/run_curl_ab.py \
|
||
--compile-db /path/to/curl-build/compile_commands.json \
|
||
--curl-binary /usr/bin/curl \
|
||
--binary-max-functions 64
|
||
```
|
||
|
||
For a stricter same-source comparison, let the harness build curl and scan the
|
||
freshly built binary instead of the Apple system binary:
|
||
|
||
```bash
|
||
python3 bench/run_curl_ab.py \
|
||
--build-curl-binary \
|
||
--source-max-files 0 \
|
||
--binary-max-functions 0
|
||
```
|
||
|
||
Additional CMake options can be passed through with repeated `--cmake-arg`, for
|
||
example `--cmake-arg=-DBUILD_TESTING=OFF`. The binary side uses fold's
|
||
`--input-type binary` path; linked `libr_core` is used when fold was configured
|
||
with radare2 support, otherwise the backend falls back to the `r2` CLI.
|
||
|
||
For Juliet slices, use file-level scoring because each gold entry names a
|
||
testcase file rather than an exact finding location. Juliet C++ files are kept
|
||
on `c++17` to match the existing Juliet runner.
|
||
|
||
```bash
|
||
python3 bench/corpora/juliet/filter_cwes.py
|
||
python3 bench/run_summary_ab.py \
|
||
--name juliet-476-summary-ab \
|
||
--corpus bench/corpora/juliet/filtered/476 \
|
||
--gold bench/gold/juliet-476.json \
|
||
--detectors null-deref \
|
||
--score-mode file \
|
||
--cxx-standard c++17
|
||
python3 bench/run_summary_ab.py \
|
||
--name juliet-121-summary-ab \
|
||
--corpus bench/corpora/juliet/filtered/121 \
|
||
--gold bench/gold/juliet-121.json \
|
||
--detectors oob \
|
||
--score-mode file \
|
||
--cxx-standard c++17 \
|
||
--max-delta-details 25
|
||
```
|
||
|
||
Gold files are currently checked in for these Juliet slices:
|
||
|
||
```text
|
||
121, 122, 124, 126, 127, 190, 191, 194, 195, 196, 197, 369, 476, 561
|
||
```
|
||
|
||
`run_fold_on_juliet.py` defaults to that full set and maps each CWE to the
|
||
Fold detector/kind expected by its gold file:
|
||
|
||
```bash
|
||
python3 bench/run_fold_on_juliet.py --processes 1
|
||
python3 bench/run_fold_on_juliet.py --cwes 476 369 190 121 --processes 4
|
||
```
|
||
|
||
Long Juliet runs can be resumed without rerunning completed variants:
|
||
|
||
```bash
|
||
python3 bench/run_summary_ab.py \
|
||
--out-dir bench/results/<timestamp>_juliet-190-summary-ab \
|
||
--reuse-existing \
|
||
--corpus bench/corpora/juliet/filtered/190 \
|
||
--gold bench/gold/juliet-190.json \
|
||
--detectors int-overflow \
|
||
--score-mode file \
|
||
--cxx-standard c++17
|
||
```
|
||
|
||
## Literature and Ceiling Comparison
|
||
|
||
`literature_baselines.json` records paperbridge-resolved references and broad
|
||
external baseline metrics for Juliet scanner comparisons, NIST SATE/SARD,
|
||
SV-COMP, production-code SAST studies, CVEfixes, and DiverseVul. Use it for
|
||
context and expansion planning, not as a direct head-to-head score unless a
|
||
baseline row explicitly uses the same corpus, scoring unit, and detector scope.
|
||
|
||
`compare_literature.py` consumes Fold benchmark outputs and writes a Markdown
|
||
and JSON report with:
|
||
|
||
- all scored Fold rows from `run_summary_ab.py`, `run_fold_on_juliet.py`, or comparator payloads,
|
||
- best Fold variant per CWE/kind,
|
||
- F1 ceilings for perfect precision, full recall at current FP count, alert-budget oracle triage, and perfect oracle scoring,
|
||
- optimization signals that rank summary variants by TP delta, FP delta, elapsed cost, and safe next action,
|
||
- literature baseline and benchmark-expansion tables.
|
||
|
||
```bash
|
||
python3 bench/compare_literature.py \
|
||
--fold-run bench/results/2026-05-17T20-06-05Z_juliet-476-summary-ab/summary.json \
|
||
--fold-run bench/results/2026-05-17T20-09-41Z_juliet-369-summary-ab/summary.json \
|
||
--fold-run bench/results/2026-05-17T20-19-02Z_juliet-190-summary-ab/summary.json \
|
||
--fold-run bench/results/2026-05-17T21-16-12Z_juliet-121-summary-ab/summary.json
|
||
```
|
||
|
||
Without `--fold-run`, it picks the latest `*summary-ab/summary.json` result per
|
||
run name. Outputs land in `bench/results/<timestamp>_literature-comparison/`.
|
||
|
||
## Adding a corpus
|
||
|
||
A corpus is just a directory of `.cpp` files. The runner generates a flat
|
||
`compile_commands.json` over `*.cpp` recursively. For corpora that ship
|
||
with their own build systems, supply a `corpora/<name>/fetch.sh` that
|
||
downloads and (optionally) configures them.
|
||
|
||
### POJ-104
|
||
|
||
`bench/corpora/poj-104/fetch.sh` is already wired up — it sparse-clones
|
||
just the POJ-104 subtree from Microsoft's CodeXGLUE repo and renames the
|
||
`.txt` files (CodeXGLUE's archive convention) to `.cpp`. The resulting
|
||
tree is ~50,000 submissions (~700 MB of source), so plan disk budget
|
||
accordingly.
|
||
|
||
```bash
|
||
bench/corpora/poj-104/fetch.sh
|
||
python3 bench/gold/_generate_poj104.py # → bench/gold/poj-104.json
|
||
python3 bench/run_bench.py \
|
||
--corpus bench/corpora/poj-104/dataset \
|
||
--name poj-104 \
|
||
--tools fold,clang-tidy \
|
||
--detectors equiv \
|
||
--processes 4
|
||
python3 bench/score.py \
|
||
--run-dir bench/results/<latest> \
|
||
--gold bench/gold/poj-104.json
|
||
```
|
||
|
||
Gold entries are one-per-algorithm-folder with the rule "at least one
|
||
cross-pair in this cluster must be flagged equivalent" — generous to
|
||
clustering detectors (fold) without requiring `O(n²)` redundant findings
|
||
per cluster.
|
||
|
||
### NIST SARD CWE-561 (Dead Code)
|
||
|
||
```bash
|
||
# corpora/sard-cwe-561/fetch.sh
|
||
curl -L https://samate.nist.gov/SARD/test-suite/...-CWE-561.zip -o sard.zip
|
||
unzip sard.zip -d sard-src
|
||
```
|
||
|
||
Gold standard: each annotated dead block becomes a `{"kind": "dead",
|
||
"files": ["..."], "id": "..."}` entry.
|
||
|
||
## Adding a tool
|
||
|
||
Drop a `run_<tool>(corpus_dir, out_dir) -> dict` helper into `run_bench.py`.
|
||
The dict must have `tool`, `version`, `elapsed_s`, `exit_code`, and
|
||
`findings: list[{file, line, kind, id}]`. Wire it into `main()`'s tool
|
||
dispatch.
|
||
|
||
Currently supported:
|
||
- `fold` — uses `--format sarif`, parses results out of the SARIF doc.
|
||
- `clang-tidy` — runs with a focused `-checks` set covering redundancy and
|
||
branch-clone diagnostics.
|
||
|
||
Planned: `pmd-cpd` (Java jar), `codeql` (DB build + query bundle).
|
||
|
||
## Determinism
|
||
|
||
Two consecutive runs against the same corpus must produce byte-identical
|
||
fold SARIF (see `src/report/sarif.cpp::writeSarif` sort step). The bench
|
||
runner asserts this via `diff` in `.github/workflows/bench.yml` so an
|
||
accidental nondeterminism regression breaks CI before it muddles the
|
||
precision numbers.
|
||
|
||
## Precision drift gate (the gate is local, not CI)
|
||
|
||
`bench/fp-baseline.json` is the **committed** per-detector precision/recall/F1
|
||
reference (regenerate it intentionally when a change is meant to move the numbers).
|
||
The drift gate compares a fresh run to it and **fails on regression** — the enforcement
|
||
that was previously missing (drift went unnoticed because nothing compared to a
|
||
baseline).
|
||
|
||
Run the gate locally (≈2 min; deterministic):
|
||
|
||
```sh
|
||
python3 bench/score_fp.py --gate --run --security-bugs --juliet \
|
||
--juliet-cwes 476 369 190 191 121 122 124 126 127 194 195 196 197 561 \
|
||
--max-files-per-cwe 120 --fold-bin build/release/src/cli/fold
|
||
# exit 0 = no per-detector precision/recall/F1 drop beyond --tolerance (default 0.02)
|
||
# exit 1 = regression (prints which detector and the delta)
|
||
```
|
||
|
||
`--max-files-per-cwe N` deterministically stride-samples each Juliet CWE so huge CWEs
|
||
(121/122 are ~8–10k files) stay tractable; the gold set is automatically restricted to
|
||
the scanned subset so recall stays meaningful. The same flags must be used for the gate
|
||
as for the committed baseline (they are, above).
|
||
|
||
Regenerate the committed baseline (only when the change *should* move numbers):
|
||
|
||
```sh
|
||
python3 bench/score_fp.py --run --security-bugs --juliet \
|
||
--juliet-cwes 476 369 190 191 121 122 124 126 127 194 195 196 197 561 \
|
||
--max-files-per-cwe 120 --summary --out bench/fp-baseline.json
|
||
```
|
||
|
||
This is a **local** gate (developer / pre-push), not a CI requirement.
|