trevon/fold
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
fold/bench/fp-baseline.md
trvon 080bf9655f
 feat(walker): affine (symbolic-bound) loop summarization + deterministic baseline 
Extend ir::Loop classification: a monotone single-induction-variable for-loop
with a symbolic bound (e.g. `for (i=0; i<n; ++i)`) is now Affine (was General).
The walker summarizes Affine loops like Counted ones — bind the induction
variable to a fresh symbolic, assert its start bound plus the loop condition
(encoded to a Z3 bool) as path constraints, walk the body once — exposing
high-index OOB that bounded unroll misses. Findings are over-approximations
(LoopHavoc/heuristic), never proved.

Net effect on the gate: oob recall 0.531 -> 0.559, F1 0.637 -> 0.652 (precision
dips 0.796 -> 0.782) — a contract-sanctioned over-approximation tradeoff.

Also regenerate bench/fp-baseline.json deterministically from a clean run: the
prior committed oob numbers (0.574) were not reproducible (transient warm-state
artifact); fresh runs are now bit-identical, so the gate is reliable.

Tests: test_ir_loop (affine classification), test_loop_summary (symbolic-bound
OOB exposed, not proved). Suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 08:46:16 -04:00

2.9 KiB
Raw Permalink Blame History

fold FP baseline

Generated: 2026-05-31T12-37-44Z Fold version: 0.1.0

Per-detector precision / recall / F1

Detector TP FP FN Gold Findings Precision Recall F1
dead 1 1 0 1 2 50.0% 100.0% 66.7%
div-by-zero 53 26 36 89 79 67.1% 59.6% 63.1%
int-overflow 89 34 76 165 123 72.4% 53.9% 61.8%
int-truncation 27 3 60 87 30 90.0% 31.0% 46.2%
null-deref 42 11 43 85 53 79.2% 49.4% 60.9%
oob 222 62 175 397 284 78.2% 55.9% 65.2%
sign-conversion 132 50 48 180 182 72.5% 73.3% 72.9%

Literature context

  • CodeQL FP rate: ~5%
  • Semgrep FP rate: ~12%
  • Source: Bennett et al. 2024 (EASE), Lipp et al. 2022 (ISSTA)

Per-detector per-corpus breakdown

dead

Corpus TP FP FN Gold Findings Prec Rec F1
security-bugs 1 1 0 1 2 50.0% 100.0% 66.7%

div-by-zero

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-369 51 26 36 87 77 66.2% 58.6% 62.2%
security-bugs 2 0 0 2 2 100.0% 100.0% 100.0%

int-overflow

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-190 45 16 31 76 61 73.8% 59.2% 65.7%
juliet-191 42 17 43 85 59 71.2% 49.4% 58.3%
security-bugs 2 1 2 4 3 66.7% 50.0% 57.1%

int-truncation

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-197 27 3 60 87 30 90.0% 31.0% 46.2%

null-deref

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-476 40 11 43 83 51 78.4% 48.2% 59.7%
security-bugs 2 0 0 2 2 100.0% 100.0% 100.0%

oob

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-121 35 9 41 76 44 79.5% 46.1% 58.3%
juliet-122 37 6 53 90 43 86.1% 41.1% 55.6%
juliet-124 40 5 35 75 45 88.9% 53.3% 66.7%
juliet-126 56 20 23 79 76 73.7% 70.9% 72.3%
juliet-127 52 22 23 75 74 70.3% 69.3% 69.8%
security-bugs 2 0 0 2 2 100.0% 100.0% 100.0%

sign-conversion

Corpus TP FP FN Gold Findings Prec Rec F1
juliet-194 52 24 28 80 76 68.4% 65.0% 66.7%
juliet-195 62 26 18 80 88 70.5% 77.5% 73.8%
juliet-196 18 0 2 20 18 100.0% 90.0% 94.7%