CTF Binary Re-Evaluation — Loop 2
Generated 2026-05-30. Re-evaluation after Phase 1-3 binary backend improvements.
Detection Rate Trend
| Loop |
Detection Rate |
Findings |
FPs |
Real Findings |
| Baseline |
3.7% (1/27) |
1 |
0 |
1 |
| Loop 1 (pre-FP fix) |
259% (70/27) |
70 |
~67 |
~3 |
| Loop 1 (post-FP fix) |
7.4% (2/27) |
2 |
0 |
2 |
Net improvement: 3.7% → 7.4% (2x)
What's Working
| Detector |
Real Findings |
Example |
dangerous-import |
2 |
gets() in libbof_overflow, system() in libcmd_injection |
binary-security |
1 |
gets() xref in vulnerable_gets function |
What Failed (FP Sources in Loop 1)
| Detector |
FPs |
Root Cause |
hardcoded-string |
~67 |
Build paths, system library paths in binary data |
hardcoded-string (post-fix) |
0 |
Fix was too aggressive — also drops real credential findings |
What's Still Missed (CWE gaps)
| CWE |
Expected Finding |
Status |
| CWE-121 |
Buffer overflow via strcpy |
❌ Not detected (binary strcpy import not flagged) |
| CWE-134 |
Format string via printf |
❌ Not detected |
| CWE-416 |
Use-after-free |
❌ Needs basic block analysis |
| CWE-369 |
Division by zero |
❌ Needs Z3 encoding |
| CWE-190 |
Integer overflow |
❌ Needs Z3 encoding |
| CWE-476 |
Null dereference |
❌ Needs Z3 encoding |
| CWE-78 |
Command injection |
✅ Detected (system() import) |
Next Steps for Binary Backend
| # |
Action |
Effort |
Expected Gain |
| 1 |
Wire binary_bb_encoder.cpp into scan pipeline |
S |
+6 Z3-proved findings |
| 2 |
Add strcpy/printf to dangerous import list |
S |
+2 findings |
| 3 |
Fix hardcoded-string: add security-relevant path filter |
S |
+2 real, 0 FP |
| 4 |
Basic block xref analysis (call→string association) |
M |
+3 findings |
eb85913f6e