eb85913f6e
- dangerous-import correctly flags gets() and system() imports - Hardcoded-string FP fix too aggressive (drops real findings) - CWE gaps: buffer overflow, format string, UAF, div-zero still missed - Next: wire binary_bb_encoder into pipeline for Z3-proved findings
48 lines
1.9 KiB
Markdown
48 lines
1.9 KiB
Markdown
# CTF Binary Re-Evaluation — Loop 2
|
|
|
|
> Generated 2026-05-30. Re-evaluation after Phase 1-3 binary backend improvements.
|
|
|
|
## Detection Rate Trend
|
|
|
|
| Loop | Detection Rate | Findings | FPs | Real Findings |
|
|
|------|---------------|----------|-----|---------------|
|
|
| Baseline | 3.7% (1/27) | 1 | 0 | 1 |
|
|
| Loop 1 (pre-FP fix) | 259% (70/27) | 70 | ~67 | ~3 |
|
|
| Loop 1 (post-FP fix) | 7.4% (2/27) | 2 | 0 | 2 |
|
|
|
|
**Net improvement: 3.7% → 7.4% (2x)**
|
|
|
|
## What's Working
|
|
|
|
| Detector | Real Findings | Example |
|
|
|----------|--------------|---------|
|
|
| `dangerous-import` | 2 | `gets()` in libbof_overflow, `system()` in libcmd_injection |
|
|
| `binary-security` | 1 | `gets()` xref in vulnerable_gets function |
|
|
|
|
## What Failed (FP Sources in Loop 1)
|
|
|
|
| Detector | FPs | Root Cause |
|
|
|----------|-----|------------|
|
|
| `hardcoded-string` | ~67 | Build paths, system library paths in binary data |
|
|
| `hardcoded-string` (post-fix) | 0 | Fix was too aggressive — also drops real credential findings |
|
|
|
|
## What's Still Missed (CWE gaps)
|
|
|
|
| CWE | Expected Finding | Status |
|
|
|-----|-----------------|--------|
|
|
| CWE-121 | Buffer overflow via strcpy | ❌ Not detected (binary strcpy import not flagged) |
|
|
| CWE-134 | Format string via printf | ❌ Not detected |
|
|
| CWE-416 | Use-after-free | ❌ Needs basic block analysis |
|
|
| CWE-369 | Division by zero | ❌ Needs Z3 encoding |
|
|
| CWE-190 | Integer overflow | ❌ Needs Z3 encoding |
|
|
| CWE-476 | Null dereference | ❌ Needs Z3 encoding |
|
|
| CWE-78 | Command injection | ✅ Detected (system() import) |
|
|
|
|
## Next Steps for Binary Backend
|
|
|
|
| # | Action | Effort | Expected Gain |
|
|
|---|--------|--------|--------------|
|
|
| 1 | Wire binary_bb_encoder.cpp into scan pipeline | S | +6 Z3-proved findings |
|
|
| 2 | Add strcpy/printf to dangerous import list | S | +2 findings |
|
|
| 3 | Fix hardcoded-string: add security-relevant path filter | S | +2 real, 0 FP |
|
|
| 4 | Basic block xref analysis (call→string association) | M | +3 findings |
|