fold/bench/corpora/ctf-challenges/ANALYSIS.md

Fold CTF Challenge Binary Analysis Report

Date: 2026-05-29 Tool: fold v0.1.0 Architecture: ARM64 (Apple Silicon) Platform: macOS (Darwin)

Executive Summary

Fold was evaluated against 10 CTF-style binaries containing 27 intentionally-seeded vulnerabilities across 7 CWE categories. The evaluation ran both binary-mode scans (fold scan --mode binary) and source-mode scans (fold scan --mode source) for ground truth comparison, plus fuzz-prep harness generation.

Key findings:

Metric	Binary Mode	Source Mode	Fuzz-Prep
Total findings	1	37	N/A
Ground truth vulns detected	1 (3.7%)	18 (66.7%)	N/A
False negatives	26 (96.3%)	9 (33.3%)	N/A
Fuzz harnesses generated	N/A	N/A	40 / 44 exported funcs (90.9%)

Detailed Per-Binary Results

1. libbof_overflow.dylib (CWE-121, CWE-134, CWE-242)

Vulnerability	CWE	Function	Source TP?	Binary TP?
gets() deprecated API	CWE-242	vulnerable_gets	TP (deprecated-api)	TP (taint-track: bl sym.imp.gets)
strcpy buffer overflow	CWE-121	vulnerable_strcpy	FN	FN
sprintf format string	CWE-134	vulnerable_sprintf	FN	FN
sprintf buffer overflow	CWE-121	vulnerable_sprintf	FN	FN

Source findings: 11 (1 deprecated-api, 1 oob-access, 6 uninit-read, 3 always-condition) Binary findings: 1 (taint-track: gets() import detected) Binary analysis: 4/4 exported symbols found. 0% vulnerable API detection beyond gets() import.

2. libcmd_injection.dylib (CWE-78, CWE-798)

Vulnerability	CWE	Function	Source TP?	Binary TP?
system() command injection	CWE-78	command_injection	FN	FN
Hardcoded password	CWE-798	hardcoded_credential_check	TP (hardcoded-credential)	FN
Hardcoded API key	CWE-798	hardcoded_api_key	TP (hardcoded-credential)	FN

Source findings: 3 (1 always-condition, 1 hardcoded-credential, 1 uninit-read) Binary findings: 0 Note: Strings "supersecret123", "sk-1234567890abcdef", "Using key: %s" ARE present in the binary but the hardcoded-credential binary detector did not fire. The taint-track detector noted system as a sink but declared it "not tainted" (false).

3. libdiv_by_zero.dylib (CWE-369)

Vulnerability	CWE	Function	Source TP?	Binary TP?
Literal zero divisor	CWE-369	div_by_zero_direct	TP (div-by-zero, proved)	FN
Variable zero divisor	CWE-369	div_by_zero_variable	TP (div-by-zero, proved)	FN
Modulo by zero	CWE-369	mod_by_zero	TP (div-by-zero, proved)	FN

Source findings: 6 (3 div-by-zero all proved, 2 clone, 1 int-overflow) Binary findings: 0 Analysis: Fold's Z3-backed div-by-zero detector is highly effective in source mode, catching all 3 variants including the path-dependent y==0 case. Binary mode has no equivalent since it cannot reconstruct the division operation from the dylib.

4. libformat_string.dylib (CWE-134)

Vulnerability	CWE	Function	Source TP?	Binary TP?
printf format string	CWE-134	fmt_string_direct	FN	FN
fprintf format string	CWE-134	fmt_string_fprintf	FN	FN
sprintf format string	CWE-134	fmt_string_sprintf	FN	FN

Source findings: 1 (always-condition only) Binary findings: 0 Analysis: Format string vulnerabilities are not detected by fold's current source detectors (taint-track identifies printf/fprintf/sprintf as sinks but doesn't check whether the format argument is user-controlled when all arguments come from parameters). This is a known gap in fold's taint-tracking precision.

5. libint_overflow.dylib (CWE-190, CWE-122)

Vulnerability	CWE	Function	Source TP?	Binary TP?
Signed add overflow	CWE-190	vulnerable_add	TP (int-overflow)	FN
Signed mul overflow	CWE-190	vulnerable_mul	TP (int-overflow)	FN
count*elem_size overflow	CWE-190	vulnerable_alloc	TP (int-overflow)	FN
Heap overflow via int overflow	CWE-122	heap_overflow_via_int_overflow	TP (memory-leak)	FN

Source findings: 7 (3 int-overflow all heuristic, 1 memory-leak proved, 2 clone, 1 redundant-store) Binary findings: 0 Analysis: Fold's bounded model catches signed integer overflow with heuristic confidence. The memory leak in heap_overflow_via_int_overflow is correctly flagged.

6. libnull_deref.dylib (CWE-476)

Vulnerability	CWE	Function	Source TP?	Binary TP?
Literal NULL deref	CWE-476	null_deref_direct	TP (null-deref, proved)	FN
Conditional NULL deref	CWE-476	null_deref_conditional	TP (null-deref, proved)	FN
Missing malloc null check	CWE-476	null_deref_after_allocation_failure	FN	FN

Source findings: 5 (2 null-deref proved, 1 dead-branch, 1 always-condition, 1 redundant-store) Binary findings: 0 Analysis: Fold's null-deref observer catches literal and conditional NULL dereferences with proved confidence. The nullable-allocations flag was not enabled; enabling it would likely catch the missing malloc check. Binary mode cannot track pointer values.

7. libuaf_doublefree.dylib (CWE-415, CWE-416, CWE-401)

Vulnerability	CWE	Function	Source TP?	Binary TP?
Double free	CWE-415	double_free_bug	TP (oob-access: "Double free")	FN
Use after free	CWE-416	use_after_free_bug	FN	FN
Use after free (global)	CWE-416	use_global_after_free	FN	FN
Memory leak	CWE-401	set_global_and_free	TP (memory-leak, proved)	FN

Source findings: 4 (1 memory-leak proved, 1 oob-access proved, 2 always-condition) Binary findings: 0 Analysis: Double-free detected via oob-access detector with "Double free on tracked heap region". Use-after-free is NOT detected in the source scan — the libFuzzer/ASan integration (fuzz-run) would be needed to catch these at runtime. Memory leak detected.

8. libvuln_demo.dylib (CWE-121, CWE-134, CWE-416)

Vulnerability	CWE	Function	Source TP?	Binary TP?
strcpy buffer overflow	CWE-121	vuln_buffer_overflow	FN	FN
printf format string	CWE-134	vuln_format_string	FN	FN
Use after free	CWE-416	vuln_use_after_free	TP (use-after-free)	FN

Source findings: 5 (2 null-deref, 1 always-condition, 1 uninit-read, 1 use-after-free) Binary findings: 0 Note: vuln_demo has only main exported since internal functions are static. Binary mode correctly discovered main but couldn't reach the static helper functions.

9-10. libFuzzMeSwift.dylib / libtest_fuzz.dylib (Control Binaries)

Binary	Vulns	Source Findings	Binary Findings	Harnesses
libFuzzMeSwift	0 (control)	N/A (Swift, no source scan)	0	10
libtest_fuzz	0 (control)	N/A	0	1

Both control binaries produced 0 false positives, confirming no spurious findings.

Binary Analysis Coverage

Symbol Extraction

Binary	Exported/Total	Coverage
libbof_overflow	4/4	100%
libcmd_injection	3/3	100%
libdiv_by_zero	4/4	100%
libformat_string	4/4	100%
libint_overflow	4/4	100%
libnull_deref	4/4	100%
libuaf_doublefree	4/5	80%
libvuln_demo	1/4	25%
libFuzzMeSwift	15/15	100%
libtest_fuzz	1/1	100%

Average symbol extraction: 90% (exported symbols). Internal/static functions: 0%.

Fuzzable Function Discovery

Metric	Count
Total exported functions	44
Total fuzz harnesses generated	40
Discovery rate	90.9%

The 4 missing harnesses are:

• 1 global_ptr data symbol (libuaf_doublefree) — not a function
• 3 static/internal functions (libvuln_demo) — not exported, not discoverable without --fuzz-internal debug info

String Extraction

The binary scanner successfully extracted all embedded strings but only the gets() import-based taint-track detection fired. Strings like "supersecret123", "sk-1234567890abcdef", "ping -c 1 %s" were present in the binaries but the hardcoded-credential and command-injection binary detectors did not flag them.

Gap Analysis

Binary Mode Limitations

1. No semantic analysis of machine code: Binary mode relies on symbol imports, string extraction, and section analysis. It cannot reconstruct control flow or data flow from raw machine code.
2. No taint tracking in binary mode: The system() call in libcmd_injection was identified as a sink but taint propagation failed because the binary backend doesn't track data flow.
3. Static functions invisible: vuln_demo has 3 of 4 functions as static, making them invisible to binary analysis without DWARF debug info.
4. No hardcoded-credential detection in binary mode: Despite strings being present, the syntactic pattern matcher for API keys/passwords didn't fire.
5. No format string detection in either mode: Format string vulnerabilities are a known gap — fold's taint-track detector doesn't correlate printf/fprintf calls with user-controlled format arguments.

Source Mode Limitations

1. Format string detection gap (3 FNs): taint-track sees printf as a sink but doesn't determine that the format argument is tainted when all arguments are parameters.
2. Use-after-free detection (2 FNs): Requires runtime fuzzing (fold fuzz-run). The source-level use-after-free detector is still maturing.
3. malloc null check (1 FN): Needs --nullable-allocations flag enabled.
4. Command injection (1 FN): taint-track correctly identifies system() as a sink and propagates taint through sprintf() but declares the sink "not tainted". The taint propagation from function parameter through sprintf to system() appears incomplete.
5. Buffer overflow (2 FNs): The oob-access detector fires on array indexing but not on strcpy/sprintf buffer overflow patterns.

Fuzz-Prep Harness Quality

All generated harnesses use dlopen()/dlsym() with uint64 parameter types (default). The warning "no ops in first block" indicates the pdbj disassembler could not determine parameter counts from the binary, so it defaults to 3 parameters. Harnesses compile correctly but would benefit from:

• DWARF-based parameter type inference
• --binary-param-types infer for better type reconstruction
• Clang-based compile + ASan integration for runtime crash detection

Recommendations

Short-term (improve detection without new features)

1. Enable --nullable-allocations for null-deref scans to catch unchecked malloc.
2. Use --binary-param-types infer for better harness type signatures.
3. Enable --fuzz-internal with DWARF symbols for internal function discovery.
4. Run fold fuzz run on all generated harnesses with ASan to catch UAF/BO at runtime.
5. Add binary regex patterns for hardcoded credentials (API keys, passwords in strings).

Medium-term (feature enhancements)

1. Implement format-string detector: check if first argument to printf-family functions is a non-literal when it reaches the sink.
2. Extend oob-access to detect strcpy/sprintf/memcpy buffer overflows via destination buffer size comparison.
3. Port hardcoded-credential detector to binary mode.
4. Add --binary-param-types dwarf to extract parameter types from DWARF debug info.

Long-term (architectural)

1. Implement basic data-flow tracking in binary mode via radare2 ESIL or Ghidra-based lifting.
2. Add cross-architecture support: the current harnesses are ARM64-only; support x86_64 ELF for CI/testing.
3. Integrate libFuzzer + ASan runtime into fuzz-run for concrete witness generation.

Conclusion

Fold's source-mode analysis achieves 66.7% detection rate (18/27) on CTF-style vulnerabilities with strong performance on null-deref (2/3), div-by-zero (3/3), integer overflow (3/4), and deprecated API (1/1) detection. Binary-mode analysis is currently limited to import-based heuristics and found only 1 of 27 vulnerabilities (3.7%). Fuzz-prep harness generation is robust at 90.9% coverage of exported functions, providing a solid foundation for runtime fuzzing.

The largest gaps are in format string detection (0/6 across both modes), use-after-free detection (1/3), and command injection (0/1). These represent the highest-value improvements for fold's detection pipeline.