fold/docs/research/clang-quirks.md

Clang API Quirks Encountered by fold

Catalog of Clang LibTooling APIs that assert or recurse on inputs fold has encountered in real-world C++ codebases (primarily yams, fmt, spdlog, libc++ headers). Each entry: assertion text or symptom, the fold call site that trips it, and the precondition or guard we install.

This list grows when the process-isolated TU boundary observes a new SIGABRT/SIGSEGV/SIGTRAP. Fatal signals terminate the worker; the parent uses flushed file_finished records to identify the unfinished TU and writes it to quarantine.txt. Fold never jumps across interrupted C++ frames.

1. StringLiteral::getString() — wide-char assertion

Symptom:

Assertion failed: ((isUnevaluated() || getCharByteWidth() == 1)
  && "This function is used in places that assume strings use char"),
  function getString, file Expr.h, line 1869.

Why: StringLiteral::getString() returns a StringRef over the raw byte array but assumes single-byte characters. For wchar_t/char16_t/char32_t literals (e.g. L"..."), the bytes are 2 or 4 wide and the function deliberately asserts.

Fold call sites (all guarded):

• src/ir/hash.cpp:53 — clone summary hashing.
• src/ir/hash.cpp:182 — clone debug dump.
• src/detect/oob_access.cpp:81 — sprintf format-string introspection.
• src/symbolic/encoder.cpp:431 — sprintf format-string size analysis.

Guard:

if (sl->getCharByteWidth() == 1) {
    // safe to call getString()
} else {
    // fall back to byte-length-only hashing / bail entirely
}

2. ASTContext::getTypeInfoImpl() — infinite recursion on templates

Symptom: Stack overflow with 7,000+ frames of clang::ASTContext::getTypeInfoImpl + 1204, SIGSEGV when the OS stack runs out. Observed on fmt::basic_format_parse_context instantiations.

Why: Some heavy templated record decls have dependent-base chains that Clang's type-info computation walks recursively without a cycle guard.

Fold call sites:

• src/symbolic/encoder.cpp:typeSizeBytes (getTypeSizeInChars)
• src/symbolic/encoder.cpp:sourceWidth (getTypeSize)
• src/symbolic/encoder.cpp:1170 (getASTRecordLayout) — biggest offender.

Guards (combined):

• qt.isNull() || qt->isDependentType()
• qt->isIncompleteType()
• For getASTRecordLayout: also skip when parent->isDependentContext(), parent->isLambda(), cxx_parent->getDescribedClassTemplate() != nullptr, or isa<ClassTemplateSpecializationDecl>(parent).

3. Z3_solver_set_params("sat.threads") — Z3 4.17 parameter removal

Symptom:

libc++abi: terminating due to uncaught exception of type z3::exception:
  unknown parameter 'sat.threads'

Why: Z3 4.17 dropped sat.threads from the SAT solver's param descriptor. Some internal paths still raise the exception when the param is addressed indirectly (e.g. through tactics.cpp's par_or portfolio or std::async future-bound solvers).

Guard: every direct s.set("sat.threads", ...) is wrapped in if (supportsParam(s.get_param_descrs(), "sat.threads")) { s.set(...); }. Ordinary Z3 exceptions are handled at C++ boundaries. A fatal Z3 signal terminates the isolated worker and is attributed by the parent; the process is never reused.

4. SIGTRAP from libc++ debug assertions

Symptom: Worker dies with signal 5 (SIGTRAP); no informative output. Observed on yams equiv runs.

Why: macOS libc++ in debug builds raises SIGTRAP via __builtin_trap() instead of abort() on certain hardened-mode contract violations (out-of-range iterator, etc.). Source location is lost because Clang's template instantiation hides the offending callsite.

Guard: the isolated worker terminates. The parent records the unfinished TU in quarantine.txt and continues remaining work in a fresh process.

5. ImplicitCastExpr at call args — observer gap (not a crash)

This is not a crash but a recall gap fixed in the same session as the crash work: observeImplicitConversion was only wired into VarDecl init, plain BO_Assign, and compound assignment. Lossy casts at call arguments (e.g. malloc(signed_int) widening + signedness flip to size_t) skipped the observer entirely, zero-ing recall on Juliet CWE-194/195. Fixed at src/symbolic/encoder.cpp CallExpr arg-encoding loop (Phase-2 of the session).

Adding a new entry

1. Run tools/audit-yams.sh /tmp/yams-audit-<date> with process isolation active. Crashes get logged to quarantine.txt with signal provenance.

2. Reproduce one of the crashed TUs alone:

   ./build/release/src/cli/fold scan --compile-db /tmp/one.json \
     --detectors <det> --processes 1 \
     --format sarif --output /tmp/x.sarif
   

3. Run under lldb (lldb --batch -k 'bt 50' ...) to get the offending Clang API.

4. Add a guard at the Fold call site. Process isolation and quarantine contain the failure but do not make unsupported library behavior safe.

5. Append a new section to this doc with the symptom + fix.

Related files

• src/cli/signal_isolation.cpp — sigaltstack plus process-terminating signal policy.
• src/cli/subprocess_fanout.cpp — parent watchdog, progress reconciliation, quarantine attribution, and fresh-worker recovery.
• src/cli/clang_shard_tu.cpp — per-TU timer scope; no post-signal continuation.
• tools/audit-yams.sh — passes --quarantine-dir so consecutive detector runs share a single skip list.