yams/tests/unit/content at experimental - trevon/yams - Forgejo: Beyond coding. We Forge.

History

trvon 2d4fca1d04 fix(extraction): tighten UTF-8 detection and keep units after bad surrogates EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes, surrogates and overlong sequences as UTF-8, so such files skipped conversion and reached the index as invalid UTF-8. It now uses the shared well-formedness check. convertToUtf8 consumed the unit after an unpaired high surrogate before checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it is a low surrogate.	2026-09-23 17:56:59 -06:00
..
archive_content_handler_catch2_test.cpp	fix(test): harden sanitizer merge blockers	2026-06-29 11:34:02 -04:00
content_handlers_test.cpp	fix(extraction): tighten UTF-8 detection and keep units after bad surrogates	2026-09-23 17:56:59 -06:00

trvon 2d4fca1d04

fix(extraction): tighten UTF-8 detection and keep units after bad surrogates

EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes,
surrogates and overlong sequences as UTF-8, so such files skipped
conversion and reached the index as invalid UTF-8. It now uses the shared
well-formedness check.

convertToUtf8 consumed the unit after an unpaired high surrogate before
checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it
is a low surrogate.

2026-09-23 17:56:59 -06:00

archive_content_handler_catch2_test.cpp

fix(test): harden sanitizer merge blockers

2026-06-29 11:34:02 -04:00

content_handlers_test.cpp

fix(extraction): tighten UTF-8 detection and keep units after bad surrogates

2026-09-23 17:56:59 -06:00