* fix(downloader): report a user cancel as OperationCancelled
The curl adapter returned PolicyViolation when shouldCancel() stopped a
transfer. DownloadService maps only OperationCancelled to a cancellation,
so a user's cancel reached callers as a NetworkError, and the service's
cancellation branch could never run.
The existing manager-level cancellation test used a fake adapter that
returns OperationCancelled itself, so it never exercised this path. The
new test drives the real curl adapter over a file:// URL, which runs its
write callback without a network, and fails before this change.
Found by pipe's experimental C++ front end: downloader::ErrorCode::
OperationCancelled was compared (download_service.cpp) but never produced.
* fix(downloader): emit the Resolving and Verifying progress stages
ProgressStage declares Resolving and Verifying, and the init, model and
download commands render both, but DownloadManager never emitted either:
the server probe and the integrity finalize / checksum check ran with no
progress event, so those CLI branches were unreachable and a long hash of
a large file looked like a stall.
The manager now reports Resolving before the probe and Verifying before
the digest is finalized. A new test records every stage the manager
emits for a checksummed download and requires Resolving < Connecting <
Verifying < Finalizing; it fails before this change.
Found by pipe's experimental C++ front end: both stages were matched in
CLI switches but never produced anywhere in src/.
* feat(topology): select the dirty-region expansion mode from config
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
* feat(search): apply lines: qualifiers to result snippets
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
* fix(build): map aarch64/arm64 to Conan's armv8 arch setting
build-arch-pkg.sh and the Arch PKGBUILD export YAMS_CONAN_ARCH from the
pacman/uname label (aarch64), which Conan rejects, failing the Arch
aarch64 release lane at conan install.
* fix(release): let clang find Arch ARM's GCC for Boost's arm64 target
Boost's b2 clang toolset compiles with --target=arm64-pc-linux on aarch64.
Clang cannot map that triple to Arch Linux ARM's aarch64-unknown-linux-gnu
GCC install, so every Boost configure check and compile lost libstdc++
headers and crtbeginS.o. Ship an arm64-pc-linux.cfg next to clang that sets
--gcc-triple to the host GCC.
---------
Co-authored-by: trvon <me@trevon.dev>
Co-authored-by: Trevon <git@trevon.dev>
Boost's b2 clang toolset compiles with --target=arm64-pc-linux on aarch64.
Clang cannot map that triple to Arch Linux ARM's aarch64-unknown-linux-gnu
GCC install, so every Boost configure check and compile lost libstdc++
headers and crtbeginS.o. Ship an arm64-pc-linux.cfg next to clang that sets
--gcc-triple to the host GCC.
build-arch-pkg.sh and the Arch PKGBUILD export YAMS_CONAN_ARCH from the
pacman/uname label (aarch64), which Conan rejects, failing the Arch
aarch64 release lane at conan install.
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
ProgressStage declares Resolving and Verifying, and the init, model and
download commands render both, but DownloadManager never emitted either:
the server probe and the integrity finalize / checksum check ran with no
progress event, so those CLI branches were unreachable and a long hash of
a large file looked like a stall.
The manager now reports Resolving before the probe and Verifying before
the digest is finalized. A new test records every stage the manager
emits for a checksummed download and requires Resolving < Connecting <
Verifying < Finalizing; it fails before this change.
Found by pipe's experimental C++ front end: both stages were matched in
CLI switches but never produced anywhere in src/.
The curl adapter returned PolicyViolation when shouldCancel() stopped a
transfer. DownloadService maps only OperationCancelled to a cancellation,
so a user's cancel reached callers as a NetworkError, and the service's
cancellation branch could never run.
The existing manager-level cancellation test used a fake adapter that
returns OperationCancelled itself, so it never exercised this path. The
new test drives the real curl adapter over a file:// URL, which runs its
write callback without a network, and fails before this change.
Found by pipe's experimental C++ front end: downloader::ErrorCode::
OperationCancelled was compared (download_service.cpp) but never produced.
archlinux/archlinux only publishes linux/amd64, so the native aarch64 Arch
lane failed with 'exec format error' before provisioning. Both Arch
Dockerfiles now take ARCH_BASE_IMAGE; the aarch64 matrix entry pins
ghcr.io/menci/archlinuxarm by digest and x86_64 keeps the official image.
package-validate.sh forwards ARCH_BASE_IMAGE to the systemd substrate.
The release Arch job passes --arch explicitly, but the script still ran
deb/rpm discovery against the default build/release dir. find fails on the
missing dir, pipefail + set -e turn that into a silent exit 1. Only discover
packages for selected lanes and tolerate a missing build dir.
* docs(agents): document PR stacking, promotion, and submodule merge rules
A stacked PR (#187) merged into its parent branch after the parent (#186)
had already merged into experimental, so its commits missed experimental
and the promotion PR until #189 carried them over. Record the merge order,
a git cherry check for stranded commits, and the no-squash rule for
submodule PRs.
* fix(metadata): scope migration 40 deletes to symbol extractor output
Migration 40 deleted every kg_edges row whose relation was calls,
implements, includes, ... and every node of a symbol type such as class or
interface. Graph ingest through MCP and IPC accepts any type and relation,
so an upgrade from v0.19 irreversibly removed user-modelled data like a
'class' node or an 'implements' edge.
Select the extractor's nodes by provenance and key shape instead
(<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its
kg_doc_entities rows reference) and delete edges that are tagged with the
extractor or touch those nodes. The new test fails before the change.
* fix(release): build Arch packages with the ONNX plugin
scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no
longer enables ONNX, so both Arch release targets would have shipped
without the ONNX embedding provider and Glint. Pass --with-onnx like the
other release lanes and fail packaging if the plugin is missing from the
stage, since the Arch lane skips the ONNX smoke test.
* fix(search): keep topology cluster edges out of relational graph richness
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
* fix(topology): keep rebuilding when the latest snapshot is undecodable
storeBatch loads the previous snapshot only to find documents that left
the topology, but returned early when it could not decode it, so one
corrupt payload blocked every future topology write. On a decode error,
log and store the new snapshot without clearing dropped documents' keys;
other load errors still propagate.
* fix(graph): make isolated-node queries usable after the symbol graph removal
Isolated mode lost its function/calls default in v0.20 and instead
required a node type, but MCP only sets the node type through list_type,
which also switched the request to list-by-type (checked first), so
isolated:true either errored or silently listed nodes.
Default isolated mode to document nodes (with the existing
semantic_neighbor relation default: documents no other document lists as
an embedding neighbour), keep MCP list_type from overriding isolated mode,
and update the tool schema examples that still named calls edges.
* fix(search): keep operator queries valid FTS5 instead of passing them through
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
* fix(common): make the lenient UTF-8 sanitizers emit well-formed UTF-8
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
* fix(extraction): tighten UTF-8 detection and keep units after bad surrogates
EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes,
surrogates and overlong sequences as UTF-8, so such files skipped
conversion and reached the index as invalid UTF-8. It now uses the shared
well-formedness check.
convertToUtf8 consumed the unit after an unpaired high surrogate before
checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it
is a low surrogate.
* fix(extraction): keep HTML script/style and tag scans linear
removeScriptAndStyle re-searched <script, <style and <!-- from the cursor
on every iteration and, for an unclosed opener, scanned to the end for
its closing tag once per occurrence; convertBlockTagsToNewlines rescanned
to the end for '>' at every '<'. An ingested ~1 MB file of unclosed
'<script' openers, empty comments or bare '<' took over a minute.
Cache each next-occurrence position until the cursor passes it and
remember when a closing token is absent from the rest of the document.
The same three inputs now finish in well under a second.
* fix(compression): check stored zstd sizes against the frame before allocating
decompress() trusted a caller-provided expected size (from the stored
block header, which has no checksum of its own) and allocated that many
bytes, up to 16 GB, before decoding. A corrupt size field could therefore
drive a huge allocation.
When the frame records its content size, the expected size must match
it. When it does not (streaming compressor output), decode incrementally
so the buffer grows with the real output and fail once it passes the
expected size.
* fix(setup): find the Conan toolchain for Fuzzing builds
./setup.sh Fuzzing runs Conan with the Debug build type, which writes its
toolchain under build/fuzzing/build-debug, but setup only looked under
build-fuzzing and failed with 'Conan meson toolchain file not found'.
This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug
as the Profiling mode already does.
* fix(search): make grouped FTS5 operator queries explicit (found by fuzzing)
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
* fix(build): link libFuzzer builds on Linux
With -Dbuild-fuzzers=true every object carries ASan/UBSan and
SanitizerCoverage, whose symbols only the fuzzer executable provides, but
Meson's default b_lundef=true links shared libraries such as
libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to
link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for
Fuzzing builds and warn when a libFuzzer configuration keeps it on.
* test(fuzz): harnesses for search queries, UTF-8, text extraction, compression
- fuzz_fts5_query: sanitized MCP/CLI search text must execute as MATCH on
an FTS5 table shaped like documents_fts, also after re-sanitizing.
- fuzz_utf8: sanitizer output is well-formed and preserves valid input; a
UTF-8 verdict implies well-formed bytes; UTF-16/Latin-1 conversions
produce well-formed UTF-8.
- fuzz_text_extractors: HTML/plain-text extraction, title and section
detection on arbitrary bytes, with an output-size bound; run with
-timeout to catch super-linear scans.
- fuzz_compressed_block: stored blocks decode within the header's size and
-rss_limit_mb; zstd/LZMA round-trip exactly.
Seeds in generate_corpus.sh; the Forgejo fuzz-smoke lane builds and runs
them and triggers on the parsers' sources. Five minutes per harness on a
Linux x86 host after the fixes in this series: no findings.
* fix(search): strip matched XML-style tag pairs in linear time
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
* fix(repair): refuse embedding repair without a resolved data directory
repairMissingEmbeddings built its database path as dataPath/'vectors.db'
with create_if_missing, so an unresolved (empty) data directory created a
stray vectors.db in the process working directory. The dispatcher passes
an empty path when the service manager has not resolved its data dir.
Return an error instead.
Tests that exercised repair or placed plugins under the unresolved data
directory now give the service manager a temp dir; the dispatcher suite
no longer leaves vectors.db or dispatcher_external_* in the working
directory.
* refactor(compression): remove the never-built LZMA compressor
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.
The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.
Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
* ci(mirror): expect the git.trevon.dev Forgejo SSH host
ssh.git.trevon.dev no longer resolves (the experimental mirror run failed
with 'Could not resolve hostname'), and main already switched the mirror
remote to git.trevon.dev, where the mirror push succeeds. The policy test
still required the old host and forbade the new one, failing Tests on
every platform for main. Require git.trevon.dev, forbid the dead name,
and update the mirroring doc.
---------
Co-authored-by: trvon <me@trevon.dev>
Co-authored-by: Trevon <git@trevon.dev>
ssh.git.trevon.dev no longer resolves (the experimental mirror run failed
with 'Could not resolve hostname'), and main already switched the mirror
remote to git.trevon.dev, where the mirror push succeeds. The policy test
still required the old host and forbade the new one, failing Tests on
every platform for main. Require git.trevon.dev, forbid the dead name,
and update the mirroring doc.
Bring the #188 promotion merge commit onto experimental so the next
release candidate contains main's complete history (the release
candidate validator rejects it otherwise). No content changes.
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.
The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.
Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
repairMissingEmbeddings built its database path as dataPath/'vectors.db'
with create_if_missing, so an unresolved (empty) data directory created a
stray vectors.db in the process working directory. The dispatcher passes
an empty path when the service manager has not resolved its data dir.
Return an error instead.
Tests that exercised repair or placed plugins under the unresolved data
directory now give the service manager a temp dir; the dispatcher suite
no longer leaves vectors.db or dispatcher_external_* in the working
directory.
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
- fuzz_fts5_query: sanitized MCP/CLI search text must execute as MATCH on
an FTS5 table shaped like documents_fts, also after re-sanitizing.
- fuzz_utf8: sanitizer output is well-formed and preserves valid input; a
UTF-8 verdict implies well-formed bytes; UTF-16/Latin-1 conversions
produce well-formed UTF-8.
- fuzz_text_extractors: HTML/plain-text extraction, title and section
detection on arbitrary bytes, with an output-size bound; run with
-timeout to catch super-linear scans.
- fuzz_compressed_block: stored blocks decode within the header's size and
-rss_limit_mb; zstd/LZMA round-trip exactly.
Seeds in generate_corpus.sh; the Forgejo fuzz-smoke lane builds and runs
them and triggers on the parsers' sources. Five minutes per harness on a
Linux x86 host after the fixes in this series: no findings.
With -Dbuild-fuzzers=true every object carries ASan/UBSan and
SanitizerCoverage, whose symbols only the fuzzer executable provides, but
Meson's default b_lundef=true links shared libraries such as
libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to
link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for
Fuzzing builds and warn when a libFuzzer configuration keeps it on.
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
./setup.sh Fuzzing runs Conan with the Debug build type, which writes its
toolchain under build/fuzzing/build-debug, but setup only looked under
build-fuzzing and failed with 'Conan meson toolchain file not found'.
This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug
as the Profiling mode already does.
decompress() trusted a caller-provided expected size (from the stored
block header, which has no checksum of its own) and allocated that many
bytes, up to 16 GB, before decoding. A corrupt size field could therefore
drive a huge allocation.
When the frame records its content size, the expected size must match
it. When it does not (streaming compressor output), decode incrementally
so the buffer grows with the real output and fail once it passes the
expected size.
removeScriptAndStyle re-searched <script, <style and <!-- from the cursor
on every iteration and, for an unclosed opener, scanned to the end for
its closing tag once per occurrence; convertBlockTagsToNewlines rescanned
to the end for '>' at every '<'. An ingested ~1 MB file of unclosed
'<script' openers, empty comments or bare '<' took over a minute.
Cache each next-occurrence position until the cursor passes it and
remember when a closing token is absent from the rest of the document.
The same three inputs now finish in well under a second.
EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes,
surrogates and overlong sequences as UTF-8, so such files skipped
conversion and reached the index as invalid UTF-8. It now uses the shared
well-formedness check.
convertToUtf8 consumed the unit after an unpaired high surrogate before
checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it
is a low surrogate.
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
Isolated mode lost its function/calls default in v0.20 and instead
required a node type, but MCP only sets the node type through list_type,
which also switched the request to list-by-type (checked first), so
isolated:true either errored or silently listed nodes.
Default isolated mode to document nodes (with the existing
semantic_neighbor relation default: documents no other document lists as
an embedding neighbour), keep MCP list_type from overriding isolated mode,
and update the tool schema examples that still named calls edges.
storeBatch loads the previous snapshot only to find documents that left
the topology, but returned early when it could not decode it, so one
corrupt payload blocked every future topology write. On a decode error,
log and store the new snapshot without clearing dropped documents' keys;
other load errors still propagate.
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no
longer enables ONNX, so both Arch release targets would have shipped
without the ONNX embedding provider and Glint. Pass --with-onnx like the
other release lanes and fail packaging if the plugin is missing from the
stage, since the Arch lane skips the ONNX smoke test.
