Bring the #188 promotion merge commit onto experimental so the next
release candidate contains main's complete history (the release
candidate validator rejects it otherwise). No content changes.
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.
The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.
Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
repairMissingEmbeddings built its database path as dataPath/'vectors.db'
with create_if_missing, so an unresolved (empty) data directory created a
stray vectors.db in the process working directory. The dispatcher passes
an empty path when the service manager has not resolved its data dir.
Return an error instead.
Tests that exercised repair or placed plugins under the unresolved data
directory now give the service manager a temp dir; the dispatcher suite
no longer leaves vectors.db or dispatcher_external_* in the working
directory.
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
- fuzz_fts5_query: sanitized MCP/CLI search text must execute as MATCH on
an FTS5 table shaped like documents_fts, also after re-sanitizing.
- fuzz_utf8: sanitizer output is well-formed and preserves valid input; a
UTF-8 verdict implies well-formed bytes; UTF-16/Latin-1 conversions
produce well-formed UTF-8.
- fuzz_text_extractors: HTML/plain-text extraction, title and section
detection on arbitrary bytes, with an output-size bound; run with
-timeout to catch super-linear scans.
- fuzz_compressed_block: stored blocks decode within the header's size and
-rss_limit_mb; zstd/LZMA round-trip exactly.
Seeds in generate_corpus.sh; the Forgejo fuzz-smoke lane builds and runs
them and triggers on the parsers' sources. Five minutes per harness on a
Linux x86 host after the fixes in this series: no findings.
With -Dbuild-fuzzers=true every object carries ASan/UBSan and
SanitizerCoverage, whose symbols only the fuzzer executable provides, but
Meson's default b_lundef=true links shared libraries such as
libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to
link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for
Fuzzing builds and warn when a libFuzzer configuration keeps it on.
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
./setup.sh Fuzzing runs Conan with the Debug build type, which writes its
toolchain under build/fuzzing/build-debug, but setup only looked under
build-fuzzing and failed with 'Conan meson toolchain file not found'.
This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug
as the Profiling mode already does.
decompress() trusted a caller-provided expected size (from the stored
block header, which has no checksum of its own) and allocated that many
bytes, up to 16 GB, before decoding. A corrupt size field could therefore
drive a huge allocation.
When the frame records its content size, the expected size must match
it. When it does not (streaming compressor output), decode incrementally
so the buffer grows with the real output and fail once it passes the
expected size.
removeScriptAndStyle re-searched <script, <style and <!-- from the cursor
on every iteration and, for an unclosed opener, scanned to the end for
its closing tag once per occurrence; convertBlockTagsToNewlines rescanned
to the end for '>' at every '<'. An ingested ~1 MB file of unclosed
'<script' openers, empty comments or bare '<' took over a minute.
Cache each next-occurrence position until the cursor passes it and
remember when a closing token is absent from the rest of the document.
The same three inputs now finish in well under a second.
EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes,
surrogates and overlong sequences as UTF-8, so such files skipped
conversion and reached the index as invalid UTF-8. It now uses the shared
well-formedness check.
convertToUtf8 consumed the unit after an unpaired high surrogate before
checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it
is a low surrogate.
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
Isolated mode lost its function/calls default in v0.20 and instead
required a node type, but MCP only sets the node type through list_type,
which also switched the request to list-by-type (checked first), so
isolated:true either errored or silently listed nodes.
Default isolated mode to document nodes (with the existing
semantic_neighbor relation default: documents no other document lists as
an embedding neighbour), keep MCP list_type from overriding isolated mode,
and update the tool schema examples that still named calls edges.
storeBatch loads the previous snapshot only to find documents that left
the topology, but returned early when it could not decode it, so one
corrupt payload blocked every future topology write. On a decode error,
log and store the new snapshot without clearing dropped documents' keys;
other load errors still propagate.
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no
longer enables ONNX, so both Arch release targets would have shipped
without the ONNX embedding provider and Glint. Pass --with-onnx like the
other release lanes and fail packaging if the plugin is missing from the
stage, since the Arch lane skips the ONNX smoke test.
Migration 40 deleted every kg_edges row whose relation was calls,
implements, includes, ... and every node of a symbol type such as class or
interface. Graph ingest through MCP and IPC accepts any type and relation,
so an upgrade from v0.19 irreversibly removed user-modelled data like a
'class' node or an 'implements' edge.
Select the extractor's nodes by provenance and key shape instead
(<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its
kg_doc_entities rows reference) and delete edges that are tagged with the
extractor or touch those nodes. The new test fails before the change.
A stacked PR (#187) merged into its parent branch after the parent (#186)
had already merged into experimental, so its commits missed experimental
and the promotion PR until #189 carried them over. Record the merge order,
a git cherry check for stranded commits, and the no-squash rule for
submodule PRs.
0fec31c: sqlite3_mprintf error paths used %zu, which SQLite parses as a
string conversion and crashed on invalid vec0 dimensions or corrupt HNSW
persistence; also corrects a RaBitQ test expectation.
- config_resolver: several [tuning] defaults derive from core count (e.g.
onnx_max_concurrent is 7 on a 48-thread host), so a reload is now checked
against the values observed before the override instead of != sentinel.
- service_manager: write the P2P trust files with explicit permissions so
a umask of 002 (Debian user-private groups) no longer trips the
group-writable check before the symbolic-link check under test.
~VerificationThreadPool set stop without holding queueMutex, so a worker
that had evaluated the wait predicate but not yet blocked could miss
notify_all() and join() hung forever (integrity_catch2 timed out under
parallel load; gdb showed join() waiting on workers parked in wait()).
Set the flag under the mutex. The new shutdown churn test hung in 35/40
parallel runs before the fix and passes 200/200 full-suite runs after.
Default builds no longer pull onnxruntime: conan enable_onnx and the conan
profiles default to False, meson enable-onnx defaults to disabled and skips
onnxruntime discovery entirely, and setup.sh/setup.ps1 only enable ONNX
(and the ONNX-backed Glint plugin) with --with-onnx / -WithOnnx.
YAMS_DISABLE_ONNX=true still forces it off.
This removes a build blocker on hosts where conan tries to build
onnxruntime from source (e.g. ROCm/MIGraphX on Linux). Release, Docker, and
Arch packaging pass the flag explicitly, so shipped artifacts are unchanged
until ONNX is dropped.
Bump simeon to 38a0b39 (active_simd_tier() checks CPUID; pq.cpp kernels
follow it) and sqlite-vec-cpp to d4ce0cc (target-attribute AVX2+FMA float
distance kernels behind a CPUID check).
x86 builds now compile simeon's arch/avx2.cpp alone with -mavx2 -mfma and
define SIMEON_HAS_AVX2; every other TU keeps the baseline ISA, so generic
x86-64 binaries stay portable while AVX2 hosts use SIMD for encoding, PQ
training, and sqlite-vec distances (3-4x per distance call at 384d on a
Zen 2 host). Full unit suite on Linux x86 (clang, Debug): no new failures.
Bump simeon to c846635:
- fix(tokenizer): locale-independent character classes
- perf(pq): batched/gathered inner-product scans, bit-identical
- perf(projection): Achlioptas apply visits only non-zero sketch buckets
(5.6x on short texts at 4096->384, bit-identical)
The PQ search scan now calls inner_product_batch/_gather instead of one
pimpl call per code. Scores are bit-identical, so ranking and tie-breaks
are unchanged (1.6-2.0x lower ADC scan time in simeon's pq_query_bench).
./setup.sh Profiling configured Meson as buildtype=debug (-O0) and turned
TSAN on by default, so Tracy zone timings described unoptimized,
instrumented code. Use debugoptimized and leave TSAN off unless --tsan is
passed.
static_cosine_ann_index.cpp instantiates sqlite-vec's header-only HNSW and
inner_product_distance templates but was compiled in yams_vector without
SQLITE_VEC_ENABLE_NEON, while the sqlite-vec library and backend TU were
compiled with it. The inline kernels therefore had two bodies (an ODR
violation): the ANN TU's copy was a serial scalar fadd chain, and the
linker could keep either. Build it in yams_sqlite_vec_backend, which
already carries the feature defines, and fail the build on AArch64 if the
defines are missing.
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon
resolves the Simeon scoring mode from reranker policy (ConfigResolver), so
the stat said 0 for every outer-MaxSim run and the scifact ablation could
not tell which scorer ran. setCrossReranker() now takes the mode label the
wiring resolved; the engine reports it as cross_rerank_scoring_mode and
derives the outer-MaxSim stat from it.
Graph rerank recomputes connected components over mutual
semantic_neighbor edges among the candidates on every query, reading
each candidate's edges from the KG. The resident topology snapshot
already holds corpus-wide clusters built from those edges.
Add search.topology.graph_community_source (reciprocal_edges |
topology_snapshot). topology_snapshot scores candidates that share a
snapshot cluster with the same normalization and falls back to the edge
walk when no snapshot is loaded; graph_community_source in debug stats
reports which ran. The default stays reciprocal_edges until a benchmark
gate shows the snapshot source is at least as good. The retrieval bench
can set it for ablations.
Topology wrote back only an opaque compressed snapshot node, so yams
graph, explore and graph expansion could not traverse clusters, and
every rebuild added another full snapshot node that was never removed.
Each stored snapshot now replaces the cluster graph in one KG write
batch before the latest pointer flips: topology:cluster:<id> nodes with
size, level, persistence, cohesion and density; document member_of
edges weighted by persistence, medoid_of edges, overlaps edges for
boundary-spill memberships, and subcluster_of hierarchy edges. After the
pointer flips, snapshot nodes beyond the newest three are removed; the
one the pointer names is always kept.
Every kNN edge was also written in reverse, so each semantic_neighbor
edge looked reciprocal and topology's reciprocalOnly filtering, SGC
smoothing over reciprocal edges, and graph reciprocity metrics filtered
nothing. Write only the edge from a document to each of its k nearest
neighbours; a reverse edge now exists exactly when the pair is mutual.
Semantic maintenance pruned one-way edges on the assumption that edges
were symmetric; with directed kNN edges that would erase every
non-mutual neighbour, so it now removes only self-loops (and no longer
reports a fabricated community size for the scoped pass).
Edge reads truncated by insertion order: getEdgesBidirectional sorted by
id and getEdgesTo had no order, so a LIMIT kept arbitrary or oldest
edges. Both now return strongest first, and the topology extractor reads
a document's own neighbours separately from its incoming edges so a
hub's many incoming edges cannot evict them.
Existing databases keep their mirrored edges until the semantic
neighbour graph is rebuilt.
kgEdgeDensity counted every kg_edges row, including the ~16
semantic_neighbor edges per document that 8-NN embedding neighbours
write in both directions, plus file-lineage and entity-attachment edges.
Any embedded corpus therefore looked like a rich knowledge graph, so the
SCIENTIFIC-profile gate that should zero the KG leg on flat prose never
fired and graph richness scaled from embedding edges.
Add kgRelationalEdgeCount/Density, which exclude embedding kNN, file
lineage and layout, text segments, and entity-to-document attachments,
and use it for hasRichGraphTopology() and the tuner's richness scaling.
kgEdgeCount keeps its meaning as the reconciled-graph signal. The live
overlay carries the reconciled relational count forward.
