getSnapshots, getSnapshotInfo and batchGetSnapshotInfo read only each
document's latest snapshot_id key, so a re-stored document disappeared
from every earlier snapshot's file count, and a snapshot whose documents
were all re-stored vanished from the list.
- getSnapshots lists tree_snapshots, document_snapshots and latest keys.
It deliberately skips legacy snapshot_id:<id> keys, which would be a
full-table scan on large corpora until the backfill finishes.
- batchGetSnapshotInfo counts members from document_snapshots, legacy
keys not yet moved, and latest keys, each looked up by id through an
index; label and git commit come from tree_snapshots first.
- getSnapshotInfo delegates to the batch query, removing three
near-duplicate queries.
Signed-off-by: trvon <me@trevon.dev>
P2pManager::handleInbound discarded runInboundSession's outcome, so a
rejected or failed inbound peer left no log line and no counter; the
InboundSessionStage it computed was never read.
- Count inbound sessions and failures, keep the last failure's stage,
error and time, and log a warning naming the stage.
- Expose them through P2pManager::inboundStats, the memory-sync status
(MemorySyncResponse, protobuf fields 17-21) and 'yams p2p status'
(text and --json).
Signed-off-by: trvon <me@trevon.dev>
'post-ingest success should persist extracted content' had been skipped
unconditionally since c051423e for lack of a deterministic completion
signal. It already waits on the observable result (the document's
extraction status), and post-ingest completion is fenced since #157, so
run it again.
Signed-off-by: trvon <me@trevon.dev>
961a1281 dropped 'tree' from the command registry while removing CLI
read paths that opened the metadata database locally; tree_command.cpp
still built but nothing could reach it. The path-tree statistics it
showed are not exposed through the daemon, so reconnecting it would mean
a new daemon request; remove the file until a path-tree view is added
under 'yams graph'.
Signed-off-by: trvon <me@trevon.dev>
Compressed GetResponse payloads were decoded only by 'yams get' (since
#214) and the MCP server, each with its own copy of the decoder. Any other
client of the daemon (library users, future commands) got the framed
bytes. In-process callers of DocumentService::retrieve defaulted to
acceptCompressed=true, so the daemon's CatRequest handler and the mobile
bindings would also have returned a frame instead of the document.
- DaemonClient::get decodes a compressed response once it is complete and
returns the document; the compression fields stay for display, and a
payload that does not decode is an error. The transfer stays compressed
on the wire.
- RetrieveDocumentRequest::acceptCompressed defaults to false; only
callers whose receiver decodes should opt in.
- Remove the now-redundant decoders from 'yams get' and the MCP get
handler.
- The integration test now requires a compressed request to deliver the
stored document, identical to a plain one.
Signed-off-by: trvon <me@trevon.dev>
Since v0.7.3 'yams get' has requested compressed transfers
(acceptCompressed), and the daemon has answered with a framed payload: a
CompressionHeader followed by a Zstandard body. Only the MCP server
decoded it; 'yams get' wrote the frame itself to stdout or -o, so users
saw the 'CNRK' header and a zstd frame instead of the document. 'yams cat'
was unaffected because it does not request compression.
- Move the frame format into compression/framed_payload
(encodeFramedPayload/decodeFramedPayload). DocumentService encodes with
it; 'yams get' and MCP decode with it.
- The decoder checks the header, body size and CRC, and the decoded size
and CRC. A payload that fails is an error; MCP used to fall back to
returning the framed bytes as content.
Verified end to end: 'yams get' output (stdout and -o) is byte-identical
to the added file while the transfer stays compressed (51.75x).
Signed-off-by: trvon <me@trevon.dev>
Every store added two metadata rows per snapshot to the document,
snapshot_id:<id> and snapshot_time:<id>, and nothing removed them.
Content-addressed documents are shared across paths and passes, so they
piled up: on one 93 GB database they were 43.1M of 44.0M metadata rows,
one file carried 52,367 snapshots, and 'yams list' timed out hydrating
2.69M rows for its 100 documents. Meanwhile findDocumentsBySnapshot read
only the latest snapshot_id, so a re-stored document dropped out of every
earlier snapshot.
- Migration 42 creates document_snapshots(document_id, snapshot_id,
snapshot_time), keyed on (document_id, snapshot_id). It only creates the
table, so startup stays fast.
- The insert transaction records membership; stores keep only the latest
snapshot_id/snapshot_time keys.
- findDocumentsBySnapshot and file history read membership, falling back
to legacy keys through one shared parser (legacySnapshotEntries).
- getMetadataForDocuments skips legacy keys via idx_metadata_doc_key: on
the 93 GB database, 0.21 s instead of 18.5 s for list's 100 documents.
- RepairService moves legacy keys in the background, 1000 snapshots per
batch, 200 ms apart, only below Warning pressure.
Signed-off-by: trvon <me@trevon.dev>
Read and write pools share the slow-holder threshold, but the warning
always said 'slow write-connection hold'. A slow read-only 'yams list'
(tag client_list, 42 s) was reported as a write hold, which pointed the
investigation at a nonexistent write. Say read or write from the pool's
readOnly config.
Signed-off-by: trvon <me@trevon.dev>
A single removal that failed returned before the watcher advanced its
snapshot, so every later pass saw the same files as changed, re-ran
addDirectory on all of them, and retried the removal forever. Removal
used force=false, and a path indexed more than once fails with 'Multiple
documents match', so one such path wedged the watcher permanently. A
live daemon re-indexed 1,793 files every 2 s this way and logged a
warning on each pass.
- Hold back only the paths that fail: a changed file whose indexing
failed stays changed, a removal that failed stays pending, and the
snapshot advances for everything else.
- Remove a deleted path with force: the file is gone, so every document
stored at it is stale.
- Warn once per failing path and retry quietly after that.
- Honor the watched directory's .gitignore (the unused gitignorePatterns
and gitignoreMtime state), and skip .git, so ignored build output is
not tracked, restatted, or removed.
Signed-off-by: trvon <me@trevon.dev>
Without a trusted clean-shutdown stamp, startup runs a full PRAGMA
quick_check before the database is ready. On a 93 GB database that took
over 11 minutes, and status showed 'Database: Opening' while the log
repeated 'still opening database', so a healthy scan looked like a hang.
Add the checking_integrity database phase, set it around the blocking
check, render it as 'Checking integrity ... full scan after unclean stop'
in 'yams daemon status', and name it in the startup watchdog log. The
check itself is unchanged: it still blocks readiness, and still recovers
confirmed corruption.
Signed-off-by: trvon <me@trevon.dev>
Squash-merged promotion (#203) and back-merge (#205) PRs dropped main's
commits from experimental, so Release Candidate rejected the next
candidate. Record which PR kinds need a merge commit, the back-merge
check, the BEGIN_COMMIT_OVERRIDE hazard behind the 9,908-line notes, and
Release-As for pinning a version.
Signed-off-by: trvon <me@trevon.dev>
This reverts commit 296e7320.
296e7320 moved R2 publication from wrangler (authenticated by
CLOUDFLARE_API_TOKEN) to boto3 over R2's S3 API, which needs
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (and EXPERIMENTAL_* pairs). Those
secrets were never configured, so the first stable release built after
the change (v0.20.2, run 36084509789) failed at 'Publish channel package
repositories to R2' with 'Missing channel-specific Cloudflare R2 S3
authority', and the APT/RPM/Arch repositories were not updated.
Restore the wrangler path so releases publish with the existing token.
* fix(downloader): report a user cancel as OperationCancelled
The curl adapter returned PolicyViolation when shouldCancel() stopped a
transfer. DownloadService maps only OperationCancelled to a cancellation,
so a user's cancel reached callers as a NetworkError, and the service's
cancellation branch could never run.
The existing manager-level cancellation test used a fake adapter that
returns OperationCancelled itself, so it never exercised this path. The
new test drives the real curl adapter over a file:// URL, which runs its
write callback without a network, and fails before this change.
Found by pipe's experimental C++ front end: downloader::ErrorCode::
OperationCancelled was compared (download_service.cpp) but never produced.
* fix(downloader): emit the Resolving and Verifying progress stages
ProgressStage declares Resolving and Verifying, and the init, model and
download commands render both, but DownloadManager never emitted either:
the server probe and the integrity finalize / checksum check ran with no
progress event, so those CLI branches were unreachable and a long hash of
a large file looked like a stall.
The manager now reports Resolving before the probe and Verifying before
the digest is finalized. A new test records every stage the manager
emits for a checksummed download and requires Resolving < Connecting <
Verifying < Finalizing; it fails before this change.
Found by pipe's experimental C++ front end: both stages were matched in
CLI switches but never produced anywhere in src/.
* feat(topology): select the dirty-region expansion mode from config
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
* feat(search): apply lines: qualifiers to result snippets
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
* fix(build): map aarch64/arm64 to Conan's armv8 arch setting
build-arch-pkg.sh and the Arch PKGBUILD export YAMS_CONAN_ARCH from the
pacman/uname label (aarch64), which Conan rejects, failing the Arch
aarch64 release lane at conan install.
* fix(release): let clang find Arch ARM's GCC for Boost's arm64 target
Boost's b2 clang toolset compiles with --target=arm64-pc-linux on aarch64.
Clang cannot map that triple to Arch Linux ARM's aarch64-unknown-linux-gnu
GCC install, so every Boost configure check and compile lost libstdc++
headers and crtbeginS.o. Ship an arm64-pc-linux.cfg next to clang that sets
--gcc-triple to the host GCC.
---------
Co-authored-by: trvon <me@trevon.dev>
Co-authored-by: Trevon <git@trevon.dev>
Boost's b2 clang toolset compiles with --target=arm64-pc-linux on aarch64.
Clang cannot map that triple to Arch Linux ARM's aarch64-unknown-linux-gnu
GCC install, so every Boost configure check and compile lost libstdc++
headers and crtbeginS.o. Ship an arm64-pc-linux.cfg next to clang that sets
--gcc-triple to the host GCC.
build-arch-pkg.sh and the Arch PKGBUILD export YAMS_CONAN_ARCH from the
pacman/uname label (aarch64), which Conan rejects, failing the Arch
aarch64 release lane at conan install.
parseQueryQualifiers parsed lines:, pages:, section: and selector: into
an ExtractScope "for downstream snippet shaping", and SearchService then
used only its name/ext/mime hints: the scope itself was ignored, so
`yams search "foo lines:1-20"` silently searched for "foo" and returned
ordinary snippets.
- lines:<range> now replaces each result's snippet with those lines of
its stored content (1-based, e.g. "1-3,10"), through the same
TextBasicHandler `get` uses. It runs after snippet hydration, so
nothing overwrites it, and only when the qualifier is present.
- A result with no line-range handler (binary formats such as PDF),
content over 1 MiB, or unreadable content keeps its snippet and is
counted, never presented as shaped.
- pages:, section: and selector: have no search-time handler; they are
now reported rather than silently dropped.
- The outcome ("lines:1-20 applied to 7 of 8 results (1 without
line-range text, ...)" or "pages:1-2 is not applied to search
results") goes into searchStats["extract_scope"] and queryInfo, which
the MCP response already carries.
The logic is a header-only seam, applyExtractScope
(<yams/app/services/extract_scope.h>), tested directly for shaping,
unsupported formats, the read cap, an invalid range, pages:, and no
qualifier; plus an end-to-end SearchService test that fails before this
change: every hit for "programming lines:1-1" carries exactly its
document's first line.
Cost: a lines: query reads each result's content once (bounded by the
result limit and the 1 MiB cap); queries without the qualifier are
unchanged.
The portability allowlist entry for search_service.cpp's reviewed
getenv moves from line 509 to 510 with the new include.
Found by pipe's experimental C++ front end: ExtractScopeType values were
produced by the parser but never examined anywhere.
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
ProgressStage declares Resolving and Verifying, and the init, model and
download commands render both, but DownloadManager never emitted either:
the server probe and the integrity finalize / checksum check ran with no
progress event, so those CLI branches were unreachable and a long hash of
a large file looked like a stall.
The manager now reports Resolving before the probe and Verifying before
the digest is finalized. A new test records every stage the manager
emits for a checksummed download and requires Resolving < Connecting <
Verifying < Finalizing; it fails before this change.
Found by pipe's experimental C++ front end: both stages were matched in
CLI switches but never produced anywhere in src/.
The curl adapter returned PolicyViolation when shouldCancel() stopped a
transfer. DownloadService maps only OperationCancelled to a cancellation,
so a user's cancel reached callers as a NetworkError, and the service's
cancellation branch could never run.
The existing manager-level cancellation test used a fake adapter that
returns OperationCancelled itself, so it never exercised this path. The
new test drives the real curl adapter over a file:// URL, which runs its
write callback without a network, and fails before this change.
Found by pipe's experimental C++ front end: downloader::ErrorCode::
OperationCancelled was compared (download_service.cpp) but never produced.
archlinux/archlinux only publishes linux/amd64, so the native aarch64 Arch
lane failed with 'exec format error' before provisioning. Both Arch
Dockerfiles now take ARCH_BASE_IMAGE; the aarch64 matrix entry pins
ghcr.io/menci/archlinuxarm by digest and x86_64 keeps the official image.
package-validate.sh forwards ARCH_BASE_IMAGE to the systemd substrate.
The release Arch job passes --arch explicitly, but the script still ran
deb/rpm discovery against the default build/release dir. find fails on the
missing dir, pipefail + set -e turn that into a silent exit 1. Only discover
packages for selected lanes and tolerate a missing build dir.
* docs(agents): document PR stacking, promotion, and submodule merge rules
A stacked PR (#187) merged into its parent branch after the parent (#186)
had already merged into experimental, so its commits missed experimental
and the promotion PR until #189 carried them over. Record the merge order,
a git cherry check for stranded commits, and the no-squash rule for
submodule PRs.
* fix(metadata): scope migration 40 deletes to symbol extractor output
Migration 40 deleted every kg_edges row whose relation was calls,
implements, includes, ... and every node of a symbol type such as class or
interface. Graph ingest through MCP and IPC accepts any type and relation,
so an upgrade from v0.19 irreversibly removed user-modelled data like a
'class' node or an 'implements' edge.
Select the extractor's nodes by provenance and key shape instead
(<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its
kg_doc_entities rows reference) and delete edges that are tagged with the
extractor or touch those nodes. The new test fails before the change.
* fix(release): build Arch packages with the ONNX plugin
scripts/build-arch-pkg.sh still ran a plain ./setup.sh Release, which no
longer enables ONNX, so both Arch release targets would have shipped
without the ONNX embedding provider and Glint. Pass --with-onnx like the
other release lanes and fail packaging if the plugin is missing from the
stage, since the Arch lane skips the ONNX smoke test.
* fix(search): keep topology cluster edges out of relational graph richness
Topology rebuilds now materialize member_of/overlaps/medoid_of/
subcluster_of edges in the KG. They derive from embeddings, but counted
toward kgRelationalEdgeCount, so building topology alone could push
hasRichGraphTopology() over its threshold and reopen graph-weight gates
that stay closed for corpora without real relations. Exclude them.
* fix(topology): keep rebuilding when the latest snapshot is undecodable
storeBatch loads the previous snapshot only to find documents that left
the topology, but returned early when it could not decode it, so one
corrupt payload blocked every future topology write. On a decode error,
log and store the new snapshot without clearing dropped documents' keys;
other load errors still propagate.
* fix(graph): make isolated-node queries usable after the symbol graph removal
Isolated mode lost its function/calls default in v0.20 and instead
required a node type, but MCP only sets the node type through list_type,
which also switched the request to list-by-type (checked first), so
isolated:true either errored or silently listed nodes.
Default isolated mode to document nodes (with the existing
semantic_neighbor relation default: documents no other document lists as
an embedding neighbour), keep MCP list_type from overriding isolated mode,
and update the tool schema examples that still named calls edges.
* fix(search): keep operator queries valid FTS5 instead of passing them through
Queries containing FTS5 operator syntax (a quote, uppercase AND/OR/NOT,
parentheses, column filters) were passed to MATCH almost verbatim, so
'foo "bar', '(foo OR bar', 'foo AND', 'NOT foo', 'IL-6 OR TNF' or
'ratio:2' failed with an fts5 syntax or no-such-column error instead of
returning results; MCP clients and the CLI reach this path.
Rebuild operator queries token by token: keep phrases, parentheses,
AND/OR/NOT, well-formed NEAR(...) groups and filters on real FTS columns,
quote any other non-bareword term, and fall back to the literal word
query when the structure is invalid. The existing assertion that
'hello NEAR/2 world' passes through encoded a query FTS5 rejects.
* fix(common): make the lenient UTF-8 sanitizers emit well-formed UTF-8
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
* fix(extraction): tighten UTF-8 detection and keep units after bad surrogates
EncodingDetector::detectEncoding accepted C0/C1 and F5-F7 lead bytes,
surrogates and overlong sequences as UTF-8, so such files skipped
conversion and reached the index as invalid UTF-8. It now uses the shared
well-formedness check.
convertToUtf8 consumed the unit after an unpaired high surrogate before
checking it, so UTF-16 'D800 0041' lost the 'A'. Only consume it when it
is a low surrogate.
* fix(extraction): keep HTML script/style and tag scans linear
removeScriptAndStyle re-searched <script, <style and <!-- from the cursor
on every iteration and, for an unclosed opener, scanned to the end for
its closing tag once per occurrence; convertBlockTagsToNewlines rescanned
to the end for '>' at every '<'. An ingested ~1 MB file of unclosed
'<script' openers, empty comments or bare '<' took over a minute.
Cache each next-occurrence position until the cursor passes it and
remember when a closing token is absent from the rest of the document.
The same three inputs now finish in well under a second.
* fix(compression): check stored zstd sizes against the frame before allocating
decompress() trusted a caller-provided expected size (from the stored
block header, which has no checksum of its own) and allocated that many
bytes, up to 16 GB, before decoding. A corrupt size field could therefore
drive a huge allocation.
When the frame records its content size, the expected size must match
it. When it does not (streaming compressor output), decode incrementally
so the buffer grows with the real output and fail once it passes the
expected size.
* fix(setup): find the Conan toolchain for Fuzzing builds
./setup.sh Fuzzing runs Conan with the Debug build type, which writes its
toolchain under build/fuzzing/build-debug, but setup only looked under
build-fuzzing and failed with 'Conan meson toolchain file not found'.
This is the path the Forgejo fuzz-smoke lane uses. Look under build-debug
as the Profiling mode already does.
* fix(search): make grouped FTS5 operator queries explicit (found by fuzzing)
fuzz_fts5_query found two inputs the operator-query normalizer still
rejected at MATCH time:
- FTS5 applies implicit AND only between phrases, so '(foo OR bar) baz',
'bar (foo)' and 'NEAR(a b) c' are syntax errors; insert an explicit AND
next to a parenthesized or NEAR group.
- 'NEAR/<x>(...)' was treated as a NEAR group; FTS5 accepts only 'NEAR('.
* fix(build): link libFuzzer builds on Linux
With -Dbuild-fuzzers=true every object carries ASan/UBSan and
SanitizerCoverage, whose symbols only the fuzzer executable provides, but
Meson's default b_lundef=true links shared libraries such as
libyams_onnx_resource with --no-undefined, so ./setup.sh Fuzzing failed to
link on Linux (the Forgejo fuzz-smoke path). Set -Db_lundef=false for
Fuzzing builds and warn when a libFuzzer configuration keeps it on.
* test(fuzz): harnesses for search queries, UTF-8, text extraction, compression
- fuzz_fts5_query: sanitized MCP/CLI search text must execute as MATCH on
an FTS5 table shaped like documents_fts, also after re-sanitizing.
- fuzz_utf8: sanitizer output is well-formed and preserves valid input; a
UTF-8 verdict implies well-formed bytes; UTF-16/Latin-1 conversions
produce well-formed UTF-8.
- fuzz_text_extractors: HTML/plain-text extraction, title and section
detection on arbitrary bytes, with an output-size bound; run with
-timeout to catch super-linear scans.
- fuzz_compressed_block: stored blocks decode within the header's size and
-rss_limit_mb; zstd/LZMA round-trip exactly.
Seeds in generate_corpus.sh; the Forgejo fuzz-smoke lane builds and runs
them and triggers on the parsers' sources. Five minutes per harness on a
Linux x86 host after the fixes in this series: no findings.
* fix(search): strip matched XML-style tag pairs in linear time
stripXmlTags removed '<name>...</name>' pairs with a lazy backreference
std::regex; every unmatched opener rescanned the rest of the query, so a
32 KB query of '<a>x' took 4.4 s (libstdc++, -O2) and time grew 4x per
doubling. Search text comes from MCP clients and the CLI.
Index closing tags once and find each opener's first closer by binary
search, preserving the regex's left-to-right shortest-match semantics.
The suspected stack overflow did not reproduce (libc++ or libstdc++, down
to 256 KB thread stacks under ASan); the new test guards the time bound.
* fix(repair): refuse embedding repair without a resolved data directory
repairMissingEmbeddings built its database path as dataPath/'vectors.db'
with create_if_missing, so an unresolved (empty) data directory created a
stray vectors.db in the process working directory. The dispatcher passes
an empty path when the service manager has not resolved its data dir.
Return an error instead.
Tests that exercised repair or placed plugins under the unresolved data
directory now give the service manager a temp dir; the dispatcher suite
no longer leaves vectors.db or dispatcher_external_* in the working
directory.
* refactor(compression): remove the never-built LZMA compressor
lzma_compressor.cpp targets the 7-Zip LZMA SDK (LzmaEnc.h/Alloc.h), but
the build only finds xz/liblzma, which does not ship those headers, so
every build (release logs: 'Has header "LzmaEnc.h" ... NO') compiled it
out. No build could read or write LZMA blocks, and its decoder never
bounded output by the expected size.
The policy still chose LZMA for cold/archival and large inactive data;
compression then failed ('Compressor not available') and batch stores
kept those blocks uncompressed. Select Zstandard, which already uses the
archive level for old data.
Remove the compressor, its test, the enable-lzma option and the build
flags that passed it. The LZMA algorithm id stays in the header format,
and compression.lzma_level remains accepted in config (now unused).
* ci(mirror): expect the git.trevon.dev Forgejo SSH host
ssh.git.trevon.dev no longer resolves (the experimental mirror run failed
with 'Could not resolve hostname'), and main already switched the mirror
remote to git.trevon.dev, where the mirror push succeeds. The policy test
still required the old host and forbade the new one, failing Tests on
every platform for main. Require git.trevon.dev, forbid the dead name,
and update the mirroring doc.
---------
Co-authored-by: trvon <me@trevon.dev>
Co-authored-by: Trevon <git@trevon.dev>
ssh.git.trevon.dev no longer resolves (the experimental mirror run failed
with 'Could not resolve hostname'), and main already switched the mirror
remote to git.trevon.dev, where the mirror push succeeds. The policy test
still required the old host and forbade the new one, failing Tests on
every platform for main. Require git.trevon.dev, forbid the dead name,
and update the mirroring doc.
