Migration 40 deleted every kg_edges row whose relation was calls,
implements, includes, ... and every node of a symbol type such as class or
interface. Graph ingest through MCP and IPC accepts any type and relation,
so an upgrade from v0.19 irreversibly removed user-modelled data like a
'class' node or an 'implements' edge.
Select the extractor's nodes by provenance and key shape instead
(<kind>:<qualified>@<file>[@snap:<hash>], symbol_ref:<name>, or nodes its
kg_doc_entities rows reference) and delete edges that are tagged with the
extractor or touch those nodes. The new test fails before the change.
A stacked PR (#187) merged into its parent branch after the parent (#186)
had already merged into experimental, so its commits missed experimental
and the promotion PR until #189 carried them over. Record the merge order,
a git cherry check for stranded commits, and the no-squash rule for
submodule PRs.
0fec31c: sqlite3_mprintf error paths used %zu, which SQLite parses as a
string conversion and crashed on invalid vec0 dimensions or corrupt HNSW
persistence; also corrects a RaBitQ test expectation.
- config_resolver: several [tuning] defaults derive from core count (e.g.
onnx_max_concurrent is 7 on a 48-thread host), so a reload is now checked
against the values observed before the override instead of != sentinel.
- service_manager: write the P2P trust files with explicit permissions so
a umask of 002 (Debian user-private groups) no longer trips the
group-writable check before the symbolic-link check under test.
~VerificationThreadPool set stop without holding queueMutex, so a worker
that had evaluated the wait predicate but not yet blocked could miss
notify_all() and join() hung forever (integrity_catch2 timed out under
parallel load; gdb showed join() waiting on workers parked in wait()).
Set the flag under the mutex. The new shutdown churn test hung in 35/40
parallel runs before the fix and passes 200/200 full-suite runs after.
Default builds no longer pull onnxruntime: conan enable_onnx and the conan
profiles default to False, meson enable-onnx defaults to disabled and skips
onnxruntime discovery entirely, and setup.sh/setup.ps1 only enable ONNX
(and the ONNX-backed Glint plugin) with --with-onnx / -WithOnnx.
YAMS_DISABLE_ONNX=true still forces it off.
This removes a build blocker on hosts where conan tries to build
onnxruntime from source (e.g. ROCm/MIGraphX on Linux). Release, Docker, and
Arch packaging pass the flag explicitly, so shipped artifacts are unchanged
until ONNX is dropped.
Bump simeon to 38a0b39 (active_simd_tier() checks CPUID; pq.cpp kernels
follow it) and sqlite-vec-cpp to d4ce0cc (target-attribute AVX2+FMA float
distance kernels behind a CPUID check).
x86 builds now compile simeon's arch/avx2.cpp alone with -mavx2 -mfma and
define SIMEON_HAS_AVX2; every other TU keeps the baseline ISA, so generic
x86-64 binaries stay portable while AVX2 hosts use SIMD for encoding, PQ
training, and sqlite-vec distances (3-4x per distance call at 384d on a
Zen 2 host). Full unit suite on Linux x86 (clang, Debug): no new failures.
Bump simeon to c846635:
- fix(tokenizer): locale-independent character classes
- perf(pq): batched/gathered inner-product scans, bit-identical
- perf(projection): Achlioptas apply visits only non-zero sketch buckets
(5.6x on short texts at 4096->384, bit-identical)
The PQ search scan now calls inner_product_batch/_gather instead of one
pimpl call per code. Scores are bit-identical, so ranking and tie-breaks
are unchanged (1.6-2.0x lower ADC scan time in simeon's pq_query_bench).
./setup.sh Profiling configured Meson as buildtype=debug (-O0) and turned
TSAN on by default, so Tracy zone timings described unoptimized,
instrumented code. Use debugoptimized and leave TSAN off unless --tsan is
passed.
static_cosine_ann_index.cpp instantiates sqlite-vec's header-only HNSW and
inner_product_distance templates but was compiled in yams_vector without
SQLITE_VEC_ENABLE_NEON, while the sqlite-vec library and backend TU were
compiled with it. The inline kernels therefore had two bodies (an ODR
violation): the ANN TU's copy was a serial scalar fadd chain, and the
linker could keep either. Build it in yams_sqlite_vec_backend, which
already carries the feature defines, and fail the build on AArch64 if the
defines are missing.
cross_rerank_simeon_outer_maxsim read SearchEngineConfig, but the daemon
resolves the Simeon scoring mode from reranker policy (ConfigResolver), so
the stat said 0 for every outer-MaxSim run and the scifact ablation could
not tell which scorer ran. setCrossReranker() now takes the mode label the
wiring resolved; the engine reports it as cross_rerank_scoring_mode and
derives the outer-MaxSim stat from it.
Graph rerank recomputes connected components over mutual
semantic_neighbor edges among the candidates on every query, reading
each candidate's edges from the KG. The resident topology snapshot
already holds corpus-wide clusters built from those edges.
Add search.topology.graph_community_source (reciprocal_edges |
topology_snapshot). topology_snapshot scores candidates that share a
snapshot cluster with the same normalization and falls back to the edge
walk when no snapshot is loaded; graph_community_source in debug stats
reports which ran. The default stays reciprocal_edges until a benchmark
gate shows the snapshot source is at least as good. The retrieval bench
can set it for ablations.
Topology wrote back only an opaque compressed snapshot node, so yams
graph, explore and graph expansion could not traverse clusters, and
every rebuild added another full snapshot node that was never removed.
Each stored snapshot now replaces the cluster graph in one KG write
batch before the latest pointer flips: topology:cluster:<id> nodes with
size, level, persistence, cohesion and density; document member_of
edges weighted by persistence, medoid_of edges, overlaps edges for
boundary-spill memberships, and subcluster_of hierarchy edges. After the
pointer flips, snapshot nodes beyond the newest three are removed; the
one the pointer names is always kept.
Every kNN edge was also written in reverse, so each semantic_neighbor
edge looked reciprocal and topology's reciprocalOnly filtering, SGC
smoothing over reciprocal edges, and graph reciprocity metrics filtered
nothing. Write only the edge from a document to each of its k nearest
neighbours; a reverse edge now exists exactly when the pair is mutual.
Semantic maintenance pruned one-way edges on the assumption that edges
were symmetric; with directed kNN edges that would erase every
non-mutual neighbour, so it now removes only self-loops (and no longer
reports a fabricated community size for the scoped pass).
Edge reads truncated by insertion order: getEdgesBidirectional sorted by
id and getEdgesTo had no order, so a LIMIT kept arbitrary or oldest
edges. Both now return strongest first, and the topology extractor reads
a document's own neighbours separately from its incoming edges so a
hub's many incoming edges cannot evict them.
Existing databases keep their mirrored edges until the semantic
neighbour graph is rebuilt.
kgEdgeDensity counted every kg_edges row, including the ~16
semantic_neighbor edges per document that 8-NN embedding neighbours
write in both directions, plus file-lineage and entity-attachment edges.
Any embedded corpus therefore looked like a rich knowledge graph, so the
SCIENTIFIC-profile gate that should zero the KG leg on flat prose never
fired and graph richness scaled from embedding edges.
Add kgRelationalEdgeCount/Density, which exclude embedding kNN, file
lineage and layout, text segments, and entity-to-document attachments,
and use it for hasRichGraphTopology() and the tuner's richness scaling.
kgEdgeCount keeps its meaning as the reconciled-graph signal. The live
overlay carries the reconciled relational count forward.
The retired symbol stage never activates, so tuning.post_ingest.
symbol_concurrent and YAMS_POST_SYMBOL_CONCURRENT had no effect, and
status reported symbol_* queue metrics and post_symbol_limit that were
always zero. Remove the config key and environment override, the status
keys and CLI rows, the metrics snapshot fields, and the event-bus symbol
counters. The internal stage slot stays so budget and limiter indices
are unchanged; benchmark JSON keeps its symbol fields at zero.
Since v0.20 the graph stage only records knowledge-graph completion
(NL entities are written by the post-ingest title+NL stage), yet it
still read every supported document from the content store and copied
it into a job that ignored it, and logged each one at info level. Queue
the completion job without content.
The isolated-node graph query now defaults to the semantic_neighbor
relation instead of the removed calls edges, so it reports documents
with no embedding neighbours.
SymbolEnricher boosted results by looking up code-symbol nodes that no
longer exist since v0.20, so it ran KG queries for the top results and
never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the
symbol_rank_* search stats, and the symbolRank option on the service,
retrieval, MCP and CLI request types. The IPC field stays in the wire
format but is ignored, and --symbol-rank/--no-symbol-rank remain
accepted as hidden no-ops so existing scripts keep working.
Fuzzing belongs on our own compute rather than GitHub-hosted runners.
Move the smoke job to .forgejo/workflows on the self-hosted label, pin
actions to code.forgejo.org, and add a preflight that fails clearly when
the runner lacks clang's libFuzzer runtime or the build tools.
Note that once .forgejo/workflows exists, Forgejo runs only those
workflows and stops executing the .github/workflows copies.
Build the libFuzzer targets with clang through setup.sh Fuzzing and run
each for 60 seconds over the generated seeds on pull requests touching
the P2P, codec or fuzzing sources, nightly, and on demand. Reproducers
are uploaded when a harness fails.
New libFuzzer/AFL++ harnesses, each trapping on an oracle violation:
p2p_json (nesting bound, round trip), p2p_connstr (endpoint round trip,
pin idempotence, no silent re-pin), p2p_handshake (validators and the
full acceptor bind the TLS peer), p2p_delta_stream (batch, session and
bootstrap receive paths stay inside count and byte budgets), p2p_inbound
(the manager inbound session through registry update) and
topology_codec (decoded snapshots re-encode losslessly). p2p_delta now
draws its exchange limits from the input.
p2p_seeds.py writes framed-session seeds for FuzzedDataProvider layouts,
dicts/p2p.dict carries the frame vocabulary, generate_corpus.sh accepts a
local seedgen, and fuzz.sh no longer fails on macOS bash 3.2 when a
target has no dictionary. The generated corpus stays untracked.
Found by fuzz_p2p_connstr. Hosts containing brackets, spaces, control
or non-ASCII bytes were accepted and could not be rendered back through
endpoint(). Accept only hostname/IPv4 characters for plain hosts and
hex, ':', '.' plus an optional %zone inside brackets.
Found by fuzz_p2p_handshake. A pre-authentication hello whose integer
field exceeded uint64 was stored by the JSON parser as a double and then
read with get<std::uint32_t>(), an undefined float-to-integer
conversion. A value just above 2^32 was also silently truncated, so
protocol 2^32+4 passed the protocol-4 version check, and negative values
wrapped to huge sizes.
Reject any frame containing a float or a negative number at the frame
boundary (the wire carries none), which also keeps the memory-sync
record deserializers defined, and read p2p numeric fields through a
range-checked unsignedField<T>.
The handshake, delta-exchange and inbound-session state machines read
frames straight from the TLS P2pConnection, so none of the multi-frame
parsing or payload accounting could be exercised without a socket.
Introduce a private FrameSource seam with a production adapter over
P2pConnection and an in-memory BufferFrameSource, pass TLS-derived
identity values explicitly, and expose the validators and receive paths
in p2p_fuzz.h. Behaviour is unchanged; the transport, protocol, delta,
registry and manager suites pass.
The fuzzing subdirectory skipped itself unless the compiler was AFL++,
so fuzzers could only be built inside the Docker image and no quick
local or CI run was possible. With -Dbuild-fuzzers=true a clang that
links the libFuzzer runtime now selects libFuzzer: project objects get
-fsanitize=fuzzer-no-link plus ASan and UBSan unless b_sanitize is set.
AFL++ builds keep their previous flags.
fuzz.sh gains a 'local' command that runs a libFuzzer target over its
seed corpus and passes a dictionary when tools/fuzzing/dicts has one.
README, AGENTS.md, the yams skill guide and the engineering prompt
still promised symbol extraction, calls/defined_in edges, and graph
--impact for callers. Point callers and blast radius at yams grep and
describe the relations the graph still holds.
Graph explore looked up entry points in symbol_metadata, which no longer
exists. Resolve them from node labels, aliases, and ingest path nodes
instead, gather a wider candidate pool, and rank it by query-term
coverage per file before applying the symbol budget so broad early
terms cannot crowd out the file that answers the query.
Impact and affected-test analysis walked code-symbol edges that are no
longer produced; they now return a warning saying the symbol graph was
removed in v0.20 instead of silently empty results.
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.
Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.
Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.
BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
Report topology_route_bq_used / topology_route_bq_candidates in search
debug stats so benchmarks can tell BQ-shortlisted routing apart from
ANN and exhaustive routing. Let retrieval_quality_bench write
bq_candidate_limit, bq_prefix_dim, sgc_hops and
simeon_rerank_outer_maxsim into its per-run config from YAMS_BENCH_*
harness settings, read through getenv_optional.
src/topology and src/vector were compiled without TRACY_ENABLE even in
-Denable-profiling builds, so their YAMS_* zones were no-ops. Mirror the
src/search setup in both libraries, then add zones for snapshot
encode/decode, store and load, SGC smoothing, route-index build and
routing, BQ search, the snapshot cache, and outer MaxSim, plus plots for
snapshot payload bytes, membership count and BQ index bytes.
