1,824 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
trvon
|
c766a17bd1
|
Merge branch 'refactor/remove-symbol-abi' into release/v0.20-integration
# Conflicts: # tests/scripts/portability_allowlist.txt |
||
trvon
|
66689cd7a3
|
refactor(daemon): remove the symbol_concurrent tuning knob and stage metrics
The retired symbol stage never activates, so tuning.post_ingest. symbol_concurrent and YAMS_POST_SYMBOL_CONCURRENT had no effect, and status reported symbol_* queue metrics and post_symbol_limit that were always zero. Remove the config key and environment override, the status keys and CLI rows, the metrics snapshot fields, and the event-bus symbol counters. The internal stage slot stays so budget and limiter indices are unchanged; benchmark JSON keeps its symbol fields at zero. |
||
|
trvon
|
0707331a8d
|
perf(graph): stop loading document content for completion-only jobs
Since v0.20 the graph stage only records knowledge-graph completion (NL entities are written by the post-ingest title+NL stage), yet it still read every supported document from the content store and copied it into a job that ignored it, and logged each one at info level. Queue the completion job without content. The isolated-node graph query now defaults to the semantic_neighbor relation instead of the removed calls edges, so it reports documents with no embedding neighbours. |
||
|
trvon
|
ac070d8d2a
|
refactor(search): remove symbol-based result ranking
SymbolEnricher boosted results by looking up code-symbol nodes that no longer exist since v0.20, so it ran KG queries for the top results and never boosted anything. Remove it, the YAMS_SYMBOL_WEIGHT variable, the symbol_rank_* search stats, and the symbolRank option on the service, retrieval, MCP and CLI request types. The IPC field stays in the wire format but is ignored, and --symbol-rank/--no-symbol-rank remain accepted as hidden no-ops so existing scripts keep working. |
||
|
trvon
|
25d171dd62
|
ci(fuzz): run the fuzz lane on the self-hosted Forgejo runner
Fuzzing belongs on our own compute rather than GitHub-hosted runners. Move the smoke job to .forgejo/workflows on the self-hosted label, pin actions to code.forgejo.org, and add a preflight that fails clearly when the runner lacks clang's libFuzzer runtime or the build tools. Note that once .forgejo/workflows exists, Forgejo runs only those workflows and stops executing the .github/workflows copies. |
||
|
trvon
|
c6e960eed7
|
Merge branch 'test/p2p-fuzz-lane' into release/v0.20-integration | ||
|
trvon
|
e6de1978c2
|
Merge branch 'refactor/remove-symbol-abi' into release/v0.20-integration | ||
|
trvon
|
34b9f7346e
|
docs(p2p): describe the fuzz lane, its oracles and remaining gaps | ||
|
trvon
|
6065798e65
|
ci(fuzz): run the P2P and topology harnesses on relevant changes
Build the libFuzzer targets with clang through setup.sh Fuzzing and run each for 60 seconds over the generated seeds on pull requests touching the P2P, codec or fuzzing sources, nightly, and on demand. Reproducers are uploaded when a harness fails. |
||
|
trvon
|
2a63eb1af8
|
test(fuzz): add a direct-P2P fuzz lane and a topology codec harness
New libFuzzer/AFL++ harnesses, each trapping on an oracle violation: p2p_json (nesting bound, round trip), p2p_connstr (endpoint round trip, pin idempotence, no silent re-pin), p2p_handshake (validators and the full acceptor bind the TLS peer), p2p_delta_stream (batch, session and bootstrap receive paths stay inside count and byte budgets), p2p_inbound (the manager inbound session through registry update) and topology_codec (decoded snapshots re-encode losslessly). p2p_delta now draws its exchange limits from the input. p2p_seeds.py writes framed-session seeds for FuzzedDataProvider layouts, dicts/p2p.dict carries the frame vocabulary, generate_corpus.sh accepts a local seedgen, and fuzz.sh no longer fails on macOS bash 3.2 when a target has no dictionary. The generated corpus stays untracked. |
||
|
trvon
|
b702cb47e1
|
fix(p2p): validate connection-string hosts
Found by fuzz_p2p_connstr. Hosts containing brackets, spaces, control or non-ASCII bytes were accepted and could not be rendered back through endpoint(). Accept only hostname/IPv4 characters for plain hosts and hex, ':', '.' plus an optional %zone inside brackets. |
||
|
trvon
|
de9e3245f8
|
fix(p2p): admit only non-negative integers in control frames
Found by fuzz_p2p_handshake. A pre-authentication hello whose integer field exceeded uint64 was stored by the JSON parser as a double and then read with get<std::uint32_t>(), an undefined float-to-integer conversion. A value just above 2^32 was also silently truncated, so protocol 2^32+4 passed the protocol-4 version check, and negative values wrapped to huge sizes. Reject any frame containing a float or a negative number at the frame boundary (the wire carries none), which also keeps the memory-sync record deserializers defined, and read p2p numeric fields through a range-checked unsignedField<T>. |
||
|
trvon
|
dc8dc76ce3
|
refactor(p2p): run handshake and delta exchange over a FrameSource
The handshake, delta-exchange and inbound-session state machines read frames straight from the TLS P2pConnection, so none of the multi-frame parsing or payload accounting could be exercised without a socket. Introduce a private FrameSource seam with a production adapter over P2pConnection and an in-memory BufferFrameSource, pass TLS-derived identity values explicitly, and expose the validators and receive paths in p2p_fuzz.h. Behaviour is unchanged; the transport, protocol, delta, registry and manager suites pass. |
||
|
trvon
|
5cf6e7119f
|
build(fuzz): build fuzzers with plain clang and libFuzzer
The fuzzing subdirectory skipped itself unless the compiler was AFL++, so fuzzers could only be built inside the Docker image and no quick local or CI run was possible. With -Dbuild-fuzzers=true a clang that links the libFuzzer runtime now selects libFuzzer: project objects get -fsanitize=fuzzer-no-link plus ASan and UBSan unless b_sanitize is set. AFL++ builds keep their previous flags. fuzz.sh gains a 'local' command that runs a libFuzzer target over its seed corpus and passes a dictionary when tools/fuzzing/dicts has one. |
||
|
trvon
|
ac60e84f57
|
fix(build): include <algorithm> where std algorithms are used
db_salvage.cpp and the SQLite KG store used std algorithms through transitive includes that a plain clang libFuzzer build does not provide. |
||
|
trvon
|
8faa5f43c3
|
docs: describe the graph without code-symbol nodes
README, AGENTS.md, the yams skill guide and the engineering prompt still promised symbol extraction, calls/defined_in edges, and graph --impact for callers. Point callers and blast radius at yams grep and describe the relations the graph still holds. |
||
|
trvon
|
ec5b78980e
|
refactor(graph): resolve explore through graph nodes and report removed impact
Graph explore looked up entry points in symbol_metadata, which no longer exists. Resolve them from node labels, aliases, and ingest path nodes instead, gather a wider candidate pool, and rank it by query-term coverage per file before applying the symbol budget so broad early terms cannot crowd out the file that answers the query. Impact and affected-test analysis walked code-symbol edges that are no longer produced; they now return a warning saying the symbol graph was removed in v0.20 instead of silently empty results. |
||
|
trvon
|
c39ed84bf7
|
refactor(plugins)!: remove the symbol_extractor_v1 plugin ABI
Migration 40 dropped the symbol tables and graph edges, but the daemon could still adopt an external symbol_extractor_v1 plugin (for example a 0.19 plugin left in a user plugin directory) and rebuild them. Remove the interface header, the ABI adapter, plugin adoption, the EntityGraphService extract-and-write path, the post-ingest symbol channel, the write-coordinator symbol ops, and the KG-store symbol metadata and extraction-state API. The ABI loader and the external host now refuse a plugin that declares the interface (or the legacy symbol_extraction capability) with a one-time warning. Status no longer reports symbol extractors, generated configs no longer write [plugins.symbol_extraction] (an existing section still parses), and the benchmark suite drops symbol_extraction_bench. Tests that exercised the dropped tables or the v39 schema template are removed; the rest run against the current schema. BREAKING CHANGE: code-symbol extraction is gone. Removed: the symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download, path, auto-enable, auto-disable), the grammar steps in `yams init`, the meson option `plugin-symbols`, the conan option `enable_symbol_extraction`, and the Windows installer SymbolPlugin component. Databases migrate to schema 41; a v0.19 binary refuses to open them, and symbol data dropped by migration 40 cannot be recovered. |
||
|
trvon
|
8912d70e5c
|
perf(bench): expose BQ routing work and new topology levers to benchmarks
Report topology_route_bq_used / topology_route_bq_candidates in search debug stats so benchmarks can tell BQ-shortlisted routing apart from ANN and exhaustive routing. Let retrieval_quality_bench write bq_candidate_limit, bq_prefix_dim, sgc_hops and simeon_rerank_outer_maxsim into its per-run config from YAMS_BENCH_* harness settings, read through getenv_optional. |
||
|
trvon
|
4bc65fe3b8
|
perf(topology): emit profiling zones from topology and vector code
src/topology and src/vector were compiled without TRACY_ENABLE even in -Denable-profiling builds, so their YAMS_* zones were no-ops. Mirror the src/search setup in both libraries, then add zones for snapshot encode/decode, store and load, SGC smoothing, route-index build and routing, BQ search, the snapshot cache, and outer MaxSim, plus plots for snapshot payload bytes, membership count and BQ index bytes. |
||
|
trvon
|
7eb64fa8a2
|
test(search): include the BQ index header in the snapshot cache test
The BQ prefix assertions dereference centroidBqIndex, which the routing headers only forward-declare. |
||
|
Trevon
|
af6f89fc6e |
Merge pull request #181 from trvon/ci/release-promotion-safeguards
Some checks failed
Secret scan / gitleaks (push) Failing after 9s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
ci(release): decouple draft promotion and enforce non-squash merge policy |
||
|
trvon
|
715a0428df
|
fix(rerank): sample MaxSim fragments across the whole document
Outer MaxSim scored only the first eight sentences of each document, so a relevant passage later in a long document was never seen, which defeats the point of fragment scoring. Sample sentences evenly across the document (first and last kept), always reserve a document-head fragment, and bound per-call fragment embeddings so large rerank windows shrink the per-document budget instead of growing unbounded. |
||
|
trvon
|
c990b4878d
|
fix(topology): make SGC smoothing deterministic and skip ragged inputs
Edges were emitted in unordered_map iteration order, so float accumulation, and the smoothed embeddings feeding snapshot fingerprints, varied with input order. Order each neighbor list by document hash. Documents whose embedding is missing or of a different dimension were treated as zero vectors yet still added degree, damping their neighbors. Exclude them from the propagation graph. Hop 0 now reads the inputs directly, dropping one n*dim buffer for single-hop smoothing. Add closed-form checks against D^-1/2 (A+I) D^-1/2 for one and two hops. |
||
|
trvon
|
b9ec471c4c
|
feat(topology): wire SGC smoothing and BQ routing through typed config
sgcHops, sgcNormalize, bqCandidateLimit and bqPrefixDimension existed only as struct fields, so no production path could enable them. Resolve topology.sgc_hops / topology.sgc_normalize into the TopologyManager build config (hops capped at 4 to bound over-smoothing) and search.topology.bq_candidate_limit / bq_prefix_dim into the routing options, request, and fingerprint. The snapshot cache rebuilds only the route index when the BQ prefix changes. All default to off. Drop the snapshot interner, which was allocated but never used. |
||
|
trvon
|
da3e398b5c
|
fix(topology): keep exhaustive routing when BQ cannot score a query
The BQ branch cleared every route candidate before searching, so a query the index could not quantize (shorter than the indexed dimension) left only sparse-seeded clusters. The BQ fallback runs whenever the dense ANN index is absent, which made this reachable in production. Only narrow candidates when BQ returns a shortlist, matching the dense ANN branch, and document that the SimHash similarity estimate is an ordering proxy for raw sign bits. |
||
|
trvon
|
438f8a8a7b
|
ci(release): refuse stable releases from non-version refs
A stable workflow_dispatch took the release tag from GITHUB_REF_NAME without checking it, so dispatching from main published a release tagged main. Require refs/tags/v* or refs/tags/yams-v*. Fetch main and experimental with explicit refspecs in the candidate preflight so a tag named main cannot shadow the branch, and drop the workflow_run trust path from draft reconciliation now that the workflow is dispatch-only. |
||
|
trvon
|
c401e045d0
|
fix(metadata): guard snapshot pruning and topology count triggers
Migration 40 pruned every topology snapshot when the latest pointer was missing or named a snapshot that did not exist, and aborted on malformed pointer JSON. Prune only when the pointer is valid JSON naming an existing snapshot node. Migration 41 matched topology keys with LIKE, which is case-insensitive in SQLite and also stopped counting user keys such as Topology.*. Use GLOB, and rebuild the topology counts on rollback so down leaves metadata_value_counts consistent. Neither migration has shipped in a release or nightly. |
||
|
trvon
|
fff24242fd
|
fix(topology): make the latest pointer the storeBatch commit point
storeBatch wrote per-document keys before the snapshot node, so a failure in between left documents pointing at a snapshot that was never persisted. Write the snapshot node first and the latest pointer last so a partial store leaves the previous snapshot authoritative. Remove only the two per-document keys still written, in one transaction via a new removeMetadataBatch, and stop the membership slow path from reading structural keys that migration 41 prunes. |
||
|
trvon
|
522c33a9ca
|
fix(topology): share one resident snapshot behind a lock
The metadata store cached the latest batch in an unguarded mutable optional that search threads read and wrote, and handed out a full copy on every loadLatest, so search kept the snapshot resident twice. Publish the latest batch and its membership index as one immutable shared snapshot under a mutex, add loadLatestShared to hand it out without copying, and feed the routing snapshot cache from it. Historical loads no longer displace the resident latest snapshot, and a reader that raced a write never installs the older batch. |
||
|
trvon
|
5267b32775
|
fix(topology): bound snapshot decode against corrupt payloads
Validate every element count against the remaining payload before reserving, reject trailing bytes, type-check envelope fields, bound the untrusted uncompressed_bytes hint, read the magic little-endian, and convert legacy JSON type errors into Result errors instead of throwing. |
||
|
trvon
|
8a72da96ff
|
test(metadata): align legacy symbol test cases with schema v39
- Use v39 metadata DB template for legacy symbol metadata test cases in metadata_schema_catch2_test.cpp and kg_store_catch2_test.cpp to avoid running against schema v40 where symbol tables are dropped. - Use yams::test::make_temp_sqlite_path instead of raw getenv in metadata_schema_catch2_test.cpp and prune the stale portability allowlist entry. - Add site directory to SKIP_PARTS in check_repository_metadata.py to prevent local docs builds from triggering retired marker warnings. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
2bfc5334eb
|
ci(release): decouple draft promotion and enforce non-squash merge policy
- Decouple draft-promotion.yml from automated Tests workflow_run triggers, making promotion draft creation manual via workflow_dispatch to prevent GitHub from auto-chaining active experimental feature PRs into a stack targeting main. - Enforce and document the mandatory Promotion Merge Policy across reconcile_promotion_pr.py, validate_release_candidate.py, and draft-promotion.yml step summary: promotions must use a merge commit (--no-ff) or fast-forward, never squash-merge into main, preserving conventional commit history and unbroken git ancestry. - Unhide refactor commits in release-please-config.json so architectural refactorings are tracked under Changed in CHANGELOG.md. - Update unit tests in test_reconcile_promotion_pr.py and test_validate_release_candidate.py for the new policy invariants. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
3f3b94f972 |
Merge pull request #175 from trvon/perf/topology-embedding-memory
Some checks failed
Secret scan / gitleaks (push) Failing after 8s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
perf(topology): binary snapshot compression and BQ routing |
||
|
trvon
|
c3b667ee40
|
feat(retrieval): SGC topology smoothing, Matryoshka BQ prefix, and fragment outer-MaxSim
Implement three research-backed retrieval optimizations connecting
paperbridge literature findings with YAMS subsystems:
1. SGC Feature Smoothing (Proposal A):
- Add sgcHops and sgcNormalize to TopologyBuildConfig.
- Diffuse document embeddings across reciprocal nearest-neighbor graph
edges prior to clustering, formalizing Lean SGC.lean.
- Re-normalize diffused embeddings to preserve unit hypersphere metric geometry.
- Wire into ConnectedComponent, Louvain, and KMeans topology engines.
2. Matryoshka Prefix Binary Quantization (Proposal C):
- Add maxDimensions parameter to BinaryQuantizer::quantize and
maxPrefixDimension to BinaryQuantizedIndex::build.
- Support fast 64-bit word-aligned prefix Hamming shortlisting in
SparseGuidedClusterRouter via bqPrefixDimension.
3. Fragment Outer-MaxSim Reranker (Proposal B):
- Implement SimeonScoringMode { SingleVectorCosine, FragmentOuterMaxSim }
in SimeonModelProvider.
- Add extractDocumentFragments and late-interaction outer-MaxSim scoring
inspired by ColBERTv2/PLAID/SPLATE to prevent relevance dilution in long docs.
- Expose simeonRerankOuterMaxSim on SearchEngineConfig and ConfigResolver.
- Add Catch2 test coverage and verify BEIR scifact benchmark metrics.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
trvon
|
052031a484
|
feat(vector): binary quantization (BQ) pre-filter for cluster routing
- Add BinaryQuantizer and BinaryQuantizedIndex in include/yams/vector/binary_quantization.h and src/vector/binary_quantization.cpp implementing 1-bit sign quantization, packed 64-bit word storage, hardware popcount Hamming distance, and cosine similarity estimation. - Integrate centroidBqIndex into SparseRouteIndex, constructing compact BQ centroids alongside dense ANN indices during SparseGuidedClusterRouter::buildRouteIndex. - Add bqCandidateLimit to TopologyRouteRequest and tracking metrics (bqUsed, bqCandidates, bqDistanceEvaluations) to SparseRouteWork. - Enable fast candidate pre-filtering in SparseGuidedClusterRouter::route with 1-bit Hamming distance scans, providing sub-millisecond coarse candidate shortlisting and automatic fallback when the dense ANN index is omitted. - Add Catch2 unit test suites in tests/unit/vector/binary_quantization_catch2_test.cpp and tests/unit/topology/topology_baseline_catch2_test.cpp. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
1550dad3be
|
perf(search): flyweight string interning in TopologyRoutingSnapshotCache
- Add StringInterner in include/yams/search/string_interner.h for reusable, thread-compatible flyweight string deduplication with heterogeneous string_view lookups. - Update TopologyRoutingSnapshot in include/yams/search/topology_routing_session.h to store zero-copy string_view keys for clustersById and membershipsByDocumentHash, pointing directly into immutable TopologyArtifactBatch strings. - Add shared_ptr<StringInterner> to TopologyRoutingSnapshot to intern repeated lookup and dynamically generated strings. - Eliminate ~75,000 duplicate heap-allocated std::string objects per cached snapshot, reducing cache footprint by ~70%. - Add Catch2 tests in tests/unit/search/string_interner_catch2_test.cpp covering interning, deduplication, lifetime guarantees, and zero-copy pointer preservation across snapshot upgrades. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
dd8105f25d
|
perf(topology): bypass metadata table triggers during snapshot sync and prune legacy metadata
- Add Migration 41 to drop and recreate metadata_value_counts triggers with 'WHEN ... AND key NOT LIKE topology.%' condition, eliminating ~650,000 trigger executions per topology rebuild in a 72k document corpus. - Purge legacy continuous float/structural keys from metadata table (persistence_score, cohesion_score, bridge_score, role, cluster_level, parent_cluster_id, overlap_cluster_ids_json) in Migration 41. - In MetadataKgTopologyArtifactStore::storeBatch, replace N individual getDocumentByHash calls with single batchGetDocumentsByHash and write only essential snapshot_id and cluster_id to metadata table. - In MetadataKgTopologyArtifactStore::loadMemberships, add in-memory fast path serving memberships directly from cached latest snapshot batch. - Update serializeTopologyBatchCompressed to handle uncompressed tiny batches. - Add Catch2 unit tests in tests/unit/topology/topology_metadata_store_catch2_test.cpp validating trigger bypass, metadata pruning, and fast-path membership loading. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8fdb4df27e
|
feat(topology): binary snapshot serialization and zstd compression
Replace 960+ MB raw JSON snapshot serialization in MetadataKgTopologyArtifactStore with a compact, string-interned binary format compressed with Zstandard. - Add yams::topology::serializeTopologyBatchBinary and deserializeTopologyBatchBinary with integer length packing and string deduplication. - Wrap binary payload in a backward-compatible JSON envelope with format: zstd_binary_v1 and base64 encoded payload, maintaining SQLite json_valid(properties) == 1. - Fall back gracefully to legacy raw JSON deserializer when reading pre-existing uncompressed snapshots. - Add Catch2 unit test suite verifying binary roundtrip fidelity, compression ratio, backward-compatibility with legacy JSON snapshots, and corrupted payload resilience. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
32e57d758d
|
refactor(symbols): remove tree-sitter from conan and fix legacy symbol tests
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py. - Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile. - Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests. - Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test fixtures to run symbol-specific test cases against schema v39. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
ee51abcb1a
|
refactor(symbols): remove in-tree tree-sitter symbol extractor plugin and tests
Remove legacy Tree-sitter symbol extractor plugin (plugins/symbol_extractor_treesitter), the unused plugin-symbols build option, and associated tests and benchmark targets: - Drop plugins/symbol_extractor_treesitter/ (grammar_loader, symbol_extractor, plugin_api). - Remove plugin-symbols Meson option and packaging references. - Remove dedicated symbol plugin tests and benchmark targets (symbol_extraction_bench, symbol_extractor_all_langs_catch2_test, symbol_extractor_edge_cases_catch2_test, symbol_extractor_solidity_catch2_test, symbol_extractor_query_test, symbol_extractor_validation_test). - Clean up dead symbol extraction plugin helper and test from hnsw_entity_search_bench. - Prune obsolete portability allowlist entries. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
9c7f780d4c
|
refactor(symbols): remove symbol extraction subsystem and add migration 40
Drop legacy Tree-sitter symbol extraction subsystem, associated database tables, triggers, and AST edge types: - Migration 40 drops symbol_metadata, document_symbol_extraction_state, symbol_metadata_fts, associated triggers, and prunes historical bloated topology snapshots while preserving active snapshot topology, semantic_neighbor edges, and non-symbol observed_as provenance edges. Exclude nl_entity:% nodes from kg_nodes / kg_edges pruning to preserve canonicalized method-type technology entities. Provides complete downSQL restoring symbol tables, indexes, FTS virtual table, and sync triggers for clean v39 rollbacks. - In EntityGraphService, ensure locally resolved documentDbId propagates to WriteBatch for knowledgeGraphDocumentId to ensure accurate enrichment token completion, and only populate batch->documentIdToDelete for snapshot documents with a resolved document database ID. - In SymbolEnricher and SearchService, restore one-directional symbol-contains-query matching with non-empty checks to prevent short substrings matching broad queries. - In tests/unit/metadata/symbol_metadata_schema_test.cpp, migrate test DB up to version 39 so tests pass against their target version prior to v40 table drops. - Remove Tree-sitter grammar download and compile workflows from CLI init - Default plugin-symbols to false. - Fix macOS daemon client and CLI executable resolution using _NSGetExecutablePath with candidate fallbacks. - Add migration validation tests and symbol enrichment unit tests. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
fb5270e5a9
|
fix(bus): explicitly include <algorithm> for std::erase_if
Add missing #include <algorithm> in include/yams/daemon/components/InternalEventBus.h to ensure std::erase_if is declared across all standard library implementations without relying on transitive includes. Addresses Copilot review finding on PR #173. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
cfaff9480e
|
fix(bus): keep the channel-registry mutation off the production API
remove_channel() was added as a test helper but left publicly callable in every build. Existing holders keep their shared_ptr, so calling it while producers or consumers are live creates two live queues for one name and silently splits or drops events between the old holders and the next get_or_create_channel(). Copilot raised this on #170. Gate remove_channel() and the test-facing capacity_mismatches() accessor behind YAMS_TESTING, matching the convention already used by other production headers (retrieval_service.h, memory_sync.h, mcp_server.h). The production-facing part of the diagnostic - the warn emitted from get_or_create_channel - stays enabled in every build; only the registry mutation and the query accessor become test-only, and neither has a caller outside the test tree. Verified: grep finds callers only in tests/unit/daemon/daemon_background_processing_test.cpp; the production daemon library compiles without YAMS_TESTING; the test binary is 307/307. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
b6f5faba06
|
test(bus): make channel capacity reuse mismatches observable
InternalEventBus::get_or_create_channel returns an existing channel for a name and silently discards the requested capacity, so the first requester sizes a channel for the whole process. Several call sites depend on that silence, and five of them survived an earlier fix precisely because nothing reported it. Record each distinct (name, actual, requested) disagreement and warn once. The warning is emitted outside the lock so a sink cannot re-enter the bus while mu_ is held. remove_channel now drops a name's records too, so the list describes live disagreements and an assertion cannot become order-dependent. The spdlog include is guarded with __has_include, matching pressure_limited_poller.h, since this is an installed public header. The recorded capacity is stored beside the type-erased channel pointer rather than read back through static_pointer_cast<SpscQueue<T>>. A name is NOT guaranteed to map to one T: "model.events" is registered as SpscQueue<ModelReadyEvent> at plugins/onnx/model_provider.cpp:1076 and as SpscQueue<ModelLoadFailedEvent> at :1089/:1266/:1503, so reading capacity() through the cast would be a wrong-type member access on a reachable production path. The cast itself is still unvalidated, but that risk is unchanged from what callers already accepted for a doubly-registered name. On the test side the drain helpers no longer pin a capacity: they drain and remove, so each case's own declared capacity applies. The two embed_jobs sites that want a small ring drain first, so they actually receive 2048 instead of silently inheriting a larger registered ring, and a ScopedChannelRemoval guard drops that ring at scope exit. The throughput case asserts that no post_ingest disagreement was recorded, guarding the drain-then-remove behaviour its ring depends on. Evidence: with the pre-fix pinning helper restored, the diagnostic reports "channel 'post_ingest' already exists with capacity 32 but 64 was requested" and the new assertion fails (10 assertions, 1 failed); with the fix the case passes 10/10 and the binary is 307/307. The 4-batch outcome itself is load-dependent and did not reproduce on this machine, so the guard's value is that it detects the precondition deterministically rather than waiting for a loaded runner. Two defects in an earlier revision of this commit were found by the entry-review workflow (Gemini 3.8 Flash + Claude Opus, adjudicated) before the push, and are fixed here: one embed_jobs site never established the 2048 its comment claimed, so the comment and this message were false; and the capacity read went through the cast while claiming a one-T-per-name invariant that is already false in production. This does not reduce the mismatch count in this binary (9 -> 10 -> 9 distinct after these fixes, with the false direction gone). The remaining records are deliberate test-local rings plus production disagreements that need separate fixes: embed_jobs (8192 vs 1000) and post_ingest (1000 vs 4096 in RepairService). The diagnostic also surfaced a family that was not previously known: PostIngestQueue derives symbol_extraction, entity_extraction, title_extraction and kg_jobs from its per-instance capacity_ via boundedStageChannelCapacity, while those channels are process-wide singletons, so the first-constructed queue sizes them for every later queue (32 vs 512 and 32 vs 64 both observed). Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
2264cb9776
|
Merge branch 'main' into experimental
Some checks failed
Secret scan / gitleaks (push) Failing after 8s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
- Integrate socket shutdown notify fix (#165) from main. - Reconcile portability allowlist. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
1189161edb |
ci(tests): free unused toolchains on the macOS runner (#174)
Some checks failed
Secret scan / gitleaks (push) Failing after 8s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* ci(tests): free unused toolchains on the macOS runner The macOS ARM64 lane hit "zstd: error 70 ... No space left on device" followed by a linker failure. Free multi-gigabyte toolchains this build never uses before anything is built: Android SDK, dotnet, PowerShell and Mono. Deliberately keep Xcode/clang, Homebrew, cmake, ccache, zig, boost and the Rust toolchain: meson.build builds the libSQL subproject only when cargo is found and silently falls back to plain sqlite3 otherwise, so deleting ~/.cargo would switch this lane database backend without failing anything. This does not address the nightly release gate, which currently fails on an unrelated flaky batch-count assertion in daemon_background_processing. Signed-off-by: trvon <me@trevon.dev> * ci(tests): clarify SQLite warning-only fallback in macOS free-disk step Update comment in .github/workflows/tests.yml to accurately describe that when Cargo is missing, meson.build:555-572 emits warnings before falling back to standard SQLite3 without failing the build. Addresses Copilot review comment on PR #174. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
27f133f595 |
test(daemon): stop pinning the shared post_ingest channel capacity (#170)
Some checks failed
Secret scan / gitleaks (push) Failing after 7s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Candidate / Validate promotion and preview release (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
The case "keeps multi-doc batches when extraction concurrency is low" asserts an exact batch count (extractionBatches == 3) and produced 4 on a loaded macOS runner (16+16+15+1). The binding constraint was not the coalescing window: the drain helper created the shared post_ingest channel with capacity 32 before any queue was constructed, and get_or_create_channel ignores the requested capacity on a name hit, so the queue declared capacity 64 but got a 31-slot ring. With 48 tasks the producer had to block on the channel until the poller drained, which made batch boundaries a function of runner load. Drain and then drop the channel so each case capacity applies, and add the bus remove_channel() that makes that possible. The exact-count assertions are kept, not relaxed. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
cac8c42634 |
fix(daemon): keep socket shutdown notify inside the lock (#165)
Some checks failed
Build and Push Docker Images / Build Docker Image (linux/amd64) (push) Failing after 12s
Secret scan / gitleaks (push) Failing after 7s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Please / release-please (push) Failing after 7s
Build and Push Docker Images / Build Docker Image (linux/arm64 native) (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Build and Push Docker Images / Create Multi-arch Manifest (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
execute_on_io_context posted a handler that set "done" under the lock and then called cv.notify_one() after releasing it. The mutex and condition variable are stack locals of the calling frame, so the waiter could wake in that window, return, and destroy them -- leaving the IO thread signalling a destroyed condition variable in reused stack memory. On shutdown that surfaced as an ASan BUS with a corrupted return address from SocketServer::stop(). Notify while still holding the lock, so the waiter cannot return before the notify completes. Release-As: 0.20.0 Signed-off-by: trvon <me@trevon.dev> |
||
github-actions[bot]
|
c178f98619 |
chore: promote experimental to main (#156)
Some checks failed
Build and Push Docker Images / Build Docker Image (linux/amd64) (push) Failing after 58s
Secret scan / gitleaks (push) Failing after 7s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Please / release-please (push) Failing after 14s
Build and Push Docker Images / Build Docker Image (linux/arm64 native) (push) Has been cancelled
Build and Push Docker Images / Create Multi-arch Manifest (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* fix(storage): retry reference counter init on transient DB lock The add-sandbox smoke test (and CLI startup generally) can fail with 'database is locked' when a prior async batch commit holds the reference database longer than ReferenceDB's short retry budget, which CI runners hit consistently while local runs pass. Retry the whole init (schema + migrations) with a bounded, growing backoff (up to ~12s total) so transient contention no longer fails startup. Adds a Catch2 test that holds a write lock across init and releases it mid-retry. * refactor: deduplicate trim/escapeRegex/hasWildcard helpers The same string helpers were copy-pasted across CLI and service code: trim/trimCopy in 5 files, escapeRegex in 3, hasWildcard in 2, and the regex meta-char table in 2. Move canonical versions to yams::common::string_utils.h (trimCopy, hasWildcard, escapeRegex, isRegexMetaChar) and wire all call sites to them; daemon_helpers' trim_copy now delegates. Add a Catch2 test for the shared helpers. Kept isTransientMetadataError and annotateResultRelations untouched: they have semantically different implementations and need a separate review, not a mechanical merge. * refactor(cli): share time filters between get/list Same 6-field time filter logic copy-pasted in GetCommand and ListCommand (~72 lines each). Extract to yams::cli::applyDocTimeFilters + DocTimeFilters, wire both commands, add unit test. * refactor: dedup retry helpers + test fixtures - isTransientMetadataError + MetadataTelemetry: 2 copies (grep/search) -> shared service_utils.hpp - grep_service local string_contains -> shared features::string_contains - ArtifactGuard: 5 identical copies in corruption tests -> shared tests/common/sqlite_corruption.h retryMetadataOp loops stay split (async coroutine vs sync sleep - real difference). annotateResultRelations/applyFileTypeFilters/formatSnippet deferred: behavior parity question, not mechanical dedup. * perf(storage): slice-by-8 CRC32 on all crc paths, 13.7x faster Profiling running daemon: bit-by-bit CRC32 top CPU consumer (2289 samples/15s during grep). Same algorithm/polynomial, byte-identical output (oracle test 0..512 len), stored checksums stay valid. Applies slice-by-8 everywhere crc ran: - yams::common::crc32 (+ crc32Update seeded variant for mid-stream) - compression::calculateCRC32/updateCRC32 delegate to it - storage-local duplicate copy removed (dead scope) - WAL entry checksum copy removed Also restructures sizeof arithmetic in WAL size math so opengrep does not misread it as pointer arithmetic. * fix(daemon): clarify IPC response mismatches * test: isolate integration state * refactor(plugins): centralize lifecycle policy * perf(topology): streamline artifact construction Reuse admitted relations and dense splitter state while preserving deterministic identities, centroids, and the 64-document safety cap. Add corpus-shaped benchmarks so construction regressions stay measurable. * perf(daemon): trust clean metadata restarts * perf(vector): persist reusable warm starts * bench(ingestion): validate Simeon lifecycle * fix(daemon): preserve ingestion admission * fix(daemon): propagate startup failures * refactor(config): centralize runtime paths * refactor(config): unify environment policy * test: isolate process environment * refactor(config): unify flat TOML lookups * fix(mcp): restore WASI final linkage Keep config and native subsystems out of the minimal archive and export the profile definition so consumer ABI matches. * refactor(config): consolidate embedding policy * refactor(bench): isolate benchmark controls * refactor(metadata): extract content transaction seams * refactor(test): consolidate daemon harness * refactor(config): consolidate runtime tuning * refactor(bench): type run configuration * perf(search): optimize concept mining warmup * test(policy): gate configuration authority * fix(config): stack daemon environment leases * fix(config): revoke removed tuning overrides * fix(bench): reject unknown arguments * fix(build): restore clean test and benchmark links * build(deps): update Simeon concept mining Point the submodule at the reviewed, published optimization so clean CI clones can resolve it. * fix(ci): stabilize sanitizer validation Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free. * style(tests): align validation formatting * fix(plugins): honor lifecycle RPC timeout Use ExternalPluginHostConfig::defaultRpcTimeout for discovery, load, initialization, health, restart, and shutdown calls so sanitizer-slow plugin startup does not fall back to JsonRpcClient's hidden three-second default. * fix(config): distinguish empty from unset env on Windows _putenv_s removes a variable for "" and returns EINVAL for NULL, so the CRT environment cannot represent an explicitly empty value; the boundary collapsed unset and empty into one state. Read/write through the Win32 environment (SetEnvironmentVariableA/GetEnvironmentVariableA) instead, keeping a best-effort _putenv_s sync so legacy std::getenv readers stay coherent, and treat the empty-vs-unset read race fail-closed. * test(metadata): clone DB template via VACUUM INTO copy_file of the migrated WAL-mode template plus -wal/-shm sidecars corrupts WAL replay on Windows (stale -shm), losing kg_nodes. VACUUM INTO produces a self-contained single-file copy; a probe verifies the KG schema survived the clone. * test(portability): add cross-platform lint gate Catch Windows/Linux/macOS-only defect classes before push: raw std::jthread, sqlite3_open* via bare path.c_str(), and raw getenv reads outside the config boundary. Wired into pre-commit and the unit/policy suite; 575 reviewed sites allowlisted. * ci(cross): add compile-only cross-compile smoke Detects mingw-w64/linux-gnu cross compilers and compiles a portability probe with -fsyntax-only to catch narrow-vs-wide API and std::jthread issues before GitHub CI. Skips cleanly when no toolchain is present; --self-test proves dispatch and failure propagation with a fake compiler. * ci(gate): distinguish transient infra from real failures Retry setup.sh dependency resolution up to 3 times on Conan network timeouts with backoff; report a distinct infra-retry (exit 77) when the network failure persists, while real compile/test failures still fail hard. Prevents a transient center2.conan.io timeout masquerading as a code failure. * test(storage): tag refcount stress cases slow Stress cases with 20 threads / 200 iterations run in the default unit lane because only ~[slow] is filtered. Tag them [slow] so they move to the storage_submodule_slow suite. * test: move stress cases to slow suite Stress-tagged cases ran in the default unit lane because only ~[slow] was filtered. Tag them [slow] and add ~[slow]/[slow] splits to the five binaries (mcp_submodule, work_coordinator, daemon_background, plugin_infrastructure, hybrid_search) so they run in the slow suite. * feat(memory-sync): add version-vector + LWW data model Version vector (element-wise max merge, dominates/concurrent partial order) plus hybrid timestamp and LWW resolution: causally-later writes win, concurrent writes resolve deterministically by (ts, origin). JSON serialization via nlohmann. Header-only, no library deps. * feat(memory-sync): add backend-agnostic sync loop Publish memory records as content-addressed blobs plus a version-vector-tagged index record per write; sync lists the index, pulls records, and LWW-merges to a deterministic winner per logical key. Backend injected (filesystem / S3-R2 interchangeable). * feat(memory-sync): add periodic sync service Daemon-facing service owns the backend + sync loop and reconciles on a configurable interval via a stoppable worker thread. Typed MemorySyncConfig (nodeId, syncIntervalMs); start/stop idempotent and stop noexcept-safe. Backend injected so filesystem and S3/R2 resolution stay with the caller. * refactor(cli): extract already-initialized init path Factor the alreadyInitialized && !force_ branch out of the monolithic execute() into a private handleAlreadyInitialized() helper. Behavior preserved; setup decision/try nesting reduced. Portability allowlist line numbers updated for the shift. * refactor(cli): extract testable post-init guidance Factor the post-setup 'what next / how to configure' guidance into pure formatLaterCommand/formatInitSummary builders so each decision branch's help is testable without stdout, and add a focused test asserting the later-command and ready-summary text. Output is byte-identical. Portability allowlist line numbers updated for the shift. * feat(memory-sync): add daemon config + service factory Typed MemorySyncDaemonConfig parsed from the [memory_sync] TOML section (enabled, node_id, backend, path, sync_interval_ms) plus createMemorySyncService() which resolves filesystem/s3 backends and constructs the service. No YAMS_* env knobs. * feat(memory-sync): wire service into daemon config + lifecycle Add a typed [memory_sync] TOML section (enabled, node_id, backend, path, sync_interval_ms) parsed into DaemonConfig, and start/stop the MemorySyncService within ServiceManager: start after content-store init (best-effort, relative paths resolve against the data dir), stop noexcept in Phase 8 shutdown. No YAMS_* env knobs. * feat(cli): add yams p2p publish/read/status Thin p2p command over the memory-sync service: publish <key> <value|@file>, read <key>, and status (merged index summary + backend identity). Binary values are hex-encoded in JSON output; config resolves from [memory_sync] with --backend/--path/--node-id overrides. * test(p2p): add container convergence harness docker-compose with two saged daemons + MinIO (R2 stand-in) + shared volume, a multi-stage saged.Dockerfile, and a validation script driving filesystem<->filesystem convergence and causal-ordering determinism (scenarios 1 + 3). * fix(memory-sync): serialize loop access + resume causal history Add a mutex around all MemorySyncLoop access so the periodic sync worker does not race publish/read/syncOnce on the loop's mutable version-vector and logical-clock state. Reconcile before publish so a fresh loop instance (e.g. a stateless CLI) resumes its node's causal history. Reject a zero sync interval. * fix(memory-sync): parse s3 URL via storage resolver Route the s3 backend through StorageBackendFactory::parseURL so scheme detection, authority credentials, and query params flow through the same path as the daemon's storage resolver, and reject a backend=s3 config whose path is not an s3:// URL. * test(daemon): make lock-holder test fork-safe The child forked by 'Daemon does not terminate unverified data-dir lock holder' built an fs::path and called json::dump(), both of which allocate; a fork in a multithreaded process can leave the malloc lock held, so the child could fail before signaling readiness. Pre-compute the lock path in the parent, format the payload with snprintf into a stack buffer, signal a failure byte on open/flock error, and retry the parent's read on EINTR. Portability allowlist line numbers updated for the shift. * feat(storage): add sync backend primitives * feat(memory-sync): authenticate records * fix(memory-sync): bound reconciliation * feat(memory-sync): manage session lifecycle * feat(memory-sync): add metadata adapters * feat(memory-sync): add vector adapter * feat(daemon): wire bounded sync lifecycle * feat(p2p): expose sync protocol status * fix(vector): bind rebuild readiness to epochs * docs(p2p): define local validation contract * fix(daemon): harden async shutdown * fix(metadata): normalize extraction status * test(portability): reject analyzer shims * test(ci): stabilize platform matrix * test(ci): fix remaining platform builds * ci(sourcehut): stream runtime artifact * fix(config): preserve empty Windows values * fix(memory-sync): escape acknowledgement keys * test(ci): normalize Windows expectations * fix(config): track empty Windows values * test(daemon): synchronize repair hysteresis setup * chore(tooling): mark dependency-managed includes * style(ci): format artifact collector * ci(github): bound apt mirror retries * test(ci): isolate ARM smoke lifecycles * fix(daemon): serialize socket accept shutdown * fix(daemon): preserve stamp claim ownership * test(memory-sync): scale convergence waits * fix(daemon): bypass strand close during global teardown * fix(storage): sign S3 wire path verbatim * fix(cli): discover daemon log via status * feat(memory-sync): log quarantine reasons and config summary * fix(cli): render disabled vectors and quiet data-dir alarm * chore(storage): annotate reviewed env reads * feat(cli): install daemon as systemd service * fix(memory-sync): include spdlog in public header * fix(memory-sync): log quarantine from daemon, not header * fix(cli): discover live daemon socket for p2p * test(extraction): poll for launcher response lines * fix(daemon): complete pool futures on cancellation * test(daemon): fix pool teardown race and scale hysteresis * fix(daemon): keep p2p status off sync lock * test(daemon): harden background wait, slow-lane hysteresis * chore(topology): remove parked join knob (#62) Signed-off-by: trvon <git@trevon.dev> * fix(graph): scope health checks to graph data (#63) Signed-off-by: trvon <git@trevon.dev> * fix(retrieval): harden one-shot output (#64) Deduplicate unary results, verify direct blobs before writing, and surface stream finalization failures instead of reporting partial output as successful. Signed-off-by: trvon <git@trevon.dev> * perf(search): favor lexical mixed defaults (#65) Signed-off-by: trvon <git@trevon.dev> * perf(vector): harden portable backends (#66) * perf(vector): update portable backends Keep generic release binaries on baseline ISA until runtime CPU dispatch can guard AVX and FMA paths. Signed-off-by: trvon <git@trevon.dev> * test(vector): isolate backend measurements Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> * feat(storage): expose typed disk pressure policy Signed-off-by: trvon <git@trevon.dev> * fix(mcp): preserve embedded stdio buffers Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order. * feat(p2p): add authenticated direct replication (#68) * fix(memory-sync): keep replication responsive Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle. Reconstructed from ff62d3f8, 8e232721, and e60f513d. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated direct transport Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged. Reconstructed from 96f9db88 and c46de77b. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): exchange causal deltas Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies. Reconstructed from 1b8be146 and 41883338. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): integrate direct sync services Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport. Reconstructed from a19aaa44. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): apply replicated document state Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths. Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5. Signed-off-by: trvon <git@trevon.dev> * test(p2p): register direct sync gates Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout. Reconstructed from 360b9115 and 89416ccb. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): require bounded peer enrollment Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity. Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): persist quarantine and deletes Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects. Reconstructed from 74ce01e0 and 19888604. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated cold bootstrap Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume. Reconstructed from 8a9f5acf. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> * fix(p2p): replicate content deletion (#69) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): resume bounded history catch-up (#70) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden transport availability (#71) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden bootstrap trust (#72) Require authenticated direct stores and explicit operator enrollment while hardening private keys and trust files across Unix and Windows. Sign local control artifacts, bind them to a protected store generation, and serialize async initialization shutdown. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): serialize sync application (#73) Signed-off-by: trvon <git@trevon.dev> * test(p2p): add direct control fuzz harnesses (#74) Add libFuzzer/AFL++ harnesses for the direct-P2P handshake/state/history and delta/bootstrap control parsers. Fuzzers invoke the same production parser seams as the live wire path, so coverage cannot drift from real validation. A protocol-v3 seed exercises exact-version rejection after the v4 upgrade. Bound noninteractive AFL++ campaigns with AFL_FUZZ_SECONDS and register focused fuzz-smoke tests without duplicating broad protocol/delta registrations. Clarify that peer/cycle status is local health telemetry rather than global convergence, and document the remaining decision-grade live-performance gate. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden direct-P2P framing and ingress admission (#75) Bound pre-authentication JSON nesting depth to 32 so a pathologically nested control frame cannot overflow the session-thread stack before any trust decision. Raise the control-frame cap to 512 KiB to cover the worst-case authenticated state frame (~300 KiB across 256 writers x 256-byte ids). Gate MemorySyncLoop::applyDeltas through the resource governor so a peer cannot flood deltas on the direct replication path. Account delta/bootstrap wire bytes using the exact received frame size instead of a compact re-serialization. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden CLI status and operator/key-rotation flows (#76) Keep `p2p status` text output machine-parseable by removing ANSI severity colorization, and stop connect/disconnect/forget from throwing on malformed daemon responses (safe .value defaults, uint64 counters). Bound handshake and delta timeouts to kP2pMaxOperationTimeout up front, and ceiling direct sync_interval_ms to the 5-minute reconnect bound so invalid config fails early instead of aborting daemon startup. Degrade memory-sync status to peerCount=0 on a transient peer-registry read instead of failing the whole status IPC. Add an explicit `p2p forget` operation (proto FORGET=9 through dispatcher, manager, and registry) so operators can rotate a peer's pinned key while preserving fail-closed enrollment. Signed-off-by: trvon <git@trevon.dev> * refactor(p2p): tighten tests, share helpers, and guard test hooks (#77) Compile the direct-P2P test-only machinery out of production builds (YAMS_DAEMON_TEST_HOOKS_ENABLED now wraps the reconnect-loop hook storage, invoker, and call site, matching the already-guarded setter). Replace the single shared writer key in p2p_manager tests with one distinct key pair per writer identity so origin-binding regressions (a record claiming origin X but signed by Y's key) fail verification. Extract the copy-pasted bytes/text/TempDir helpers into tests/common/p2p_test_helpers.h and reuse them across the transport/delta/manager suites. Replace a timing-sensitive fixed sleep in the transport session-expiry test with a bounded poll that accepts both deadline manifestations, and gate the memory-sync-service concurrent-lifecycle stress case behind YAMS_RUN_STRESS_TESTS=1 per the existing FTS5 pattern. Deduplicate the byte-identical commitment-JSON serialization inside MemorySyncLoop (coldBootstrapRoot / persistReplicationCheckpoint) into one static helper. Signed-off-by: trvon <git@trevon.dev> * refactor(daemon): decompose ServiceManager responsibilities (#79) * fix(ci): enable OpenGrep custom ignore support Signed-off-by: trvon <me@trevon.dev> * fix(p2p): centralize backoff cap and bound test polling Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): split ServiceManager responsibilities Signed-off-by: trvon <me@trevon.dev> * fix(daemon): narrow ServiceManager test-hook declarations Signed-off-by: trvon <me@trevon.dev> * chore(architecture): ignore local Brick artifacts Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): apply Brick audit fixes to the PR #79 decomposition (#80) * feat(storage): expose typed disk pressure policy Signed-off-by: trvon <git@trevon.dev> * fix(mcp): preserve embedded stdio buffers Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order. * refactor(daemon): scope shared P2P protected-file accessors for cross-TU use The protected-file accessors (readProtectedP2pPrivateKey, readProtectedP2pTrustFile, writeExclusiveP2pPrivateKey) are shared by the P2P identity-key paths and the memory-sync bootstrap. Keep their platform-specific definitions in p2p_integration.cpp unchanged and scope them into yams::daemon::service_manager_detail, declared by the private, non-installed header p2p_protected_file.h. Qualify the internal callsites. No behavior change. Prepares moving ServiceManager::initializeMemorySync into the memory-sync translation unit (Brick audit cohesion note). Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): place initializeMemorySync with memory-sync TU Brick structural audit of PR #79 flagged that ServiceManager::initializeMemorySync lives in p2p_integration.cpp while the rest of the memory-sync surface lives in memory_sync.cpp. Move it — together with its exclusive helper ensureAuthenticatedDirectStore — into memory_sync.cpp. The shared protected-file accessors remain defined in p2p_integration.cpp (see the preceding service_manager_detail commit); moved function bodies are unchanged and ServiceManager.h is untouched. Brick evidence (experimental pre-79 |