Commit graph

55 commits

Author SHA1 Message Date
trvon
c39ed84bf7
refactor(plugins)!: remove the symbol_extractor_v1 plugin ABI
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.

Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.

Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.

BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
2026-09-22 13:30:35 -06:00
trvon
32e57d758d
refactor(symbols): remove tree-sitter from conan and fix legacy symbol tests
- Remove enable_symbol_extraction option and tree-sitter requirement from conanfile.py.
- Remove YAMS_DISABLE_SYMBOL_EXTRACTION toggles from setup.ps1, setup.sh, and ubuntu.Dockerfile.
- Add v39_metadata_db_template helper to MetadataDbTemplate for legacy symbol tests.
- Update GraphComponent, DeleteCascade, GraphContextService, and KGStore test
  fixtures to run symbol-specific test cases against schema v39.

Signed-off-by: trvon <me@trevon.dev>
2026-09-22 07:37:30 -06:00
Trevon
af609a1e63
refactor(p2p): tighten tests, share helpers, and guard test hooks (#77)
Compile the direct-P2P test-only machinery out of production builds
(YAMS_DAEMON_TEST_HOOKS_ENABLED now wraps the reconnect-loop hook storage,
invoker, and call site, matching the already-guarded setter). Replace the
single shared writer key in p2p_manager tests with one distinct key pair per
writer identity so origin-binding regressions (a record claiming origin X but
signed by Y's key) fail verification. Extract the copy-pasted bytes/text/TempDir
helpers into tests/common/p2p_test_helpers.h and reuse them across the
transport/delta/manager suites. Replace a timing-sensitive fixed sleep in the
transport session-expiry test with a bounded poll that accepts both deadline
manifestations, and gate the memory-sync-service concurrent-lifecycle stress
case behind YAMS_RUN_STRESS_TESTS=1 per the existing FTS5 pattern. Deduplicate
the byte-identical commitment-JSON serialization inside MemorySyncLoop
(coldBootstrapRoot / persistReplicationCheckpoint) into one static helper.

Signed-off-by: trvon <git@trevon.dev>
2026-08-30 11:27:12 -06:00
trvon
9a234f449f
test(metadata): clone DB template via VACUUM INTO
copy_file of the migrated WAL-mode template plus -wal/-shm sidecars
corrupts WAL replay on Windows (stale -shm), losing kg_nodes. VACUUM
INTO produces a self-contained single-file copy; a probe verifies the
KG schema survived the clone.
2026-08-12 19:32:48 -06:00
trvon
9c9e19199e
refactor(test): consolidate daemon harness 2026-08-05 20:53:18 -04:00
trvon
5066fc0893
test: isolate process environment 2026-08-04 21:31:39 -04:00
trvon
4c3eb3d018
refactor: dedup retry helpers + test fixtures
- isTransientMetadataError + MetadataTelemetry: 2 copies (grep/search)
  -> shared service_utils.hpp
- grep_service local string_contains -> shared features::string_contains
- ArtifactGuard: 5 identical copies in corruption tests -> shared
  tests/common/sqlite_corruption.h

retryMetadataOp loops stay split (async coroutine vs sync sleep - real
difference). annotateResultRelations/applyFileTypeFilters/formatSnippet
deferred: behavior parity question, not mechanical dedup.
2026-08-01 00:12:32 -04:00
trvon
1ef8d91be9
fix: windows ci fix 2026-07-03 09:03:30 -04:00
trvon
903e29ab55
test(metadata): add corruption fixture + suites: page smash (header
+ cell tail, not free space), torn WAL, truncation, crash snapshot,
salvage row-by-row fallback, writer promise draining, pool behavior.
Runs green on libsql and stock sqlite.
2026-07-01 20:11:03 -04:00
Trevon
ca604bbcdf
fix: windows build and test fixes 2026-06-27 10:51:12 -04:00
Trevon
090a5ca69c
fix(testing): stabilize TSAN coverage, harden daemon shutdown, and speed up KG/storage paths (#53)
* perf(search, metadata): add KG write buffer, score cache, adaptive budgets

* perf(indexing): wire KGWriteBuffer into indexing service

   Converts per-document addEdgesUnique calls into batched flush via
   KGWriteBuffer.  500-doc ingest: 500 SQL round-trips → 1 batch.

   Verified: 14.2× edge insert gap (14.8K→195K edges/s) in kg_edge bench.
   Tests: 10/10 Catch2 pass (105 assertions).

* perf(storage): add fsync durability, parallel batch, read integrity

* test(storage): add stress, crash-sim, cold-cache, batch benchmarks

* fix: CI optimizations and work coordinator shutdown fix

* fix(testing): stabilize TSAN coverage gate

* fix: harden KG flush, score cache, and benches

   - keep KG buffers intact when batch commit fails
   - count flushed edges per batch instead of lifetime totals
   - guard the Simeon score cache and clear it on rebuild
   - fall back to fsync when F_FULLFSYNC fails on macOS
   - remove interactive sudo from cold-cache benchmarks
2026-06-26 17:57:02 -04:00
Trevon
7076e4b283
feat(daemon): shutdown hardening + test helper linkage fixes (#44)
* perf(daemon): coalesce metadata writes

* refactor(metadata): split repository ops

* fix(metadata): harden repository invariants

* feat: test coverage and continued refactor work in metadata code

* fix(graph): handle path-like explore queries

- prefer file path symbol lookup for graph explore\n- sanitize alias FTS queries before MATCH

* fix(daemon): quiesce shutdown background work

- Interrupt pending DB acquires before worker shutdown
- Wait for tracked background coroutines and timers to drain

* fix(daemon): cancel repair work on shutdown

Blocked embed and post-ingest queues could keep salvage and\nstuck-doc repair alive during shutdown. Consume cancellation in\nrepair retry loops and lock the integrated ServiceManager path\nwith regressions so shutdown fails fast instead of drifting.

* perf(metadata): reduce sqlite lock contention

* perf(metadata): flatten batch content retries

* fix(daemon): align shutdown wait budgets

A low shutdown timeout override could expire before ServiceManager\nfinished its 5s+30s WorkCoordinator join path. Share the join\nbudget constants, clamp graceful lifecycle waits to that floor,\nand lock the behavior with a daemon regression.

* perf(metadata): bound feedback write waits

* perf(metadata): prep batch content before tx

* fix(daemon): honor no-embeddings tasks

Post-ingest queue tasks dropped the per-task noEmbeddings flag,\nso focused extraction tests still emitted embed jobs and Windows\nCI exercised extra async paths. Preserve the flag through the bus,\nskip embed dispatch when requested, and wake the extraction poller\nimmediately after successful queueing.

* perf(daemon): batch fallback metadata writes

* feat(perf,docs): optimizations for metadata writes, prompt updates and test improvements

* perf(metadata): flatten insert retries

* fix(metadata): clear CI portability blockers

Use the shared portable ScopedEnvVar in metadata search-query helper tests, add mutators for test reuse, and normalize Statement::bindAll integral dispatch so long long arguments bind consistently across platforms. Add focused regression coverage for the bindAll long long path.

* perf(metadata): batch writes and steady bench

* fix(daemon): skip inactive post-ingest pollers

* perf(metadata): tighten tag and path caches

* perf(metadata): cache small batch upserts

* fix(daemon): harden embedded host teardown

* feat(docs): updating docs policies and language

* perf(metadata): tune doc update retries

* perf(metadata): bound download retries

* perf(metadata): bound download metadata waits

* bench(metadata): scope post-ingest content path

* fix(daemon): export embedded test hooks

* fix(daemon): harden test helper linkage

* fix(build): add missing standard includes
2026-06-08 16:29:44 -04:00
trvon
d67d5d63ac
Remediate Fold SAST findings
Centralize duplicated helpers and harden numeric, timing, IPC, SQLite retry, config, plugin, metrics, and async content-store paths identified during Fold SAST triage.\n\nAdd targeted tests for shared utilities and preserve per-task YAMS checkpoints.
2026-05-27 22:41:44 -04:00
trvon
4df44e0c2f
fix(tests): use portable getpid shim 2026-05-20 15:54:13 -04:00
trvon
69f6f63f55
fix(security): harden fold-flagged guards
Signed-off-by: trvon <me@trevon.dev>
2026-05-17 23:50:32 -04:00
trvon
e2c8ba043b
fix: removing gtest and continued cleanup work of test suite 2026-04-14 12:33:17 -04:00
trvon
b6b6ba9384
feat: additional test clean up and expansion for benchmarks 2026-04-14 11:39:53 -04:00
trvon
a930776655
feat: cleaning up test suite 2026-04-14 10:25:54 -04:00
trvon
0beaf58b18
feat: test cleanups 2026-03-23 19:43:09 -04:00
trvon
f4b3576e8e
fix: bug fixes from test regressions and test infrastructure improvements 2026-03-21 19:52:39 -04:00
trvon
91e212389c
feat: fixing TSAN issues in MCP. implementing cloudflare code mode https://blog.cloudflare.com/code-mode-mcp/ 2026-02-20 20:39:12 -05:00
trvon
fc284f3b37
fix: resolving TSAN issues and expanding graph building to use gliner data 2026-02-19 14:52:01 -05:00
trvon
0703a28188
fix: removing duplicate shim logic 2026-02-14 19:39:51 -05:00
trvon
94a49472f1
feat: fixing bug with multiple processes access gpu, read/write pools now added in metadata repository, and updated plugins for opencode and vscode 2026-02-14 18:16:57 -05:00
trvon
2e966c4f1b
fix: windows build fixes 2026-02-09 23:26:33 -05:00
trvon
f4f3c055fe
feat: benchmark expansion and normalization. Imrpoving metrics for repair and models 2026-02-08 20:35:39 -05:00
trvon
8c9cd40c38
bug fixes and metrics exposure improvements 2026-01-27 21:35:21 -05:00
trvon
80ea3ca869
testing improvements 2026-01-14 16:27:42 -05:00
trvon
523ab8d3b7
ingestion optimizations 2026-01-06 02:02:16 -05:00
trevon
48dc0dfa89
removing gtest migrated tests and continuing with test migration 2025-12-16 10:50:18 -05:00
trevon
86378b8cb6
testing improvements, bug fixes for plugin loading and logic. Search engine bug fix 2025-12-12 04:54:25 -05:00
trevon
e60e82c4f0
improvements for v0.7.10 2025-12-11 22:07:22 -05:00
trvon
c16939f482
linter 2025-11-28 20:30:41 -05:00
trevon
ebe81823db
improvements for daemon lifecycle, tree repair, and mcp regression 2025-11-26 14:22:35 -05:00
trvon
70e88af608
running linter 2025-11-25 13:10:29 -05:00
trevon
244cf02822
windows build stable on dev machine 2025-11-23 23:22:30 -05:00
trvon
2913adcd20
pre-release v0.7.8 2025-11-14 11:58:19 -05:00
trvon
b06e9335b9
staging v0.7.8. running benchmarks on updated search engine and improving code for testing/benchmarking 2025-11-13 10:27:10 -05:00
trvon
352ee45a69
updating files with linter run 2025-11-11 10:11:27 -05:00
trvon
12fc1b8a50
resolved concurrency bugs in daemon. added fuzzing work in progress per issue #8 2025-11-11 10:10:07 -05:00
trvon
166d8c520d
staging v0.7.7 2025-11-01 16:02:12 -04:00
trvon
8d23211af2
improvements for ci, setup.sh and updates to new symbol extraction plugin 2025-10-21 19:38:45 -04:00
trvon
5bb124a3c1
v0.7.3 2025-10-08 17:02:37 -04:00
trvon
e04effe2d2
fixing conan file so ft5s builds 2025-10-04 21:12:19 -04:00
trvon
e586892e06
checkpoint. Expanded tests and resolved minor issues with grep, ci/cd and daemon protocol add spec 2025-09-29 14:38:04 -04:00
Trevon
186a2e6b5a
ci and testing improvements 2025-09-25 12:43:25 -04:00
Trevon
20fcaca931
ci imrpovements, prompt update and test improvements 2025-09-25 12:15:10 -04:00
Trevon
0f5b58df48
expanding ci platform testes, improving test coverage and resolve platform bugs related to migrated implementations and sanitization 2025-09-25 11:06:54 -04:00
Trevon
117927eee0
graph improvements before plugin update 2025-09-22 10:31:08 -04:00
Trevon
43699f99a3
v0.7.0-prerelease 2025-09-20 22:55:34 -04:00