Commit graph

10 commits

Author SHA1 Message Date
trvon
86fe2ff6c9
feat(memory-sync): manage session lifecycle 2026-08-17 21:43:10 -06:00
trvon
91fb1a32d9
fix(rm): make deletion safe and observable 2026-07-29 10:46:44 -04:00
Trevon
ae15bf317a
perf(ingest): harden daemon batching and shutdown (#58)
* fix(daemon): harden model state ownership

Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.

* refactor(daemon): consolidate ingest pollers

Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.

* refactor(daemon): unify repair execution

Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.

* refactor(daemon): unify runtime tuning

Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.

* refactor(daemon)!: remove dead IPC surfaces

Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.

BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.

* refactor(daemon): hide test hooks

Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.

* fix(daemon): avoid replaying successful adds

A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.

* bench(ingest): enforce lossless pipeline runs

Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.

* bench(search): expose cold seed ingest timings

Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.

* perf(ingest): batch deferred content indexing

Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.

* feat(metadata): expose insert writer metrics

Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.

* fix(daemon): retry transient refused connects

The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.

* bench(ingest): attribute submission path costs

Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.

* perf(ingest): batch durable document storage

Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.

* perf(ingest): coalesce bulk post-processing

Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.

* bench(ingest): record post-ingest batch size

Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.

* perf(ingest): batch queued storage waves

Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.

* perf(ingest): route bulk enrichment batches

Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.

* test(daemon): initialize task fixtures

* perf(ingest): batch post-ingest dispatch

* perf(ingest): widen directory storage waves

Borrow from a full wave when one document would otherwise fall through to single-document storage.

* perf(storage): bound batch writer fanout

Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.

* perf(ingest): align storage batch drains

Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.

* fix(ingest): commit metadata state atomically

Fold first-version, extracted-title, and embedding-completion state
into existing transactions.

Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.

* bench(ingest): measure production submission

Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.

* fix(xplan): preserve native failures

Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.

* fix(daemon): preserve embedding success hashes

* fix(daemon): serialize pooled socket shutdown

* fix(daemon): harden batch publication shutdown

* fix(storage): use compat jthread shim

* fix(daemon): avoid poller template shadowing

* fix(search): keep vector pipeline test enabled

* fix: harden shutdown and fallback paths

Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
2026-07-17 07:04:30 -04:00
Trevon
4228a0c5e9
chore: housekeeping — rm dead code, consolidate MCP tools, add tests, clean docs (#50)
* implemented stubbed IGraphContextService methods

* feat: wire graph communication updates in ipc, mcp and cli

* fix: fix call graph recall and add profiling coverage to graph calls sites

* fix: fixes for work where we added agent navigation to graph tooling

* fix(corpus): clean FTS and KG orphans on document delete

* fix(repair): clean orphans, honor cancellation, surface stats

* fix(rm): order content removal before metadata/KG delete. keep manifest while content is still referenced

* fix(daemon): seed built-in content extractor for readiness

* chore(cli): remove dead stubs and unwired helpers

* feat(search): named-collection scoping + provable relevance gate

* fix(refactor): rm dead stubs, inline simd, add proto/metrics tests

* chore: rm dead interfaces/enums, consolidate guards, add tests

* refactor: rm dr vaporware, collapse download handlers, clean list flags

* docs: rm stale testing investigation logs

* fix: address Copilot review findings from PR #50

- relationship_ops: return existing id when INSERT OR IGNORE skips
- document_crud_ops: propagate FTS5 cleanup error (was silent skip)
- builtin_text_content_extractor: unsigned char cast for tolower, add cctype
- mcp_server download_jobs: validate job_id for status/cancel actions
- list_command tests: unsigned char cast for isspace, add cctype include
2026-06-16 19:37:24 -04:00
Trevon
08322369cb
Search/grep UX, runtime hardening, and CI stabilization (#39)
* test(storage): add crash-replay and handle-failure coverage for ReferenceCounter

Adds 6 test cases:
- Recovery after forced close preserves committed data
- Uncommitted transaction rolled back after forced close
- Corrupted database file rejected on open
- Consistent stats after interrupted commit recovery
- Multiple PENDING transactions survive force close
- WAL checkpoint survives forced close

* fix(ci): remove ccache — conflicts with OpenBLAS ASM as direct compiler

* fix(ci): use explicit CC/CXX paths to prevent Conan from using ccache directly

ccache was picked up by Conan/CMake as the compiler itself (/usr/bin/ccache)
rather than as a transparent wrapper, causing OpenBLAS ASM compilation to fail
with 'ccache: invalid option -- D'. Setting explicit compiler paths prevents this.

* fix(ci): add YAMS_SKIP_CCACHE to avoid OpenBLAS ASM ccache conflict

setup.sh wraps CC/CXX with ccache for faster incremental builds, but
this causes Conan/CMake to use ccache directly as the ASM compiler for
OpenBLAS, where it fails on -D flags. Add an opt-out gate in setup.sh
and set YAMS_SKIP_CCACHE=1 in .build.yml for sourcehut CI VMs.

* fix(ci): add libomp-dev, libfaiss-dev, libopenblas-dev for FAISS build

* fix(ci): limit meson compile to -j1 to prevent OOM on sourcehut VMs

* fix(build): restore hooks and shrink srht

Compile the storage library with YAMS_TESTING in test builds so
debug-only hooks link again.

Include the POSIX chmod header used by the storage utility
Catch2 coverage cases.

Prune SourceHut fallback stage archives to runtime files so
branch artifacts stay under the upload limit.

* refactor(storage): drop dead code, add coverage

* refactor(cli): polish grep/search graph UX

* fix(daemon): stabilize shutdown and CI paths

- force vector-backed tests to opt out of CI-wide disable flags
- reset reference prune statements between deletes
- make optional-plugin and POSIX-only test assumptions portable
- extract ServiceManager open/shutdown helpers for safer WAL recovery work

* test(daemon): add db vacuum coverage

* test(daemon): add WAL startup recovery

- cover stale WAL startup recovery in ServiceManager
- cover repeated restart cycles with seeded WAL sidecars
- assert daemon recovery reporting for corrupt data-dir DBs
- verify oversized metadata DBs shrink after WAL recovery shutdown

* refactor(grep): consolidate indexed content scan

* fix(grep): remove pre-scan candidate truncation

* refactor(runtime): split startup, harden storage

* feat(bench): add manifest retrieval corpus

* fix(bench): run fusion experiment on simeon

* fix(cli): sharpen search graph hints

* fix(build): refresh version metadata

* fix(ci): stabilize sync search and srht packaging

* fix(ci): unblock windows and shrink artifacts

* fix(ci): stabilize linux runtime checks

* fix(build): shrink srht manifest
2026-05-27 12:15:17 -04:00
Trevon
bfe7cd94ca
test: expand storage-readiness coverage — 48.1%→59.8% broad, 74.3% narrow (#38)
* fix(daemon): avoid socket cancel on stop

Signal active sockets with shutdown only so pending I/O wakes without entering Boost.Asio's reactor cancel path during teardown.

* fix(metadata): open read pools read-only

* ci: fix coverage report generation

* fix(metadata): tag unscoped pool holders

* test(daemon): avoid ipc teardown takeover

* ci: skip plugin coverage objects

* fix(daemon): repair completed FTS5 ghosts

* build(vector): guard unusable FAISS headers

* ci: use clang gcov for coverage

* test(daemon): stabilize arm CI flakes

* Harden search and compression numeric config

* fix(daemon): reconcile orphan vector embeddings

* fix: address vector cleanup review comments

* chore(main): release 0.16.1

* Harden search ranking divisors

* Harden snippet distance and command registration

* test(storage): add readiness coverage gate

Document the distributed sync readiness surface and add direct adapter, refcount, and GC failure-path coverage.

* test(storage): cover strict resolver fallback

* fix(search): tighten retrieval ranking

Honor component weights in default RRF, over-fetch vector chunks before document aggregation, and persist topology centroids for sparse-guided routing.

* fix(search): route weak queries through topology

* fix(search): improve rerank and weak queries

Wire provider-backed cross reranking, add direct Simeon candidates for weak lexical pools, and report stage-level retrieval metrics.

* fix(storage): honor explicit plugin names

Add readiness coverage for factory, URL backend, and compressed storage failure paths.

* fix(storage): avoid remote size listing

* fix(api): rollback failed content adds

* test(storage): cover engine readiness

* test(storage): cover compressed readiness

* fix(core): harden delete and recovery paths

Route prune and local delete fallback through DocumentService so content, metadata, vectors, and KG cleanup stay consistent.

Treat transient SQLite quick_check lock failures as ResourceBusy, refresh IPC activity on streaming writes, and preserve reranker config in tuner seeding.

* test(storage): cover URL backend readiness

* fix(api): harden content rollback

* feat(storage): add list seam, URL listing, and write-edge coverage

* feat(api): reconcile ambiguous remote manifest writes

* feat(storage): add deep compressed scan validation

* test(storage): add plugin loader error path coverage

* fix(gc): prune reference rows after collection

* test(api,storage): add health, compact, gc, and verify edge coverage

* docs: update storage-readiness coverage to 71.8%

* test(metadata): add connection pool shutdown-while-leased coverage

* test(metadata): add tree builder snapshot root stability coverage

* test(vector): add before-init and after-close operation rejection coverage

* test(metadata): add database post-close, open-failure, and cache-clear coverage

* test(integrity): add P2P wrong-hash rejection + batch matrix resolution

* docs: audit and resolve all matrix rows — 18 fully covered

* test(metadata): add failed-migration and rollback coverage

* test(storage): add getBlockSize-invalid-key, empty-verify, PUT/DELETE retry coverage

* feat(storage): add file-open and rename failure hooks + ContentStore retrieve edge coverage

* docs: add broad-surface coverage with per-subsystem breakdown

* test(integrity): add chunk validator manifest validation coverage

* test(vector): add batch insert, delete, and update operation coverage

* docs: update integrity subsystem coverage to 46.4%

* docs: consolidate coverage report — clean matrix, remove boilerplate

* docs: update coverage with complete-suite numbers — 59.8% broad, 74.3% narrow

* test(storage): add URLBackend HEAD exists with 2xx coverage

* bump submodules

* fix: address PR review — curl_slist leak, ec ordering, CAS rollback, production hooks

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-25 22:58:15 -04:00
trvon
d0e33f4fe0
fix: addressing gaps and todo in storage implementation. Improving tests 2026-05-02 00:20:43 -04:00
trvon
3b5fb8fba8
fix: improvements for laoding handling in UX response 2026-04-09 16:40:04 -04:00
trvon
e24a85651a
plugin install path update, daemon status fixes, lint run 2025-12-20 18:13:54 -05:00
trevon
48dc0dfa89
removing gtest migrated tests and continuing with test migration 2025-12-16 10:50:18 -05:00