Commit graph

20 commits

Author SHA1 Message Date
trvon
be24d07c27
fix(ci): stabilize sanitizer validation
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
2026-08-08 11:42:33 -04:00
Trevon
0dd3985db4
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs

Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.

* refactor(vector): retire dormant backends

Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.

Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.

* refactor(plugins): remove legacy SDK tooling

Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.

* build(vector): remove stale FAISS overrides

* refactor(ipc): remove dormant FSM snapshots

* refactor(ingest): unify bounded KG enqueue

* fix(plugins): harden allocation contracts

Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).

* fix(storage): define empty object reads

Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.

* refactor(storage): trim through string views

* test: prune dormant harness helpers

Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.

* chore(cpp): initialize analyzer-facing locals

* build(simeon): advance analyzer cleanup

* refactor(config): drop dead daemon client keys

* refactor(ipc): inject daemon client transport

Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.

* refactor(retrieval): route CLI through daemon

Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.

* refactor(graph): move CLI scope into daemon

Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.

* build: isolate test-only runtime controls

* refactor(metadata): decouple lock telemetry

* fix(services): return on request timeout

* fix(retrieval): enforce workspace scope

* fix(graph): scope nodes before pagination

* fix(grep): drop binary result lines

* fix(daemon): bound checkpoint wait

* fix(repair): use KG health checks

* fix(storage): always verify compressed CRC

* perf(vector): skip unchanged PQ snapshots

* build(zyp): exclude bench from default build

* build: protect Meson files from formatter

* refactor: remove unreachable production APIs

* refactor(daemon): remove dormant runtime state

* fix(vector): harden daemon provider calls

Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.

* refactor(storage): remove unusable adapters

* fix(cli): unify in-process add requests

* fix(tsan): preserve async worker progress

* test(cli): isolate daemon-dependent suites

* fix(repair): expose queue backoff progress
2026-07-31 14:16:45 -04:00
trvon
4e566f6387
fix: harden buffer and parser contracts
Validate Simeon buffer boundaries and replace ZYP reverse pointer arithmetic with bounded indexing. Keep release SIMD kernels unchecked while asserting contracts in debug builds.
2026-07-29 19:00:10 -04:00
trvon
a217a41ee7
test: lift coverage for core paths
Add focused API, CLI, doctor, downloader, search, and vector
   infrastructure coverage.

   Cover retry seams, path-scoped search fallback, renderer fail-soft
   paths, router helpers, and model infrastructure without changing
   production behavior.

   Keep coverage gates stable by skipping Zyp on Darwin coverage builds
   and ignoring SIGPIPE in daemon protocol tests.
2026-06-22 10:25:28 -04:00
Trevon
771deebc84
fix: harden CI and metadata stability (#48)
* feat: dead code removal, profling foudnation improvements, addressing code smells in reference db implementation

* fix: test fixes and optimmizations for benchmarks

* fix: harden audit findings

   Reject invalid tokenizer ids, fix move-order hazards, bound manifest
   and audio IO, and guard numeric/process edge cases found by Fold.

* fix: route SymSpell writes through coordinator

* test: fix batch content initializers

* fix(ci): restore cross-platform builds

* fix: address Copilot review findings
2026-06-11 16:42:54 -04:00
Trevon
6d8877a6d4
feat(search): add Simeon bandit tuning (#33)
* feat(search): per-profile simeon bandit with training-free self-tuning

* fix(zyp): free parallel buffers correctly

* fix: code cleanup and simeon version bump

* fix(search): guard Simeon bandit state
2026-05-17 04:08:50 -04:00
trvon
65564158a9
perf(zyp): true parallel PDF extraction + throughput benchmark
- Implement Document::extractAllParallelized() using std::async
  to distribute pages across threads with per-thread Document
  instances, achieving true parallelism (not the old fake alias)

- zyp_plugin.cpp now uses extractAllParallelized() by default
  for PDF extraction during ingestion

- Add zyp_extraction_bench standalone benchmark measuring
  sequential vs 2/4/N-thread extraction throughput (pages/s, MB/s)

Build: ninja bench_zyp
Run:   build/debug/plugins/zyp/zyp_extraction_bench <pdf>
2026-05-15 12:13:33 -04:00
trvon
66c21c0b25
perf(ingest): bypass GLiNER for zyp PDFs, preserve plugin metadata
- Add IContentExtractor::extractTextAndMetadata() to carry plugin metadata
- Add extractDocumentContent() in extraction_util for text+metadata
- PIQ uses plugin-provided title from PDF metadata, skipping GLiNER
  entirely when a title already exists (e.g. from zyp PDF parsing)
- Fix misleading 'parallel' naming in zyp wrapper/comments
  (zpdf_extract_all_parallel is actually sequential)

This removes the dominant slow-write source for paperbridge PDF
ingestion — title metadata batches that previously backed up the
WriteCoordinator with 34s queue waits.
2026-05-15 11:52:28 -04:00
trvon
bc100efcdc
feat: updating zyp pdf extractor to use v0.16.0 compat in plugin build 2026-05-05 19:37:50 -04:00
trvon
8772cc8e54
fix(docs,ci): addressing breaking changes in zig v0.16.x in Dockerfile, docs update and code cleanup 2026-04-25 11:27:18 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos 2026-04-12 18:16:10 -04:00
trvon
081fb740b4
fix: improving test coverage and addressign ASAN issues 2026-03-19 06:32:13 -04:00
trvon
df1348f3c7
fix: tuning improvements and better tests. Now working address memory issue of huge queue when ingesting allot of files 2026-02-06 04:31:09 -05:00
trvon
df3997545b
search benchmarking fixes 2026-01-19 02:50:05 -05:00
trvon
83bd1b7fc1
updating entity interface and adding new plugin for entity extraction 2026-01-08 15:06:58 -05:00
trevon
4be4216aa4
windows build fixes for zyp 2026-01-06 23:56:06 -05:00
trvon
5eadb107d4
test improvements and search bugs fixes 2025-12-31 18:46:32 -05:00
trevon
6c7e4069ef
windows build fixes 2025-12-31 15:38:25 -05:00
trvon
bc71d36534
migrated pdf plugin, daemon shutdown logic fix 2025-12-31 14:46:07 -05:00
trvon
0557c9b33b
test improvements and linter run 2025-12-31 12:51:28 -05:00