trvon
65564158a9
perf(zyp): true parallel PDF extraction + throughput benchmark
...
- Implement Document::extractAllParallelized() using std::async
to distribute pages across threads with per-thread Document
instances, achieving true parallelism (not the old fake alias)
- zyp_plugin.cpp now uses extractAllParallelized() by default
for PDF extraction during ingestion
- Add zyp_extraction_bench standalone benchmark measuring
sequential vs 2/4/N-thread extraction throughput (pages/s, MB/s)
Build: ninja bench_zyp
Run: build/debug/plugins/zyp/zyp_extraction_bench <pdf>
2026-05-15 12:13:33 -04:00
trvon
66c21c0b25
perf(ingest): bypass GLiNER for zyp PDFs, preserve plugin metadata
...
- Add IContentExtractor::extractTextAndMetadata() to carry plugin metadata
- Add extractDocumentContent() in extraction_util for text+metadata
- PIQ uses plugin-provided title from PDF metadata, skipping GLiNER
entirely when a title already exists (e.g. from zyp PDF parsing)
- Fix misleading 'parallel' naming in zyp wrapper/comments
(zpdf_extract_all_parallel is actually sequential)
This removes the dominant slow-write source for paperbridge PDF
ingestion — title metadata batches that previously backed up the
WriteCoordinator with 34s queue waits.
2026-05-15 11:52:28 -04:00
trvon
bc100efcdc
feat: updating zyp pdf extractor to use v0.16.0 compat in plugin build
2026-05-05 19:37:50 -04:00
trvon
a7ea3da82a
updating boost variable exposure
2026-05-03 17:03:34 -04:00
trvon
8772cc8e54
fix(docs,ci): addressing breaking changes in zig v0.16.x in Dockerfile, docs update and code cleanup
2026-04-25 11:27:18 -04:00
trvon
df0c78cf29
feat: test fixes, metric expansion, build improvements for gliner
2026-04-23 08:48:33 -04:00
trvon
2b60bed67b
feat(breaking): breaking change. removing onnxruntime for embeddings in favor of simeon. embeddings will now use less system resources, run faster but require avx/neon support per library
2026-04-21 12:23:38 -04:00
trvon
801980ce11
fix: fixing onnx model logic so that memory does not hold indefinitely. Removing re-ranker logic from plugin
2026-04-21 11:49:12 -04:00
trvon
d7ab4844a1
fix: small fixes for benchmarks and documentation update
2026-04-17 22:32:04 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos
2026-04-12 18:16:10 -04:00
trvon
5795a0f360
fix: continued correctness improvements. Added vec0 search mode and benchmark plumbing for YAMS retrieval
2026-04-12 13:32:33 -04:00
trvon
ef3d925797
feat: continued corectness improvements, ANN / vectordb improvements
2026-04-12 12:53:29 -04:00
trvon
2d0df863b7
fix: improved plugin locking around onnx session state and ownership improvements
2026-04-12 00:33:08 -04:00
trvon
120d98bc77
fix: fixing scaling logic of embedding generation that fallback to CPU
2026-04-10 18:54:46 -04:00
trvon
fff1ca19ac
fix: address regressions in onnx runtime generation from previous commit. Addressing critical bugs
2026-04-07 21:24:59 -04:00
trvon
814672f4dd
feat: removing unnecessary abstraction and adding some optimizations for embeddings generation
2026-04-07 18:04:34 -04:00
trvon
e09778b7fe
fix: improvements for embedding generation (embedding gemma specific issue) and Dockerfile regression fix
2026-04-07 15:54:52 -04:00
trvon
677f67e799
feat: search improvements, UBSan meson option, CI quality job, and fix clang-tidy findings
2026-04-06 23:11:03 -04:00
trvon
e8a6296797
fix: re-ranker embedding generation fix, test failures drift fix, search improvements for graph query expansion and gliner improvements for building graph
2026-04-06 18:38:54 -04:00
trvon
3f3ba0582d
fix: release please fix, improvements for embedding generation (check if model is loading an query that). continue work on search retrieval improvement related to on going graph improvements
2026-04-06 07:07:36 -04:00
trvon
d5967deedf
fix: make Gemma CoreML fallback automatic
2026-04-02 13:56:42 -04:00
trvon
a0b4915bcd
fix: bypass broken Gemma CoreML path and restore ANN/TurboQuant search
2026-04-01 21:09:22 -04:00
trvon
58a8cf8efe
feat: homebrew fixes for nightly formula install
2026-03-24 09:29:16 -04:00
trvon
2381c90c86
feat: cleaning up codebase and improving treesitter plugin
2026-03-23 16:57:12 -04:00
trvon
0261b90a7b
refactor(build): centralize dependency discovery and trim Meson duplication
2026-03-23 15:41:13 -04:00
trvon
9c496e674e
fix: correctness fixes for stale sqlite handles (apart of previous work), improvements in test suite and continue work for search optimizations
2026-03-22 20:06:29 -04:00
trvon
2e20416e96
feat: adding fuzz test for new download logic, removing r2 endpoint from tests, decoupling embedding / onnxruntime logic for work from previous commit
2026-03-21 18:08:23 -04:00
trvon
86ec015eea
feat: massive commit. Download logic updates with expanding controls (stop, start, resume etc.) from daemon control. This is used in my archive openclaw agent. This commit also includes work to harden onnxruntime abi access. this should allow loading newer abi's when builds are pinned to 1.23.0.
2026-03-21 17:12:47 -04:00
trvon
2a11749c57
fix: plugin bug fix for reranker
2026-03-19 16:16:41 -04:00
trvon
081fb740b4
fix: improving test coverage and addressign ASAN issues
2026-03-19 06:32:13 -04:00
trvon
bed12ed2d2
fix: fixing stop regression when socket exists to fast, fixing bug in binary detection where agent added files were seen as text. Added repair fix to improve binary classification of files without extensions
2026-03-15 08:13:41 -04:00
trvon
1c2bd4f058
feat: experimenting with parallelization of onnx reranker
2026-03-12 11:21:25 -04:00
trvon
513d43098c
feat: hardening client reponses and improving search engine precision through optimization loop
2026-03-12 11:07:18 -04:00
trvon
7a8c26110c
feat: early embeddinggemma support
2026-03-11 16:43:29 -04:00
trvon
f3399c001e
feat: adding perf optimizations for search and grep based on previous commit. Improving model logic for future embeddinggemma and improved colbert support. Re-ranking fixes for search to improve retrieval accuracy.
2026-03-11 14:15:33 -04:00
trvon
a3fd21766c
feat: reliability improvements for daemon lifecycle, submodule bumps, plugin improvements and search tuning optimizations based on benchmarks
2026-03-09 19:38:15 -04:00
trvon
41a2282691
feat: expanding storage system to work with r2 backends via object storage plugin. Updating benchmarks and documentation
2026-03-07 19:43:14 -05:00
trvon
92bb0c5bfd
fix: fixing s3 signing bug
2026-03-06 16:51:58 -05:00
trvon
31eb110391
fix: hardening release CI, addressing issues with onnxruntime linking on macos and linux and expanding benchmark
2026-03-06 08:30:14 -05:00
trvon
fdf2c323f7
fix: fixing onnxruntime linking isuses which result in degraded state on linux
2026-03-05 09:49:03 -05:00
trvon
8523b77169
feat: linking fix for onnx plugin for windows builds
2026-02-28 10:36:04 -05:00
trvon
099bbef70d
feat: large update. Documentation update, config and plugin hardning and improvements related to loading trust. Exposed trust state in the doctor command with hints on improvements. Bug fixes and test improvements to reduce flakiness and TSAN issues
2026-02-26 11:21:29 -05:00
0xkite
56cb6a66c5
Recreate last 7 commits (signed)
2026-02-24 14:32:58 -05:00
trvon
91e212389c
feat: fixing TSAN issues in MCP. implementing cloudflare code mode https://blog.cloudflare.com/code-mode-mcp/
2026-02-20 20:39:12 -05:00
trvon
218499386b
fix: pipeline scaling fixes, post ingestion benchmark updates
2026-02-19 12:16:47 -05:00
trvon
a9bad18fa4
feat: adding additional tests, making the aggressive profile scaling better
2026-02-17 19:15:26 -05:00
trvon
6a90f8c934
feat: test improvements, ASAN fixes, and onnx model performance improvements
2026-02-17 08:59:53 -05:00
trvon
33821d93f1
feat: test hardening, added benchmark for multiclient and improve stability to support up to 16 clients before performance degrades, improved parallization and request handling in daemon ingestion pipeline
2026-02-16 15:46:16 -05:00
trvon
c4ae922b7a
feat: mobile bindings expansion, update mcp for graph operations and download tool hardening
2026-02-15 17:50:14 -05:00
trvon
94a49472f1
feat: fixing bug with multiple processes access gpu, read/write pools now added in metadata repository, and updated plugins for opencode and vscode
2026-02-14 18:16:57 -05:00
