Commit graph

14 commits

Author SHA1 Message Date
trvon
65564158a9
perf(zyp): true parallel PDF extraction + throughput benchmark
- Implement Document::extractAllParallelized() using std::async
  to distribute pages across threads with per-thread Document
  instances, achieving true parallelism (not the old fake alias)

- zyp_plugin.cpp now uses extractAllParallelized() by default
  for PDF extraction during ingestion

- Add zyp_extraction_bench standalone benchmark measuring
  sequential vs 2/4/N-thread extraction throughput (pages/s, MB/s)

Build: ninja bench_zyp
Run:   build/debug/plugins/zyp/zyp_extraction_bench <pdf>
2026-05-15 12:13:33 -04:00
trvon
66c21c0b25
perf(ingest): bypass GLiNER for zyp PDFs, preserve plugin metadata
- Add IContentExtractor::extractTextAndMetadata() to carry plugin metadata
- Add extractDocumentContent() in extraction_util for text+metadata
- PIQ uses plugin-provided title from PDF metadata, skipping GLiNER
  entirely when a title already exists (e.g. from zyp PDF parsing)
- Fix misleading 'parallel' naming in zyp wrapper/comments
  (zpdf_extract_all_parallel is actually sequential)

This removes the dominant slow-write source for paperbridge PDF
ingestion — title metadata batches that previously backed up the
WriteCoordinator with 34s queue waits.
2026-05-15 11:52:28 -04:00
trvon
bc100efcdc
feat: updating zyp pdf extractor to use v0.16.0 compat in plugin build 2026-05-05 19:37:50 -04:00
trvon
8772cc8e54
fix(docs,ci): addressing breaking changes in zig v0.16.x in Dockerfile, docs update and code cleanup 2026-04-25 11:27:18 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos 2026-04-12 18:16:10 -04:00
trvon
081fb740b4
fix: improving test coverage and addressign ASAN issues 2026-03-19 06:32:13 -04:00
trvon
df1348f3c7
fix: tuning improvements and better tests. Now working address memory issue of huge queue when ingesting allot of files 2026-02-06 04:31:09 -05:00
trvon
df3997545b
search benchmarking fixes 2026-01-19 02:50:05 -05:00
trvon
83bd1b7fc1
updating entity interface and adding new plugin for entity extraction 2026-01-08 15:06:58 -05:00
trevon
4be4216aa4
windows build fixes for zyp 2026-01-06 23:56:06 -05:00
trvon
5eadb107d4
test improvements and search bugs fixes 2025-12-31 18:46:32 -05:00
trevon
6c7e4069ef
windows build fixes 2025-12-31 15:38:25 -05:00
trvon
bc71d36534
migrated pdf plugin, daemon shutdown logic fix 2025-12-31 14:46:07 -05:00
trvon
0557c9b33b
test improvements and linter run 2025-12-31 12:51:28 -05:00