yams/tests/benchmarks
Trevon ae15bf317a
perf(ingest): harden daemon batching and shutdown (#58)
* fix(daemon): harden model state ownership

Keep ModelPoolConfig single-defined and serialize telemetry snapshots to avoid ODR drift and torn concurrent reads.

* refactor(daemon): consolidate ingest pollers

Require the wake-timer contract so every post-ingest stage shares one event-driven poller configuration.

* refactor(daemon): unify repair execution

Share admission, operation ordering, cancellation, progress, and result aggregation between synchronous and asynchronous repair paths.

* refactor(daemon): unify runtime tuning

Resolve startup and reload settings through one typed path so parsing, precedence, and range validation cannot drift.

* refactor(daemon)!: remove dead IPC surfaces

Delete unreferenced framing, cancellation, and compatibility paths while keeping the active streaming protocol intact.

BREAKING CHANGE: callers using connectAsync or legacy MessageFramer and FrameReader helpers must migrate to connect, frame_message_into, feed, and get_frame.

* refactor(daemon): hide test hooks

Preserve test-library linkage without exposing daemon-only probes through production headers or external symbols.

* fix(daemon): avoid replaying successful adds

A fresh-connection transport retry reuses the response handler. Clear its stale EOF error when the retry returns a valid add response so the outer retry loop does not enqueue the document again.

* bench(ingest): enforce lossless pipeline runs

Separate admission, storage, drain, enrichment, and searchability timing. Decision-grade plans now reject drops, duplicate processing, incomplete pipelines, and experiment identity drift.

* bench(search): expose cold seed ingest timings

Pair generalized-memory quality arms with admission, storage, drain, searchability, and enrichment timing from their shared cold seed. Keep the observation distinct from repeated throughput measurements.

* perf(ingest): batch deferred content indexing

Directory ingestion already schedules content work through post-ingest. Avoid duplicate inline indexing and coalesce normal-priority trickles within a bounded window.

* feat(metadata): expose insert writer metrics

Separate queue, batch-apply, and transaction time so ingestion runs can attribute connection pressure without changing write semantics.

* fix(daemon): retry transient refused connects

The daemon socket can be bound before its listener is ready during startup. Reuse the existing bounded retry budget for that refusal window.

* bench(ingest): attribute submission path costs

Decision-grade runs need stable coalescing identity and separate queue, apply, transaction, and estimated connection-wait costs.

* perf(ingest): batch durable document storage

Commit chunks, manifests, references, and metadata in bounded batches while preserving publication only after durable reference success.

* perf(ingest): coalesce bulk post-processing

Use a 16-item bulk cap while keeping latency-sensitive daemon ingestion on the existing four-item priority quota.

* bench(ingest): record post-ingest batch size

Include the effective cap in metrics and experiment identity so runs with different batching policies cannot be compared as one system.

* perf(ingest): batch queued storage waves

Preserve single-document fallback semantics and execute inline when the coordinator has one worker to avoid self-deadlock.

* perf(ingest): route bulk enrichment batches

Keep quota-sized interactive groups on the RPC lane while sending larger lossless groups through bounded coalescing.

* test(daemon): initialize task fixtures

* perf(ingest): batch post-ingest dispatch

* perf(ingest): widen directory storage waves

Borrow from a full wave when one document would otherwise fall through to single-document storage.

* perf(storage): bound batch writer fanout

Reuse a fixed worker set so batch size no longer creates one thread and payload copy per object.

* perf(ingest): align storage batch drains

Dispatch one configured batch through storage so intermediate waves do not fragment downstream metadata work.

* fix(ingest): commit metadata state atomically

Fold first-version, extracted-title, and embedding-completion state
into existing transactions.

Read committed extraction state so duplicate-content tasks cannot
double-count corpus counters.

* bench(ingest): measure production submission

Drive pipelined files through DocumentIngestionService.
Export dispatch, fingerprint, and enqueue costs through xplan.

* fix(xplan): preserve native failures

Report child signals and bounded stderr before readiness checks so
assertions are not masked as missing traces.

* fix(daemon): preserve embedding success hashes

* fix(daemon): serialize pooled socket shutdown

* fix(daemon): harden batch publication shutdown

* fix(storage): use compat jthread shim

* fix(daemon): avoid poller template shadowing

* fix(search): keep vector pipeline test enabled

* fix: harden shutdown and fallback paths

Demote expected shutdown SQL interruptions, ignore readonly FTS integrity noise, avoid stopped-executor socket hangs, and preserve metadata writer fallback outcomes.
2026-07-17 07:04:30 -04:00
..
baseline feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
configs feat: instrumentation improvements for benchmarks, shell completition improvements, packaging updates for completions and docs updates 2026-04-13 21:34:18 -04:00
data Search/grep UX, runtime hardening, and CI stabilization (#39) 2026-05-27 12:15:17 -04:00
relevance feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
scripts feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
search perf(ingest): harden daemon batching and shutdown (#58) 2026-07-17 07:04:30 -04:00
xplan perf(ingest): harden daemon batching and shutdown (#58) 2026-07-17 07:04:30 -04:00
api_benchmarks.cpp Remediate Fold SAST findings 2026-05-27 22:41:44 -04:00
beir_loader.h feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
bench_utils.h fix: harden CI and metadata stability (#48) 2026-06-11 16:42:54 -04:00
benchmark_base.h fix: small fixes for benchmarks and documentation update 2026-04-17 22:32:04 -04:00
benchmark_cli.h fix: tree-diff benchmark regression accounting and stabilize subtree fast-path measurement 2026-04-03 20:45:09 -04:00
core_benchmarks.cpp security: expanding opengrep suite and addressing issues 2026-05-02 21:33:22 -04:00
daemon_socket_accept_bench.cpp fix: improving repair check 2026-04-17 18:47:24 -04:00
daemon_warm_latency_main.cpp fix: harden CI and metadata stability (#48) 2026-06-11 16:42:54 -04:00
entity_extraction_bench.cpp perf: improve ingestion optimization and profiling infrastructure (#52) 2026-06-19 17:32:19 -04:00
fts5_indexing_bench.cpp perf: improve ingestion optimization and profiling infrastructure (#52) 2026-06-19 17:32:19 -04:00
grep_benchmarks.cpp fix(security): harden fold-flagged guards 2026-05-17 23:50:32 -04:00
hnsw_entity_search_bench.cpp Remediate Fold SAST findings 2026-05-27 22:41:44 -04:00
ingestion_throughput_bench.cpp fix: harden CI and metadata stability (#48) 2026-06-11 16:42:54 -04:00
ipc_streaming_bench.cpp v0.7.3 staging 2025-10-07 11:18:35 -04:00
kg_edge_insert_bench.cpp feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
large_scale_ingestion_bench.cpp fix(tests): use portable getpid shim 2026-05-20 15:54:13 -04:00
meson.build perf(ingest): harden daemon batching and shutdown (#58) 2026-07-17 07:04:30 -04:00
metadata_contention_bench.cpp perf(metadata): profile write-path batching (#49) 2026-06-12 14:11:49 -04:00
metadata_path_query_bench.cpp feat(daemon): shutdown hardening + test helper linkage fixes (#44) 2026-06-08 16:29:44 -04:00
multi_client_ingestion_bench.cpp feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
repair_ability_bench.cpp feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
retrieval_service_benchmarks.cpp feat: improved search fusion logic and tuning parameters. CI improvements, reverting to push with wrangler in CI. Experimenting with vector rebuild to address additional lost overhead in search 2026-05-05 08:58:20 -04:00
run_multi_client_optimization_loop.sh feat: expanding storage system to work with r2 backends via object storage plugin. Updating benchmarks and documentation 2026-03-07 19:43:14 -05:00
run_multi_client_sweep.sh feat: expanding storage system to work with r2 backends via object storage plugin. Updating benchmarks and documentation 2026-03-07 19:43:14 -05:00
search_bandit_bench.cpp feat(search): add Simeon bandit tuning (#33) 2026-05-17 04:08:50 -04:00
search_benchmarks.cpp feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
simeon_lexical_score_bench.cpp perf: coalesce KG writes and document live-mirror drift 2026-07-02 09:12:23 -04:00
single_client_pipeline_bench.cpp fix: harden CI and metadata stability (#48) 2026-06-11 16:42:54 -04:00
storage_throughput_bench.cpp fix(cli): improve graph-guided grep 2026-06-04 22:46:10 -04:00
symbol_extraction_bench.cpp fix: windows test fixes 2026-02-20 22:51:41 -05:00
symbol_ground_truth_v2.jsonl feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
topology_ablation_bench.cpp feat(search): make topology-assisted retrieval the default (#57) 2026-07-13 20:26:43 -04:00
topology_topic_vocab.h feat: expanding topology system and search for better testing and simeon integration 2026-04-19 09:25:28 -04:00
TRACY_PROFILING.md perf(metadata): profile write-path batching (#49) 2026-06-12 14:11:49 -04:00
tree_build_bench.cpp Remediate Fold SAST findings 2026-05-27 22:41:44 -04:00
tree_diff_benchmarks.cpp feat: optimizations for data returns and harden daemon status and normalize MCP/version responses 2026-04-04 17:33:59 -04:00
tree_list_filter_bench.cpp experimental → main: MCP timeout fix + doctor integrity checks + search/perf improvements (#31) 2026-05-14 23:29:44 -04:00
vector_backend_engine_compare.cpp fix: harden CI and metadata stability (#48) 2026-06-11 16:42:54 -04:00
vector_memory_scaling_bench.cpp Assertion hardening, metadata correctness, & daemon lifecycle fixes (#41) 2026-06-04 12:53:42 -04:00
write_coordinator_bench.cpp perf: coalesce KG writes and document live-mirror drift 2026-07-02 09:12:23 -04:00
zyp_extraction_bench.cpp perf(zyp): true parallel PDF extraction + throughput benchmark 2026-05-15 12:13:33 -04:00