118 lines
4.3 KiB
Markdown
118 lines
4.3 KiB
Markdown
# Multi-Client Optimization Loop
|
|
|
|
This runbook captures the optimization cycle for bursty CLI/MCP/daemon workloads.
|
|
|
|
It executes:
|
|
|
|
1. phase 2 (throughput-first)
|
|
2. phase 1 (stability-first)
|
|
|
|
and prepares outputs for phase 3 (tuning sweeps) and phase 4 (regression gating).
|
|
|
|
## Prerequisites
|
|
|
|
- Build benchmark target:
|
|
|
|
```bash
|
|
meson compile -C build/debug bench_multi_client
|
|
```
|
|
|
|
- Optional but recommended: point to an existing corpus for read-path tests.
|
|
|
|
## Run
|
|
|
|
From repo root:
|
|
|
|
```bash
|
|
tests/benchmarks/run_multi_client_optimization_loop.sh \
|
|
--bench-bin build/debug/tests/benchmarks/multi_client_ingestion_bench \
|
|
--data-dir <corpus-dir>
|
|
```
|
|
|
|
By default, `--data-dir` is copied to an isolated temporary snapshot (`--data-dir-mode snapshot`)
|
|
before benchmark execution to avoid `.yams-lock` contention with an already-running daemon.
|
|
|
|
Use the source directory directly only when you explicitly want that behavior:
|
|
|
|
```bash
|
|
tests/benchmarks/run_multi_client_optimization_loop.sh \
|
|
--bench-bin build/debug/tests/benchmarks/multi_client_ingestion_bench \
|
|
--data-dir <corpus-dir> \
|
|
--data-dir-mode direct
|
|
```
|
|
|
|
The run emits per-phase JSONL, per-phase summaries, a combined summary, a case manifest, and a run manifest.
|
|
|
|
The summary schema is versioned (`multi_client_summary_v2`) and now includes:
|
|
|
|
- compatibility KPIs used by legacy regression checks
|
|
- per-op (`search/list/grep/status/get/cat`) throughput/fail-rate/p95 aggregates
|
|
- grep server telemetry aggregates (`execution_time_ms`, `phase_worker_scan_ms`,
|
|
`content_retrieval_ms`, scanned volume counters)
|
|
- case-level breakdown keyed by phase + transport + usage profile
|
|
|
|
If `--data-dir` is omitted, large-corpus runs are skipped and contention runs still execute.
|
|
|
|
## Summarize Existing JSONL
|
|
|
|
```bash
|
|
python3 tests/scripts/summarize_multi_client_jsonl.py \
|
|
--input <phase2.jsonl> \
|
|
--input <phase1.jsonl> \
|
|
--output <combined-summary.json>
|
|
```
|
|
|
|
## Regression Check
|
|
|
|
Compare current vs baseline summaries:
|
|
|
|
```bash
|
|
python3 tests/scripts/check_multi_client_regression.py \
|
|
<current-summary.json> \
|
|
<baseline-summary.json> \
|
|
--output <regression-report.json> \
|
|
--fail-on-regression
|
|
```
|
|
|
|
`check_multi_client_regression.py` is backward compatible with older baseline summaries
|
|
while using the richer v2 metrics when present.
|
|
|
|
## KPI Targets
|
|
|
|
- `large_corpus_reads.*.ops_per_sec`: non-regressing or improving
|
|
- `large_corpus_reads.*.fail_rate`: non-regressing
|
|
- `large_corpus_reads.*.connection_drop_rate`: non-regressing
|
|
- `connection_contention.fail_rate`: non-regressing
|
|
- `connection_contention.retry_after_rate`: stable or decreasing
|
|
|
|
## Current contention policy notes
|
|
|
|
- Peak-load CLI probe failures should now be classified, not treated as opaque harness failures.
|
|
- `timeout_under_load` is the only accepted CLI failure mode during the saturation window.
|
|
- Post-stress CLI recovery must pass for `status` and `list` before a contention run is considered healthy.
|
|
- In the current reduced contention scenario, `cli_search` now succeeds instead of timing out.
|
|
- Fresh ASAN validation remains the conservative safety lane; current ASAN mixed-workload runs are clean through 48 total clients and first degrade at 56.
|
|
|
|
## Mixed stress interpretation
|
|
|
|
- The mixed-client benchmark is a **closed-loop** workload with fixed per-client op budgets.
|
|
- Aggregate ops/s can rise at higher client counts even while the system is under *more* stress.
|
|
- For scaling decisions, use the first appearance of **nonzero failures** and rising **p95/p99 latencies** as the primary boundary markers.
|
|
- In the fresh Debug validation on the tuned defaults, **64 total mixed clients** is the clean tier and **68+** is the first degraded tier.
|
|
|
|
## Phase1 Daemon Scheduling Defaults
|
|
|
|
For large-corpus mixed read stability, prefer `[tuning]` keys over ad-hoc env overrides and keep the model hardware-aware:
|
|
|
|
```toml
|
|
[tuning]
|
|
grep_inflight_limit = 1
|
|
grep_admission_wait_ms = 20000
|
|
```
|
|
|
|
Notes:
|
|
|
|
- Read-path worker capacity, CLI request pool size, search concurrency, and read DB pool cap are now derived centrally in `TuneAdvisor`; avoid hard-coding machine-specific thread counts in docs or tests.
|
|
- `grep_inflight_limit = 1` is the recommended phase1 default for lower variance and stable tails.
|
|
- If your corpus is grep-heavy and stable, test `grep_inflight_limit = 2` for higher peak throughput.
|
|
- Keep timeout knobs unchanged while validating scheduler changes.
|