72647bbaf7- CI warm: ccache dep, continue-on-error, 60min timeout, YAMS_DISABLE_FAISS - Faiss: YAMS_HAS_FAISS guard, Apple Clang disable, OpenMP detection - Metadata: retry constraint failures, deduplicate batch UPSERT entries - Docker: wget for Zig, gfortran, YAMS_DISABLE_FAISS=1 - CI config: free-disk-space v1.3.1, libomp-18-dev, experimental trigger - Weekly benchmarks: faiss disable, ccache, diagnostics capture - Simeon: bump to 7763a97, EntropyRouter, concept_count - CorpusStats: extraction readiness fields - Remove committed artifacts (.wrangler, .artifacts, dispatcher_external) - Fix .vscode/settings.json hardcoded path Signed-off-by: trvon <git@trevon.dev>
125 lines
5.3 KiB
Markdown
125 lines
5.3 KiB
Markdown
# Storage Backends Benchmark
|
|
|
|
This page tracks local vs S3-compatible backend behavior for CLI-level CRUD workflows.
|
|
|
|
## Scope
|
|
|
|
- Compare backend behavior for `add`, `get`, `search`, and `delete`.
|
|
- Measure end-to-end CLI latency (includes command startup + daemon path), not storage microbenchmarks.
|
|
- Keep embeddings/model loading disabled for stable backend-path comparisons.
|
|
|
|
## Provider validation status
|
|
|
|
- **Cloudflare R2**: validated in this repository, including `temp_credentials` flow.
|
|
- **AWS S3**: supported by the same S3-compatible path, but not currently part of automated validation here.
|
|
|
|
## Latest validated run (2026-03-08)
|
|
|
|
- Backends: `local,r2`
|
|
- Params: `iterations=1`, `files=60`, `file_size_kb=8`, `retrieve_count=20`
|
|
- R2 auth mode: `temp_credentials` (bearer token + account id bootstrap)
|
|
|
|
| Backend | Store (files/s) | Retrieve mean (ms) | Retrieve ops/s | Search mean (ms) | Search qps | Delete mean (ms) | Delete ops/s |
|
|
|---|---:|---:|---:|---:|---:|---:|---:|
|
|
| local | 40.34 | 1675.76 | 0.60 | 1519.15 | 0.66 | 1500.77 | 0.67 |
|
|
| r2 | 1.14 | 5126.69 | 0.20 | 4398.37 | 0.23 | 4190.88 | 0.24 |
|
|
|
|
## Validation notes
|
|
|
|
- Remote fallback guard stayed clean: R2 iterations wrote `0` local object files.
|
|
- CRUD gate passed (`PUT/GET/UPDATE/DELETE` semantics validated by harness checks).
|
|
- List output currently reports duplicate rows (`120` rows for `60` unique paths) for both backends; benchmark normalizes to unique paths for timing.
|
|
|
|
## Compression-mode speed track
|
|
|
|
R2/S3 writes now use the same transparent compression wrapper as local storage: content is hashed
|
|
as original bytes, then compressible chunks are compressed before the object-storage backend sees
|
|
the payload. This affects both upload bandwidth/cost and retrieval latency, so live backend runs
|
|
should compare both modes.
|
|
|
|
Harness:
|
|
|
|
```bash
|
|
YAMS_BENCH_BIN=$PWD/build/debug/tools/yams-cli/yams-cli \
|
|
YAMS_BENCH_R2_BUCKET=<bucket> \
|
|
YAMS_BENCH_R2_ENDPOINT=<account-id>.r2.cloudflarestorage.com \
|
|
YAMS_BENCH_R2_ACCESS_KEY=<r2-s3-access-key> \
|
|
YAMS_BENCH_R2_SECRET_KEY=<r2-s3-secret-key> \
|
|
YAMS_BENCH_R2_REGION=auto \
|
|
python3 tests/benchmarks/run_local_vs_r2_benchmark.py \
|
|
--backends r2 \
|
|
--compression-modes both \
|
|
--iterations 3 \
|
|
--files 60 \
|
|
--file-size-kb 64 \
|
|
--retrieve-count 20 \
|
|
--require-remote \
|
|
--output-dir bench_results/storage_backends/r2-compression-$(date -u +%Y%m%dT%H%M%SZ)
|
|
```
|
|
|
|
For temporary Cloudflare R2 credentials instead of direct S3 keys:
|
|
|
|
```bash
|
|
YAMS_BENCH_R2_AUTH_MODE=temp_credentials
|
|
YAMS_BENCH_R2_API_TOKEN=<cloudflare-api-token>
|
|
YAMS_BENCH_R2_ACCOUNT_ID=<account-id> # optional if encoded in endpoint
|
|
```
|
|
|
|
Latest local harness smoke (2026-05-02, credentials not present in this shell for live R2):
|
|
|
|
- Backends: `local`
|
|
- Compression modes: `compressed,raw`
|
|
- Params: `iterations=1`, `files=8`, `file_size_kb=4`, `retrieve_count=3`
|
|
|
|
| Backend | Store (files/s) | Retrieve mean (ms) | Retrieve ops/s | Search mean (ms) | Search qps | Delete mean (ms) | Delete ops/s |
|
|
|---|---:|---:|---:|---:|---:|---:|---:|
|
|
| local_compressed | 7.61 | 2171.09 | 0.46 | 1064.63 | 0.94 | 1066.49 | 0.94 |
|
|
| local_raw | 7.60 | 2197.76 | 0.46 | 1067.01 | 0.94 | 1071.99 | 0.93 |
|
|
|
|
Live R2 result status:
|
|
|
|
- Not run in this shell: required `YAMS_BENCH_R2_*` credentials were absent.
|
|
- The harness now fails fast with `--require-remote` when credentials are missing, and it uses
|
|
distinct object prefixes for `compressed` vs `raw` runs.
|
|
|
|
## Multi-client backend benchmark track
|
|
|
|
- Scope: agent-like read-heavy workloads.
|
|
- Profiles: `mixed`, `external_agent_churn`.
|
|
- Transports: `daemon_ipc`, `mcp`.
|
|
- Client counts: `4`, `8` total concurrent clients.
|
|
- Backends: `local`, `r2` (R2 in `temp_credentials` mode).
|
|
|
|
### Latest validated multi-client run (2026-03-08)
|
|
|
|
- Parameters: `iterations=1`, `files=120`, `file_size_kb=8`, `ops_per_client=12`
|
|
|
|
| Backend | Profile | Transport | Clients | Ops/s | Fail rate | Search p95 (ms) |
|
|
|---|---|---|---:|---:|---:|---:|
|
|
| local | mixed | daemon_ipc | 4 | 23.80 | 0.0000 | 5.67 |
|
|
| local | mixed | mcp | 4 | 23.83 | 0.0000 | 5.75 |
|
|
| local | mixed | daemon_ipc | 8 | 47.56 | 0.0000 | 5.75 |
|
|
| local | mixed | mcp | 8 | 47.63 | 0.0000 | 5.79 |
|
|
| local | external_agent_churn | daemon_ipc | 4 | 23.82 | 0.0000 | 5.72 |
|
|
| local | external_agent_churn | mcp | 4 | 23.81 | 0.0000 | 7.56 |
|
|
| local | external_agent_churn | daemon_ipc | 8 | 47.52 | 0.0000 | 5.69 |
|
|
| local | external_agent_churn | mcp | 8 | 47.63 | 0.0000 | 7.41 |
|
|
| r2 | mixed | daemon_ipc | 4 | 19.32 | 0.2308 | 5.76 |
|
|
| r2 | mixed | mcp | 4 | 19.35 | 0.2308 | 5.89 |
|
|
| r2 | mixed | daemon_ipc | 8 | 34.28 | 0.3913 | 5.72 |
|
|
| r2 | mixed | mcp | 8 | 34.19 | 0.3913 | 5.90 |
|
|
| r2 | external_agent_churn | daemon_ipc | 4 | 20.85 | 0.1429 | 5.70 |
|
|
| r2 | external_agent_churn | mcp | 4 | 20.87 | 0.1429 | 7.30 |
|
|
| r2 | external_agent_churn | daemon_ipc | 8 | 38.61 | 0.2308 | 5.70 |
|
|
| r2 | external_agent_churn | mcp | 8 | 38.75 | 0.2308 | 7.34 |
|
|
|
|
Key findings:
|
|
|
|
- Local backend scaled near-linearly from 4 to 8 clients, with zero observed failures in this run.
|
|
- R2 had lower throughput and non-zero failure rates under this load pattern (roughly 14% to 39%).
|
|
- MCP and daemon IPC showed similar throughput bands within each backend/profile/client combination.
|
|
|
|
## Notes
|
|
|
|
- Benchmark harness scripts are internal and are not shipped as part of public docs.
|
|
- Storage backend validation remains a stabilization priority before expanding benchmark scope.
|