213 lines
6.1 KiB
Markdown
213 lines
6.1 KiB
Markdown
# Performance Profiling & Analysis
|
|
|
|
YAMS supports multiple profiling and analysis tools to help identify performance bottlenecks and optimize critical paths.
|
|
|
|
## Profiling Build
|
|
|
|
The `Profiling` build type enables instrumentation for various profiling tools:
|
|
|
|
```bash
|
|
# Build with profiling support
|
|
./setup.sh Profiling
|
|
|
|
# Compile
|
|
meson compile -C build/profiling
|
|
```
|
|
|
|
This build:
|
|
- Uses Debug as base (symbols + assertions enabled)
|
|
- Enables Tracy profiler integration
|
|
- Includes test binaries for benchmarking
|
|
- Outputs to `build/profiling/` directory
|
|
|
|
## Tracy Profiler
|
|
|
|
Tracy provides real-time, low-overhead performance profiling with frame-by-frame analysis.
|
|
|
|
### Setup
|
|
|
|
1. **Download Tracy**: Get the profiler from https://github.com/wolfpld/tracy/releases
|
|
2. **Build with Tracy**: `./setup.sh Profiling`
|
|
3. **Start Tracy Server**: Run the Tracy GUI application
|
|
4. **Run YAMS**: Tracy automatically connects and captures data
|
|
|
|
### Usage Example
|
|
|
|
```bash
|
|
# Build with profiling
|
|
./setup.sh Profiling
|
|
meson compile -C build/profiling
|
|
|
|
# Run commands (Tracy server should be running)
|
|
build/profiling/tools/yams-cli/yams search "query" --limit 20
|
|
build/profiling/tools/yams-cli/yams add large_corpus/ --recursive
|
|
|
|
# Observe in Tracy:
|
|
# - Function execution times
|
|
# - Call stack flamegraphs
|
|
# - Memory allocations
|
|
# - Lock contention
|
|
```
|
|
|
|
### Key Areas to Profile
|
|
|
|
For **hierarchical search**:
|
|
- `twoStageVectorSearch()` total overhead
|
|
- Document grouping (hash map operations)
|
|
- Hierarchical boosting calculation
|
|
- Final result sorting
|
|
|
|
For **General Performance**:
|
|
- Search query execution paths
|
|
- Document ingestion throughput
|
|
- Vector index operations
|
|
- Metadata repository queries
|
|
|
|
## Valgrind
|
|
|
|
Valgrind provides detailed memory profiling, leak detection, and cache analysis.
|
|
|
|
### Memory Leak Detection
|
|
|
|
```bash
|
|
# Build profiling binary
|
|
./setup.sh Profiling
|
|
meson compile -C build/profiling
|
|
|
|
# Run with Valgrind
|
|
valgrind --leak-check=full \
|
|
--show-leak-kinds=all \
|
|
--track-origins=yes \
|
|
--verbose \
|
|
--log-file=valgrind-out.txt \
|
|
build/profiling/tools/yams-cli/yams search "test query"
|
|
|
|
# Check results
|
|
less valgrind-out.txt
|
|
```
|
|
|
|
### Callgrind (CPU Profiling)
|
|
|
|
```bash
|
|
# Run with callgrind
|
|
valgrind --tool=callgrind \
|
|
--dump-instr=yes \
|
|
--collect-jumps=yes \
|
|
build/profiling/tools/yams-cli/yams search "query"
|
|
|
|
# Visualize with kcachegrind
|
|
kcachegrind callgrind.out.*
|
|
```
|
|
|
|
### Massif (Heap Profiling)
|
|
|
|
```bash
|
|
# Track heap usage over time
|
|
valgrind --tool=massif \
|
|
build/profiling/tools/yams-cli/yams add large_directory/ --recursive
|
|
|
|
# Visualize with massif-visualizer
|
|
massif-visualizer massif.out.*
|
|
```
|
|
|
|
#### Daemon under Massif (startup + ingest)
|
|
|
|
1. Build the profiling daemon: `./setup.sh Profiling && meson compile -C build/profiling`
|
|
2. Run the daemon under Massif and give it a few minutes to bring up plugins, repair, and model loads:
|
|
```bash
|
|
timeout 600s valgrind --tool=massif --stacks=yes --time-unit=ms --detailed-freq=10 \
|
|
--max-snapshots=400 --massif-out-file=/tmp/massif.yams-daemon.out \
|
|
build/profiling/src/daemon/yams-daemon --foreground --log-level=info
|
|
```
|
|
3. While Massif runs, drive a workload from another shell (e.g., ingest a tree):
|
|
`build/profiling/tools/yams-cli/yams add /path/to/data --recursive`
|
|
4. Inspect results: `ms_print /tmp/massif.yams-daemon.out | less` or open in `massif-visualizer`.
|
|
|
|
Tips:
|
|
- Increase `timeout` and `--max-snapshots` for longer captures.
|
|
- `--time-unit=ms` helps correlate snapshots with daemon logs.
|
|
- If the profiling build is not available, you can substitute the installed daemon binary: replace the last line with `/usr/local/bin/yams-daemon --foreground --log-level=info` (features like TSAN may be disabled).
|
|
|
|
### Cachegrind (Cache Analysis)
|
|
|
|
```bash
|
|
# Analyze cache behavior
|
|
valgrind --tool=cachegrind \
|
|
build/profiling/tools/yams-cli/yams search "query"
|
|
|
|
# View cache statistics
|
|
cg_annotate cachegrind.out.*
|
|
```
|
|
|
|
## Perf (Linux Profiling)
|
|
|
|
For system-wide profiling on Linux:
|
|
|
|
```bash
|
|
# Record performance data
|
|
perf record -g build/profiling/tools/yams-cli/yams search "query"
|
|
|
|
# Generate report
|
|
perf report
|
|
|
|
# Generate flamegraph (requires flamegraph tools)
|
|
perf script | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg
|
|
```
|
|
|
|
## Fuzzing (AFL++)
|
|
|
|
YAMS includes AFL++ fuzz harnesses and a Docker-based runner:
|
|
|
|
```bash
|
|
./tools/fuzzing/fuzz.sh build
|
|
./tools/fuzzing/fuzz.sh fuzz ipc_roundtrip
|
|
```
|
|
|
|
This setup provides:
|
|
- AFL++ instrumentation for coverage-guided fuzzing
|
|
- libFuzzer-compat mode harnesses (`-fsanitize=fuzzer`)
|
|
- Sanitizer integration via `AFL_USE_ASAN=1` (ASan)
|
|
- Corpus + findings directories under `data/fuzz/`
|
|
|
|
Details and additional targets live in `tools/fuzzing/README.md`.
|
|
|
|
Target areas for fuzzing:
|
|
- Query parser (malformed queries)
|
|
- Content handlers (malicious file formats)
|
|
- IPC protocol (invalid protobuf messages)
|
|
- Metadata repository (SQL injection vectors)
|
|
|
|
## Build Comparison
|
|
|
|
| Build Type | Use Case | Overhead | Tools |
|
|
|------------|----------|----------|-------|
|
|
| `Debug` | Development | Low | GDB, basic profiling |
|
|
| `Release` | Production | None | Minimal instrumentation |
|
|
| `Profiling` | Performance analysis | Low-Medium | Tracy, Valgrind, Perf |
|
|
| `Fuzzing` | Security testing | High | AFL++ (Docker-based harnesses) |
|
|
|
|
## Best Practices
|
|
|
|
1. **Start with Tracy** for interactive, real-time profiling
|
|
2. **Use Valgrind** for deep memory analysis and leak detection
|
|
3. **Use Perf** for system-wide CPU profiling on Linux
|
|
4. **Profile realistic workloads** - synthetic benchmarks may not represent production
|
|
5. **Focus on hot paths** - optimize where it matters most
|
|
6. **Measure before and after** - validate optimizations with data
|
|
|
|
## Disabling Profiling
|
|
|
|
Regular builds have no profiling overhead:
|
|
|
|
```bash
|
|
./setup.sh Debug # Standard debug build
|
|
./setup.sh Release # Optimized production build
|
|
```
|
|
|
|
## References
|
|
|
|
- **Tracy**: https://github.com/wolfpld/tracy
|
|
- **Valgrind**: https://valgrind.org/docs/manual/
|
|
- **Perf**: https://perf.wiki.kernel.org/index.php/Tutorial
|
|
- **AFL++**: https://github.com/AFLplusplus/AFLplusplus
|
|
- **Flamegraphs**: https://www.brendangregg.com/flamegraphs.html
|