299 lines
8.4 KiB
Markdown
299 lines
8.4 KiB
Markdown
# YAMS Stress Tests
|
|
|
|
This directory contains comprehensive stress tests for validating YAMS behavior under extreme load conditions.
|
|
|
|
## Overview
|
|
|
|
The stress tests are designed to validate:
|
|
|
|
1. **Compression Stress Tests** (`compression_stress_test.cpp`)
|
|
- Multiple compression algorithms (Zstandard, LZ4, Gzip)
|
|
- Various compression levels (1, 3, 6, 9)
|
|
- Different data patterns (random, zeros, repetitive)
|
|
- Large file handling (1MB to 100MB)
|
|
- Sustained high-throughput compression
|
|
- Resource exhaustion scenarios
|
|
- Rapid compression/decompression cycles
|
|
- Memory leak detection
|
|
- Error injection and recovery
|
|
- Compression bomb protection
|
|
|
|
2. **Storage Concurrency Tests** (`storage/concurrent_test.cpp`)
|
|
- Concurrent readers (100, 500, 1000 threads)
|
|
- Concurrent writers (up to 100 threads)
|
|
- Mixed read/write workloads (80/20, 90/10, 50/50, 25/75 ratios)
|
|
- High contention scenarios (many threads, few objects)
|
|
- Variable object sizes (100B to 1MB)
|
|
- Burst traffic patterns
|
|
- Long-running sustained load (30+ seconds)
|
|
- Create/delete cycles
|
|
- Latency under load measurements
|
|
- Connection pool scaling
|
|
|
|
## Building Stress Tests
|
|
|
|
Stress tests are **disabled by default** to avoid long CI times. Enable them during build:
|
|
|
|
```bash
|
|
# Configure with stress tests enabled
|
|
meson setup builddir -Denable-stress-tests=true
|
|
|
|
# Or reconfigure existing build
|
|
meson configure builddir -Denable-stress-tests=true
|
|
|
|
# Build
|
|
meson compile -C builddir
|
|
```
|
|
|
|
## Running Stress Tests
|
|
|
|
### Basic Execution
|
|
|
|
Run all stress tests once:
|
|
|
|
```bash
|
|
meson test -C builddir stress --suite stress
|
|
```
|
|
|
|
### Running Tests Multiple Times
|
|
|
|
Meson provides a `--repeat` option to run tests in a loop:
|
|
|
|
```bash
|
|
# Run stress tests 10 times
|
|
meson test -C builddir stress --suite stress --repeat 10
|
|
|
|
# Run 100 times to detect intermittent issues
|
|
meson test -C builddir stress --suite stress --repeat 100
|
|
|
|
# Run indefinitely until failure (Ctrl+C to stop)
|
|
meson test -C builddir stress --suite stress --repeat 999999
|
|
```
|
|
|
|
### Advanced Options
|
|
|
|
```bash
|
|
# Run with verbose output
|
|
meson test -C builddir stress --suite stress --repeat 10 -v
|
|
|
|
# Stop on first failure
|
|
meson test -C builddir stress --suite stress --repeat 50 --maxfail 1
|
|
|
|
# Run without rebuilding (faster for repeated runs)
|
|
meson test -C builddir stress --suite stress --repeat 100 --no-rebuild
|
|
|
|
# Run specific test patterns
|
|
meson test -C builddir "ConcurrentStorageTest.ThousandConcurrentReaders" --repeat 20
|
|
|
|
# Run multiple specific tests
|
|
meson test -C builddir "CompressionStressTest.*" --repeat 10
|
|
```
|
|
|
|
### Parallel Test Execution
|
|
|
|
Run multiple test processes in parallel (use with caution on shared resources):
|
|
|
|
```bash
|
|
# Run with 4 parallel jobs
|
|
meson test -C builddir stress --suite stress --repeat 10 --num-processes 4
|
|
```
|
|
|
|
### Timeout Configuration
|
|
|
|
Stress tests have a 1800 second (30 minute) timeout by default. Adjust if needed:
|
|
|
|
```bash
|
|
# Extend timeout to 1 hour
|
|
meson test -C builddir stress --suite stress --repeat 10 --timeout-multiplier 2
|
|
```
|
|
|
|
## Test Categories
|
|
|
|
### Quick Stress (< 5 minutes each)
|
|
- `CompressionStressTest.ExtremelyLargeFiles`
|
|
- `CompressionStressTest.RapidCycles`
|
|
- `ConcurrentStorageTest.ThousandConcurrentReaders`
|
|
- `ConcurrentStorageTest.HundredConcurrentWriters`
|
|
|
|
### Moderate Stress (5-15 minutes each)
|
|
- `CompressionStressTest.SustainedHighThroughput`
|
|
- `ConcurrentStorageTest.MixedReadWriteWorkload`
|
|
- `ConcurrentStorageTest.HighContentionSameObjects`
|
|
- `ConcurrentStorageTest.VariableObjectSizesConcurrent`
|
|
|
|
### Heavy Stress (15+ minutes each)
|
|
- `CompressionStressTest.ResourceExhaustion`
|
|
- `ConcurrentStorageTest.LongRunningSustainedLoad`
|
|
- `ConcurrentStorageTest.ConnectionPoolWorkerSweep`
|
|
|
|
## Interpreting Results
|
|
|
|
### Success Criteria
|
|
|
|
All tests should:
|
|
- Complete without crashes or hangs
|
|
- Report zero data corruption or integrity failures
|
|
- Meet performance targets (logged in output)
|
|
- Show consistent behavior across multiple runs
|
|
|
|
### Performance Metrics
|
|
|
|
Key metrics reported:
|
|
- **Throughput**: Operations per second, MB/s
|
|
- **Latency**: Average, P50, P95, P99 response times
|
|
- **Concurrency**: Number of simultaneous operations
|
|
- **Resource usage**: Memory consumption, thread count
|
|
- **Error rates**: Failed operations, conflicts, retries
|
|
|
|
### Common Issues
|
|
|
|
**Intermittent Failures**: Run with `--repeat 100` to identify race conditions
|
|
|
|
**Timeout**: Increase with `--timeout-multiplier` or reduce test parameters
|
|
|
|
**Resource Exhaustion**: Reduce concurrent thread counts in test configuration
|
|
|
|
**High Error Rates**: Check system resources (memory, file descriptors, disk space)
|
|
|
|
## CI/CD Integration
|
|
|
|
For continuous integration, use selective stress testing:
|
|
|
|
```bash
|
|
# Quick smoke test (5 iterations, critical tests only)
|
|
meson test -C builddir "ConcurrentStorageTest.ThousandConcurrentReaders" --repeat 5
|
|
|
|
# Nightly full stress test
|
|
meson test -C builddir stress --suite stress --repeat 50 --maxfail 3
|
|
|
|
# Pre-release validation (extensive)
|
|
meson test -C builddir stress --suite stress --repeat 200 --maxfail 1
|
|
```
|
|
|
|
## System Resource Requirements
|
|
|
|
Stress tests are designed to **push system limits**. Some tests may exhaust system resources (this is intentional validation, not a bug):
|
|
|
|
### File Descriptor Limits
|
|
|
|
The `LongRunningSustainedLoad` test may exhaust file descriptors on systems with default limits. This validates that the storage engine properly handles resource exhaustion.
|
|
|
|
**Symptoms**: Errors like "Failed to create temp file" after thousands of operations
|
|
|
|
**Performance achieved before exhaustion**: ~500,000 ops/sec (excellent!)
|
|
|
|
**Solutions**:
|
|
```bash
|
|
# Check current limit
|
|
ulimit -n
|
|
|
|
# Temporarily increase (for current session)
|
|
ulimit -n 1048576
|
|
|
|
# Permanently increase (add to /etc/security/limits.conf)
|
|
* soft nofile 1048576
|
|
* hard nofile 1048576
|
|
```
|
|
|
|
### Expected Resource Usage
|
|
|
|
- **Memory**: Up to 4-8GB during sustained load tests
|
|
- **File Descriptors**: Up to 10,000+ for concurrent write tests
|
|
- **CPU**: Will saturate all available cores
|
|
- **Disk I/O**: Sustained writes of 100+ MB/s
|
|
|
|
## Debugging Failed Stress Tests
|
|
|
|
1. **Enable verbose logging**:
|
|
```bash
|
|
SPDLOG_LEVEL=debug meson test -C builddir stress --suite stress --repeat 10 -v
|
|
```
|
|
|
|
2. **Run under debugger**:
|
|
```bash
|
|
meson test -C builddir "TestName" --gdb
|
|
```
|
|
|
|
3. **Isolate the failing test**:
|
|
```bash
|
|
# Run just the failing test with high repeat count
|
|
meson test -C builddir "FailingTest" --repeat 100 --verbose
|
|
```
|
|
|
|
4. **Check system resources**:
|
|
```bash
|
|
# Monitor during test execution
|
|
watch -n 1 'free -h; echo; ps aux | grep yams_stress | head -20'
|
|
|
|
# Check file descriptor usage
|
|
lsof -p $(pgrep yams_stress) | wc -l
|
|
```
|
|
|
|
5. **Resource exhaustion is a success**: If tests fail due to "out of file descriptors" or similar OS limits after completing millions of operations, **this validates the stress test is working correctly**. The system reached its limit before the code did.
|
|
|
|
## Customizing Stress Tests
|
|
|
|
### Environment Variables
|
|
|
|
Some tests support environment variables for customization:
|
|
|
|
```bash
|
|
# FTS5 performance tests
|
|
export YAMS_RUN_STRESS_TESTS=1
|
|
export YAMS_STRESS_DOC_COUNT=100000
|
|
export YAMS_STRESS_TIMEOUT_MS=10000
|
|
|
|
meson test -C builddir "FTS5StressTest.*"
|
|
```
|
|
|
|
### Modifying Test Parameters
|
|
|
|
Edit test source files to adjust:
|
|
- Thread counts
|
|
- Operation counts
|
|
- Data sizes
|
|
- Test durations
|
|
- Workload ratios
|
|
|
|
After modifications, rebuild and rerun:
|
|
|
|
```bash
|
|
meson compile -C builddir
|
|
meson test -C builddir stress --suite stress --repeat 10
|
|
```
|
|
|
|
## Performance Baselines
|
|
|
|
Expected performance on a modern workstation (adjust for your hardware):
|
|
|
|
- **Compression**:
|
|
- Zstandard level 1: ~500 MB/s
|
|
- Zstandard level 6: ~100 MB/s
|
|
- LZ4: ~1000 MB/s
|
|
|
|
- **Storage**:
|
|
- Concurrent reads: >10,000 ops/sec with 1000 threads
|
|
- Concurrent writes: >1,000 ops/sec with 100 threads
|
|
- Mixed workload: >8,000 ops/sec
|
|
|
|
- **Latency**:
|
|
- P99 read latency: < 10ms under load
|
|
- P99 write latency: < 50ms under load
|
|
|
|
## Contributing
|
|
|
|
When adding new stress tests:
|
|
|
|
1. Place in appropriate subdirectory (`compression`, `storage`, etc.)
|
|
2. Add to `tests/stress/meson.build` or subdirectory build file
|
|
3. Use descriptive test names: `ComponentName.TestScenario`
|
|
4. Test multiple scenarios with loops/iterations within the test
|
|
5. Log meaningful metrics with `spdlog::info()`
|
|
6. Document in this README
|
|
7. Validate with `--repeat 100` before committing
|
|
|
|
## See Also
|
|
|
|
- [Main test README](../README.md)
|
|
- [Benchmark documentation](../../docs/benchmarks/README.md)
|
|
- [Test helpers](../common/)
|