yams/tests/stress/README.md

YAMS Stress Tests

This directory contains comprehensive stress tests for validating YAMS behavior under extreme load conditions.

Overview

The stress tests are designed to validate:

1. Compression Stress Tests (compression_stress_test.cpp)

   • Multiple compression algorithms (Zstandard, LZ4, Gzip)
   • Various compression levels (1, 3, 6, 9)
   • Different data patterns (random, zeros, repetitive)
   • Large file handling (1MB to 100MB)
   • Sustained high-throughput compression
   • Resource exhaustion scenarios
   • Rapid compression/decompression cycles
   • Memory leak detection
   • Error injection and recovery
   • Compression bomb protection

2. Storage Concurrency Tests (storage/concurrent_test.cpp)

   • Concurrent readers (100, 500, 1000 threads)
   • Concurrent writers (up to 100 threads)
   • Mixed read/write workloads (80/20, 90/10, 50/50, 25/75 ratios)
   • High contention scenarios (many threads, few objects)
   • Variable object sizes (100B to 1MB)
   • Burst traffic patterns
   • Long-running sustained load (30+ seconds)
   • Create/delete cycles
   • Latency under load measurements
   • Connection pool scaling

Building Stress Tests

Stress tests are disabled by default to avoid long CI times. Enable them during build:

# Configure with stress tests enabled
meson setup builddir -Denable-stress-tests=true

# Or reconfigure existing build
meson configure builddir -Denable-stress-tests=true

# Build
meson compile -C builddir

Running Stress Tests

Basic Execution

Run all stress tests once:

meson test -C builddir stress --suite stress

Running Tests Multiple Times

Meson provides a --repeat option to run tests in a loop:

# Run stress tests 10 times
meson test -C builddir stress --suite stress --repeat 10

# Run 100 times to detect intermittent issues
meson test -C builddir stress --suite stress --repeat 100

# Run indefinitely until failure (Ctrl+C to stop)
meson test -C builddir stress --suite stress --repeat 999999

Advanced Options

# Run with verbose output
meson test -C builddir stress --suite stress --repeat 10 -v

# Stop on first failure
meson test -C builddir stress --suite stress --repeat 50 --maxfail 1

# Run without rebuilding (faster for repeated runs)
meson test -C builddir stress --suite stress --repeat 100 --no-rebuild

# Run specific test patterns
meson test -C builddir "ConcurrentStorageTest.ThousandConcurrentReaders" --repeat 20

# Run multiple specific tests
meson test -C builddir "CompressionStressTest.*" --repeat 10

Parallel Test Execution

Run multiple test processes in parallel (use with caution on shared resources):

# Run with 4 parallel jobs
meson test -C builddir stress --suite stress --repeat 10 --num-processes 4

Timeout Configuration

Stress tests have a 1800 second (30 minute) timeout by default. Adjust if needed:

# Extend timeout to 1 hour
meson test -C builddir stress --suite stress --repeat 10 --timeout-multiplier 2

Test Categories

Quick Stress (< 5 minutes each)

• CompressionStressTest.ExtremelyLargeFiles
• CompressionStressTest.RapidCycles
• ConcurrentStorageTest.ThousandConcurrentReaders
• ConcurrentStorageTest.HundredConcurrentWriters

Moderate Stress (5-15 minutes each)

• CompressionStressTest.SustainedHighThroughput
• ConcurrentStorageTest.MixedReadWriteWorkload
• ConcurrentStorageTest.HighContentionSameObjects
• ConcurrentStorageTest.VariableObjectSizesConcurrent

Heavy Stress (15+ minutes each)

• CompressionStressTest.ResourceExhaustion
• ConcurrentStorageTest.LongRunningSustainedLoad
• ConcurrentStorageTest.ConnectionPoolWorkerSweep

Interpreting Results

Success Criteria

All tests should:

• Complete without crashes or hangs
• Report zero data corruption or integrity failures
• Meet performance targets (logged in output)
• Show consistent behavior across multiple runs

Performance Metrics

Key metrics reported:

• Throughput: Operations per second, MB/s
• Latency: Average, P50, P95, P99 response times
• Concurrency: Number of simultaneous operations
• Resource usage: Memory consumption, thread count
• Error rates: Failed operations, conflicts, retries

Common Issues

Intermittent Failures: Run with --repeat 100 to identify race conditions

Timeout: Increase with --timeout-multiplier or reduce test parameters

Resource Exhaustion: Reduce concurrent thread counts in test configuration

High Error Rates: Check system resources (memory, file descriptors, disk space)

CI/CD Integration

For continuous integration, use selective stress testing:

# Quick smoke test (5 iterations, critical tests only)
meson test -C builddir "ConcurrentStorageTest.ThousandConcurrentReaders" --repeat 5

# Nightly full stress test
meson test -C builddir stress --suite stress --repeat 50 --maxfail 3

# Pre-release validation (extensive)
meson test -C builddir stress --suite stress --repeat 200 --maxfail 1

System Resource Requirements

Stress tests are designed to push system limits. Some tests may exhaust system resources (this is intentional validation, not a bug):

File Descriptor Limits

The LongRunningSustainedLoad test may exhaust file descriptors on systems with default limits. This validates that the storage engine properly handles resource exhaustion.

Symptoms: Errors like "Failed to create temp file" after thousands of operations

Performance achieved before exhaustion: ~500,000 ops/sec (excellent!)

Solutions:

# Check current limit
ulimit -n

# Temporarily increase (for current session)
ulimit -n 1048576

# Permanently increase (add to /etc/security/limits.conf)
* soft nofile 1048576
* hard nofile 1048576

Expected Resource Usage

• Memory: Up to 4-8GB during sustained load tests
• File Descriptors: Up to 10,000+ for concurrent write tests
• CPU: Will saturate all available cores
• Disk I/O: Sustained writes of 100+ MB/s

Debugging Failed Stress Tests

1. Enable verbose logging:

   SPDLOG_LEVEL=debug meson test -C builddir stress --suite stress --repeat 10 -v
   

2. Run under debugger:

   meson test -C builddir "TestName" --gdb
   

3. Isolate the failing test:

   # Run just the failing test with high repeat count
   meson test -C builddir "FailingTest" --repeat 100 --verbose
   

4. Check system resources:

   # Monitor during test execution
   watch -n 1 'free -h; echo; ps aux | grep yams_stress | head -20'
   
   # Check file descriptor usage
   lsof -p $(pgrep yams_stress) | wc -l
   

5. Resource exhaustion is a success: If tests fail due to "out of file descriptors" or similar OS limits after completing millions of operations, this validates the stress test is working correctly. The system reached its limit before the code did.

Customizing Stress Tests

Environment Variables

Some tests support environment variables for customization:

# FTS5 performance tests
export YAMS_RUN_STRESS_TESTS=1
export YAMS_STRESS_DOC_COUNT=100000
export YAMS_STRESS_TIMEOUT_MS=10000

meson test -C builddir "FTS5StressTest.*"

Modifying Test Parameters

Edit test source files to adjust:

• Thread counts
• Operation counts
• Data sizes
• Test durations
• Workload ratios

After modifications, rebuild and rerun:

meson compile -C builddir
meson test -C builddir stress --suite stress --repeat 10

Performance Baselines

Expected performance on a modern workstation (adjust for your hardware):

• Compression:

  • Zstandard level 1: ~500 MB/s
  • Zstandard level 6: ~100 MB/s
  • LZ4: ~1000 MB/s

• Storage:

  • Concurrent reads: >10,000 ops/sec with 1000 threads
  • Concurrent writes: >1,000 ops/sec with 100 threads
  • Mixed workload: >8,000 ops/sec

• Latency:

  • P99 read latency: < 10ms under load
  • P99 write latency: < 50ms under load

Contributing

When adding new stress tests:

1. Place in appropriate subdirectory (compression, storage, etc.)
2. Add to tests/stress/meson.build or subdirectory build file
3. Use descriptive test names: ComponentName.TestScenario
4. Test multiple scenarios with loops/iterations within the test
5. Log meaningful metrics with spdlog::info()
6. Document in this README
7. Validate with --repeat 100 before committing

See Also

• Main test README
• Benchmark documentation
• Test helpers