Soak Testing

Soak testing, also called endurance testing, evaluates system stability and reliability over extended periods under sustained load. It identifies issues like memory leaks, resource exhaustion, and gradual performance degradation.

Purpose

Soak tests help you:

Identify memory leaks and resource exhaustion
Detect gradual performance degradation over time
Verify database connection pool management
Test log rotation and disk space management
Validate that systems remain stable over hours or days
Find issues that only appear after extended runtime

Soak tests require significant time investment but are crucial for production readiness. They catch issues that shorter tests miss.

Configuration Pattern

Soak tests use moderate load for extended duration:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },  // Ramp up to 100 users
    { duration: '3h56m', target: 100 }, // Stay at 100 for ~4 hours
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function() {
  const res = http.get('https://quickpizza.grafana.com');
  check(res, {
    'status is 200': (r) => r.status === 200,
  });
  sleep(1);
}

Load Level

80-100% of normal capacity

Duration

Minimum 4-8 hours

Goal

Stable performance throughout

Using the Constant VUs Executor

The constant-vus executor maintains steady load:

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-vus',
      vus: 100,
      duration: '4h',
    },
  },
};

Using the Ramping VUs Executor

For controlled ramp-up and ramp-down:

export const options = {
  scenarios: {
    soak: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '5m', target: 100 },   // Ramp up
        { duration: '8h', target: 100 },   // Soak
        { duration: '5m', target: 0 },     // Ramp down
      ],
      gracefulRampDown: '30s',
    },
  },
};

Soak Test Stages

Ramp Up

Gradually increase to target load over 5-10 minutes.

{ duration: '5m', target: 100 },

Sustained Load

Maintain constant load for extended period (4-24 hours).

{ duration: '8h', target: 100 },

Ramp Down

Gracefully reduce load and monitor cleanup.

{ duration: '5m', target: 0 },

Soak Test Duration Guidelines

Minimum

4 hours - catches most common issues

Extended

24-72 hours - production simulation

Weekend

Full weekend - ultimate confidence

Realistic Soak Test Example

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Counter, Trend } from 'k6/metrics';

const errorRate = new Counter('errors');
const responseTime = new Trend('response_time');

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-vus',
      vus: 100,
      duration: '6h',
    },
  },
  thresholds: {
    errors: ['count<100'],
    response_time: ['p(95)<1000', 'p(99)<2000'],
    http_req_duration: ['p(95)<500'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function() {
  // Simulate realistic user behavior
  
  // Browse homepage
  let res = http.get('https://quickpizza.grafana.com');
  check(res, { 'homepage loaded': (r) => r.status === 200 });
  sleep(2);
  
  // View product list
  res = http.get('https://quickpizza.grafana.com/api/pizza');
  if (!check(res, { 'pizza list loaded': (r) => r.status === 200 })) {
    errorRate.add(1);
  }
  responseTime.add(res.timings.duration);
  sleep(3);
  
  // View specific product
  res = http.get('https://quickpizza.grafana.com/api/pizza/1');
  check(res, { 'pizza detail loaded': (r) => r.status === 200 });
  sleep(2);
  
  // Add to cart
  res = http.post('https://quickpizza.grafana.com/api/cart',
    JSON.stringify({ pizzaId: 1, quantity: 2 }),
    { headers: { 'Content-Type': 'application/json' } }
  );
  check(res, { 'added to cart': (r) => r.status === 200 });
  sleep(1);
}

Include realistic think times (sleep) between requests to simulate actual user behavior and avoid overwhelming the system unnecessarily.

Using Constant Arrival Rate

Maintain consistent request rate:

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: 50,             // 50 iterations per second
      timeUnit: '1s',
      duration: '8h',
      preAllocatedVUs: 50,
      maxVUs: 200,          // Allow scaling if needed
    },
  },
};

What to Monitor

Time-Based Metrics

Track how metrics change over time:

import { Trend, Counter, Gauge } from 'k6/metrics';

const memoryUsage = new Gauge('memory_usage');
const activeSessions = new Gauge('active_sessions');
const cumulativeErrors = new Counter('cumulative_errors');

export default function() {
  // Your test logic
  
  // Track metrics over time
  // (These would typically come from system monitoring)
}

Critical Indicators

Memory Usage

Should remain stable, not gradually increase

Response Times

Should not degrade over time

Error Rates

Should remain consistently low

Resource Utilization

CPU, memory, disk should be stable

Best Practices

Choose Appropriate Load

Don’t use maximum capacity for soak tests:

// Good: 80% of normal capacity
export const options = {
  stages: [
    { duration: '5m', target: 80 },
    { duration: '8h', target: 80 },
    { duration: '5m', target: 0 },
  ],
};

// Too aggressive: May cause premature failures
export const options = {
  stages: [
    { duration: '5m', target: 200 },  // Peak capacity
    { duration: '8h', target: 200 },  // Too stressful for soak
    { duration: '5m', target: 0 },
  ],
};

Soak tests should use sustainable load levels (80-100% of normal capacity), not maximum stress levels.

Monitor Throughout the Test

Track metrics at regular intervals:

import { check, sleep } from 'k6';
import http from 'k6/http';

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-vus',
      vus: 100,
      duration: '6h',
    },
  },
};

export default function() {
  const res = http.get('https://quickpizza.grafana.com');
  
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
    'response time < 1s': (r) => r.timings.duration < 1000,
    'response time < 2s': (r) => r.timings.duration < 2000,
  });
  
  sleep(1);
}

Compare First and Last Hours

Key analysis: Does performance degrade?

export const options = {
  thresholds: {
    // Compare early vs late performance
    'http_req_duration{phase:early}': ['p(95)<500'],
    'http_req_duration{phase:late}': ['p(95)<500'],
    
    // Error rates should be consistent
    'http_req_failed{phase:early}': ['rate<0.01'],
    'http_req_failed{phase:late}': ['rate<0.01'],
  },
};

Common Issues Found

Memory Leaks

Memory leaks manifest as:

Gradually increasing memory usage
Eventual OutOfMemory errors
System becomes unresponsive
Requires restart to recover

Resource Exhaustion

Watch for:

Database connection pool exhaustion
File descriptor limits
Thread pool saturation
Disk space depletion from logs

Gradual Performance Degradation

Signs of degradation:

Response times increase over hours
Throughput decreases over time
Error rates gradually increase
Resource utilization grows unbounded

Analysis Tips

Create Time-Series Graphs

Visualize metrics over the full duration:

Response time (p50, p95, p99) over time
Error rate over time
Throughput over time
Active VUs over time

Look for Patterns

Hour 1

Baseline performance after initial ramp-up

Hours 2-4

Should remain stable and consistent

Hours 4-8

Watch for gradual degradation

Final Hour

Compare against Hour 1 baseline

Success Criteria

export const options = {
  thresholds: {
    // Performance remains consistent
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
    
    // Low error rate throughout
    http_req_failed: ['rate<0.01'],
    
    // High check success rate
    checks: ['rate>0.99'],
  },
};

A successful soak test means:

Performance metrics remain stable throughout
No memory leaks or resource exhaustion
Error rates stay consistently low
System recovers cleanly after test ends

When to Use

Pre-production: Before major releases
After infrastructure changes: Validate stability
Performance regression testing: Compare releases
Production simulation: Weekend-long tests
Compliance requirements: Prove reliability

Shorter Soak Tests

If 8+ hours isn’t feasible, run shorter tests:

export const options = {
  scenarios: {
    short_soak: {
      executor: 'constant-vus',
      vus: 100,
      duration: '2h', // Minimum 2 hours
    },
  },
};

Even a 2-hour soak test is better than no soak test. It will catch many common issues, though longer tests provide more confidence.

Production-Like Soak Test

import http from 'k6/http';
import { check, sleep } from 'k6';
import { randomIntBetween } from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

export const options = {
  scenarios: {
    soak: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '10m', target: 100 },   // Morning ramp-up
        { duration: '6h', target: 100 },    // Day time load
        { duration: '2h', target: 150 },    // Afternoon peak
        { duration: '2h', target: 80 },     // Evening decline
        { duration: '10m', target: 0 },     // Night shutdown
      ],
    },
  },
};

export default function() {
  // Vary behavior to simulate different users
  const scenario = randomIntBetween(1, 3);
  
  if (scenario === 1) {
    // Browser user
    http.get('https://quickpizza.grafana.com');
    sleep(randomIntBetween(2, 5));
  } else if (scenario === 2) {
    // API user
    http.get('https://quickpizza.grafana.com/api/pizza');
    sleep(randomIntBetween(1, 3));
  } else {
    // Heavy user
    http.batch([
      ['GET', 'https://quickpizza.grafana.com'],
      ['GET', 'https://quickpizza.grafana.com/api/pizza'],
    ]);
    sleep(randomIntBetween(3, 7));
  }
}

​Purpose

​Configuration Pattern

Load Level

Duration

Goal

​Using the Constant VUs Executor

​Using the Ramping VUs Executor

​Soak Test Stages

​Soak Test Duration Guidelines

Minimum

Recommended

Extended

Weekend

​Realistic Soak Test Example

​Using Constant Arrival Rate

​What to Monitor

​Time-Based Metrics

​Critical Indicators

Memory Usage

Response Times

Error Rates

Resource Utilization

​Best Practices

​Choose Appropriate Load

​Monitor Throughout the Test

​Compare First and Last Hours

​Common Issues Found

​Memory Leaks

​Resource Exhaustion

​Gradual Performance Degradation

​Analysis Tips

​Create Time-Series Graphs

​Look for Patterns

​Success Criteria

​When to Use

​Shorter Soak Tests

​Production-Like Soak Test

Purpose

Configuration Pattern

Using the Constant VUs Executor

Using the Ramping VUs Executor

Soak Test Stages

Soak Test Duration Guidelines

Realistic Soak Test Example

Using Constant Arrival Rate

What to Monitor

Time-Based Metrics

Critical Indicators

Best Practices

Choose Appropriate Load

Monitor Throughout the Test

Compare First and Last Hours

Common Issues Found

Memory Leaks

Resource Exhaustion

Gradual Performance Degradation

Analysis Tips

Create Time-Series Graphs

Look for Patterns

Success Criteria

When to Use

Shorter Soak Tests

Production-Like Soak Test