Load Testing Procedures

Test types and when to use them:

Step 2 — Tool Selection

Instinct SRE standard: k6

Standard k6 install:

# Windows
choco install k6
# Or via npm (project-local)
npm install -g k6

Step 3 — Establish Baseline

Run a baseline before any optimisation or change. The baseline is the control measurement.

// baseline.js — k6 smoke test
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 1,          // 1 virtual user
  duration: '30s', // for 30 seconds
  thresholds: {
    http_req_duration: ['p99<500'],  // 99th percentile under 500ms
    http_req_failed: ['rate<0.01'],  // <1% error rate
  },
};

export default function () {
  const res = http.get('https://dev.instinctsre.ai/api/health');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 200ms': (r) => r.timings.duration < 200,
  });
  sleep(1);
}

k6 run baseline.js

Record the baseline: note p50, p95, p99, error rate, and RPS achieved. Store in .squad/docs/performance/baselines/{service}-{date}.md.

Step 4 — Define Thresholds

Thresholds must be derived from SLO targets (see slo-sli-sla-definitions skill):

export const options = {
  thresholds: {
    // From SLO: p99 latency < 500ms for SRE Agent API
    http_req_duration: ['p(99)<500', 'p(95)<200', 'p(50)<100'],
    // From SLO: < 0.1% error rate
    http_req_failed: ['rate<0.001'],
    // From SLO: > 99.9% requests handled
    checks: ['rate>0.999'],
  },
};

Threshold failure = SLO risk. A load test that fails its thresholds means the service will breach its SLO under that traffic level.

Step 5 — Execute the Load Test

Standard load test ramp pattern:

export const options = {
  stages: [
    { duration: '2m', target: 10 },   // Ramp up to 10 VUs
    { duration: '5m', target: 10 },   // Hold at 10 VUs
    { duration: '2m', target: 50 },   // Ramp to 50 VUs (stress)
    { duration: '5m', target: 50 },   // Hold at 50 VUs
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: { /* from step 4 */ },
};

Execution:

# Run and output results to JSON for reporting
k6 run --out json=results.json load-test.js

# Run with real-time metrics (Azure-compatible)
k6 run --out statsd load-test.js

Step 6 — Analyse and Report

Metrics to always include in the report:

Report format:

# Load Test Report — {Service} — {Date}

**Test type:** Load / Stress / Soak / Spike
**Duration:** X minutes
**Peak VUs:** X
**Tool:** k6 vX.X.X

## Results Summary
| Metric | Target | Actual | Pass/Fail |
...

## Observations
- [Notable behaviour at specific load levels]
- [Any errors observed and their types]
- [Performance cliff (if found)]

## Recommendations
- [Changes needed to meet SLO under target load]
- [Infrastructure scaling required]

## Attached
- results.json — full k6 output

Store report in .squad/docs/performance/reports/{service}-{date}.md.

Examples

Full k6 Load Test for SRE Agent API

// sre-agent-load.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';

const errorRate = new Rate('errors');

export const options = {
  stages: [
    { duration: '1m', target: 5 },
    { duration: '3m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(99)<500'],
    errors: ['rate<0.01'],
  },
};

const BASE_URL = __ENV.BASE_URL || 'https://api.instinctsre.ai';

export default function () {
  // Health check
  const health = http.get(`${BASE_URL}/health`);
  errorRate.add(health.status !== 200);
  check(health, { 'health 200': (r) => r.status === 200 });

  sleep(0.5);
}

Gotchas

• Do not run load tests against prod without explicit approval — load tests can trigger autoscaling costs, trigger alert firing, and consume error budget. Always test in dev/lab first
• Azure SWA does not need load testing — it's a fully managed CDN-backed service. Load testing it is testing Azure's infrastructure, not ours
• Thresholds without a baseline are arbitrary — always establish a baseline before setting thresholds. "p99 < 500ms" means nothing if the service has never been measured
• k6 VUs ≠ concurrent users — 1 k6 VU runs one iteration at a time (with sleep()); 10 VUs with no sleep can generate 100+ RPS. Always define think time with sleep()
• Load test environment must match prod — running against a Free-tier dev service will produce results that are not predictive of Standard-tier prod performance. Match SKU for meaningful results
• Soak tests need monitoring — a 4-hour soak test without monitoring is useless. Stream metrics to Azure Monitor or at minimum check memory/CPU at the end and at the midpoint