Usability Testing

Derivation Logic

TL;DR

1. AI-First Principles

Traditional vs AI-First

Speed Advantages

AI handles:
✅ Real-time transcription
✅ Automatic summarization
✅ Friction pattern detection
✅ Fix suggestions

Humans focus on:
✅ Strategy and interpretation
✅ Nuanced insights
✅ Product decisions
✅ Stakeholder communication

2. Tools 2025-2026

Unmoderated Testing

Heuristic Evaluation

A/B Testing

3. Unmoderated Testing

When to Use

✅ Good for:
- Task completion testing
- Navigation and findability
- First-click testing
- Preference testing
- Quick feedback on prototypes

❌ Not for:
- Complex emotional insights
- "Why" questions
- Exploratory research

Maze AI Workflow

1. Upload prototype (Figma, URL, or images)
2. Define tasks and success paths
3. Write clear task prompts
4. Set completion criteria
5. Launch (results in 1-24 hours)
6. AI summarizes findings
7. Fix issues same day
8. Retest

Task Prompt Template:

You are booking a flight for next Friday.

Starting from the homepage, find a flight from 
New York to Los Angeles and complete the booking.

When you're done, click "Task Complete."

Key Metrics

4. Moderated Testing

When to Use

✅ Good for:
- Complex flows
- Sensitive topics
- Need follow-up questions
- Exploratory research
- Early concept testing

AI Assistance During Sessions

Tools

• Lookback — AI-powered note taking
• Zoom + Otter.ai — Transcription
• Dovetail — Analysis and synthesis
• UserZoom — Enterprise moderated

Session Script Template

## Intro (2 min)
- Thanks for joining
- Recording notice + consent
- No right/wrong answers
- Think aloud

## Warm-up (3 min)
- Tell me about your experience with [domain]
- How do you currently [solve problem]?

## Tasks (20-30 min)
- Task 1: [Scenario and goal]
  - Probing: What did you expect to happen?
  - Follow-up: What would make this easier?

- Task 2: [Scenario and goal]
  - ...

## Wrap-up (5 min)
- Overall impression?
- Anything confusing?
- What would you change?
- Any questions for me?

5. Heuristic Evaluation

Nielsen's 10 Heuristics (Quick Reference)

AI-Enhanced Workflow

1. Screenshot prototype
2. Run through Attention Insight (heatmap prediction)
3. Identify attention gaps
4. Ask Claude for heuristic review:

   "Review this UI screenshot against Nielsen's 10 
   usability heuristics. For each issue found, rate 
   severity (1-4) and suggest a fix."

5. Prioritize fixes by severity
6. Test with real users to validate

6. A/B Testing

When to Use

✅ Good for:
- Quantitative validation
- High-traffic features
- Before major launches
- Comparing two clear options

❌ Not for:
- Low traffic (no statistical power)
- Radically different concepts
- When you need "why" insights

Implementation (PostHog)

// Feature flag setup
import { usePostHog } from 'posthog-js/react'

function CheckoutButton() {
  const posthog = usePostHog()
  
  // Get variant
  const variant = posthog.getFeatureFlag('checkout-button-experiment')
  
  if (variant === 'control') {
    return <button>Checkout</button>
  }
  
  if (variant === 'treatment') {
    return <button className="bg-green-500">Complete Purchase →</button>
  }
}

// Track conversion
function handlePurchase() {
  posthog.capture('purchase_completed', {
    revenue: cart.total,
    items: cart.items.length,
  })
}

Statistical Significance

Minimum sample size = 16 × (p × (1-p)) / MDE²

Where:
- p = baseline conversion rate
- MDE = minimum detectable effect

Example:
- 5% baseline conversion
- Want to detect 10% improvement (0.5% absolute)
- Need ~6,000 per variant

7. Testing AI Features

Unique Considerations

What to Measure

interface AIFeatureMetrics {
  // Task success
  taskCompletion: number       // % who achieved goal
  timeToComplete: number       // Seconds
  aiAssistUsage: number        // % who used AI feature
  
  // Quality
  outputAccuracy: number       // % correct outputs
  userSatisfaction: number     // 1-5 rating
  trustScore: number           // "I trust this AI" 1-5
  
  // Error handling
  errorRate: number            // % that showed errors
  errorRecoveryRate: number    // % who recovered
  
  // Comparison
  aiVsManual: {
    speed: number              // AI time / manual time
    accuracy: number           // AI accuracy / manual accuracy
  }
}

Test Scenarios

1. Happy Path
   - AI gives correct answer first try
   - User achieves goal quickly

2. Partial Success
   - AI gives mostly correct answer
   - User needs to edit/refine

3. AI Failure
   - AI gives wrong or no answer
   - Can user recover?

4. Edge Cases
   - Unusual inputs
   - Long/short prompts
   - Ambiguous requests

8. Rapid Iteration Protocol

Same-Day Cycle

Morning: