Ab Test Store Listing

Limitations:

• Only tests against organic App Store traffic
• Minimum 90% confidence required to declare winner
• Tests run for 7-90 days
• Can only run one test at a time
• Traffic split is automatic (not configurable)

Custom Product Pages (CPP)

35 custom product pages per app, each with unique:

• Screenshots
• App preview videos
• Promotional text

Use for:

• Different audiences (from different ad campaigns)
• Different value propositions
• Seasonal messaging
• Localized creative for specific markets

Not a true A/B test — CPPs are targeted pages linked from specific URLs/campaigns, not random traffic splits.

Test Prioritization

Impact × Effort Matrix

What to Test First

Always start with the first screenshot. It has the highest impact because:

• It's the first thing users see in search results
• 80% of users never scroll past the first 3 screenshots
• Small improvements here affect every visitor

Test Design Framework

Step 1: Hypothesis

Write a clear hypothesis before each test:

If we [change], then [metric] will [improve/increase] because [reason].

Examples:

• "If we add social proof ('5M+ users') to the first screenshot, conversion rate will increase because it builds trust"
• "If we change the icon from blue to orange, tap-through rate will increase because it stands out more in search results"
• "If we show the app's AI feature first instead of the basic editor, conversion will increase because AI is the key differentiator"

Step 2: Variants

Design 2-3 variants (including control):

Rules for good variants:

• Change ONE thing per test (isolate the variable)
• Make the change significant enough to detect (don't test subtle color shifts)
• Each variant should have a clear hypothesis
• Don't test more than 3 variants (dilutes traffic)

Step 3: Sample Size

Calculate required test duration:

Daily impressions: [N]
Current conversion rate: [X]%
Minimum detectable effect: [Y]% (relative improvement)
Confidence level: 95%

Required sample per variant: ~[N] impressions
Estimated duration: [N] days

Rules of thumb:

• < 1000 daily impressions: Tests take 30-90 days (consider if worth it)
• 1000-5000 daily impressions: Tests take 14-30 days
• 5000+ daily impressions: Tests take 7-14 days
• Need at least 1000 impressions per variant for meaningful results

Step 4: Run the Test

In App Store Connect:

1. Go to Product Page Optimization
2. Create a new test
3. Upload variant assets
4. Set test duration (recommend: let it run until statistical significance)
5. Monitor but don't stop early

Step 5: Interpret Results

Statistical significance:

• Apple requires 90% confidence minimum
• Aim for 95% confidence before making decisions
• Look at the confidence interval, not just the point estimate

What to look for:

• Conversion rate lift (primary metric)
• Impression-to-tap rate (for icon tests)
• Download rate (for screenshot/video tests)
• Segment differences (new vs returning, country, source)

Common Test Ideas

Icon Tests

Screenshot Tests

Video Tests

Output Format

Test Plan

Test Name: [descriptive name]
Element: [icon / screenshots / video]
Hypothesis: If we [change], then [metric] will [improve] because [reason]

Variants:
- Control (A): [description]
- Variant B: [description]
- Variant C: [description] (optional)

Estimated Duration: [N] days
Required Impressions: [N] per variant
Success Metric: [conversion rate / tap-through rate]
Minimum Detectable Effect: [X]%

Test Results Interpretation

When the user shares results:

1. Is it statistically significant? (confidence level)
2. What's the actual lift? (with confidence interval)
3. Are there segment differences?
4. What's the next test to run?
5. Estimated annual impact (downloads × lift)

Testing Roadmap

Provide a 3-month testing calendar:

• Month 1: [highest impact test]
• Month 2: [second priority test]
• Month 3: [third priority test]

Related Skills

• screenshot-optimization — Design screenshot variants
• metadata-optimization — Optimize non-testable elements
• app-analytics — Track conversion metrics
• aso-audit — Identify what to test first