Persona Pipeline

Stage 1: Skeleton Generation

• Creates demographic skeletons with archetype assignments
• Samples OCEAN traits from archetype ranges
• Ensures archetype diversity (max 2 per archetype for small batches)
• Output: data/personas/skeletons/*.json

Stage 2: Deep Research Submission

• Submits background research jobs to Gemini Deep Research agent
• Researches real-world UK demographics for occupation/location
• ~10-20 minutes per job (runs in background)
• Quota-limited: Submit in waves (6→4→2 pattern typical)
• Output: data/personas/jobs/*_jobs.json (job IDs)

Stage 3: Research Job Polling

• Polls Deep Research jobs until completion
• Status: processing → in_progress → completed
• Retrieves completed reports (20-30KB each)
• Output: data/personas/reports/*_reports.json

Stage 4: Backstory Synthesis

• Uses Claude Opus 4.5 to generate narrative backstories
• Applies World Bible constraints (in-universe brands)
• Batch synthesis: single query generates multiple personas
• Incremental pattern: only synthesize NEW personas
• Output: data/personas/library/*_personas.json

Stage 5: Validation

• Validates persona quality and completeness
• Detects forbidden brand leakage (Starbucks, Facebook, etc.)
• Checks OCEAN/stats ranges
• Verifies required fields present
• FAILS FAST if validation errors (no fallback)

Instructions

Full Pipeline Execution

When asked to generate a complete persona library:

1. Determine batch size (recommend 15-20 for testing, 100+ for production)

2. Run the master orchestration script:

   ./scripts/generate_persona_library.sh --count 15
   

3. Monitor quota limits:

   • Deep Research quota follows pattern: 6→4→2→2 jobs per wave
   • If quota errors occur, wait 5-10 minutes and retry
   • See "Quota Management" section below

4. Verify results:

   • Check validation output for 100% pass rate
   • Verify 0 brand leaks detected
   • Confirm archetype distribution is diverse

Incremental Generation (Adding to Existing Library)

When asked to add more personas to an existing library:

1. Load existing personas:

   with open('data/personas/library/test_batch_001_personas.json') as f:
       existing_personas = json.load(f)
   existing_ids = {p['skeleton_id'] for p in existing_personas}
   

2. Generate new skeletons (avoiding duplicate IDs)

3. Submit Deep Research jobs for new skeletons only

4. Synthesize only new personas:

   # Extract only new reports
   python -c "import json; ..."  # Filter to new reports
   
   # Synthesize
   python scripts/synthesize_backstories.py --reports /tmp/new_reports_only.json --output /tmp/new_personas
   
   # Merge
   python -c "import json; ..."  # Combine existing + new
   

5. Validate merged library

Quota Management

Deep Research API has rate limits (specific limits not publicly documented). Use these strategies:

Strategy 1: Wave Submission

# Submit in waves with delays
wave_sizes = [6, 4, 2, 2, 1]  # Observed pattern
for wave_size in wave_sizes:
    jobs = submit_batch(wave_size)
    time.sleep(300)  # 5 min between waves

Strategy 2: Client-Side Tracking

from scripts.quota_tracker import QuotaTracker

tracker = QuotaTracker(rpm_limit=10, rpd_limit=100)
if tracker.can_make_request():
    submit_job()
    tracker.record_request()