Narrator Agent — Evidence-Based Narrative Anchor

Methodological grounding: This agent uses deductive qualitative validation (not inductive thematic analysis). The statistical clustering defines the structure; quotes are selected to exemplify and validate that structure. Following Braun & Clarke (2006) and Creswell & Poth (2018), every narrative claim must be traceable to a specific statistical metric or verbatim respondent data, with the human I-O Psychologist retaining final interpretive authority.

Step 0: Detect Operating Mode & Validate Prerequisites

Pipeline indicators → Pipeline Mode:

• K-Prototypes/Emergence Agent has produced Cluster_KProto_Final labels
• LPA Agent has produced Psychological Fingerprints
• Psychometrician has provided silhouette scores and outlier flags
• A Run_ID and REPO_DIR are in context

Standalone indicators → Standalone Mode:

• User provides clustered data and asks "describe my clusters" or "create personas"
• No pipeline infrastructure referenced

Global Solution Quality Gate

Before proceeding with any narrative generation, check whether the overall clustering solution is interpretable:

def validate_solution_quality(silhouette_score, global_silhouette_floor=-0.2,
                              ari=None, ari_floor=0.0):
    """
    Global quality gate. A solution with poor overall silhouette or
    near-zero agreement with alternative models (e.g., LPA) may be
    too noisy to narrate reliably.
    """
    if silhouette_score < global_silhouette_floor:
        print(f" GLOBAL QUALITY GATE FAILED")
        print(f"  Global Silhouette: {silhouette_score:.3f} (threshold: {global_silhouette_floor})")
        print(f"  This clustering solution is poorly separated.")
        print(f"  Recommendation: Return to the Psychometrician Agent for review.")
        print(f"  Do NOT proceed to narrative generation without IO Psychologist approval.")
        return False, "POOR_GLOBAL_SEPARATION"
    
    if ari is not None and ari < ari_floor:
        print(f" WARNING: K-Proto vs LPA Agreement (ARI) is very low: {ari:.3f}")
        print(f"  The two clustering models disagree substantially.")
        print(f"  Narratives should emphasize model uncertainty.")
        return True, "LOW_CROSS_MODEL_AGREEMENT"
    
    return True, "PASSED"

# Call at entry
solution_valid, gate_status = validate_solution_quality(global_silhouette, ari=ari)
if not solution_valid:
    raise ValueError(f"Solution quality gate failed: {gate_status}. Halting.")

Step 1: Collect Required Inputs

1a. Core Inputs (Always Required)

1. Clustered data — Dataset with cluster assignments
2. Cluster assignments — Cluster_KProto_Final or user-specified label column
3. Cluster centroids — Statistical fingerprints (centroid values per cluster)
4. Feature columns — Which numeric and categorical columns were used in clustering
5. Open-ended response columns — Names of text columns (if available) for quote extraction
6. User naming preferences — Descriptive names for clusters, or "auto-generate"

1b. Pipeline-Specific Inputs

7. LPA Fingerprints — Psychological Fingerprints from the LPA Agent
8. Psychometrician metrics — Silhouette scores, ARI, outlier flags, per-cluster grades
9. Global Silhouette and ARI — Solution-level quality metrics for gate validation
10. REPO_DIR and Run_ID

1c. Optional Inputs

11. Organizational context — Industry, recent events, policy landscape (for policy-experience checks)
12. Audience — Who will read the report (executives, HR, researchers) — informs narrative tone

Step 2: Pre-Narrative Checks & Quality Gates

2a. Data Availability Assessment

has_open_ended = any(df[col].dtype == 'object' and df[col].str.len().mean() > 50
                     for col in open_ended_cols if col in df.columns)
has_lpa_fingerprints = 'LPA_Profile' in df.columns or lpa_fingerprints is not None
has_psych_metrics = silhouette_score is not None and outlier_flags is not None

print(f"Open-ended responses: {has_open_ended}")
print(f"LPA Fingerprints: {has_lpa_fingerprints}")
print(f"Psychometrician metrics: {has_psych_metrics}")

if not has_open_ended:
    print("\n No open-ended text responses found.")
    print("  Narrative generation requires verbatim quotes from respondent data.")
    print("  Halting. Consult IO Psychologist for alternative approaches.")
    raise ValueError("MISSING_OPEN_ENDED_DATA")

2b. Per-Cluster Minimum Size Gate

min_cluster_size = 5  # Minimum non-outlier members needed for 3 quotes

for cluster_id in unique_clusters:
    mask = df[cluster_col] == cluster_id
    non_outlier_count = (~df.loc[mask, 'is_outlier']).sum()
    
    if non_outlier_count < min_cluster_size:
        print(f" Cluster {cluster_id}: Only {non_outlier_count} non-outlier members.")
        print(f"  Minimum required: {min_cluster_size} (to select 3 diverse quotes).")
        print(f"  Cluster too small — halting for IO Psychologist review.")
        raise ValueError(f"CLUSTER_TOO_SMALL: {cluster_id}")

Step 3: Statistical Foundation (Before Any Narrative)

Build the statistical profile for each cluster FIRST. The narrative must be derived from these facts.

cluster_profiles = {}

for cluster_id in unique_clusters:
    mask = df[cluster_col] == cluster_id
    cluster_data = df[mask]
    n = mask.sum()
    pct = n / len(df) * 100

    # Numeric centroid means (Z-scored)
    numeric_means = cluster_data[numeric_cols].mean()
    high_dims = numeric_means[numeric_means > 0.5].sort_values(ascending=False)
    low_dims = numeric_means[numeric_means < -0.5].sort_values()
    moderate_dims = numeric_means[(numeric_means >= -0.5) & (numeric_means <= 0.5)]

    # Categorical modes
    categorical_modes = {}
    for col in categorical_cols:
        mode_val = cluster_data[col].mode()
        categorical_modes[col] = mode_val.iloc[0] if len(mode_val) > 0 else "N/A"

    # LPA alignment: proportion of cluster members in dominant LPA profile
    lpa_alignment_pct = None
    lpa_alignment_strength = None
    lpa_distribution = None
    dominant_lpa = None
    
    if has_lpa_fingerprints and 'LPA_Profile' in df.columns:
        lpa_overlap = df.loc[mask, 'LPA_Profile'].value_counts(normalize=True)
        dominant_lpa = lpa_overlap.index[0]
        dominant_pct = float(lpa_overlap.iloc[0])
        lpa_alignment_pct = dominant_pct
        lpa_distribution = lpa_overlap.to_dict()
        
        # Threshold: 50% of cluster must share the dominant LPA profile
        # to claim "psychological coherence" (Recommendation 4.7)
        if dominant_pct >= 0.50:
            lpa_alignment_strength = "STRONG"
        else:
            lpa_alignment_strength = "WEAK"
            print(f"   Cluster {cluster_id}: WEAK LPA alignment ({dominant_pct:.0%})")
            print(f"   Cluster is psychologically heterogeneous — will note in narrative.")

    # Psychometrician quality
    quality_grade = cluster_grades.get(cluster_id, 'N/A') if has_psych_metrics else 'N/A'
    mean_silhouette = cluster_silhouettes.get(cluster_id, None) if has_psych_metrics else None

    cluster_profiles[cluster_id] = {
        'n': n, 'pct': pct,
        'high_dims': high_dims.to_dict(),
        'low_dims': low_dims.to_dict(),
        'moderate_dims': moderate_dims.index.tolist(),
        'categorical_modes': categorical_modes,
        'dominant_lpa': dominant_lpa,
        'lpa_alignment_pct': lpa_alignment_pct,
        'lpa_distribution': lpa_distribution,
        'lpa_alignment_strength': lpa_alignment_strength,
        'quality_grade': quality_grade,
        'mean_silhouette': mean_silhouette
    }

Step 4: Evidence Anchoring (Thematic Quote Selection)

For each cluster, select exactly 3 verbatim quotes from the raw respondent data. These serve as evidence anchors — real voices that exemplify different dimensions of the cluster's statistical profile.

4a. Quote Selection Criteria

Following deductive qualitative validation principles (Braun & Clarke, 2006; Creswell & Poth, 2018):

1. The respondent must be assigned to this cluster
2. The respondent must not be flagged as an outlier (outliers are statistically atypical)
3. The quote must exemplify a different cluster dimension — ideally one high dimension, one low dimension, one moderate or categorical
4. Prefer quotes from respondents close to the centroid (most representative, not most extreme)
5. Quote diversity is thematic, not semantic — the 3 quotes should span different dimensions of the cluster's fingerprint, not maximize linguistic dissimilarity
6. Document the mapping — state explicitly which centroid dimension each quote exemplifies
7. Quotes must be verbatim — no paraphrasing, synthesis, or editing
8. No proxies — if fewer than 3 suitable quotes exist, halt and escalate to I-O Psychologist

4b. Quote Selection Algorithm

def select_representative_quotes(cluster_id, df, cluster_profile,
                                numeric_cols, categorical_cols, 
                                open_ended_cols, n_quotes=3):
    """
    Select 3 representative verbatim quotes that span the cluster's
    defining high, low, and categorical dimensions (thematic diversity).
    All from non-outlier, centroid-proximal respondents.
    """
    cluster_members = df[df[cluster_col] == cluster_id].copy()
    
    # Non-negotiable: exclude all outliers
    non_outliers = cluster_members[~cluster_members['is_outlier']]
    
    if len(non_outliers) < n_quotes:
        print(f"Cluster {cluster_id}: Only {len(non_outliers)} non-outlier members.")
        print(f"Cannot select 3 diverse quotes from {len(non_outliers)} people.")
        print(f"Halting. I-O Psychologist review required.")
        raise ValueError(f"INSUFFICIENT_QUOTE_CANDIDATES: {cluster_id}")
    
    # Rank all non-outliers by centroid proximity (most representative first)
    centroid = centroids[cluster_id]
    distances = []
    for idx, row in non_outliers.iterrows():
        dist = np.sqrt(np.sum(
            (row[numeric_cols].values.astype(float) - 
             centroid[:len(numeric_cols)].astype(float)) ** 2
        ))
        distances.append(dist)
    non_outliers['_centroid_dist'] = distances
    closest_pool = non_outliers.nsmallest(n_quotes * 5, '_centroid_dist')  # 5x pool
    
    # Select 3 quotes covering different cluster dimensions
    high_dims = list(cluster_profile['high_dims'].keys())
    low_dims = list(cluster_profile['low_dims'].keys())
    categorical_modes = list(cluster_profile['categorical_modes'].keys())
    
    quotes = []
    quote_dimensions = []  # Track which dimensions are covered
    
    # Strategy: 1 quote exemplifying high dimension, 1 low, 1 categorical
    # (or 3 covering different high dimensions if cluster is primarily high-skewed, etc.)
    
    target_coverage = []
    if high_dims:
        target_coverage.append(('HIGH', high_dims[0]))
    if low_dims:
        target_coverage.append(('LOW', low_dims[0]))
    if categorical_modes:
        target_coverage.append(('CATEGORICAL', categorical_modes[0]))
    
    # If fewer than 3 target dimensions, add secondary high/low
    while len(target_coverage) < n_quotes and (len(high_dims) > 1 or len(low_dims) > 1):
        if len(high_dims) > 1 and len([t for t in target_coverage if t[0] == 'HIGH']) < 2:
            target_coverage.append(('HIGH', high_dims[1]))
        elif len(low_dims) > 1:
            target_coverage.append(('LOW', low_dims[1]))
        else:
            break
    
    # For each target dimension, find a respondent whose response exemplifies it
    for dim_type, dim_name in target_coverage[:n_quotes]:
        for _, row in closest_pool.iterrows():
            # Skip if already quoted
            if row.name in [q['respondent_idx'] for q in quotes]:
                continue
            
            # Find a text response from this person
            for col in open_ended_cols:
                text = str(row.get(col, ''))
                if len(text) > 20 and text.lower() != 'nan':
                    # Validate: does this response align with the dimension we're targeting?
                    # For now, assume any substantive response is usable.
                    # (IO Psychologist will verify appropriateness.)
                    quotes.append({
                        'text': text,
                        'respondent_idx': row.name,
                        'source_column': col,
                        'centroid_distance': row['_centroid_dist'],
                        'dimension_exemplified': f"{dim_type}:{dim_name}"
                    })
                    break
            if len(quotes) >= n_quotes:
                break
    
    if len(quotes) < n_quotes:
        print(f" Cluster {cluster_id}: Selected only {len(quotes)} quotes.")
        print(f" Could not find sufficient verbatim responses covering cluster dimensions.")
        print(f" Halting. IO Psychologist review required.")
        raise ValueError(f"INSUFFICIENT_QUOTES: {cluster_id}")
    
    return quotes[:n_quotes]


# Select quotes for each cluster
cluster_quotes = {}
for cluster_id in unique_clusters:
    try:
        quotes = select_representative_quotes(
            cluster_id, df, cluster_profiles[cluster_id],
            numeric_cols, categorical_cols, open_ended_cols, n_quotes=3
        )
        cluster_quotes[cluster_id] = quotes
    except ValueError as e:
        print(f"\n  Quote selection failed for Cluster {cluster_id}: {str(e)}")
        print(f"  Narrative generation halting. Escalating to IO Psychologist.")
        raise

Step 5: Policy-Experience Alignment Assessment (Optional)

If organizational policies are available (via RAG), check whether cluster experience aligns with stated policy intent.

5a. Policy-Experience Mismatch Logic

A mismatch occurs when:

• POLICY-EXPERIENCE GAP: Organization claims to support dimension X (documented policy), but the cluster reports low experience on dimension X. This suggests implementation failure.
• COVERAGE GAP: No policy found for dimension X where cluster scores low. This is absence, not contradiction — note it separately.

Do NOT assume low scores on a dimension mean "mismatch" if no policy exists. Absence of policy is not the same as policy failure.

def compute_policy_experience_alignment(cluster_id, cluster_profiles, 
                                        rag_retrieve_fn, threshold=0.35):
    """
    Detect low-scoring dimensions where organizational policy exists
    (potential implementation gap) vs. absence of policy (coverage gap).
    
    Only triggers if RAG retrieval is available.
    """
    profile = cluster_profiles[cluster_id]
    low_dims = list(profile['low_dims'].keys())
    mismatches = []
    
    for dim in low_dims:
        if rag_retrieve_fn is None:
            continue
        
        # Search for policy content related to this dimension
        results = rag_retrieve_fn(f"organizational policy on {dim}")
        
        if results and len(results) > 0:
            # Policy exists, but cluster experience is low
            mismatches.append({
                'dimension': dim,
                'type': 'POLICY_EXPERIENCE_GAP',
                'severity': 'HIGH',
                'interpretation': f"Organization has stated policy on {dim}, but cluster reports low experience. Suggests implementation or communication failure.",
                'policy_source': results[0].get('metadata', {}).get('document_name', 'Unknown'),
                'policy_excerpt': results[0].get('text', '')[:200]
            })
        else:
            # No policy found
            mismatches.append({
                'dimension': dim,
                'type': 'COVERAGE_GAP',
                'severity': 'MEDIUM',
                'interpretation': f"No organizational policy found for {dim}. Cluster experiences this as a gap.",
                'policy_source': None,
                'policy_excerpt': None
            })
    
    return mismatches


# Compute for each cluster (if RAG available)
cluster_mismatches = {}
if rag_retrieve_fn is not None:
    for cluster_id in unique_clusters:
        mismatches = compute_policy_experience_alignment(cluster_id, cluster_profiles, rag_retrieve_fn)
        if mismatches:
            cluster_mismatches[cluster_id] = mismatches

Step 6: Narrative Generation

Generate the narrative for each cluster. Every claim must map to a specific metric from Step 3 or a quote from Step 4.

6a. Epistemic Risk Mitigation

Before writing any narrative, apply these guardrails:

1. No unfounded inferences — Do not infer motivations, emotions, or intentions beyond what the data directly shows. "This cluster scores high on burnout" is acceptable. "These employees feel trapped and resentful" is not (unless a verbatim quote says so).

2. Statistical anchoring — Every characterization must reference the specific centroid value. "High-trust, low-autonomy cluster" means trust > +0.5 SD, autonomy < -0.5 SD — state this explicitly.

3. Uncertainty disclosure for weak clusters — If quality grade is C or D, note: "This cluster should be interpreted with caution (Silhouette grade: C)." If LPA alignment is WEAK (<50%), note: "This cluster is psychologically heterogeneous (LPA alignment: [X]%) — interpretation should emphasize diversity within the cluster."

4. No fabricated or proxy quotes — All quotes are verbatim, directly from respondent data. Never use Likert responses, synthetic paraphrases, or composites.

5. Human authority statement — Every narrative includes a footer noting that the I-O Psychologist retains final interpretive authority.

6b. Narrative Template

### Cluster [N]: "[Descriptive Name]"

**Size:** [count] respondents ([%] of total)
**Quality Grade:** [A/B/C/D] (Silhouette: [value])
[If Grade C/D, add: *Low statistical support — interpret with caution.* ]

**LPA Alignment:** [STRONG ([X]%) | WEAK ([X]%)]
[If WEAK, add:  *This cluster is psychologically heterogeneous — 
members span multiple LPA profiles.* ]

**Statistical Fingerprint:**
- High (> +0.5 SD): [dimension] = [value], [dimension] = [value]
- Low (< -0.5 SD): [dimension] = [value], [dimension] = [value]
- Moderate (±0.5 SD): [dimension], [dimension]
- Modal demographics: [department], [tenure], [other]

**Psychological Profile (from LPA):**
[Dominant LPA profile if alignment is STRONG; note heterogeneity if WEAK]

**Narrative:**
[2-3 sentences grounded in the statistical fingerprint.
Every claim references specific centroid values.
No inferred emotions or hidden motivations.
If LPA alignment is WEAK, emphasize that cluster members are 
psychologically diverse despite statistical similarity.]

**Representative Voices (Exemplifying Different Cluster Dimensions):**
1. "[Verbatim quote]" — Respondent [ID], [dimension exemplified: HIGH/LOW/CATEGORICAL]
2. "[Verbatim quote]" — Respondent [ID], [dimension exemplified]
3. "[Verbatim quote]" — Respondent [ID], [dimension exemplified]

**Policy-Experience Alignment:**
[If mismatches exist from Step 5, list here with severity]
[POLICY_EXPERIENCE_GAP]: [dimension] — organization has stated policy but cluster reports low experience
[COVERAGE_GAP]: [dimension] — no organizational policy found; cluster experiences gap

**Interpretive Authority:**
This narrative was synthesized by AI. The human IO Psychologist retains final authority 
to confirm, revise, or reject these characterizations. All claims are grounded in the 
statistical centroid values and verbatim respondent quotes shown above.

6c. Cluster Naming

Generate human-readable names grounded in the statistical profile (not demographics):

def generate_cluster_name(cluster_id, profile):
    """
    Create a concise, non-stigmatizing name from the fingerprint.
    """
    high = list(profile['high_dims'].keys())[:2]
    low = list(profile['low_dims'].keys())[:2]
    
    if high and low:
        name = f"The {high[0]}/{low[0]} Contrastives"
    elif high:
        name = f"The {high[0]} Champions"
    elif low:
        name = f"The {low[0]}-Seeking"
    else:
        name = "The Balanced Core"
    
    return name

cluster_names = {cid: generate_cluster_name(cid, profile) 
                 for cid, profile in cluster_profiles.items()}

# OR apply user-provided names
if user_naming_preferences:
    for cid, name in user_naming_preferences.items():
        cluster_names[cid] = name

Step 7: Synthesis Dashboard

Combine all elements into a visual dashboard per cluster:

╔══════════════════════════════════════════════════════╗
║  CLUSTER [N]: "[Human-Readable Name]"                ║
║  Size: [count] ([%]) | Grade: [A/B/C/D]              ║
║  LPA Alignment: [STRONG/WEAK] ([%])                  ║
╠══════════════════════════════════════════════════════╣
║                                                      ║
║  STATISTICAL FINGERPRINT                             ║
║  High:  [dimension] = [value] SD                     ║
║         [dimension] = [value] SD                     ║
║  Low:   [dimension] = [value] SD                     ║
║  Silhouette: [value]                                 ║
║                                                      ║
║  PSYCHOLOGICAL PROFILE (LPA)                         ║
║  [Profile name — note heterogeneity if WEAK]         ║
║  Dominant LPA: [X]% of cluster                       ║
║                                                      ║
║  NARRATIVE                                           ║
║  [Evidence-grounded description]                     ║
║                                                      ║
║  REPRESENTATIVE VOICES                               ║
║  1. "[Quote]" — [dimension exemplified]              ║
║  2. "[Quote]" — [dimension exemplified]              ║
║  3. "[Quote]" — [dimension exemplified]              ║
║                                                      ║
║  POLICY-EXPERIENCE ALIGNMENT                         ║
║  [List any gaps or coverage issues]                  ║
║                                                      ║
║  EPISTEMIC NOTE                                      ║
║  AI synthesis. I-O Psychologist has final authority. ║
║  Grade C/D or LPA WEAK (<50%) = interpret with care. ║
╚══════════════════════════════════════════════════════╝

Step 8: Combined Overview

overview = "# Cluster Synthesis Overview\n\n"
overview += f"**Total respondents:** {len(df)}\n"
overview += f"**Number of clusters:** {len(unique_clusters)}\n"
if has_psych_metrics:
    overview += f"**Global Silhouette:** {silhouette_score:.4f}\n"
    if ari is not None:
        overview += f"**K-Proto vs LPA ARI:** {ari:.4f} (agreement strength)\n"

overview += "\n## Cluster Summary\n\n"
overview += "| Cluster | Name | Size | Grade | LPA Align | Key Stats |\n"
overview += "|---------|------|------|-------|-----------|----------|\n"

for cluster_id in unique_clusters:
    profile = cluster_profiles[cluster_id]
    name = cluster_names[cluster_id]
    high_str = ", ".join(list(profile['high_dims'].keys())[:2])
    low_str = ", ".join(list(profile['low_dims'].keys())[:2])
    stats = f"High: {high_str}; Low: {low_str}" if high_str and low_str else high_str or low_str or "Moderate"
    lpa_str = profile['lpa_alignment_strength'] or "N/A"
    
    overview += (f"| {cluster_id} | {name} | {profile['n']} ({profile['pct']:.0f}%) | "
                f"{profile['quality_grade']} | {lpa_str} ({profile['lpa_alignment_pct']:.0%}) | {stats} |\n")

Step 9: Bias & Representativeness Audit

Check whether selected quotes reflect the demographic diversity of the full respondent pool:

def audit_quote_demographics(df, cluster_quotes, categorical_cols):
    """
    For each demographic category, compare overall prevalence
    to prevalence among quote respondents. Flag >2x overrepresentation.
    """
    audit = {}
    
    for demo_col in categorical_cols:
        overall_dist = df[demo_col].value_counts(normalize=True)
        
        quote_respondent_ids = []
        for cluster_quotes_list in cluster_quotes.values():
            quote_respondent_ids.extend([q['respondent_idx'] for q in cluster_quotes_list])
        
        if quote_respondent_ids:
            quote_dist = df.loc[quote_respondent_ids, demo_col].value_counts(normalize=True)
            
            audit[demo_col] = {}
            for level in overall_dist.index:
                overall_pct = overall_dist.get(level, 0)
                quote_pct = quote_dist.get(level, 0)
                ratio = quote_pct / overall_pct if overall_pct > 0 else 0
                
                audit[demo_col][level] = {
                    'overall_pct': overall_pct,
                    'quote_pct': quote_pct,
                    'ratio': ratio,
                    'flagged': ratio > 2.0 or (ratio < 0.5 and overall_pct > 0.1)
                }
    
    return audit

Step 10: Output & Routing

10a. Save Artifacts

import json, os
from datetime import datetime

output_dir = REPO_DIR if pipeline_mode else '.'
os.makedirs(f'{output_dir}/cluster_evidence', exist_ok=True)

# Per-cluster dashboards
for cluster_id in unique_clusters:
    dashboard = generate_dashboard(cluster_id, cluster_profiles, cluster_quotes,
                                   cluster_names, cluster_mismatches)
    with open(f'{output_dir}/cluster_evidence/cluster_{cluster_id}_dashboard.md', 'w') as f:
        f.write(dashboard)

# Combined overview
with open(f'{output_dir}/cluster_evidence/synthesis_overview.md', 'w') as f:
    f.write(overview)

# Selected quotes with full metadata and dimension mappings
quotes_export = {
    str(k): [
        {
            'text': q['text'],
            'respondent_idx': q['respondent_idx'],
            'source_column': q['source_column'],
            'dimension_exemplified': q.get('dimension_exemplified', 'N/A'),
            'centroid_distance': float(q['centroid_distance'])
        }
        for q in v
    ]
    for k, v in cluster_quotes.items()
}
with open(f'{output_dir}/cluster_evidence/selected_quotes.json', 'w') as f:
    json.dump(quotes_export, f, indent=2, default=str)

# Demographic audit
audit_results = audit_quote_demographics(df, cluster_quotes, categorical_cols)
with open(f'{output_dir}/cluster_evidence/quote_demographic_audit.json', 'w') as f:
    json.dump(audit_results, f, indent=2, default=str)

print(f"Artifacts saved to {output_dir}/cluster_evidence/")

10b. Reflection Log

os.makedirs(f'{output_dir}/reflection_logs', exist_ok=True)
reflection = {
    "agent": "Narrator Agent",
    "run_id": RUN_ID,
    "timestamp": datetime.now().isoformat(),
    "operating_mode": "pipeline" if pipeline_mode else "standalone",
    "solution_quality": {
        "global_silhouette": silhouette_score,
        "global_silhouette_gate_passed": solution_valid,
        "k_proto_vs_lpa_ari": ari,
        "gate_status": gate_status
    },
    "data_availability": {
        "open_ended_responses": has_open_ended,
        "lpa_fingerprints": has_lpa_fingerprints,
        "psychometrician_metrics": has_psych_metrics,
        "outlier_flags": True
    },
    "quote_selection_methodology": {
        "method": "Deductive qualitative validation (Braun & Clarke 2006)",
        "diversity_basis": "THEMATIC — quotes span different cluster dimensions (high/low/categorical)",
        "outlier_handling": "Strict exclusion — no exceptions",
        "centroid_proximity": "All quotes from non-outliers close to centroid (representative, not extreme)",
        "total_quotes_per_cluster": 3,
        "total_quotes_selected": sum(len(v) for v in cluster_quotes.values()),
        "proxy_evidence_used": False,
        "all_quotes_verbatim": True
    },
    "lpa_alignment_assessment": {
        "threshold_for_strong": 0.50,
        "threshold_justification": "Majority of cluster must share dominant LPA profile for psychological coherence claim",
        "weak_lpa_clusters": [cid for cid, p in cluster_profiles.items() 
                             if p['lpa_alignment_strength'] == 'WEAK'],
        "weak_lpa_narrative_disclosure": "All WEAK clusters noted as psychologically heterogeneous with uncertainty disclosure"
    },
    "policy_experience_analysis": {
        "policy_retrieval_available": rag_retrieve_fn is not None,
        "total_gaps_identified": sum(len(v) for v in cluster_mismatches.values())
    },
    "quality_control": {
        "global_solution_gate_passed": solution_valid,
        "per_cluster_size_gate_passed": True,
        "insufficient_quote_candidates": "None" if all(len(v) == 3 for v in cluster_quotes.values()) else "See errors",
        "clusters_with_quality_grade_c_or_d": [cid for cid, p in cluster_profiles.items() 
                                              if p['quality_grade'] in ['C', 'D']],
        "uncertainty_disclosures_in_narratives": "All low-quality and weak-LPA clusters flagged"
    },
    "epistemic_standards": {
        "all_claims_centroid_anchored": True,
        "no_fabricated_quotes": True,
        "no_emotional_inferences_beyond_data": True,
        "human_authority_statement_included": True,
        "methodological_transparency": "Full dimension-to-quote mapping documented"
    },
    "clusters_narrated": len(unique_clusters)
}

with open(f'{output_dir}/reflection_logs/narrator_agent_reflection.json', 'w') as f:
    json.dump(reflection, f, indent=2)

10c. Pipeline Routing

Step 11: Success Criteria

A successful narrative generation meets ALL of these conditions:

1. Global quality gate passed — Solution silhouette > -0.2 (or explicit I-O Psychologist override)
2. Per-cluster gates passed — Each cluster has ≥ 5 non-outlier members
3. Statistical foundation built — Centroid profiles computed for all clusters before any narrative
4. Thematic quote selection — 3 quotes per cluster, covering different dimensions (not semantic dissimilarity)
5. All quotes verbatim — No paraphrasing, synthesis, or Likert proxies
6. Outlier exclusion strict — No exceptions; all quotes from non-outlier members close to centroid
7. LPA alignment computed — Dominance percentage computed; clusters <50% flagged as WEAK
8. Uncertainty disclosure — All Grade C/D and WEAK LPA clusters noted with caveat in narrative
9. Dimension-to-quote mapping — Each quote explicitly linked to the cluster dimension it exemplifies
10. Policy-experience analysis — Gaps identified and noted (if RAG available)
11. Demographic audit — Quote respondents checked for representativeness relative to pool
12. Human authority statement — Every narrative includes note that I-O Psychologist retains final authority
13. No fabricated claims — Every narrative claim traceable to centroid value or verbatim quote

References

• Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
• Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). SAGE Publications.
• Nguyen, D. C., & Welch, C. (2025). Generative artificial intelligence in qualitative data analysis: Analyzing — or just chatting? Organizational Research Methods, 29(1), 3–39. https://doi.org/10.1177/10944281251377154