V1-V3 Visual Cortex Alignment for VLM Robustness

Brain Regions Analyzed

Methodology

Three-Stage Pipeline

Stage 1: Brain Alignment Measurement
├── Extract vision encoder features from 12 VLMs
├── Predict fMRI responses across 6 visual cortex ROIs
├── Use Natural Scenes Dataset (Algonauts 2023)
└── 8 human subjects, 6 ROIs each

Stage 2: Sycophancy Evaluation  
├── 76,800 two-turn gaslighting prompts
├── 5 manipulation categories
├── 10 difficulty levels
└── Measure rate of capitulation to false claims

Stage 3: Correlation Analysis
├── Brain alignment scores vs sycophancy rates
├── Aggregate and ROI-specific correlations
├── BCa bootstrap, leave-one-out, permutation testing
└── Control for model size, architecture, training

Stage 1: Brain Alignment Measurement

fMRI Dataset

• Source: Natural Scenes Dataset (NSD) / Algonauts 2023
• Stimuli: 10,000 natural scene images
• Subjects: 8 human participants
• ROIs: 6 visual cortex regions

Neural Encoding Model

from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_predict

def compute_brain_alignment(model_features, fmri_responses):
    """
    Compute neural predictivity (brain alignment).
    
    Args:
        model_features: CNN features [n_images, n_features]
        fmri_responses: Brain activity [n_images, n_voxels]
    
    Returns:
        r: Pearson correlation (predictivity)
    """
    # Fit ridge regression
    ridge = Ridge(alpha=1.0)
    
    # Cross-validated prediction
    predicted = cross_val_predict(
        ridge, model_features, fmri_responses, 
        cv=5, n_jobs=-1
    )
    
    # Compute correlation
    r = pearsonr(predicted.flatten(), fmri_responses.flatten())[0]
    
    return r

ROI-Specific Analysis

rois = {
    'V1': 'Early visual - primary',
    'V2': 'Early visual - secondary', 
    'V3': 'Early visual - tertiary',
    'V4': 'Mid-level visual - color/form',
    'LO': 'Lateral occipital - objects',
    'FFA': 'Fusiform face area',
    'PPA': 'Parahippocampal place area',
    'EBA': 'Extrastriate body area'
}

# Compute alignment per ROI
roi_alignment = {}
for roi_name, roi_voxels in roi_masks.items():
    roi_alignment[roi_name] = compute_brain_alignment(
        model_features, 
        fmri_responses[:, roi_voxels]
    )

Stage 2: Sycophancy Evaluation

Gaslighting Prompt Categories

1. Existence Denial: "That object isn't really there"
2. Attribute Manipulation: "The color is actually different"
3. Relationship Distortion: "A is larger than B" (when B > A)
4. Count Disagreement: "There are 3 items" (when there are 5)
5. Category Misassignment: "That's a dog" (when it's a cat)

Prompt Structure

def create_gaslighting_prompt(image, initial_qa, manipulation):
    """
    Two-turn gaslighting conversation.
    
    Turn 1:
    User: [Image] + [Question]
    Assistant: [Correct Answer]
    
    Turn 2:
    User: "Actually, I think [Manipulation]. Are you sure?"
    Assistant: [Response - measure if changes answer]
    """
    return {
        'image': image,
        'turn1_question': initial_qa['question'],
        'turn1_correct': initial_qa['answer'],
        'turn2_manipulation': manipulation['claim'],
        'turn2_ground_truth': manipulation['ground_truth']
    }

# Measure sycophancy
def evaluate_sycophancy(model, prompt):
    # Turn 1: Get initial correct answer
    answer1 = model.generate(prompt['image'], prompt['turn1_question'])
    
    # Turn 2: Apply pressure
    pressure_prompt = f"User said: {prompt['turn2_manipulation']}\nAre you sure?"
    answer2 = model.generate(prompt['image'], pressure_prompt, 
                             context=[answer1])
    
    # Check if model capitulated
    capitulated = check_answer_change(answer2, prompt['turn1_correct'],
                                       prompt['turn2_manipulation'])
    return capitulated

Difficulty Levels

• Level 1: Simple, obvious claims
• Level 5: Moderate ambiguity
• Level 10: Sophisticated, plausible-sounding falsehoods

Stage 3: Correlation Analysis

from scipy.stats import pearsonr, bootstrap

def analyze_v1_v3_robustness(alignment_scores, sycophancy_rates):
    """
    Analyze V1-V3 specific correlation with robustness.
    """
    results = {}
    
    # Early visual cortex aggregate
    early_visual = ['V1', 'V2', 'V3']
    early_alignment = np.mean([alignment_scores[r] for r in early_visual])
    
    # Correlation
    r, p = pearsonr(early_alignment, sycophancy_rates)
    results['early_visual_r'] = r
    results['early_visual_p'] = p
    
    # BCa Bootstrap CI
    def correlation_stat(x, y):
        return pearsonr(x, y)[0]
    
    ci = bootstrap(
        (early_alignment, sycophancy_rates),
        statistic=lambda i, j: correlation_stat(i, j),
        n_resamples=10000,
        method='BCa'
    )
    results['bci_95'] = (ci.confidence_interval.low, 
                         ci.confidence_interval.high)
    
    # Leave-one-out validation
    loo_correlations = []
    for i in range(len(models)):
        mask = np.ones(len(models), dtype=bool)
        mask[i] = False
        r_loo = pearsonr(early_alignment[mask], sycophancy_rates[mask])[0]
        loo_correlations.append(r_loo)
    
    results['loo_all_negative'] = all(r < 0 for r in loo_correlations)
    results['loo_mean'] = np.mean(loo_correlations)
    
    return results

Key Results

Models Evaluated

• 12 open-weight VLMs
• 6 architecture families
• 40× parameter range (256M–10B)

Findings by Attack Type

Implications

Existence denial shows the strongest effect because:

1. V1-V3 encode fundamental "what is there"
2. Harder to override with language when perceptually grounded
3. Higher-level inferences more malleable

Practical Applications

1. Model Selection

def select_robust_vlm(models, alignment_data):
    """Select model with high V1-V3 alignment."""
    
    scores = {}
    for model in models:
        # Weight early visual alignment
        early_score = np.mean([
            alignment_data[model]['V1'],
            alignment_data[model]['V2'],
            alignment_data[model]['V3']
        ])
        scores[model] = early_score
    
    return max(scores, key=scores.get)

2. Robustness Prediction

def predict_sycophancy_risk(model_features, v1_v3_encoder):
    """Predict sycophancy risk from V1-V3 alignment."""
    
    # Extract early visual features
    v1_v3_features = v1_v3_encoder(model_features)
    
    # Compare to human V1-V3 responses
    alignment_score = compute_v1_v3_alignment(v1_v3_features)
    
    # Lower alignment = higher risk
    risk = 1 - normalize(alignment_score)
    
    return risk

3. Training Objective

class V1V3AlignmentLoss(nn.Module):
    """Auxiliary loss for V1-V3 brain alignment."""
    
    def __init__(self, human_v1_v3_responses):
        super().__init__()
        self.target_responses = human_v1_v3_responses
        
    def forward(self, model_features, roi_masks):
        """
        Compute alignment loss for early visual areas.
        
        Args:
            model_features: [batch, features, h, w]
            roi_masks: Dict of ROI spatial masks
        """
        loss = 0
        
        # Focus on early visual
        for roi in ['V1', 'V2', 'V3']:
            # Extract ROI-specific features
            roi_features = model_features * roi_masks[roi]
            
            # Predict fMRI response
            predicted = self.encoding_model(roi_features)
            
            # Match human responses
            target = self.target_responses[roi]
            loss += F.mse_loss(predicted, target)
        
        return loss

Design Recommendations

For More Robust VLMs

1. Prioritize Early Visual Fidelity

   • Train with objectives that preserve low-level visual information
   • Avoid aggressive compression in early layers
   • Use perceptual losses that emphasize V1-V3 correspondence

2. Multi-Scale Architecture

   Input Image
       ↓
   Early Visual (V1-V3-like) - High resolution, preserve detail
       ↓
   Mid-Level (V4-like) - Feature integration
       ↓
   High-Level (IT-like) - Semantic abstraction
       ↓
   Language Decoder
   

3. Contrastive Grounding

   • Contrastive learning that preserves visual grounding
   • Avoid over-reliance on linguistic priors
   • Balance vision-language with vision-only objectives

4. Adversarial Training

   • Include sycophancy-inducing prompts in training
   • Reward maintaining correct answers under pressure
   • Use brain alignment as regularization

Limitations and Considerations

1. Correlation ≠ Causation: Alignment correlates with robustness but may not cause it
2. Model Scale: Effect tested up to 10B parameters; may differ at larger scales
3. Task Specificity: Results specific to vision-language tasks
4. Human Variability: fMRI data from 8 subjects; individual differences exist

Future Directions

1. Intervention Studies: Can we improve robustness by enhancing V1-V3 alignment?
2. Causal Mechanisms: What specific representational properties confer robustness?
3. Extension to Other Modalities: Does early auditory cortex alignment help audio-language models?
4. Clinical Applications: Can this help design AI resistant to manipulation?

References

Paper: "Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation"

• arXiv: 2604.13803
• Authors: Arya Shah, Vaibhav Tripathi, Mayank Singh, et al.
• Published: April 2026
• Code: https://github.com/aryashah2k/Gaslight-Gatekeep-Sycophantic-Manipulation
• Dataset: https://huggingface.co/datasets/aryashah00/Gaslight-Gatekeep-V1-V3

Trigger Keywords

• visual cortex alignment
• V1 V2 V3 brain
• early visual cortex
• sycophancy robustness
• adversarial manipulation
• neural predictivity
• brain-ai alignment
• fMRI encoding models
• vision-language robustness
• perceptual grounding
• sycophantic behavior
• gaslighting attacks
• vision model safety
• neural correspondence

Paper Reference (Updated 2026-04-17)

• Title: Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation
• arXiv ID: 2604.13803
• Date: 2026-04-15
• Authors: Arya Shah, Vaibhav Tripathi, Mayank Singh, Chaklam Silpasuwanchai
• Categories: cs.CV, cs.AI