Vibe Research

- Research question (specific, testable)
- Hypotheses (what you expect and why)
- Evaluation metrics (defined before seeing results)
- Baselines (what you're comparing against)
- Scope boundaries (what this study does NOT claim)

Read it before every major decision. For long sessions, also keep findings.md for key paper notes and decisions — prevents goal drift across iterations.

Why this matters: Metrics defined after seeing results = p-hacking. Baselines chosen after seeing your method = cherry-picking. Lock both before you look.

2. Parallel Literature Search

Run independent searches simultaneously, not serially:

Agent 1: Keyword cluster A (e.g., "gender bias medical QA LLM")
Agent 2: Keyword cluster B (e.g., "fairness healthcare NLP benchmark")
Agent 3: Citation network of seed paper
Agent 4: Related datasets / benchmarks

Different sources in parallel: arXiv, Semantic Scholar, ACL Anthology, OpenAlex. Synthesize after all return — never wait for one before starting another.

3. Primary Source Verification

Never trust Claude's interpretation. Always go to the source.

Claude summarizes paper → read the actual numbers in the paper
Claude reports experiment result → check the metric in W&B / log file
Claude says "significant" → check p-value and CI yourself
Claude lists citations → verify each one exists and says what Claude claims

Citation verification is mandatory. No exceptions.

Claude hallucinates citations. For every paper Claude names:

• Confirm it exists (search arXiv / Semantic Scholar by title + authors)
• Open the actual PDF — do not trust the abstract
• Find the specific claim Claude attributes to it (table, page, section)
• Note the exact location: "[Author Year, Table 3, p.8]"

A citation that can't be verified to page level should not be used.

EDA before statistics:

1. Plot raw data distributions
2. Look at failure cases before aggregate metrics
3. Check for data leakage, label imbalance, demographic skew
4. Then run statistical tests
5. Then interpret

Aggregate metrics hide what matters. Never skip to interpretation.

Forbidden shortcuts:

• Do NOT report a metric without first plotting its distribution
• Do NOT call a result "significant" without p-value + CI + sample size
• Do NOT skip failure case analysis ("the average looks fine" is not enough)

4. Research Knowledge Base (RESEARCH.md)

One file checked into the project repo, shared across sessions:

## Dead Ends (do not repeat)
- GPT-2 on MedQA: too small, results not publishable (tried 2026-01)
- WinoBias for medical domain: domain mismatch, reviewers will flag it

## Confirmed Findings
- Llama-3 shows 12% gap on female pronouns in clinical notes (our result)

## Key Papers (verified)
- [Author, Year, Venue] — one-line contribution summary

Every failed experiment → RESEARCH.md immediately. Negative results documented prevent the same mistake next week, next month, by a collaborator.

5. Model Selection

A flawed research design costs weeks of compute. Wrong fast reasoning is slower than right slow reasoning.

Quick Reference

Tool Stack

Red Flags — Stop and Verify

• "Claude said this paper found X, so let's build on that"
• "The results look good" (without checking raw metrics)
• "I'll document this dead end later"
• "Let me search one more keyword before designing the experiment"
• "These baselines make sense" (chosen after seeing your method's results)
• "I remember what we decided — no need to check the plan"

All of these introduce errors that compound. Stop and verify.