Gap Identify

Workflow

Step 1: Parse Taxonomy

1. Read taxonomy.md
2. Extract all Method Categories and Submethods
3. Extract all Training Paradigms
4. Extract all Core Challenges
5. Extract all Evaluation Focuses
6. Extract all paper IDs and their classifications

Step 2: Analyze Coverage Matrix

Build and analyze:

Method-Challenge Matrix:

Identify: Cells with 0 or very few papers = Unexplored Combinations

Method-Paradigm Matrix:

Identify: Missing cells = Unexplored Combinations

Step 3: Identify Benchmark Gaps

Check for each method category:

• Are standard benchmarks used?
• Are benchmarks consistent across papers?
• Are important benchmarks missing?
  • e.g., "No standardized robustness benchmark under realistic shift settings"
  • e.g., "No cross-architecture/cross-domain generalization benchmark"

Step 4: Identify Methodological Gaps

Check:

• Are there methods that address multiple challenges but aren't combined?
• Are there successful approaches from related fields not applied here?
• Are there theoretical foundations missing?

Step 5: Identify Scale Gaps

Check:

• Model size coverage (e.g., 7B tested, but 70B+ not tested)
• Dataset size coverage
• Regime/setting coverage (e.g., one setting tested, adjacent settings missing)
• Training data scale coverage

Step 6: Identify Generalization Gaps

Check:

• Cross-model generalization (tested on LLaMA, but not on Mistral)
• Cross-task generalization (tested on NLP, but not on multimodal)
• Cross-domain generalization
• Cross-bit-width generalization (method works at 4-bit but not 2-bit)

Step 7: Prioritize Gaps

For each identified gap:

1. Severity: How important is this gap?
2. Confidence: How certain are we that this is a real gap?
3. Supporting Evidence: Which papers support this gap identification?

Step 8: Generate Research Opportunities

For high-priority gaps:

1. Describe the opportunity
2. Gap type
3. Potential impact (High/Med/Low)
4. Difficulty (High/Med/Low)
5. Suggested approach

Step 9: Write Output

Write gap_analysis.md following templates/GAP_ANALYSIS_TEMPLATE.md

Gap Identification Heuristics

Unexplored Combinations

• Method-Challenge cell = 0 papers
• Method-Training Paradigm cell = 0 papers
• Cross-cutting dimension pairs not explored

Benchmark Gaps

• "No standard benchmark for X"
• "Inconsistent evaluation protocols across papers"
• "Important benchmark not used by most methods"

Methodological Gaps

• "No unified framework for X"
• "Successful approach Y from related field not applied here"
• "Theoretical analysis missing for X"

Scale Gaps

• "Only tested on models up to XB params"
• "Not tested at sub-N bit"
• "Training data scale not explored"

Generalization Gaps

• "Not tested on cross-architecture"
• "Not tested on cross-modal"
• "Not tested on cross-setting transfer"

Key Rules

• Evidence-based: Gap identification must be grounded in actual taxonomy analysis
• Cite supporting papers: Each gap should cite papers that support the identification
• Be specific: Don't say "more research needed" - specify what research
• Prioritize: Not all gaps are equally important
• Actionable: Suggest concrete future directions

Example Output Structure

# Gap Analysis: [Survey Topic]

## Gap 1: Unexplored Combinations

### Description
No papers explore the combination of Method A + Paradigm B under Setting C.

### Evidence
- Paper 2401.12345 uses Method A + Paradigm X
- Paper 2402.23456 uses Method B + Paradigm B
- No paper combines both

### Recommended Action
Evaluate whether Method A + Paradigm B can outperform single-axis variants under Setting C.

---

## Prioritized Gap List

| Priority | Gap Type | Gap Description | Confidence |
|----------|----------|------------------|------------|
| 1 | Unexplored Combination | Method A + Paradigm B under Setting C | High |
| 2 | Scale Gap | Not tested on larger-scale or cross-domain settings | Med |
...

For survey construction workflow, see /survey-pipeline For survey writing, see /survey-write