Idea Gen

Workflow

Phase 1: Landscape Verification (~2 min — NOT a full literature search)

The purpose of this phase is to establish enough context for high-quality idea generation. It is NOT a replacement for /lit-survey.

1. Check for existing landscape artifacts:

   • Read outputs/LANDSCAPE.json if it exists
   • Read outputs/LANDSCAPE.md if it exists
   • These files are produced by the /lit-survey skill

2. If landscape files exist:

   • Verify the research direction in the landscape matches the user's current direction (fuzzy match is acceptable — e.g., "efficient transformers" matches "transformer efficiency")
   • Extract the Gap Identification Matrix or equivalent gap listing from the landscape
   • Extract the key papers list (titles + one-line summaries)
   • Extract any open problems or future work themes
   • Store these as landscape_summary, identified_gaps, and key_papers for use in Phase 2

3. If landscape files do NOT exist (fallback — abbreviated inline survey):

   • Print a notice: "No landscape files found. Running abbreviated inline survey. For better results, run /lit-survey [direction] first."
   • Run 3-5 quick WebSearch queries:
     • "[direction] survey" site:arxiv.org
     • "[direction] benchmark" NeurIPS OR ICML OR ICLR 2024 2025
     • "[direction] limitations" OR "future work"
     • "[direction]" state-of-the-art
     • One more query based on a specific sub-aspect of the direction
   • For the top 5-8 results, use WebFetch to read abstracts/introductions
   • Build a mini landscape map:
     • Group findings into 2-4 sub-themes
     • List 5-10 key papers (title, year, one-sentence summary)
     • Identify 3-5 gaps or open questions
   • Store as landscape_summary, identified_gaps, and key_papers

4. Direction specificity check:

   • Auto-narrowing for broad directions: If the user's direction is very broad (e.g., just "NLP" or "computer vision"), do NOT stop to ask. Instead:
     1. Identify the top 3 most promising sub-directions based on the landscape
     2. Generate ideas for each sub-direction (3-4 ideas each)
     3. Merge all ideas into a single pool and apply normal filtering
     4. Log in outputs/PIPELINE_LOG.md: "⚠️ Direction was broad, auto-narrowed to: [sub1], [sub2], [sub3]"
   • A good direction is 1-2 sentences specifying the problem, domain, and constraint — e.g., "factorized gap in discrete diffusion LMs" or "sample efficiency of offline RL with image observations"
   • If the direction is broad, auto-narrowing handles it autonomously; the pipeline never stops to ask

Phase 2: Idea Generation via External LLM

Use the external LLM via Codex MCP for divergent, high-quality brainstorming. This is the creative core of the skill.

Codex MCP failure handling: If mcp__codex__codex call fails (tool unavailable, timeout, or error):

1. Fall back to Claude's own brainstorming capability
2. Use the same prompt structure but execute it directly (not via external LLM)
3. Log: "⚠️ Codex MCP unavailable. Brainstorming performed by Claude (single-model mode, reduced diversity)."
4. Continue the pipeline normally — do NOT stop or ask the user.

Call mcp__codex__codex with the following parameters:

• model: REVIEWER_MODEL (i.e., gpt-5.4)
• config: {"model_reasoning_effort": "xhigh"}
• prompt: Construct the prompt below, filling in the bracketed sections with data from Phase 1:

You are a senior ML/systems researcher. Your task is to generate research ideas
that are NOVEL, SPECIFIC, and EVALUATABLE.

Research direction: [user's direction from $ARGUMENTS]

Current landscape (from systematic survey):
[paste landscape_summary — either from LANDSCAPE.md or the mini-survey]

Identified gaps:
[paste identified_gaps — either the Gap Identification Matrix or the mini-survey gaps]

Generate 8-12 concrete research ideas. For each idea, provide:

1. **Title**: A concise, descriptive title (as it would appear on a paper)
2. **One-sentence thesis**: The core claim, stated as "We show that X by Y"
3. **Problem it solves**: Which specific gap from the landscape does this address?
4. **Core mechanism**: The key technical insight (not just "apply X to Y")
5. **Why it is non-obvious**: What would a skeptic's first objection be, and why is it wrong?
6. **Expected contribution type**: empirical finding / new method / theoretical result / diagnostic / new formulation
7. **Risk level**: LOW / MEDIUM / HIGH (with 1-sentence justification)
8. **Estimated effort**: person-weeks to a publishable result
9. **Closest existing work**: The single most similar paper and the precise delta

Quality criteria for generated ideas:
- REJECT "apply X to Y" unless the application reveals a genuinely surprising mechanism
- REJECT ideas where the outcome does not matter (if +3% or -3%, who cares?)
- PREFER ideas where a NEGATIVE result is equally publishable
- PREFER ideas that challenge an assumption the field takes for granted
- PREFER ideas with a clear "skeleton experiment" that takes < 1 week
- Each idea must be differentiated from the landscape papers above

Generate diverse ideas: at least 2 should be HIGH risk / high reward,
at least 2 should be LOW risk / solid contribution, and the rest MEDIUM.

After the call:

• Parse the response to extract individual ideas into a structured list
• Verify that at least MIN_IDEAS (8) ideas were generated; if fewer, call mcp__codex__codex-reply on the same thread asking for additional ideas to reach the minimum
• Verify that no more than MAX_IDEAS (12) ideas are kept; if more were generated, keep all but note the count
• Save the threadId — it will be used for potential follow-up in later skills (e.g., /idea-screen)
• Assign each idea an identifier: IDEA-01, IDEA-02, etc.

Phase 3: First-Pass Filtering

For each generated idea, perform three quick evaluations. The goal is to eliminate clearly non-viable ideas before investing time in deeper scoring.

3a. Feasibility Check

For each idea, evaluate:

• Compute requirements: Estimate GPU-hours needed for the minimum viable experiment. Skip ideas requiring > 1 month of GPU time.
• Data availability: Is the required data publicly available or obtainable? Skip ideas requiring proprietary or non-existent datasets.
• Implementation complexity: Can this be implemented in a reasonable timeframe by a small team (1-3 researchers)?
• Dependency risk: Does this require access to specific models, APIs, or infrastructure that may not be available?

Mark each idea as: FEASIBLE, FEASIBLE WITH CAVEATS, or INFEASIBLE. Eliminate INFEASIBLE ideas. Note the reason for elimination.

3b. Novelty Quick-Check

For each remaining idea, run 2-3 targeted WebSearch queries to check if it has already been done:

• Search for the idea's title or close paraphrase
• Search for the core mechanism + domain combination
• Search for the closest existing work mentioned in the idea + the proposed delta

For each idea, assign a novelty status:

• LIKELY NOVEL — No close matches found
• NEEDS DEEPER CHECK — Tangentially related work exists, but the exact angle appears unexplored
• ALREADY DONE — A paper doing essentially the same thing was found

Eliminate ALREADY DONE ideas. Note the paper that already covers it.

3c. Impact Estimation ("So What?" Test)

For each remaining idea, evaluate:

• If the experiment succeeds with a positive result, does it change how people think or work?
• If the experiment produces a negative result, is that equally informative and publishable?
• Is the finding actionable (leads to better methods, new understanding) or just academically interesting?
• Would a reviewer at a top venue find the contribution significant?

Mark each idea as: HIGH IMPACT, MEDIUM IMPACT, or LOW IMPACT. Eliminate LOW IMPACT ideas where neither a positive nor negative result would be interesting.

After Phase 3: Typically 8-12 ideas reduce to 5-8 survivors. Record all eliminated ideas and their elimination reasons.

Phase 4: Prof. He's 4-Dimension Filter

Apply Prof. Bingsheng He's research idea evaluation framework. This is a structured scoring system that captures dimensions often missed by pure novelty/feasibility analysis.

For each surviving idea from Phase 3, score on four dimensions (1-5 scale each):

Composite score = Longevity + Passion + Application + Uniqueness (out of 20).

Elimination rule: Ideas scoring below FILTER_THRESHOLD (12/20) are eliminated. Record the scores and the reason (which dimension(s) dragged the score down).

Dynamic threshold adjustment: If ALL ideas score below FILTER_THRESHOLD (12/20):

1. Lower the threshold to 10/20
2. Keep the top 3 ideas regardless of score
3. Log: "⚠️ All ideas below threshold 12/20. Lowered to 10/20, keeping top 3."
4. If still no ideas survive at 10/20, keep the single highest-scoring idea and log: "⚠️ Emergency: keeping highest-scoring idea (score: X/20) despite low score"

After Phase 4: Target SURVIVING_TARGET (4-6) ideas. If more than 6 survive, keep all but note that the top 6 by composite score are recommended. If fewer than 4 survive, revisit eliminated ideas from Phase 3 that scored MEDIUM IMPACT or FEASIBLE WITH CAVEATS and re-evaluate with the He framework — some may pass on a second look.

Phase 5: Anti-Pattern Check

Before finalizing the output, check each surviving idea against four common anti-patterns. This is a quality gate to catch ideas that look good on paper but have structural problems.

For each surviving idea, check:

1. "Overly trendy" — If 5+ papers in the landscape already address this exact angle, flag it. The space is crowded and differentiation will be hard.

2. "Overly niche" — If 0 papers in the landscape are even tangentially related, flag it. The idea may be too far from the current discourse to get reviewer buy-in, or it may indicate an underdeveloped landscape search.

3. "A+B stitching" — If the idea is essentially "combine method A and method B" without a clear new mechanism or insight that explains why the combination is non-trivially better, flag it. This is the most common anti-pattern in mediocre research.

4. "Scale-dependent" — If the expected result only holds at a computational scale the researcher cannot reproduce (e.g., "this works at 100B parameters but we can only test at 1B"), flag it. The contribution becomes unverifiable.

Flagging rules:

• Flagged ideas get a warning label and a one-sentence explanation
• Flagged ideas are NOT automatically eliminated — the user may disagree with the flag or have additional context
• If an idea has 2+ flags, add a strong caution note
• Display flags prominently in the output

Phase 6: Output

Write two output files. Ensure the outputs/ directory exists before writing (create it if needed).

File 1: outputs/IDEAS_RAW.md

This file contains ALL generated ideas before any filtering, serving as a complete record.

# Generated Research Ideas (Raw)

**Direction**: [research direction from $ARGUMENTS]
**Date**: [today's date]
**Model**: gpt-5.4
**Landscape source**: [LANDSCAPE.md / abbreviated inline survey]
**Ideas generated**: [N]
**Codex thread ID**: [threadId for follow-up]

---

## IDEA-01: [title]
- **Thesis**: We show that X by Y
- **Gap addressed**: [specific gap from landscape, e.g., "G3: No existing work on Z"]
- **Core mechanism**: [the key technical insight]
- **Non-obvious because**: [skeptic's objection + rebuttal]
- **Contribution type**: [empirical finding / new method / theoretical result / diagnostic / new formulation]
- **Risk**: [LOW / MEDIUM / HIGH] — [1-sentence justification]
- **Effort**: [N] person-weeks
- **Closest work**: [paper title + authors/year] — delta: [what is specifically different]

---

## IDEA-02: [title]
[same structure]

---

[repeat for all ideas]

File 2: outputs/IDEAS_FILTERED.md

This file contains the filtered, scored, and ranked ideas — the actionable output.

# Filtered Research Ideas

**Direction**: [research direction from $ARGUMENTS]
**Date**: [today's date]
**Pipeline**: Generated [X] ideas -> Feasibility filter -> Novelty quick-check -> Impact filter -> Prof. He 4-dimension filter -> [Y] surviving
**Landscape source**: [LANDSCAPE.md / abbreviated inline survey]
**Codex thread ID**: [threadId for follow-up]

---

## Surviving Ideas (ranked by Prof. He composite score, descending)

### Rank 1: [title] (IDEA-XX)
- **Thesis**: We show that X by Y
- **Gap addressed**: [specific gap]
- **Core mechanism**: [technical insight]
- **Non-obvious because**: [skeptic's objection + rebuttal]
- **Contribution type**: [type]
- **Risk**: [level] — [justification]
- **Effort**: [N] person-weeks
- **Closest work**: [paper] — delta: [difference]
- **He Score**: Longevity [X] + Passion [X] + Application [X] + Uniqueness [X] = [XX]/20
- **Anti-pattern flags**: [none / list of flags with explanations]
- **Quick novelty**: [LIKELY NOVEL / NEEDS DEEPER CHECK]
- **Why this ranks #1**: [1-2 sentences explaining why this is the top recommendation]

---

### Rank 2: [title] (IDEA-XX)
[same structure]

---

[repeat for all 4-6 surviving ideas]

---

## Eliminated Ideas

| # | Idea | Stage | Reason |
|---|------|-------|--------|
| IDEA-XX | [title] | Feasibility | [e.g., Requires unavailable dataset (ImageNet-22k with annotations)] |
| IDEA-XX | [title] | Novelty | [e.g., Already published: "Paper Title" (Author et al., 2025)] |
| IDEA-XX | [title] | Impact | [e.g., Neither positive nor negative result would change practice] |
| IDEA-XX | [title] | He Filter | [e.g., Score 10/20 — Longevity 2 (trend-dependent), Uniqueness 2 (easily replicated)] |

---

## Risk Distribution of Survivors
| Risk Level | Count | Ideas |
|------------|-------|-------|
| HIGH | [N] | [IDEA-XX, IDEA-XX] |
| MEDIUM | [N] | [IDEA-XX, IDEA-XX] |
| LOW | [N] | [IDEA-XX, IDEA-XX] |

---

## Suggested Next Steps
1. Run `/idea-screen` on the top 2-3 ideas for deep multi-dimensional screening
2. Run `/idea-refine` on the #1 ranked idea to iteratively sharpen it
3. If no landscape was available, run `/lit-survey "[direction]"` and re-run `/idea-gen` for better results

---

## Methodology Notes
- Brainstorming model: gpt-5.4 with xhigh reasoning effort
- Filtering pipeline: Feasibility -> Novelty quick-check -> Impact -> Prof. He 4-dimension (threshold: 12/20) -> Anti-pattern check
- Novelty checks are quick (2-3 searches per idea); run `/idea-screen` for deep novelty verification
- Prof. He scores reflect researcher-agnostic assessment unless user provided preference information

Writing procedure:

1. Ensure the outputs/ directory exists: mkdir -p outputs/
2. Write outputs/IDEAS_RAW.md using the Write tool
3. Write outputs/IDEAS_FILTERED.md using the Write tool
4. Large file fallback: If the Write tool fails due to file size, immediately retry using Bash:

   cat << 'FILEEOF' > outputs/IDEAS_RAW.md
   [content]
   FILEEOF
   

   Do NOT ask the user for permission — just do it silently.

Key Rules

1. The user provides a DIRECTION, not an idea. Your job is to generate the ideas. Do not ask the user "what idea do you want to explore?" — that is your task.

2. Quantity first, quality second. Brainstorm broadly in Phase 2, then filter ruthlessly in Phases 3-5. The external LLM should generate freely without over-constraining.

3. A good negative result is just as publishable as a positive one. Prioritize ideas where the answer matters regardless of which way it goes. An idea where only one outcome is interesting is a weaker idea.

4. Don't fall in love with any idea before validating it. Be willing to kill ideas that don't pass the filters, even if they sound exciting.

5. "Apply X to Y" is the lowest form of research idea. Push for deeper questions: Why does X work? When does X fail? What assumption does X make that is wrong?

6. Include eliminated ideas in the report. They save future time by documenting what was considered and why it was rejected. A researcher returning to this direction later will benefit from seeing the dead ends.

7. If the user's direction is too broad, auto-narrow it. Do not stop or ask the user to clarify. Instead, identify the top 3 sub-directions from the landscape and generate ideas for each (see Phase 1, step 4). Log the auto-narrowing decision to outputs/PIPELINE_LOG.md.

8. Respect the phase boundaries. Do not skip phases or combine them. Each phase has a distinct purpose and rushing through produces lower-quality output.

9. Track provenance. Every claim about the landscape, every novelty assessment, and every gap reference should be traceable back to a specific search result or landscape file entry.

10. Be transparent about confidence. If the landscape data is thin (abbreviated inline survey rather than full /lit-survey), say so. If a novelty check is inconclusive, say NEEDS DEEPER CHECK rather than guessing.

Composing with Other Skills

This skill is designed to work as part of a larger research idea pipeline:

/lit-survey "direction"    -> landscape (run first for best results)
/idea-gen "direction"      <- you are here
/idea-screen               -> deep multi-dimensional screening of top ideas
/idea-refine               -> iterative refinement of top ideas
/idea-pipeline             -> full automated workflow (runs the above in sequence)

Upstream: /lit-survey produces outputs/LANDSCAPE.json and outputs/LANDSCAPE.md which this skill consumes. Running /lit-survey first significantly improves idea quality because the landscape data is more comprehensive.

Downstream: /idea-screen takes the surviving ideas from outputs/IDEAS_FILTERED.md and performs deep multi-dimensional screening (novelty verification, critical review, competitive analysis). /idea-refine then iteratively sharpens the top ideas based on screening feedback.