Run Experiments

Never ignore contradictory evidence. If the experiment doesn't support the claim, update the claim — don't cherry-pick results.

The 4 Experiment Stages

Work through these stages sequentially. Each stage builds on the previous one's results.

Stage 1: Initial Implementation (get it working)

Goal: Produce a correct, running implementation of the proposed method.

1. Read idea.md (check version in idea.json) to understand what needs to be implemented
2. Write runfile.py (or a more descriptively named script) that:
   • Loads or generates the required data
   • Implements the proposed method
   • Trains/runs the model
   • Evaluates on relevant metrics
   • Prints results to stdout as METRIC_NAME: VALUE
   • Saves numerical results as .npy files in experiment_results/
   • Generates basic diagnostic plots (loss curves, sample outputs, etc.)
3. Run the script and check the output
4. If there are errors, debug and re-run (up to 3 debug attempts per issue)
5. Verify the results make basic sense (sanity checks)
6. → Log & Evaluate: Append experiment_completed event to research_log.jsonl. Assess if initial results are consistent with the hypothesis.

Output format for metrics (print to stdout):

train_loss: 0.342
test_accuracy: 0.891
val_f1: 0.756

Output format for data (save to experiment_results/):

np.save("experiment_results/train_losses.npy", train_losses)
np.save("experiment_results/test_accuracies.npy", test_accuracies)

Stage 2: Baseline Comparison (establish context)

Goal: Implement baselines and run fair comparisons.

0. Read memory: Check research_log.jsonl for Phase 1 findings. Re-read idea.md in case it was updated.
1. Implement each baseline method described in idea.md
2. Run all methods (proposed + baselines) under identical conditions:
   • Same data splits
   • Same evaluation protocol
   • Same number of seeds (at least 3)
   • Fair hyperparameter tuning for all methods
3. Save per-method results:

   np.save("experiment_results/baseline_random_accuracies.npy", results)
   np.save("experiment_results/proposed_method_accuracies.npy", results)
   

4. Generate comparison plots (bar charts, learning curves)
5. Write a summary:

   summary = {
       "methods": {
           "proposed": {"accuracy": {"mean": 0.89, "std": 0.02}},
           "baseline_1": {"accuracy": {"mean": 0.82, "std": 0.03}},
       },
       "best_method": "proposed",
       "analysis": "Proposed method outperforms baseline by 7% on accuracy..."
   }
   with open("baseline_summary.json", "w") as f:
       json.dump(summary, f, indent=2)
   

6. → Log & Evaluate: This is the most critical feedback point. If baselines outperform the proposed method, update the idea:
   • Shift the claim (e.g., from "higher accuracy" to "better efficiency-accuracy tradeoff")
   • Add new metrics that capture the actual advantage
   • Or pivot the approach entirely
   • Document the change in idea_evolution.md and research_log.jsonl

Stage 3: Full Experiments (comprehensive results)

Goal: Run full-scale experiments with thorough evaluation.

0. Read memory: Re-read idea.md — it may have been updated after Phase 2. Check idea_evolution.md to understand what changed and why.
1. Scale up from Stage 2:
   • Full dataset (not subsampled)
   • More evaluation metrics
   • Multiple random seeds (3-5)
   • Statistical significance tests if applicable
2. Generate comprehensive results:
   • Learning curves (loss/metric vs. epoch for all methods)
   • Performance tables (mean +/- std across seeds)
   • Visualization of model outputs (if applicable: generated samples, attention maps, etc.)
   • Distribution plots for key metrics
3. Save everything:

   np.save("experiment_results/full_learning_curves.npy", curves)
   np.save("experiment_results/full_comparison_results.npy", results)
   

4. Write research summary:

   summary = {
       "key_findings": ["..."],
       "best_configuration": {...},
       "statistical_tests": {...},
       "analysis": "..."
   }
   with open("research_summary.json", "w") as f:
       json.dump(summary, f, indent=2)
   

5. → Log & Evaluate: Full-scale results may reveal scaling behaviors not visible in Phase 2. If findings change the story, update the idea.

Stage 4: Ablation Studies (validate components)

Goal: Systematically validate which components of the proposed method matter.

0. Read memory: Re-read idea.md — ablate the current version of the method, which may have evolved since Stage 1.
1. Identify the key components to ablate (from idea.md and Stage 3 results)
2. For each ablation:
   • Remove or replace one component at a time
   • Run with the same setup as Stage 3
   • Record the impact on all metrics
3. Common ablation patterns:
   • Remove a module → measure degradation
   • Replace a component with a simpler alternative
   • Vary a key hyperparameter across a range
   • Test with different dataset sizes (scaling behavior)
4. Save ablation data:

   np.save("experiment_results/ablation_no_attention.npy", results)
   np.save("experiment_results/ablation_lr_sweep.npy", results)
   

5. Write ablation summary:

   summary = {
       "ablations": [
           {
               "name": "without_attention",
               "change": "Removed self-attention layer",
               "impact": {"accuracy": -0.05, "loss": +0.12},
               "conclusion": "Attention is important for ..."
           }
       ],
       "analysis": "The most critical component is ..."
   }
   with open("ablation_summary.json", "w") as f:
       json.dump(summary, f, indent=2)
   

6. → Log & Evaluate: If ablations show a component is unnecessary, consider simplifying the method and updating idea.md.

Iteration and Debugging

For each experiment run:

1. Execute the script via python runfile.py (or the relevant script)
2. Check output: Read stdout for metrics and stderr for errors
3. If error occurs:
   • Read the traceback carefully
   • Identify the root cause (shape mismatch, missing import, data issue, etc.)
   • Fix the code
   • Re-run (max 3 debug attempts per issue)
4. If results look wrong (NaN, zero accuracy, loss exploding):
   • Check for common issues: learning rate too high, data not normalized, loss function mismatch
   • Add debugging prints if needed
   • Fix and re-run
5. If results look reasonable: Save and proceed to next stage

Key Principles

• Reproducibility: Always set random seeds. Save exact configurations alongside results.
• Fair comparison: Give baselines the same tuning budget as the proposed method.
• Statistical rigor: Use multiple seeds. Report mean +/- std. Note if differences are significant.
• Save everything: Any number that might appear in the paper should be saved as a .npy file.
• Diagnostic plots: Generate plots as you go — they help catch bugs early.

Final Output Structure

After all 4 stages, the workspace should contain:

experiments/{idea_name}/
├── idea.md                       # Current idea (final version, may differ from v1)
├── idea.json                     # Metadata with version number
├── idea_evolution.md             # Complete history of idea changes
├── research_log.jsonl            # Full event log (all experiments + idea updates)
├── config.yaml
├── runfile.py                    # Main experiment script(s)
├── baseline_summary.json         # Stage 2 summary
├── research_summary.json         # Stage 3 summary
├── ablation_summary.json         # Stage 4 summary
├── experiment_results/
│   ├── train_losses.npy
│   ├── test_accuracies.npy
│   ├── baseline_*.npy
│   ├── ablation_*.npy
│   ├── full_*.npy
│   └── *.png                     # Intermediate diagnostic plots
├── figures/                      # (populated by aggregate-plots skill)
└── logs/
    └── experiment_log.txt