Reproducibility Check

Usage Examples

Check your own paper before submission

reproducibility_check.check_manuscript(
    manuscript_text="[paste manuscript]",
    target_venue="NeurIPS",
    include_checklist=True
)

Check if an external paper can be reproduced

reproducibility_check.check_external(
    doi="10.48550/arXiv.2310.xxxxx",
    check_code_availability=True,
    check_data_availability=True
)

Generate reproducibility checklist for a venue

reproducibility_check.checklist(
    venue="NeurIPS",
    year=2024,
    format="markdown_checklist"
)

Score reproducibility

reproducibility_check.score(
    doi="10.48550/arXiv.2310.xxxxx",
    rubric="ml_reproducibility_challenge"
)
# Returns: {"score": 6.5/10, "breakdown": {...}, "critical_gaps": [...]}

Output Format

Returns structured checklist with pass/fail per item, critical gaps flagged in red, and an overall reproducibility score. Fix recommendations are actionable: "Add random seed to Section 3.2" not "improve reproducibility."

Best Paired With

• method_reviewer for missing technical detail checks
• experiment_skeptic for submission-readiness and reviewer-risk checks

Notes

• Code availability is checked by attempting to resolve GitHub/Zenodo links — link rot is flagged
• Dataset availability is verified against known registries (HuggingFace, Zenodo, paperswithcode)
• NeurIPS 2024+ requires a mandatory reproducibility checklist at submission — this skill generates it
• Well-logged experiments and explicit implementation notes make reproducibility checks much easier to pass