Study Quality Appraiser

This is a strong checkpoint before:

• claim-strength judgments
• evidence synthesis
• research recommendations based on published studies

Scope Boundaries

In scope:

• identify study type and evidence level
• assess obvious strengths and weaknesses
• flag bias, sample, measurement, and applicability concerns
• estimate how much confidence the study deserves

Out of scope by default:

• full statistical reanalysis
• exhaustive risk-of-bias frameworks by default
• synthesizing many studies into one final narrative
• pretending one study can settle a broad question

Appraisal Workflow

1. State the claim the study is being used to support.
2. Identify the study type:
   • systematic review / meta-analysis
   • randomized trial
   • cohort / case-control / observational
   • cross-sectional
   • case series / anecdotal
   • preclinical / animal / mechanistic
   • benchmark / simulation / theoretical
   • preprint or non-peer-reviewed item
3. Assess core quality signals:
   • design strength
   • sample adequacy
   • outcome relevance
   • confounding or bias risk
   • fit between claim and measured result
4. Note major limitations.
5. Judge applicability to the user's actual question.
6. Return a conservative confidence assessment.

Escalation Conditions

Pause and keep the appraisal narrow when:

• the study is being used for a claim broader than it actually tests
• the paper details are too thin to assess responsibly
• multiple studies conflict and cross-study synthesis is now required
• the real issue is search coverage, not study quality

If those conditions persist, recommend literature-search, evidence-synthesizer, or a narrower claim before stronger conclusions.

Appraisal Heuristics

Prefer higher confidence when:

• the design directly fits the claim
• outcomes are relevant and clearly measured
• limitations are known but bounded
• the study is peer-reviewed and methodologically appropriate

Reduce confidence when:

• the evidence is preclinical or indirect
• the sample is tiny or obviously unrepresentative
• the measured outcome is only a weak proxy for the claim
• major confounding, bias, or missing controls are likely
• the claim being made is much stronger than the study result
• the source is a preprint with no reviewed corroboration

Certainty Lens

When quality matters, assess these certainty dimensions explicitly:

• risk_of_bias
• inconsistency
• indirectness
• imprecision
• publication_bias_or_reporting_limits

Use them as a lightweight certainty check, not as a formal full-review claim.

Preprint-Aware Appraisal

If the source is a preprint or otherwise non-peer-reviewed:

• treat it as potentially valuable but not fully stabilized evidence
• separate methodological promise from evidentiary confidence
• ask whether stronger reviewed support exists
• avoid letting a single preprint carry a broad claim

Output Contract

Always return:

1. study_or_source
2. target_claim
3. study_type
4. quality_signals
5. major_limitations
6. applicability
7. evidence_strength (strong, moderate, weak, very-weak)
8. certainty_dimensions
9. peer_review_status
10. confidence_notes
11. recommended_followup
12. next_step

Guardrails

• Do not equate publication with strong evidence.
• Do not treat preclinical or mechanistic work as direct proof for broad human claims.
• Do not upgrade confidence because findings are exciting or intuitive.
• Keep claim fit separate from study quality; a decent study can still be a poor match for the claim.
• If one study is clearly insufficient, say so.
• If the source is preprint-only, do not hide that the confidence ceiling is lower.

Anti-Patterns

• abstract-only certainty
• one-paper overreach
• ignoring proxy outcomes
• treating observational association as causal proof
• using weak studies to force a yes/no conclusion

Example

User request:

We found a small study suggesting cold exposure improves mood. How much should we trust it?

Expected shape of response:

1. study_or_source: small mood-related cold exposure study
2. target_claim: cold exposure improves mood or mental health
3. study_type: likely small human intervention or observational study
4. quality_signals: direct intervention but limited sample and likely short-term outcomes
5. major_limitations: low power, heterogeneous protocol, short follow-up, possible proxy outcome issues
6. applicability: partial for short-term mood effects, weak for broad mental health claims
7. evidence_strength: weak
8. confidence_notes: interesting signal but far from settled support for broad claims
9. recommended_followup: evidence-synthesizer or broader literature search
10. next_step: compare it against higher-quality or larger studies before making a claim-strength judgment