Argument Audit

Step 1.5: Internal Consistency Scan

Before evaluating individual arguments, systematically check that the paper does not contradict itself. Contradictions separated by many paragraphs are easy to miss during linear reading — this step forces an exhaustive cross-referencing pass.

1.5.1 Claim-Level Consistency

For each core claim or design principle extracted in Step 1:

• Search the full paper for language that contradicts, weakens, or qualifies the claim
• Pay special attention to the gap between principles (often stated in the introduction) and descriptions (often stated in methods/design sections) — a principle declared as foundational in §1 must not be described as optional in §3
• Flag any claim whose status shifts between sections (required ↔ optional, universal ↔ conditional, validated ↔ hypothetical) without explicit acknowledgment of the shift

1.5.2 Terminology Consistency

• Identify key terms that carry architectural or argumentative weight (e.g., "content-agnostic," "required," "optional," "validated," "provenance")
• For each term, check that its meaning and scope remain stable across all uses
• Flag equivocation: same term used with different scopes or implications in different sections

1.5.3 Dependency Integrity

• Using the argument dependency graph from Step 1, trace each critical-path argument to its dependencies
• For each dependency, verify that the depended-upon claim is not undermined elsewhere in the paper
• Specifically check: if Claim A depends on Component B, does the paper anywhere describe Component B as optional, unimplemented, or unnecessary?

1.5.4 Abstract–Body Alignment

• Compare the abstract's framing of each major claim against how that claim appears in the body
• Flag any claim that is stronger in the abstract than its body treatment supports, or that uses different framing language (e.g., abstract says "requires" but body says "can optionally use")

1.5.5 Invariant Compliance

If a domain model with invariants exists for the project being audited (./docs/domain-model.md, § Invariants):

• Read the invariants before auditing any document
• For every document being audited, check that no claim contradicts a current invariant
• Invariant violations are constitutional violations — higher severity than normal internal contradictions, because they mean the document is operating on superseded assumptions
• Flag these in the Internal Consistency Report as a separate category

## Internal Consistency Report

### Contradictions Found
| # | Claim A (Location) | Claim B (Location) | Nature of Contradiction |
|---|---|---|---|
| 1 | ... | ... | ... |

### Terminology Shifts
| Term | Meaning in [Section] | Meaning in [Section] | Problematic? |
|---|---|---|---|
| ... | ... | ... | Yes/No |

### Broken Dependency Chains
| Conclusion | Depends On | But Paper Says | Location |
|---|---|---|---|
| ... | ... | ... | ... |

### Abstract–Body Mismatches
| Abstract Claim | Body Treatment | Gap |
|---|---|---|
| ... | ... | ... |

### Invariant Violations (if domain model exists)
| Document | Claim (Location) | Contradicted Invariant | Severity |
|---|---|---|---|
| ... | ... | ... | Constitutional |

Note: Contradictions found here are often the highest-impact findings in the entire audit. A paper with locally valid arguments that globally contradict each other has a more serious problem than a paper with a weak individual inference — the former suggests the author hasn't fully worked out their own position.

Step 2: Logical Audit

For each argument in the map, evaluate:

2.1 Validity of Inference

Check for:

• Non sequitur: Conclusion doesn't follow from premises
• Affirming the consequent: "If A then B; B; therefore A"
• Hasty generalization: Small sample → broad claim
• False dichotomy: Presenting two options when more exist
• Equivocation: Same term used with different meanings
• Circular reasoning: Conclusion assumed in premises
• Post hoc ergo propter hoc: Temporal sequence treated as causation
• Appeal to authority: Citation used as proof rather than evidence
• Straw man: Misrepresenting opposing views to dismiss them
• Composition/division: Attributing properties of parts to whole or vice versa

2.2 Premise Evaluation

For each premise:

• Stated or unstated? Unstated premises (hidden assumptions) are the most dangerous
• Empirically supported? Is evidence provided, and is it adequate?
• Contested? Would reasonable scholars dispute this premise?
• Scope-appropriate? Does the premise actually apply to the context claimed?

2.3 Evidence-Claim Alignment

For each claim backed by evidence:

• Does the evidence actually support the specific claim made (not just a related claim)?
• Is the evidence sufficient (not just a single study or anecdote)?
• Are there alternative explanations for the evidence that the paper doesn't consider?
• Is there known contradictory evidence the paper ignores?

Step 3: Gap Analysis

Identify:

## Argument Gaps

### Unsupported Claims
[Claims made without evidence or reasoning]
- **Claim:** [quote] — **Location:** [section/paragraph]
- **What's needed:** [what evidence or argument would support this]

### Hidden Assumptions
[Premises the argument requires but never states]
- **Assumption:** [what's being assumed]
- **Where it operates:** [which arguments depend on it]
- **Risk:** [what happens to the argument if this assumption is wrong]

### Missing Counterarguments
[Objections a critical reader would raise that the paper doesn't address]
- **Objection:** [what a skeptic would say]
- **Applies to:** [which argument]
- **Severity:** [would this undermine a minor point or the central thesis?]

### Scope Overreach
[Where conclusions go beyond what the evidence supports]
- **Claim:** [what the paper says]
- **Evidence supports:** [what the evidence actually shows]
- **Gap:** [the distance between evidence and claim]

### Inferential Leaps
[Where the paper jumps from A to C without establishing B]
- **From:** [established point]
- **To:** [claimed conclusion]
- **Missing step:** [what needs to be argued]

Step 4: Strength Assessment

Not just weaknesses — identify what the argument does well:

## Argument Strengths
- [Well-constructed arguments, elegant reasoning, effective use of evidence]

## Strongest Links
[The most well-supported inferences in the paper]

## Weakest Links
[The inferences most vulnerable to challenge — if these fail, what collapses?]

Step 5: Synthesis Report

## Argument Audit Summary

**Overall logical coherence:** [Strong / Moderate / Weak]
**Critical vulnerabilities:** [count] — issues that threaten the central thesis
**Moderate issues:** [count] — weaken individual arguments but don't collapse the thesis
**Minor issues:** [count] — presentation or precision problems

### The Strongest Version of This Argument
[Restate the paper's argument in its strongest possible form — steelman it.
What would the paper look like if all gaps were filled?]

### What Must Be Fixed
[Prioritized list of logical repairs, ordered by impact on the central thesis]

### Recommended Revisions
[Specific, actionable suggestions for strengthening the argument]
1. **[Issue]** — [What to do about it, with specific location in paper]

IMPORTANT PRINCIPLES

• Steelman first: Before criticizing, make sure you understand the argument in its strongest form. Misrepresenting the argument to find flaws is itself a logical error.
• Distinguish logical from empirical: A valid argument can have false premises. An invalid argument can have true premises. Separate these assessments.
• Hidden assumptions matter most: The claims the paper doesn't realize it's making are more dangerous than the ones it makes explicitly.
• Not all gaps are fatal: Some gaps are easily filled, some are standard practice in the field. Focus on gaps that actually threaten the argument.
• Be specific: "The logic is weak" is useless. "The inference from X to Y on page 4 assumes Z, which is contested by [source]" is useful.