Adversarial verification for AI-generated legal content. Use when fact-checking legal documents, validating citations, detecting hallucinations, scoring document quality, or assessing distribution readiness.
Production-ready adversarial verification framework for AI-generated legal content. Covers factual accuracy, citation validation, arithmetic checking, speculation detection, and distribution readiness scoring.
Every AI-generated legal document must be checked across 6 categories.
| # | Category | What to Check | Red Flags |
|---|---|---|---|
| 1 | Factual Accuracy | Dates, references, numbers, entity names, timelines | Wrong effective dates, confused entity names, incorrect amounts |
| 2 | Legal Authority Citations | Primary/secondary sources, format, hierarchy, currency | Non-existent articles, wrong section numbers, outdated citations |
| 3 | Arithmetic Validation | Timelines, percentages, financial calculations, deadlines | Date math errors, percentage miscalculations, compounding mistakes |
| 4 | Source Verification | Verifiable claims, official sources, cross-referencing | Unverifiable assertions stated as fact, single-source claims |
| 5 | Speculation Detection | Opinion vs fact, uncertainty language, predictive claims | Predictions stated as certainty, guidance treated as binding law |
| 6 | Disclaimer Adequacy | Legal advice disclaimers, jurisdiction, date, professional consultation | Missing disclaimers, overly broad claims, no jurisdiction limits |
Scans legal text for verifiable claims and flags potential hallucination patterns.
# Check a legal document
python scripts/legal_fact_checker.py --input document.txt
# Check with JSON output
python scripts/legal_fact_checker.py --input memo.txt --json
# Check inline text
python scripts/legal_fact_checker.py --text "Under GDPR Article 83(5), fines can reach EUR 20 million..."
# Save verification report
python scripts/legal_fact_checker.py --input document.txt --output report.json
Scores legal document quality across all 6 verification categories.
# Score a document
python scripts/legal_quality_scorer.py --input document.txt
# Score with JSON output
python scripts/legal_quality_scorer.py --input document.txt --json
# Score with detailed breakdown
python scripts/legal_quality_scorer.py --input document.txt --verbose
# Save quality assessment
python scripts/legal_quality_scorer.py --input document.txt --output assessment.json
Read the entire document with an adversarial mindset. For each claim, ask:
Mark every factual assertion, citation, date, number, and predictive statement.
For every verifiable claim, attempt to verify against official sources.
| Source Type | Verification Method | Examples |
|---|---|---|
| EU legislation | EUR-Lex official database | eur-lex.europa.eu |
| US federal law | congress.gov, govinfo.gov | Official code and statutes |
| US regulations | eCFR, Federal Register | ecfr.gov |
| UK legislation | legislation.gov.uk | Official statute database |
| Court decisions | Court databases, Westlaw, LexisNexis | Official reporters |
| Agency guidance | Agency official website | Direct download from .gov/.europa.eu |
| International treaties | UN Treaty Collection | treaties.un.org |
Rule: If a claim cannot be verified from an official source, flag it. Do not assume accuracy.
Check every calculation, date computation, and numerical claim.
| Check Type | Method |
|---|---|
| Timeline calculations | Count days/months/years between stated dates |
| Percentage calculations | Recalculate from base figures |
| Financial computations | Verify arithmetic and compounding |
| Deadline calculations | Confirm against statutory text |
| Penalty ranges | Cross-check against statute |
For every legal citation, verify:
| Element | Check |
|---|---|
| Source exists | Does the cited statute/article/section actually exist? |
| Content matches | Does the cited provision say what the document claims? |
| Citation format | Is the citation in correct format for the jurisdiction? |
| Currency | Is this the current, in-force version? |
| Hierarchy correct | Is the source characterized at the right authority level? |
Distinguish fact from opinion, certainty from prediction.
| Language Pattern | Classification | Action |
|---|---|---|
| "The law requires..." | Factual claim | Verify against statutory text |
| "Courts will likely..." | Speculation | Flag; add uncertainty qualifier |
| "It is recommended..." | Guidance | Verify source; clarify if binding |
| "Best practice suggests..." | Opinion | Label as opinion; cite source |
| "This means that..." | Interpretation | Flag if stated as fact without authority |
| "Companies must..." | Obligation claim | Verify statutory basis |
Every AI-generated legal document must include:
| Required Element | Description |
|---|---|
| Not legal advice | Clear statement that content is informational only |
| Jurisdiction limitations | Which jurisdictions are and are not covered |
| Date of preparation | When the content was prepared (law changes) |
| Professional consultation | Recommendation to consult qualified legal counsel |
| AI-generated disclosure | Statement that content was generated or assisted by AI |
| Accuracy limitations | Acknowledgment that verification is recommended |
| Severity | Definition | Examples | Action |
|---|---|---|---|
| CRITICAL | Factually wrong in a way that could cause legal harm | Wrong article number creating false obligation, incorrect penalty amount, non-existent legal requirement | Must fix before any distribution |
| HIGH | Materially misleading or unverifiable | Guidance stated as binding law, unverifiable timeline, confident but unsourced claim | Must fix or add prominent caveat |
| MODERATE | Imprecise or potentially confusing | Ambiguous language, minor date discrepancy, incomplete citation | Should fix; acceptable with caveat |
| LOW | Style or formatting issue | Citation format inconsistency, missing cross-reference, minor redundancy | Fix if time permits |
| Score | Rating | Distribution Status | Criteria |
|---|---|---|---|
| 5/5 | Distribution Ready | Safe to distribute | Zero CRITICAL/HIGH issues; all citations verified; disclaimers complete |
| 4/5 | Minor Revisions | Safe after small fixes | Zero CRITICAL; 1-2 HIGH issues with clear fixes; most citations verified |
| 3/5 | Moderate Revisions | Needs work before distribution | Zero CRITICAL; 3+ HIGH issues; some unverified citations |
| 2/5 | Major Revisions | Not safe to distribute | 1+ CRITICAL issues; multiple HIGH issues; significant unverified content |
| 1/5 | Not Distribution Ready | Requires complete rework | Multiple CRITICAL issues; pervasive inaccuracies; unreliable throughout |
AI models exhibit 5 recurring patterns when generating legal content.
| # | Pattern | Description | Detection Technique |
|---|---|---|---|
| 1 | Plausible but wrong article numbers | AI generates article/section numbers that sound correct but do not exist (e.g., "Article 42(5)" when only 42(1)-(4) exist) | Cross-reference every article number against official statute text |
| 2 | Confident but incorrect dates | Implementation timelines, effective dates, or deadlines stated with false confidence (off by weeks or months) | Verify every date against official timeline from the statute or implementing body |
| 3 | Mixing guidance and legal requirements | Treating non-binding recommendations as binding obligations (e.g., stating ENISA recommendations as NIS2 requirements) | Check whether cited source is binding legislation vs guidance; verify authority level |
| 4 | Outdated legal references | Citing superseded or repealed provisions without noting they are no longer in force | Verify currency of every cited provision; check for amendments and repeals |
| 5 | Arithmetic errors in timeline calculations | Miscounting days, months, or years between dates; wrong deadline calculations | Independently calculate every timeline; do not trust AI date math |
See references/hallucination_patterns.md for detailed examples and prevention strategies.
| Guide | Path | Description |
|---|---|---|
| Verification Methodology | references/verification_methodology.md | Complete 6-step methodology with source hierarchy and citation formats |
| Hallucination Patterns | references/hallucination_patterns.md | 5 patterns with examples, detection, and prevention strategies |
scripts/legal_fact_checker.py on the document.scripts/legal_quality_scorer.py for category scores.scripts/legal_fact_checker.py on the document.scripts/legal_quality_scorer.py on the final document.| Problem | Likely Cause | Resolution |
|---|---|---|
| Too many false positives | Regex patterns matching non-legal text | Narrow input to legal content only; use context-aware review |
| Cannot verify citation | Source not freely accessible | Note as "unverifiable from public sources"; do not assume correct |
| AI-generated text has no citations | Content is entirely unsourced | Flag entire document as unverified; score as 2/5 or lower |
| Hallucination pattern detected | AI confabulation of legal details | Replace with verified information from official source |
| Document mixes jurisdictions | No clear jurisdiction scope | Flag as HIGH; recommend splitting by jurisdiction |
| Quality score seems too high | Automated scoring has limits | Always supplement automated scoring with manual review |
| Criterion | Target |
|---|---|
| Citations verified | 100% of legal citations checked against official sources |
| Hallucination patterns scanned | All 5 known patterns checked |
| Arithmetic validated | Every calculation independently verified |
| Severity assigned | Every finding classified CRITICAL/HIGH/MODERATE/LOW |
| Quality score calculated | Composite score with per-category breakdown |
| Disclaimers verified | All 6 required disclaimer elements present |
| Distribution decision | Clear go/no-go recommendation with rationale |
In scope: Verifying factual claims in legal text, validating citations, detecting hallucination patterns, scoring document quality, assessing distribution readiness.
Out of scope: Verifying legal conclusions or interpretations, assessing litigation strategy, replacing professional legal review, accessing paid legal databases (Westlaw, LexisNexis).
Disclaimer: This skill provides a structured adversarial verification methodology. It catches common AI errors but cannot guarantee complete accuracy. Professional legal review remains essential for high-stakes documents.
| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Trusting AI-generated citations without verification | AI models routinely generate plausible but non-existent legal citations; unverified citations in distributed documents create serious credibility and legal risk | Verify every citation against official sources; assume wrong until proven right |
| Relying solely on automated checking | Automated tools catch patterns but miss contextual errors, mischaracterizations, and subtle hallucinations | Use automated tools for first pass, then conduct manual review of all flagged items and a sample of unflagged items |
| Skipping the "adversarial mindset" | Confirmation bias leads reviewers to accept plausible-sounding content; legal text that "sounds right" may still be wrong | Actively seek to disprove every claim; assume error until verified; question every specific number, date, and citation |
| Distributing with score 3/5 or below | MODERATE and HIGH issues in distributed documents undermine credibility and may cause legal harm | Set a firm distribution threshold at 4/5; no exceptions without documented risk acceptance by a qualified reviewer |
| Tool | Input | Output | Use Case |
|---|---|---|---|
legal_fact_checker.py | Legal document text | Verification report with flagged claims, citations, dates, hallucination alerts | First-pass automated scanning of legal content |
legal_quality_scorer.py | Legal document text | Quality score (1-5) with per-category breakdown and severity-classified findings | Pre-distribution quality gate |