Pdf Markdown Validator

Core concepts

Validation Framework

The validation framework computes a composite quality score (0–100) combining four independent metric dimensions:

FinalScore = (0.40 × TableAccuracy) + (0.40 × StyleAccuracy)
           + (0.10 × Robustness) + (0.10 × Performance)

Each dimension is independent and can be evaluated separately or together.

1. Table Accuracy (40% weight)

Measures how accurately tables are detected and their cell content extracted.

Components:

• Table Detection F1: IoU-based matching of predicted vs. gold tables (IoU ≥ 0.5 threshold)

  • Precision: correctly identified tables / all detected tables
  • Recall: correctly identified tables / all gold tables
  • F1: harmonic mean of precision and recall

• Cell Content Accuracy: Token-level F1 averaging across matched table cells

  • Matches cells by position within detected tables
  • Unmatched cells score 0.0
  • Aggregates to mean F1 across all cells

Formula:

TableAccuracy = (0.5 × TableDetectionF1) + (0.5 × CellContentAccuracy)

Interpretation:

• 90–100: Excellent table handling; tables detected and content extracted accurately
• 70–89: Good; minor detection misses or cell content variations
• 50–69: Fair; tables detected but cells have significant content errors
• Below 50: Poor; systematic table detection failures or content corruption

2. Style Accuracy (40% weight)

Measures how accurately text formatting (bold, italic, heading levels) is preserved.

Components:

• Bold F1: Token-level F1 for bold detection
• Italic F1: Token-level F1 for italic detection
• Heading F1: Heading-level accuracy (prediction must match gold level exactly; e.g., H2 ≠ H3)

Formula:

StyleAccuracy = macro_average(BoldF1, ItalicF1, HeadingF1)
              = (BoldF1 + ItalicF1 + HeadingF1) / 3

Interpretation:

• 90–100: Styles consistently preserved; formatting markup matches gold
• 70–89: Good; minor style detection misses
• 50–69: Fair; mixed style accuracy; some styles missed consistently
• Below 50: Poor; styles largely lost or misdocumented

3. Robustness (10% weight)

Measures system stability and validity across a test corpus, particularly edge cases.

Components:

• Crash-free rate: Percent of documents processed without panics or non-zero exit codes
• Markdown validity: Percent of generated Markdown files passing pandoc syntax validation
• Completeness: Percent of documents producing non-empty output

Formula:

Robustness = (CrashFreeRate + MarkdownValidityRate + CompletenessRate) / 3

Interpretation:

• 95–100: Production-ready; system handles edge cases gracefully
• 85–94: Stable; occasional edge-case issues but recovers
• 70–84: Moderate stability; known failure patterns on specific input types
• Below 70: Fragile; systematic crashes or invalid output on subset of inputs

4. Performance (10% weight)

Measures processing speed relative to a baseline; targets 1-page PDF ≈ 200–500ms.

Components:

• Median latency: P50 processing time across corpus
• P95 latency: 95th percentile processing time (captures tail behavior)

Formula:

Performance = 0.5 × min(1.0, baseline_median / run_median)
            + 0.5 × min(1.0, baseline_p95 / run_p95)

Interpretation:

• 95–100: Fast processing; meets or exceeds baseline
• 80–94: Good; acceptable slowdown (≤20%)
• 60–79: Acceptable; noticeable overhead (20–60%)
• Below 60: Slow; significant performance regression

Quick start

1. Prepare ground-truth annotations

Create .gold.md files for each test PDF:

# Copy reference Markdown next to PDF with .gold.md extension
cp reference_output.md test.gold.md

# Format: Each section annotated with metadata
# Gold format example:
# # Heading 1
# **bold text** and *italic text*
#
# | Column A | Column B |
# |----------|---------|
# | cell 1   | cell 2  |

2. Run validation

# Evaluate against ground truth
cargo run -p edgequake-pdf --example real_dataset_eval -- \
  --input crates/edgequake-pdf/test-data/real_dataset \
  --gold \
  --metrics

# Generate detailed report
python3 .github/skills/pdf-markdown-validator/scripts/validate.py \
  --pdf-dir crates/edgequake-pdf/test-data/real_dataset \
  --gold-dir . \
  --output-report metrics_report.json

3. Interpret results

# View summary scores
cat metrics_report.json | jq '.summary'

# Analyze failures by category
python3 .github/skills/pdf-markdown-validator/scripts/analyze_failures.py \
  metrics_report.json

Capabilities

Core Capabilities

1. Metric Computation

• Token-level F1 calculation: Boolean/classification metrics at granular text unit level
• Spatial matching: IoU-based table detection using bounding box coordinates
• Aggregate averaging: Macro-averaging (label-balanced) and weighted averaging strategies
• Normalization: Robust handling of missing annotations and edge cases

2. Evaluation Harness

• Batch evaluation: Process entire corpora in single invocation
• Incremental scoring: Support partial annotations and progressive refinement
• Performance profiling: Automatically capture timing with statistical summaries (median, P95)
• Error categorization: Classify failures into systematic buckets (missed tables, style loss, etc.)

3. Reporting & Visualization

• JSON reports: Machine-readable output for CI/CD integration
• Markdown summaries: Human-friendly reports with trend analysis
• Per-document breakdown: Detailed scores for each file with failure reasons
• Comparison mode: Side-by-side analysis of baseline vs. current runs

4. Validation Gates

• Hard gates: Fail evaluation if Markdown is invalid or crashes occur
• Soft gates: Warn if quality drops below threshold but allow pass
• Trend analysis: Flag regressions even if absolute score passes
• Baseline anchoring: Detect performance slowdowns vs. previous runs

Integration Points

With Cargo Testing

# Embed metrics in standard cargo test output
cargo test -p edgequake-pdf -- --nocapture

# Fail CI if composite score below threshold
cargo test -p edgequake-pdf --features ci-strict

With Python Evaluation Scripts

from pdf_validator import PDFValidator

validator = PDFValidator(
    pdf_dir="test-data/real_dataset",
    gold_dir="test-data/gold",
    metrics=["table", "style", "robustness", "performance"]
)
score = validator.evaluate()
print(f"Composite Score: {score.composite}/100")

With CI/CD Pipelines

# GitHub Actions example
- name: Validate PDF → Markdown
  run: |
    cargo run -p edgequake-pdf --example real_dataset_eval -- --metrics
    python .github/skills/pdf-markdown-validator/scripts/validate.py \
      --ci-mode --fail-below 75

Metric Definitions & Examples

Table Accuracy Example

Input PDF: 2×3 table with headers "Name, Age" and row "John, 25"

Gold Markdown:

| Name | Age |
| ---- | --- |
| John | 25  |

Generated Markdown (Perfect):

| Name | Age |
| ---- | --- |
| John | 25  |

Scores:

• TableDetectionF1: 1.0 (table detected correctly)
• CellContentAccuracy: 1.0 (all cells match token-for-token)
• TableAccuracy: 1.0

Generated Markdown (Partial Match):

| Name | Age  |
| ---- | ---- |
| John | 25.0 |

Scores:

• TableDetectionF1: 1.0 (table structure preserved)
• CellContentAccuracy: 0.8 (minor token variations: "25" vs "25.0")
• TableAccuracy: 0.9

Style Accuracy Example

Gold Markdown:

# Main Heading

This is **bold** and _italic_ text.

## Sub Heading

More content here.

Generated Markdown (Perfect):

# Main Heading

This is **bold** and _italic_ text.

## Sub Heading

More content here.

Scores:

• BoldF1: 1.0
• ItalicF1: 1.0
• HeadingF1: 1.0
• StyleAccuracy: 1.0

Generated Markdown (Partial):

# Main Heading

This is bold and italic text.

## Sub Heading

More content here.

Scores:

• BoldF1: 0.5 (missed "bold" marker)
• ItalicF1: 0.5 (missed "italic" marker)
• HeadingF1: 1.0 (headings correct)
• StyleAccuracy: 0.67

Robustness Example

Test corpus: 30 PDFs (including 5 edge cases: corrupted, multilingual, scanned, etc.)

Results:

• Crash-free rate: 29/30 = 96.7% (1 crash on scanned PDF)
• Markdown validity: 30/30 = 100% (all outputs valid)
• Completeness: 28/30 = 93.3% (2 near-empty outputs from obfuscated fonts)

Robustness Score: (96.7 + 100 + 93.3) / 3 = 96.7%

Performance Example

Baseline (previous release):

• Median: 250ms per 1-page PDF
• P95: 800ms per 1-page PDF

Current run:

• Median: 280ms per 1-page PDF
• P95: 750ms per 1-page PDF

Scores:

• Median: min(1.0, 250/280) = 0.89
• P95: min(1.0, 800/750) = 1.0
• Performance: 0.5 × 0.89 + 0.5 × 1.0 = 0.95

Workflow: Running a Full Validation

Step 1: Prepare Test Data

# Navigate to PDF crate
cd edgequake/crates/edgequake-pdf

# Ensure ground-truth annotations exist
# Files should be named: <pdf_name>.gold.md
ls -1 test-data/real_dataset/*.gold.md

Step 2: Generate Markdown Output

# Convert all PDFs to Markdown
cargo run -p edgequake-pdf --example real_dataset_eval -- --write

# Outputs written to: test-data/real_dataset/*.md

Step 3: Run Validation

# Compute all metrics
python3 ../../.github/skills/pdf-markdown-validator/scripts/validate.py \
  --pdf-dir test-data/real_dataset \
  --gold-dir test-data/real_dataset \
  --output-report validation_report.json \
  --verbose

Step 4: Analyze Results

# View summary
cat validation_report.json | jq '.summary'

# Detailed per-document breakdown
cat validation_report.json | jq '.documents | .[] | {name, scores}'

# Identify failure patterns
python3 ../../.github/skills/pdf-markdown-validator/scripts/analyze_failures.py \
  validation_report.json --group-by failure_type

Step 5: Iterative Improvement

1. Identify low-scoring dimensions (table, style, etc.)
2. Focus on highest-impact directory (processor, renderer, layout)
3. Implement targeted fix with tests
4. Re-run validation to measure improvement
5. Record baseline for next comparison

Ground-Truth Annotation Format

The .gold.md files serve as reference implementations. Use this structure:

# Document Title (H1)

## Section Heading (H2)

This paragraph contains **bold text** and _italic text_ and **_bold-italic text_**.

### Subsection (H3)

#### Sub-subsection (H4)

**Note:** Use standard Markdown syntax. Be precise with:

- Bold: **text**
- Italic: _text_
- Bold-Italic: **_text_**
- Headings: # through #### for H1–H4

### Tables

| Column 1 | Column 2 | Column 3 |
| -------- | -------- | -------- |
| Cell 1   | Cell 2   | Cell 3   |
| Cell 4   | Cell 5   | Cell 6   |

Ensure:

- Pipes align properly
- Headers separated by `---|---` row
- No trailing spaces (can affect parsing)

### Code Blocks

\`\`\`python
def hello():
print("world")
\`\`\`

Use triple backticks with language identifier.

### Lists

Bullet list:

- Item 1
- Item 2
  - Nested item
- Item 3

Numbered list:

1. First
2. Second
3. Third

### Edge Cases

- **Multi-line table cells**: Not standard Markdown; flatten to single line
- **Merged cells**: Not representable in Markdown tables; split into separate rows
- **Vertical headers**: Use first row convention (all cells with **bold**)

CLI Commands & Scripts

Validation Runner

# Full validation pipeline
python3 scripts/validate.py \
  --pdf-dir <path/to/pdfs> \
  --gold-dir <path/to/gold> \
  [--output-report <report.json>] \
  [--metrics table,style,robustness,performance] \
  [--ci-mode] \
  [--fail-below 75]

Options:

• --pdf-dir: Directory containing PDFs and generated .md files
• --gold-dir: Directory containing .gold.md reference files
• --output-report: JSON file for machine-readable results (default: validation_report.json)
• --metrics: Comma-separated metrics to compute (default: all)
• --ci-mode: Fail with non-zero exit code if score below threshold
• --fail-below: Minimum acceptable score (default: 75)

Failure Analysis

# Identify and categorize failures
python3 scripts/analyze_failures.py \
  <report.json> \
  [--group-by failure_type|document|metric] \
  [--export <output.csv>]

Comparison Tool

# Compare two validation runs
python3 scripts/compare_runs.py \
  <baseline_report.json> \
  <current_report.json> \
  [--show-improvements] \
  [--show-regressions]

Integration with CI/CD

GitHub Actions Example