PDF-to-Markdown Extraction

Step 1: Understand the Document

Determine from the user's request or from $ARGUMENTS:

• Source PDF path: Where the PDF is located
• Output directory: Where to write the extracted files
• Scope: The whole document, or specific pages/sections?
• Document type: Report, article, manual, form, presentation, etc.

If the user hasn't specified an output directory, create one next to the PDF (e.g., document_name/ alongside document_name.pdf).

Step 2: Build the Page Map

Read the PDF page by page using the Read tool (it handles PDFs natively). For large PDFs, read in batches of 15-20 pages.

Read the PDF: pages "1-15"

Verify the actual page count. Don't trust content like slide footers ("1 of 8") or table-of-contents page ranges — these can be wrong. Read the last few pages to confirm where the document actually ends. If pdftoppm fails on a page, the PDF doesn't have that page.

For each page, record:

• Page number (1-based)
• What section/chapter it belongs to
• What content is on it (headings, body text, figures, tables, appendices)
• Whether it contains images/diagrams that need extraction

Write the page map down before proceeding. Example:

Page 1: Title page
Page 2: Table of Contents
Page 3: Section 1 introduction (text only)
Page 4: Section 1 continued (FIGURE: bar chart of revenue)
Page 5: Section 2 (text + TABLE: quarterly results)
Page 6: Section 2 continued (FIGURE: process flow diagram)
Page 7: Appendix A (text only)

Step 3: Create Directory Structure

output_dir/
├── content.md           # Full document content in Markdown
├── manifest.yaml        # Structured metadata
├── extract_images.sh    # Reproducible extraction script
└── img/                 # Extracted figure PNGs

For longer documents with clear sections, you may split into multiple markdown files (e.g., section_1.md, section_2.md) — use your judgment based on document length and structure.

Step 4: Extract Text Content

Read each page and convert to Markdown. General rules:

• Headings: Map document headings to markdown headings (#, ##, ###)
• Lists: Use markdown lists (bulleted or numbered as appropriate)
• Tables: Use markdown tables. For complex tables, use HTML if needed.
• Math/formulas: Use LaTeX notation: $inline$ and $$display$$
• Emphasis: Preserve bold, italic, underline where meaningful
• Page breaks: Use --- between major sections if helpful
• Footnotes: Convert to markdown footnotes [^1] or inline notes
• Cross-references: Preserve as markdown links where possible

Preserve the document's logical structure. Don't just dump raw text — organize it so a reader (human or AI) can navigate the content.

Math-heavy documents (academic papers, textbooks):

• Use LaTeX notation consistently: $inline$ for inline, $$display$$ for display equations
• Number equations with \tag{N} to match the source document's numbering
• Preserve theorem/definition/proof structure with bold labels
• Special functions (Gamma, Beta, hypergeometric) should use standard LaTeX commands: \Gamma, \Beta, {}_2F_1, etc.
• If a document has no figures at all (common for pure-math papers), skip Steps 5-6 entirely and note images: [] in the manifest

Step 5: Extract Images and Figures

Only extract pages that contain actual figures, diagrams, charts, or images. Skip decorative elements, logos, and backgrounds unless the user specifically wants them.

The extraction process:

1. Extract full page as PNG at 300 dpi:

   pdftoppm -png -r 300 -f {PAGE} -l {PAGE} "$PDF" temp{PAGE}
   

   Use a unique prefix per page (e.g., temp2, temp3, temp6) instead of a shared temp prefix. This prevents filename collisions if you extract multiple pages before cleaning up, and makes the ls + convert commands unambiguous.

   pdftoppm page padding: The output filename uses zero-padded page numbers, but the padding width depends on the total page count of the PDF:

   • PDFs with 1-9 pages: temp-1.png, temp-2.png (no padding)
   • PDFs with 10-99 pages: temp-02.png, temp-14.png (2-digit)
   • PDFs with 100-999 pages: temp-002.png, temp-014.png (3-digit)

   Always check with ls temp*.png after extraction to see the actual filename before writing the convert command. This is a common source of bugs — don't guess the padding, verify it.

2. Determine crop coordinates. Read the extracted full-page PNG to see exactly what's on it, then crop to the figure area:

   convert temp-{PAGE_PADDED}.png -crop {W}x{H}+{X}+{Y} +repage output.png
   

   At 300 dpi, common page sizes are:

   • A4: ~2480x3508 pixels (portrait)
   • US Letter: ~2550x3300 pixels (portrait)
   • Widescreen 16:9 slides: ~3996x2250 pixels (landscape)
   • Standard 4:3 slides: ~3300x2475 pixels (landscape)

   Use this to estimate where the figure sits on the page. Always verify by reading the full-page PNG first.

3. Verify the crop. Read the cropped image to confirm:

   • The figure is fully visible (not cut off)
   • All labels, legends, and annotations are readable
   • No extraneous text from surrounding content is included
   • If the crop is wrong, adjust and re-extract

4. If the crop cuts off content, make it LARGER, not smaller. Start generous (e.g., full width, generous height) and tighten later. A too-large crop with some whitespace is far better than a too-tight crop missing labels or legends.

Naming convention for extracted images:

• Use descriptive names: fig1_revenue_chart.png, diagram_process_flow.png
• Include the figure number if the document has numbered figures
• Keep names lowercase with underscores

Step 6: Write Image References in Markdown

Use this three-part format to make images useful to both humans and AI:

<!-- image-ref: {image_id} | see manifest.yaml for full metadata -->
![{Descriptive alt text}](img/{filename}.png)

> **Figure {N}** | Type: {chart/diagram/photo/illustration} | Source: PDF p. {page}
>
> **Content:** {What the figure shows — one or two sentences}
>
> **Key elements:** {List the important visual elements}
>
> **Labels/annotations:** {Any text, numbers, or labels visible in the figure}

The blockquote captures everything an AI agent needs to understand the image without seeing the pixels. Be thorough — list every labeled element and important visual detail.

Step 7: Write the Manifest

Create manifest.yaml with structured metadata: