Pdf Extract Ordered Fallback

Key principle: Always try shell tools (run_shell) before Python sandbox (execute_code_sandbox) when both are viable options. Shell execution is more reliable in constrained environments.

Step-by-Step Instructions

Step 0: Initial Extraction Attempt with read_file Tool

First, attempt to extract PDF text using the read_file tool. This is the simplest approach and handles many PDFs correctly:

read_file filetype="pdf" file_path="path/to/document.pdf"

Expected outcomes:

• Success: Returns extracted text content - proceed to use this directly
• Binary/Image data returned: The tool failed to extract text; file content is raw binary or image data
  • Immediate action: Proceed to Step 1 (run_shell with pdftotext)
• Error returned: Tool failed entirely; proceed to Step 1 (run_shell with pdftotext)

Critical: If read_file returns binary data (PNG/JPEG headers, raw PDF bytes), do NOT attempt to parse it manually. Immediately switch to shell-based pdftotext.

Step 1: Download PDF with Browser User-Agent

Many PDF hosting sites use JavaScript-based redirects or block automated requests. Use curl with a realistic browser user-agent:

curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -o output.pdf "URL_HERE"

Key flags:

• -L: Follow redirects
• -A: Set user-agent header to mimic a real browser
• -o: Specify output filename

Step 2: Verify File Type Before Parsing

Always validate the downloaded file is actually a PDF before attempting extraction:

file output.pdf

Expected output should contain "PDF document". If not:

• The URL may have redirected to an HTML error page
• The file may be corrupted
• Access may be blocked

Step 3: Primary Shell Extraction with pdftotext via run_shell

This step takes priority over Python-based extraction. If Step 0 failed or if you're working with a newly downloaded PDF, use run_shell with pdftotext before attempting any Python libraries:

pdftotext downloaded.pdf extracted.txt

Execute via run_shell:

run_shell command="pdftotext downloaded.pdf extracted.txt"

If pdftotext is not available, install it first:

# Debian/Ubuntu
apt-get update && apt-get install -y poppler-utils

# macOS  
brew install poppler

# RHEL/CentOS
yum install -y poppler-utils

Why shell-first? Shell-based pdftotext is more reliable, faster, and avoids sandbox execution issues that can affect Python code execution in constrained environments.

Step 4: Secondary Python Fallback with PyMuPDF via execute_code_sandbox

Only if run_shell with pdftotext fails or is unavailable, fall back to Python's PyMuPDF library via execute_code_sandbox:

import fitz  # PyMuPDF

doc = fitz.open("downloaded.pdf")
text = ""
for page in doc:
    text += page.get_text()
doc.close()

with open("extracted.txt", "w") as f:
    f.write(text)

Execute within execute_code_sandbox:

execute_code_sandbox code="<Python code above>"

Install if needed:

pip install pymupdf

Note: Some environments may experience execute_code_sandbox failures (unknown errors). This is why shell-based extraction (Step 3) must be attempted first.

Step 5: Graceful Degradation to Domain Knowledge

If the PDF cannot be accessed or extracted after all attempts:

1. Document the failure mode (network issue, corrupted file, access denied, etc.)
2. Extract any partial content that was successfully retrieved
3. Supplement missing content from established domain knowledge
4. Clearly mark which portions are from source vs. generated from knowledge
5. Provide citations for any claimed requirements or specifications

Example degradation note:

NOTE: Source document [URL] was inaccessible due to [reason]. 
Content below combines partial extraction with established domain knowledge 
for [topic]. Verify against official sources when available.

Complete Workflow Script

#!/bin/bash
# pdf-extract-workflow.sh

PDF_URL="$1"
OUTPUT_PDF="downloaded.pdf"
OUTPUT_TXT="extracted.txt"

# Step 0/1: Download with browser user-agent
echo "Downloading PDF..."
curl -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" -o "$OUTPUT_PDF" "$PDF_URL"

# Step 1: Verify file type
echo "Verifying file type..."
if ! file "$OUTPUT_PDF" | grep -q "PDF document"; then
    echo "WARNING: Downloaded file is not a valid PDF"
    echo "Attempting fallback extraction anyway..."
fi

# Step 2: Try pdftotext via shell (PRIMARY EXTRACTION)
echo "Attempting pdftotext extraction..."
if command -v pdftotext &> /dev/null; then
    if pdftotext "$OUTPUT_PDF" "$OUTPUT_TXT" 2>/dev/null; then
        echo "Extraction successful with pdftotext"
        exit 0
    fi
fi

# Step 3: Fallback to PyMuPDF via Python sandbox (SECONDARY)
echo "Falling back to PyMuPDF..."
python3 << 'PYTHON_SCRIPT'
import fitz
import sys