Arxiv Latex Source

Core Endpoints

Download LaTeX Source

• URL: GET https://arxiv.org/e-print/{arxiv_id}

• Response: application/gzip — a .tar.gz archive containing the source files

• Parameters:

  Param	Type	Required	Description
  arxiv_id	string	Yes	arXiv identifier, e.g. 2301.00001 or 2301.00001v2 for a specific version

• Example:

  # Download source archive (response: 200, application/gzip, ~1.3 MB)
  curl -sL -o source.tar.gz "https://arxiv.org/e-print/2301.00001"
  
  # List archive contents
  tar tz -f source.tar.gz | head -10
  # ACM-Reference-Format.bbx
  # ACM-Reference-Format.bst
  # Image_1.jpg
  # README.txt
  # acmart.cls
  

• Content-Disposition header: attachment; filename="arXiv-2301.00001v1.tar.gz"

• ETag: SHA-256 hash provided for caching: sha256:f1ffe8ec...

Format Detection

The endpoint almost always returns a gzip-compressed tar archive. Rare cases (very old or single-file submissions) may return a single gzip-compressed .tex file without tar wrapper. Always verify format before extracting:

curl -sL "https://arxiv.org/e-print/{arxiv_id}" -o source.gz
file source.gz  # "gzip compressed data, was 'XXXX.tar', ..."

Metadata API (Companion)

Pair source downloads with the arXiv Atom API for structured metadata:

• URL: GET https://export.arxiv.org/api/query?id_list={arxiv_id}
• Response: Atom XML with <title>, <author>, <summary>, <category>, <published>
• Example: curl -s "https://export.arxiv.org/api/query?id_list=2301.00001"

LaTeX Source Parsing Guide

Locating the Main .tex File

A source archive typically contains multiple files. To find the main document:

1. Look for \documentclass in .tex files — this marks the root document
2. Check for a README.txt that may specify the main file
3. If multiple .tex files contain \documentclass, prefer the one with \begin{document}

import tarfile, re

def find_main_tex(tar_path):
    with tarfile.open(tar_path, 'r:gz') as tar:
        tex_files = [m for m in tar.getmembers() if m.name.endswith('.tex')]
        for member in tex_files:
            content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
            if r'\documentclass' in content and r'\begin{document}' in content:
                return member.name, content
    return None, None

Extracting Sections

LaTeX sections follow a predictable hierarchy:

import re

def extract_sections(tex_content):
    pattern = r'\\(section|subsection|subsubsection)\{([^}]+)\}'
    sections = re.findall(pattern, tex_content)
    return [(level, title) for level, title in sections]
    # [('section', 'Introduction'), ('section', 'Related Work'), ...]

Extracting Equations

def extract_equations(tex_content):
    patterns = [
        r'\\\[(.+?)\\\]',
        r'\\begin\{equation\}(.+?)\\end\{equation\}',
        r'\\begin\{align\*?\}(.+?)\\end\{align\*?\}',
    ]
    equations = []
    for pat in patterns:
        equations.extend(re.findall(pat, tex_content, re.DOTALL))
    return equations

Extracting Bibliography

Parse .bib files (BibTeX entries) or .bbl files (compiled \bibitem commands):

def extract_bibliography(tar_path):
    refs = []
    with tarfile.open(tar_path, 'r:gz') as tar:
        for member in tar.getmembers():
            if member.name.endswith('.bib'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'@\w+\{([^,]+),(.+?)\n\}', content, re.DOTALL))
            elif member.name.endswith('.bbl'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'\\bibitem.*?\{(.+?)\}', content))
    return refs

Rate Limits

• Maximum: 4 requests per second for automated access
• Recommended: 1 request/second with delays between sequential downloads
• Bulk access: For 1000+ papers, use the arXiv S3 bulk data mirror instead
• HTTP 429: Rate limit exceeded; implement exponential backoff
• User-Agent: Required — set a descriptive string: MyTool/1.0 (mailto:[email protected])
• Persistent abuse may result in IP-level blocks

Academic Use Cases

• Formula extraction for ML training — Build equation datasets with ground-truth LaTeX notation, free of OCR noise from PDF parsing
• Citation network analysis — Parse .bib/.bbl files for exact reference keys to construct citation graphs
• Section-level text analysis — Extract specific sections (e.g., all "Related Work" across a subfield) for systematic reviews
• Reproducibility auditing — Examine algorithm environments, hyperparameter tables, and methodology sections
• Cross-paper notation alignment — Compare and normalize equation environments across papers in a subfield

Complete Python Example

import requests, tarfile, io, re, time, gzip

def download_arxiv_source(arxiv_id, delay=1.0):
    """Download and extract all .tex files from an arXiv paper's source."""
    url = f"https://arxiv.org/e-print/{arxiv_id}"
    headers = {"User-Agent": "ResearchTool/1.0 (mailto:[email protected])"}
    resp = requests.get(url, headers=headers)
    resp.raise_for_status()
    time.sleep(delay)

    buf = io.BytesIO(resp.content)
    try:
        with tarfile.open(fileobj=buf, mode='r:gz') as tar:
            return {m.name: tar.extractfile(m).read().decode('utf-8', errors='ignore')
                    for m in tar.getmembers() if m.name.endswith('.tex') and m.isfile()}
    except tarfile.ReadError:
        buf.seek(0)
        return {"main.tex": gzip.decompress(buf.read()).decode('utf-8', errors='ignore')}

# Usage
sources = download_arxiv_source("2301.00001")
for fname, content in sources.items():
    if r'\documentclass' in content:
        sections = re.findall(r'\\section\{([^}]+)\}', content)
        equations = re.findall(r'\\begin\{equation\}(.+?)\\end\{equation\}', content, re.DOTALL)
        print(f"{fname}: {len(sections)} sections, {len(equations)} equations")

References

• arXiv e-print access: https://info.arxiv.org/help/bulk_data_s3.html
• arXiv API documentation: https://info.arxiv.org/help/api/index.html
• arXiv terms of use: https://info.arxiv.org/help/api/tou.html
• arXiv S3 bulk data: https://info.arxiv.org/help/bulk_data_s3.html