Curator

Agent(subagent_type="general-purpose", prompt="Search IA for Paracelsus works. Write importable identifiers to /tmp/agent-paracelsus.txt")

Multi-source strategy: Don't stop at Internet Archive. Search in order:

1. Internet Archive (broadest, IA API)
2. Gallica / BnF (French, Arabic, Persian MSS — use SRU API, ARK identifiers)
3. NDL Japan (Japanese Go, shogi, Buddhist texts — IIIF at dl.ndl.go.jp/api/iiif/{PID}/manifest.json)
4. Bodleian / Cambridge / Manchester (IIIF manuscripts)
5. Qatar Digital Library (Arabic MSS — blocks automation, needs manual PDF download)
6. Library of Congress (Chinese rare books, LOC API)
7. MDZ/BSB, e-rara, HAB, Vatican (European rare books)

Step 2: Evaluate & Deduplicate

Before building the script:

1. Search existing collection: curl -s "https://sourcelibrary.org/api/search?q=AUTHOR&limit=20"
2. Apply selection rules (see below)
3. Pick best edition per work (oldest original-language edition)
4. Check for work_id linking (related editions of same work)

Step 3: Determine Collection Assignment

Before importing, decide which collection(s) the batch belongs to.

Existing collections (22 curated): alchemy, hermetica, kabbalah, magic, natural-philosophy, demonology, secret-societies, astrology, mysticism, sacred-texts, theology, classical-philosophy, renaissance-philosophy, medicine, indic-traditions, chinese-classics, art-illustrated, literature, music, herbalism, leonardo-da-vinci, shwep-reading-room

Check if an existing collection fits:

curl -s "https://sourcelibrary.org/api/collections" | python3 -c "import sys,json; [print(c['slug'], '—', c['name']) for c in json.load(sys.stdin)]"

If no collection fits, create a new one using the API after import (see Step 5).

Note: Gemini auto-scores new books into collections via the pipeline. But for themed batches (e.g., "Strategy Games", "Persian Literary Tradition"), explicitly assigning a collection ensures proper grouping.

Step 4: Generate Batch Script

Write a _tmp-batch-import-{theme}.mjs script following this template:

#!/usr/bin/env node
const BASE = 'https://sourcelibrary.org';
const AUTH = `Bearer ${process.env.CRON_SECRET}`;

const imports = [
  // Internet Archive:
  // { ia_identifier: '...', title: '...', author: '...', year: NNNN, original_language: '...' },
  //
  // IIIF (NDL Japan, Bodleian, Manchester, etc.):
  // { manifest_url: 'https://dl.ndl.go.jp/api/iiif/PID/manifest.json', title: '...', author: '...', language: '...', published: '...', provider: '...' },
  //
  // Gallica: { ark: 'bpt6k...', title: '...', ... }
  // Google Books: { google_books_id: '...', title: '...', ... }
  // MDZ: { bsb_id: 'bsb...', title: '...', ... }
  // See @.claude/docs/import-apis.md for all routes
];

let imported = 0, skipped = 0, errors = 0, totalPages = 0;
const importedIds = [];

for (let i = 0; i < imports.length; i++) {
  const item = imports[i];
  const route = item.manifest_url ? 'iiif' : item.google_books_id ? 'google-books' : item.ark ? 'gallica' : item.bsb_id ? 'mdz' : 'ia';
  console.log(`[${i+1}/${imports.length}] ${item.ia_identifier || item.manifest_url?.match(/\d+/)?.[0] || item.ark || item.bsb_id || item.google_books_id}`);
  try {
    const resp = await fetch(`${BASE}/api/import/${route}`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', 'Authorization': AUTH },
      body: JSON.stringify(item),
    });
    const data = await resp.json();
    if (!resp.ok) {
      if (resp.status === 409 || (data.error && data.error.includes('already'))) {
        console.log(`  SKIP (dupe): ${item.title}`); skipped++;
      } else {
        console.log(`  ERROR: ${item.title} — ${data.error || resp.statusText}`); errors++;
      }
    } else {
      const pages = data.book?.pages_count || data.pagesCreated || 0;
      const bookId = data.bookId || data.book?.id;
      console.log(`  OK: ${item.title} — ${pages} pages`);
      imported++; totalPages += pages;
      if (bookId) importedIds.push(bookId);
    }
  } catch (err) { console.log(`  ERROR: ${item.title} — ${err.message}`); errors++; }
  if (i < imports.length - 1) await new Promise(r => setTimeout(r, 2000));
}

console.log(`\nDone: ${imported} imported, ${skipped} dupes, ${errors} errors, ${totalPages} pages`);

// === COLLECTION ASSIGNMENT ===
// Uncomment and set the collection slug to assign imported books:
//
// const COLLECTION_SLUG = 'strategy-games'; // or an existing slug
// if (importedIds.length > 0) {
//   console.log(`\nAssigning ${importedIds.length} books to collection: ${COLLECTION_SLUG}`);
//   const resp = await fetch(`${BASE}/api/collections`, {
//     method: 'PATCH',
//     headers: { 'Content-Type': 'application/json', 'Authorization': AUTH },
//     body: JSON.stringify({ slug: COLLECTION_SLUG, addBookIds: importedIds }),
//   });
//   const data = await resp.json();
//   if (resp.ok) console.log('  Collection updated.');
//   else console.log('  Collection error:', data.error);
// }

Step 5: Create New Collections (when needed)

If the batch represents a new thematic area not covered by existing collections, create one:

// Create a new collection
const resp = await fetch(`${BASE}/api/collections`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Authorization': AUTH },
  body: JSON.stringify({
    name: 'Strategy Games',
    slug: 'strategy-games',
    subtitle: 'Chess, Go, Backgammon, and the Philosophy of Play',
    description: 'Historical treatises on strategy games from chess and Go to backgammon and rithmomachia, spanning Arabic, Persian, Japanese, Sanskrit, and European traditions.',
    color: 'gold',  // 'rust' | 'sage' | 'violet' | 'gold'
    bookIds: importedIds,  // Initial books to include
  }),
});

Collection naming guidelines:

• Use clear, descriptive names (not jargon)
• Slug format: kebab-case (e.g., persian-literary-tradition)
• Colors: rust (warm/ancient), sage (natural/philosophical), violet (mystical/esoteric), gold (royal/classical)
• Write a substantive description — it appears on the public collection page

Step 6: Run

set -a; source .env.production.local; set +a; node _tmp-batch-import-{theme}.mjs

Post-import processing (archive, OCR, translation) is fully automatic via the pipeline cron. No manual action needed.

Authentication

All import and collection APIs require auth via Bearer CRON_SECRET header:

const AUTH = `Bearer ${process.env.CRON_SECRET}`;
// Use in headers: { 'Authorization': AUTH }

The CRON_SECRET is in .env.production.local. Source it with set -a; source .env.production.local; set +a before running scripts.

IIIF Imports

For libraries that serve IIIF manifests (NDL Japan, Bodleian, Manchester, Kyoto U, etc.):

{
  manifest_url: 'https://dl.ndl.go.jp/api/iiif/1183163/manifest.json',
  title: '発陽論 (Hatsuyoron)',
  author: 'Inoue Inseki',
  language: 'Japanese',
  published: '1914',
  provider: 'National Diet Library of Japan',
}

Known IIIF sources:

For QDL (Qatar Digital Library): Blocks all automated access. User must download PDF manually, then import via R2 upload + direct MongoDB insertion (see session notes for the Kitab al-Shatranj workflow).

PDF Imports (Manual Pipeline)

For large PDFs from sources without IIIF (QDL downloads, scanned books):

1. Upload PDF to R2:

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
const r2 = new S3Client({
  region: 'auto',
  endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
  credentials: { accessKeyId: process.env.R2_ACCESS_KEY_ID, secretAccessKey: process.env.R2_SECRET_ACCESS_KEY },
});

2. Extract pages with pdftoppm -r 150 -jpeg
3. Upload page images to R2 at books/{bookId}/pages/0001.jpg
4. Create book + page records in MongoDB directly (pages need an id field — use new ObjectId().toString())

Selection Rules

Edition Priority (CRITICAL)

ALWAYS prefer the oldest available edition in original language:

1. Manuscripts — highest priority (especially pre-1500)
2. Incunabula (pre-1501)
3. 16th century — first printed editions, editio princeps
4. 17th century — important scholarly editions
5. 18th century — when earlier unavailable
6. 19th century critical editions — Teubner, Loeb (pre-1929), OCT
7. Modern translations — ONLY when no original text edition exists

Language priority: Original language ALWAYS over English. Never import 20th-21st century English translations when Latin/Greek/Arabic/Persian/Hebrew originals exist.

ACQUIRE

• Original historical editions (pre-1800 primary sources)
• Illuminated manuscripts with miniatures
• Early printed books in original language
• First editions and important early printings
• Critical scholarly editions with original text
• Texts from non-Western traditions (Arabic, Persian, Sanskrit, Chinese, Japanese, Hebrew)

REJECT

• Modern translations without original text
• English-only editions when originals available
• Secondary literature and commentaries
• Facsimile reprints when original scans exist
• Anthologies that excerpt rather than present complete works
• Books already in collection

Scoring (1-10 scale)

Session Tracking

Append to curatorreports.md:

# Session [N]: [DATE] - [THEME]

## Collection: [slug] (new|existing)

## Acquired
| Title | Author | Year | Pages | Book ID | Source |
|-------|--------|------|-------|---------|--------|

## Rejected
| Title | Reason |

## Session Total: N books, N pages