Research & Literature Gathering for Engineering Domains

Additional Domains (in standards but not yet in literature tree)

Legacy Standards Storage

The og_standards corpus lives at /mnt/ace/docs/_standards/ organized by org: ABS, API, ASTM, BSI, DNV, ISO, MIL, NEMA, Norsok, OnePetro, Unknown. Inventory DB: /mnt/ace/O&G-Standards/_inventory.db (SQLite, 6.8 GB).

Step-by-Step Procedure

Step 0 — Verify Domain Directory Exists

DOMAIN="geotechnical"   # ← set your domain
LIT_DIR="/mnt/ace-data/digitalmodel/docs/domains/${DOMAIN}/literature"
mkdir -p "${LIT_DIR}"
ls -la "${LIT_DIR}"

If the domain is new, create standard subdirectories:

mkdir -p "${LIT_DIR}"/{textbooks,standards,course-notes,worked-examples}

Step 1 — Query the Standards Ledger

Find standards already tracked for this domain:

uv run --no-project python scripts/data/document-index/query-ledger.py \
  --domain ${DOMAIN} --verbose

Record each standard's status: gap, done, wrk_captured, reference. Standards with gap or wrk_captured are download candidates.

Step 2 — Query the Document Index

Search the 1M+ record index for existing documents in this domain:

uv run --no-project python -c "
import json
from collections import Counter
matches = []
with open('data/document-index/index.jsonl') as f:
    for line in f:
        rec = json.loads(line)
        path_lower = rec.get('path', '').lower()
        summary_lower = (rec.get('summary') or '').lower()
        if '${DOMAIN}' in path_lower or '${DOMAIN}' in summary_lower:
            matches.append(rec)
print(f'Found {len(matches)} documents')
by_source = Counter(r['source'] for r in matches)
for s, c in by_source.most_common():
    print(f'  {s}: {c}')
"

Prioritize og_standards and ace_standards sources — these are already local.

Step 3 — Cross-Reference Capability Map

Check what calculations exist vs. gaps in the target repo:

uv run --no-project python -c "
import yaml
with open('specs/capability-map/digitalmodel.yaml') as f:
    data = yaml.safe_load(f)
for m in data['modules']:
    if '${DOMAIN}' in m['module'].lower():
        print(f\"Module: {m['module']} ({m.get('standards_count', '?')} standards)\")
        for s in m.get('standards', [])[:30]:
            print(f\"  {s['status']:15s} {s['org']:8s} {s['id'][:70]}\")
"

Step 4 — Web Search for Open-Access Literature

Search for freely available PDFs across these source tiers:

Tier 1 — High-value free sources:

• DNV Veracity (rules.dnv.com/docs/pdf/) — free DNV-RP, DNV-OS PDFs
• API publications (some free after registration)
• DTIC (apps.dtic.mil) — US military/govt technical reports
• NASA Technical Reports (ntrs.nasa.gov)
• BOEM/BSEE (boem.gov, bsee.gov) — regulatory guidance
• University open repos (MIT OCW, TU Delft, deepblue.lib.umich.edu)

Tier 2 — Conference/journal open access:

• OnePetro open-access papers
• ISOPE proceedings (selected)
• OTC open papers
• ResearchGate/Academia.edu author copies

Tier 3 — Textbooks and course notes:

• Internet Archive (archive.org) — public domain texts
• University course note PDFs
• Open textbook initiatives

WAF/paywall notes:

Step 5 — Generate or Update the Download Script

Option A: Use the research-domain.py driver (queries all data sources, generates brief + script):

uv run --no-project python scripts/data/research-literature/research-domain.py \
  --category ${DOMAIN} --repo digitalmodel --generate-download-script

Option B: Manual script creation from template:

#!/usr/bin/env bash
# ABOUTME: Download open-access ${DOMAIN} literature
# Usage: bash download-literature.sh [--dry-run]

set -uo pipefail

DEST="/mnt/ace-data/digitalmodel/docs/domains/${DOMAIN}/literature"
LOG_DIR="$(git rev-parse --show-toplevel)/.claude/work-queue/assets"
LOG_FILE="${LOG_DIR}/download-${DOMAIN}.log"
DRY_RUN=false
[[ "${1:-}" == "--dry-run" ]] && DRY_RUN=true

mkdir -p "${DEST}"/{textbooks,standards,course-notes,worked-examples}
mkdir -p "${LOG_DIR}"

# shellcheck source=scripts/lib/download-helpers.sh
source "$(git rev-parse --show-toplevel)/scripts/lib/download-helpers.sh"

log "=== ${DOMAIN} Literature Download ==="
log "Destination: ${DEST}"
log "Dry run: ${DRY_RUN}"

# ─── TEXTBOOKS ────────────────────────────────
log "--- Textbooks ---"

download \
  "https://example.org/textbook.pdf" \
  "${DEST}/textbooks" \
  "Author-Year-Short-Title.pdf"

# ─── STANDARDS ────────────────────────────────
log "--- Standards ---"

download \
  "https://rules.dnv.com/docs/pdf/dnvpm/codes/docs/..." \
  "${DEST}/standards" \
  "DNV-RP-XXXX-Title-Year.pdf" || true

log "=== Download complete ==="
total=$(find "${DEST}" -name "*.pdf" | wc -l)
log "  Total PDFs: ${total}"

Save script to: /mnt/ace-data/digitalmodel/docs/domains/${DOMAIN}/literature/download-literature.sh

Key script patterns:

• Always source scripts/lib/download-helpers.sh for the download and log functions
• Use set -uo pipefail (NOT set -e) — download failures should log, not abort
• Guard fallible downloads with || true or || log "NOTE: ..."
• Use descriptive filenames: Author-Year-Short-Title.pdf
• The download function auto-skips existing files (resume-safe)
• Always run --dry-run first to preview

Step 6 — Execute Downloads and Validate

# Dry run first
bash download-literature.sh --dry-run

# Execute
bash download-literature.sh

# Validate all PDFs are real PDFs (not HTML/WAF responses)
find "${LIT_DIR}" -name "*.pdf" -exec file {} \; | grep -v "PDF document"

Any file that shows "HTML document" or "ASCII text" instead of "PDF document" is a WAF response. Move it to a _failed/ directory and add to pending_manual.

Step 7 — Create Catalogue YAML

Write knowledge/seeds/${DOMAIN}-resources.yaml: