Palmetto Slurm Safe Guard

Slurm Template Checklist

1. Preflight

• set -euo pipefail
• node class check (login vs compute)
• nvidia-smi exists and GPU count meets requirement
• input/output paths validated

2. Scratch Paths

• HF_HOME=/scratch/$USER/.hf_cache
• PIP_CACHE_DIR=/scratch/$USER/.pip_cache
• env under /scratch/$USER/envs/...
• optional containers under /scratch/$USER/containers
• datasets, model weights, checkpoints, and large result directories under /scratch/$USER/...
• for Codex/agent-submitted jobs, prefer SCRATCH_ROOT=/scratch/<real_user> over raw $USER expansion if $USER may resolve to coder
• checkpoint trees and checkpoint-writing output_dir under /scratch/$USER/...
• repository keeps code, configs, manifests, lightweight logs, lightweight metrics, and standard small experiment results
• if artifacts are large, append-heavy, or likely to exceed home quota, place them on scratch and keep only small summaries or pointers in the repository
• pre-warm model repos into HF_HOME before the main Python command
• set offline defaults in job scripts: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1

3. Logging and Monitoring

• print one startup block: job id, model, input, output, limits
• monitor loop every 120s by default
• do not print per-step noisy diagnostics unless requested
• write #SBATCH -o/-e and code-generated run logs to project-local directories by default

4. Saving and Recovery

• pass save controls to Python entry:
  • --save-every
  • --save-every-batches
  • --save-every-seconds
  • --progress-json
• Python entry must save atomically and flush on signals
• For training jobs, checkpoint logic must satisfy exact-training-resume-guard; model-only saves do not count as resumable training checkpoints.

Python Entry Requirements

• Add autosave policy driven by rows/batches/time.
• Save output CSV atomically.
• Save <output>.progress.json with current language/column/counters.
• Install signal handlers for graceful early termination with forced save.
• Process in batches and write translated content incrementally to memory + periodic disk flush.

Default Parameters

• MONITOR_INTERVAL=120
• SAVE_EVERY=200
• SAVE_EVERY_BATCHES=10
• SAVE_EVERY_SECONDS=120
• LOG_EVERY_BATCHES=10

Definition of Done

A job script is only done when all are true:

• preflight checks implemented
• scratch-only heavy paths enforced
• monitor loop exists and is cleanup-safe
• Python entry supports autosave and signal-safe final save
• training jobs support exact resume with full optimizer/scheduler/RNG/framework state
• output CSV and progress JSON are written during run (not only at end)
• code-generated logs and lightweight experiment artifacts stay inside the repository unless the user explicitly requested another path
• checkpoint trees and checkpoint-writing output directories stay under scratch
• large artifacts and large result directories stay under scratch
• model/tokenizer/processor repos are pre-cached under scratch and the run executes offline by default