Llm Creation

Operating procedure

1. Define architecture. Choose a decoder-only transformer configuration:

   Model size guide:
   - 1B:  num_layers=24,  hidden_dim=2048,  num_heads=16,  context=2048
   - 7B:  num_layers=32,  hidden_dim=4096,  num_heads=32,  context=4096
   - 13B: num_layers=40,  hidden_dim=5120,  num_heads=40,  context=4096
   - 70B: num_layers=80,  hidden_dim=8192,  num_heads=64,  context=8192
   

   Use RoPE positional embeddings, SwiGLU activation, RMSNorm, GQA (grouped-query attention) for ≥13B models.
2. Apply scaling laws. Chinchilla-optimal: train on ~20 tokens per parameter (a 7B model needs ~140B tokens). Estimate total FLOPs: C ≈ 6 × N × D where N = parameters, D = tokens. Budget GPU-hours: C / (GPU_TFLOPS × utilization × 3600), target 40–50% MFU.
3. Train tokenizer. Train a BPE tokenizer (SentencePiece or HuggingFace tokenizers) on a representative corpus sample. Vocab size: 32k–64k. Ensure coverage of code, multilingual text, and special tokens (<|begin_of_text|>, <|end_of_text|>, chat role markers).
4. Configure distributed pretraining. Choose framework:
   • FSDP (PyTorch native): FullyShardedDataParallel with mixed precision (bf16), activation checkpointing.
   • DeepSpeed ZeRO Stage 3: partitions optimizer states, gradients, and parameters.
   • Megatron-LM: tensor + pipeline parallelism for >70B models. Learning rate: peak 3e-4, cosine decay to 3e-5, warmup over first 2000 steps. Batch size: ramp from 256 to 4M tokens over warmup.
5. Monitor training. Log training loss, gradient norm, and learning rate every 10 steps. Evaluate perplexity on held-out validation set every 1000 steps. Run downstream benchmarks (MMLU, HellaSwag, HumanEval) at 25%, 50%, 75%, and 100% of training.
6. Post-training pipeline. After pretraining: instruction-tune with SFTTrainer (delegate to instruction-tuning skill), then align with DPO or RLHF (delegate to preference-optimization skill).
7. Document and release. Produce a model card: architecture, training data composition, compute used, benchmark results, known limitations, intended use, and license.

Decision rules

• Follow Chinchilla scaling: if compute is fixed, allocate ~50% to model size and ~50% to data tokens.
• For context length > 8k, use RoPE with NTK-aware scaling or YaRN for length extrapolation.
• Use GQA (num_kv_heads = num_heads / 8) for models ≥ 13B to reduce KV cache memory.
• If training loss spikes, reduce learning rate by 50% and resume from last stable checkpoint.
• Checkpoint every 1000 steps minimum; keep at least last 5 checkpoints for rollback.
• Do not release without running safety evaluations (ToxiGen, BBQ bias benchmark, red-teaming).

Output requirements

1. Architecture spec — model config (layers, dims, heads, context, vocab), parameter count
2. Compute budget — FLOPs estimate, GPU type/count, estimated wall-clock time, cost
3. Data plan — corpus composition, token counts, deduplication and filtering pipeline
4. Training config — optimizer, LR schedule, batch size ramp, parallelism strategy
5. Evaluation schedule — checkpoints, benchmarks, and pass/fail criteria at each stage
6. Model card — standard model card with architecture, data, benchmarks, limitations

References

• Chinchilla scaling laws: Hoffmann et al., "Training Compute-Optimal LLMs" (arXiv:2203.15556)
• LLaMA architecture: Touvron et al. (arXiv:2302.13971)
• DeepSpeed ZeRO: Rajbhandari et al. (arXiv:1910.02054)
• Megatron-LM: Shoeybi et al. (arXiv:1909.08053)
• RoPE: Su et al., "RoFormer" (arXiv:2104.09864)
• Model cards: Mitchell et al., "Model Cards for Model Reporting" (FAT* 2019)

Related skills

• model-architecture
• data-cleaning-labeling
• instruction-tuning
• fine-tuning
• preference-optimization

Failure handling

• If training loss diverges, roll back to the last stable checkpoint, halve the learning rate, and resume.
• If downstream benchmarks regress at a checkpoint, investigate data quality for the recent training window — check for duplicates or corrupted shards.
• If GPU utilization drops below 30% MFU, profile for communication bottlenecks and adjust parallelism strategy (increase tensor parallelism, reduce pipeline stages).