Peft Lora Training

Pre-Training Checklist

Before any training begins, verify each of these areas. Do not skip steps.

1. Dataset Readiness

• Images/data are curated for quality (no duplicates, no corruption, sufficient diversity)
• Captions/labels are complete and consistent
• Content-style separation is addressed (for image LoRAs)
• Dataset size is appropriate for training type (see reference guides)

2. Configuration Validated

• Training parameters selected (learning rate, epochs, rank, alpha)
• VRAM requirements estimated and confirmed to fit available GPU
• Optimizer chosen and configured
• Backend/framework selected and installed

3. Environment Verified

• GPU detected and CUDA version compatible with PyTorch
• Required libraries installed at compatible versions
• Sufficient disk space available (10 GB minimum, 20+ GB recommended)
• Dataset files accessible with correct permissions

4. Dry Run Passed

• Short test run (2-5 steps) completes without errors
• No OOM errors during test
• Loss values are reasonable (not NaN, not zero)
• Output files are being written to expected location

Workflow Overview

Phase 1: Intent & Planning

1. Understand what the user wants to train and why
2. Select appropriate method (LoRA vs QLoRA vs full fine-tune)
3. Estimate resource requirements
4. Set expectations on dataset size and training time

Phase 2: Dataset Preparation

5. Audit dataset quality
6. Remove duplicates and low-quality samples
7. Caption/label data appropriately
8. Verify diversity and balance

Phase 3: Configuration & Validation

9. Select training parameters (use presets as starting points)
10. Estimate VRAM and confirm hardware sufficiency
11. Validate environment (GPU, CUDA, libraries)
12. Run dry run to verify pipeline

Phase 4: Training Execution

13. Confirm configuration with user before starting
14. Monitor training progress (loss curves, VRAM usage)
15. Save checkpoints at regular intervals

Phase 5: Evaluation

16. Generate test outputs at multiple LoRA weights
17. Compare against baseline / original examples
18. Diagnose any issues using failure modes guide
19. Document results and recommended settings

Decision Guide: Which Method?

Is your model an LLM?
  ├─ Yes → Do you have ≥24 GB VRAM?
  │         ├─ Yes → LoRA (standard)
  │         └─ No  → QLoRA (4-bit quantized)
  └─ No (image model) → LoRA (via Kohya/SimpleTuner/ai-toolkit)

Do you need the absolute best quality?
  ├─ Yes → Consider full fine-tuning (if resources allow)
  └─ No  → LoRA/QLoRA covers 90%+ of use cases

Quick Start: LLM LoRA Fine-Tuning

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# Configure LoRA
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                          # Rank
    lora_alpha=32,                 # Alpha (typically 2x rank)
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Typical output: trainable params: 13M || all params: 8B || trainable%: 0.16%

Quick Start: QLoRA Fine-Tuning

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)

# Configure LoRA on quantized model
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

LoRA Parameter Quick Reference

Target Modules by Architecture

Image LoRA Training Presets

Quick (5-15 min)

• Epochs: 5 | Optimizer: Prodigy (LR 1.0) | Rank: 16 | Alpha: 8
• Use for: Testing, iteration, proof of concept

Standard (15-60 min)

• Epochs: 15 | Optimizer: Prodigy or AdamW (1e-4) | Rank: 32 | Alpha: 16
• Use for: Most training runs

Thorough (30-120 min)

• Epochs: 25 | Optimizer: AdamW (5e-5) | Rank: 64 | Alpha: 32
• Use for: Final quality, production LoRAs

PEFT Methods Comparison

Reference Files

Read these for detailed guidance on specific topics:

Rules

1. Verify readiness before training. Walk through the pre-training checklist. Don't skip steps.
2. Explain every recommendation. When you suggest "use rank 16," explain why. Cite trade-offs.
3. Estimate costs first. Always estimate VRAM usage and training time before proceeding.
4. Present samples for approval. For captioning, review samples with the user before batching.
5. Surface the content-style problem early. For image LoRAs, discuss content vs style separation upfront — it's the most common cause of failed LoRAs.
6. Track decisions. Document parameter choices, dataset decisions, and evaluation results.
7. Diagnose before retrying. If training fails, identify the root cause before attempting again.