Use this skill when adding support for a new model to VeOmni. Covers the full lifecycle: analyzing the HuggingFace model, creating model patches, defining parallel plans, writing configs, integrating with the trainer, and testing. Trigger: 'add model', 'support new model', 'integrate <model_name>', 'new model support'.
Use TodoWrite to track all phases:
Phase 1: Analyze HF model -> in_progress
Phase 2: Create model patch -> pending
Phase 3: Define parallel plan -> pending
Phase 4: Write training config -> pending
Phase 5: Integrate with trainer -> pending
Phase 6: Test -> pending
Identify the model on HuggingFace. Read its config.json, modeling_*.py, and any processor configs.
Determine model category:
veomni/models/transformers/<model_name>/veomni/models/transformers/<model_name>/ + veomni/data/multimodal/veomni/distributed/moe/ integrationveomni/models/diffusers/<model_name>/veomni/models/seed_omni/Check existing similar models: Find the closest existing model in veomni/models/transformers/ and use it as a reference. E.g., if adding a new Qwen variant, reference qwen3/ or qwen3_vl/.
Identify required patches: VeOmni uses a patchgen system (veomni/patchgen/) to auto-generate model patches from HuggingFace models. Check if a patch spec already exists or if one needs to be created.
Create the model directory: veomni/models/transformers/<model_name>/
Required files:
__init__.py — model registrationmodeling_<model_name>_patch.py — monkey-patches for HF model (flash attention, sequence parallel, etc.)parallel_plan.py — FSDP/FSDP2 sharding plangenerated/ — auto-generated files from patchgen (do NOT edit manually)Patch patterns — follow existing models:
veomni/ops/flash_attn/)veomni/distributed/sequence_parallel/)veomni/models/auto.pyRun patchgen if applicable: make patchgen
Create parallel_plan.py in the model directory.
Define FSDP/FSDP2 sharding strategy:
If the model is MoE, define expert parallelism plan in addition to FSDP.
Reference existing parallel plans for guidance (e.g., veomni/models/transformers/qwen3/parallel_plan.py).
Model config: Create configs/model_configs/<model_family>/<ModelName>.json matching HuggingFace format.
Training config: Create YAML in the appropriate directory:
configs/text/<model_name>.yamlconfigs/multimodal/<model_name>/<model_name>.yamlconfigs/dit/<model_name>.yamlConfig must include: model path, data config, optimizer settings, parallelism config, checkpoint settings.
Verify against existing configs — match the structure of similar model configs.
Verify the model works with the appropriate trainer:
TextTrainer (veomni/trainer/text_trainer.py)VLMTrainer (veomni/trainer/vlm_trainer.py)DitTrainer (veomni/trainer/dit_trainer.py)If the model needs custom data preprocessing:
veomni/data/data_transform.py or veomni/data/multimodal/If the model needs custom collator logic:
veomni/data/data_collator.pyCreate toy config: Add tests/toy_config/<model_name>_toy/config.json with minimal parameters for fast testing.
Unit tests: Add tests in tests/models/ to verify:
veomni.models.autoE2e tests (if feasible): Test a short training run using the toy config.
Run make quality and pytest tests/models/.
Update documentation:
docs/ (training command, config reference)..agents/knowledge/architecture.md if the model adds a new module or trainer path.README.md if applicable.__init__.py. If the model's AutoConfig type is not registered, build_foundation_model() will fail.generated/ directories — they are overwritten by patchgen. Edit the patch spec or the modeling_*_patch.py instead.veomni/data/chat_template.py.is_transformers_version_greater_or_equal_to() for v4 compat guards. v4-era APIs (e.g., AutoModelForVision2Seq) no longer exist in v5 — use v5 equivalents.