Prepare Flash Data

Do not use this skill for:

• COLMAP-only conversion workflows handled by convert.py.
• Non-flash datasets that do not use MVInverse albedo/normal or flash precompute caches.

Working Rules

• Keep convert.py unchanged; Blender workflows must stay separate.
• Prefer the repo workflow script tools/blender_flash_dataset_workflow.py.
• Use only ref_zjr Python for repo scripts, MVInverse inference, and training.
• Keep scene naming consistent: the prepared dataset folder basename and the training model_path basename should match.
  • Example: /home/wdh/Project/Data-Scaffold/office_clear pairs with ../output/office_clear.
• Before precompute or resume, check free disk space. Treat < 30G free as risky.
• If cache EXRs are zero-byte or unreadable, treat them as missing and repair them instead of forcing a full restart.

Execution Flow

1. Locate the repo root.

   • Find the directory that contains both train.py and tools/blender_flash_dataset_workflow.py.
   • If not obvious, use REFGS_REPO_ROOT.

2. Confirm dataset shape.

   • Read references/dataset_contract.md.
   • Verify the raw Blender scene has image/, segment/, transforms_train.json, and usually a .blend file.
   • Confirm the split rule: first N frames are nonflash, last N frames are flash.

3. Prepare the dataset.

   • Run scripts/prepare_blender_flash_dataset.sh.
   • Typical flow: render missing raw frames, build nonflash/flash, run MVInverse, normalize points3d.ply, and emit workflow_report.json.
   • After preparation, verify counts, dimensions, and loader behavior.

4. Validate before training.

   • Use scripts/check_precompute_cache.py for cache health when resuming.
   • Confirm:
     • nonflash == N
     • flash == N
     • albedo == total
     • normal == total
     • segment == total
     • points3d.ply contains x y z nx ny nz red green blue

5. Train or resume.

   • Use the scene-specific train shell when it exists.
   • If the scene has no train shell yet, create one from assets/train_flash_scene.template.sh or call scripts/run_flash_training.sh.
   • Keep the scene name aligned across paths:
     • dataset root basename: office_clear
     • model output basename: office_clear
   • For precompute-stage resume, prefer --load_iteration 19999 when flash_indirect_from_iter == 20000.
   • Read references/recovery_playbook.md before touching cache directories.

6. Repair precompute failures.

   • If ind_diff_precomp/*.exr is unreadable or zero-byte:
     • clear only the bad cache files
     • keep valid spec caches
     • rerun the train shell
   • The repo now auto-detects incomplete caches and backfills missing diffuse caches on resume.
   • Use scripts/repair_and_resume_flash.sh for the standard repair path.

Output Standard

When using this skill, always report:

• Scene root and prepared dataset root.
• Whether missing Blender frames were rendered.
• Final counts for nonflash, flash, albedo, normal, segment.
• Whether loader smoke-check passed.
• Whether points3d.ply needed normalization.
• Current disk free space before and after large precompute or resume actions.
• If training was resumed: the loaded iteration, cache counts, and whether the run advanced past the previous failure point.

Resources

• Dataset format and validation rules:
  • references/dataset_contract.md
• Resume and precompute repair procedure:
  • references/recovery_playbook.md
• Canonical commands and environment mapping:
  • references/command_recipes.md
• Automation wrappers:
  • scripts/prepare_blender_flash_dataset.sh
  • scripts/run_flash_training.sh
  • scripts/check_precompute_cache.py
  • scripts/repair_and_resume_flash.sh
• Reusable shell template:
  • assets/train_flash_scene.template.sh
  • assets/train_flash_resume.template.sh