Pt2 Bug Basher

Investigation Strategy

Prefer direct tools over meta_codesearch

Use Grep, Glob, and Read directly for code exploration. Do not spawn meta_codesearch agents — they are slow and expensive. The Architectural Knowledge and Key Source Files sections below should give you enough context to know where to look. A targeted Grep for a function name is always faster.

Know which compilation mode you're in

Before reading implementation code, determine the compilation mode. These share code but diverge in important ways:

• torch.compile -- Dynamo + Inductor. tx.export=False, no _compiling_state_context().
• torch.export (strict) -- tx.export=True, _compiling_state_context() active.
• torch.export (non-strict, the default) -- Uses Dynamo via fullgraph_capture but tx.export may differ from strict. _compiling_state_context() active. Check torch._export.config.use_new_tracer_experimental — it changes which code path is used.

Distinguish trace-time vs runtime

Many PT2 bugs come from confusing these two:

• Trace-time: Inside Dynamo's symbolic interpreter. Dynamo intercepts function calls and may constant-fold them (e.g., is_exporting() → ConstantVariable(True)).
• Runtime: Real tensors, real Python calls, module-level flags like torch.compiler._is_exporting_flag.

When debugging, add temporary print() statements directly in the source file rather than monkey-patching from outside — dispatch chains make monkey-patching unreliable.

Gathering Information

Pick the right diagnostic tool based on the error category:

• Quick overview: TORCH_LOGS="+dynamo,graph_breaks,recompiles" python your_script.py
• Full debug artifacts: TORCH_COMPILE_DEBUG=1 python your_script.py — creates torch_compile_debug/ with FX graphs, Inductor IR, and generated code
• Generated code only: TORCH_LOGS="output_code" python your_script.py
• Structured tracing: TORCH_TRACE=/path/to/trace python your_script.py then tlparse /path/to/trace
• Single-threaded (for pdb): TORCHINDUCTOR_COMPILE_THREADS=1 python your_script.py

Error Triage

Classify the failure using the error message and traceback:

Debugging by Category

Graph Breaks

Graph breaks split the compiled graph into smaller subgraphs, often causing performance regressions or unexpected behavior.

Diagnosis:

TORCH_LOGS="graph_breaks" python your_script.py

Key files:

• torch/_dynamo/exc.py -- Unsupported exception class
• torch/_dynamo/variables/ -- where most graph break decisions happen

Common causes:

• Unsupported Python constructs (data-dependent control flow, unsupported builtins)
• Tensor operations that can't be traced (in-place ops on inputs, unsupported dtypes)
• Calls to non-traceable functions

Fix approach:

1. Read the graph break message to identify the unsupported operation
2. Check if there's a decomposition or supported alternative
3. If the operation genuinely can't be traced, consider torch._dynamo.allow_in_graph or restructuring user code

Backend Compiler Failures

BackendCompilerFailed means Inductor (or another backend) crashed during compilation.

Diagnosis:

TORCHDYNAMO_REPRO_AFTER=aot TORCHDYNAMO_REPRO_LEVEL=2 python your_script.py

This generates minifier_launcher.py that isolates the minimal failing graph.

Key files:

• torch/_dynamo/repro/after_aot.py -- repro/minifier for post-AOT failures
• torch/_inductor/ -- the backend itself

Fix approach:

1. Run the minifier to get a minimal reproduction
2. Inspect the FX graph (TORCH_COMPILE_DEBUG=1) to understand what ops are involved
3. Check if it's a lowering issue (torch/_inductor/lowering.py), scheduling issue, or codegen issue
4. Look at the generated output code if the error is in codegen

Recompilation Issues

Excessive recompilation happens when guards are too specific, causing cache misses.

Diagnosis:

TORCH_LOGS="recompiles,recompiles_verbose,guards" python your_script.py

Key config:

• torch._dynamo.config.recompile_limit (default: 8)
• torch._dynamo.config.fail_on_recompile_limit_hit -- set to True to get a hard error

Common causes:

• Changing tensor shapes without marking them dynamic
• Python scalar values that change between calls
• Global state mutations between calls

Fix approach:

1. Read the recompilation reason from logs
2. Identify the failing guard
3. Either mark the relevant dimension as dynamic with torch._dynamo.mark_dynamic() or fix the source of guard instability

Accuracy Issues

The compiled model produces different numerical results than eager mode.

Diagnosis:

TORCHDYNAMO_REPRO_AFTER=aot TORCHDYNAMO_REPRO_LEVEL=4 python your_script.py

This compares compiled vs. eager with an fp64 reference and dumps a repro if accuracy fails.

Key utilities:

• torch/_dynamo/debug_utils.py -- same_two_models(), backend_accuracy_fails(), cast_to_fp64()
• torch._dynamo.config.repro_tolerance (default: 1e-3)

Fix approach:

1. Get the minimal failing graph from the minifier
2. Compare eager vs. compiled output at fp64 precision
3. Binary search through ops to find the diverging operation
4. Check for known numerical issues (reduction order, fused kernels, dtype promotions)

Internal Dynamo Errors

InternalTorchDynamoError indicates a bug in Dynamo itself.

Diagnosis:

TORCHDYNAMO_VERBOSE=1 python your_script.py
# or equivalently:
TORCH_LOGS="+dynamo" python your_script.py

Key files:

• torch/_dynamo/symbolic_convert.py -- bytecode interpreter
• torch/_dynamo/variables/ -- variable tracking system
• torch/_dynamo/guards.py -- guard generation

Fix approach:

1. Get the full stack trace with TORCHDYNAMO_VERBOSE=1
2. Identify which bytecode instruction or variable type caused the crash
3. Create a minimal repro (the error message often includes a minifier path)
4. Debug with TORCHINDUCTOR_COMPILE_THREADS=1 and pdb if needed

Runtime Crashes

Segfaults and CUDA illegal memory access errors during execution of compiled code.

Diagnosis (make crash deterministic):

PYTORCH_NO_CUDA_MEMORY_CACHING=1 CUDA_LAUNCH_BLOCKING=1 python your_script.py

For CUDA IMA, add NaN checks:

TORCHINDUCTOR_NAN_ASSERTS=1 python your_script.py

For Inductor-level sync debugging:

torch._inductor.config.triton.debug_sync_kernel = True  # sync after every kernel
torch._inductor.config.triton.debug_sync_graph = True   # sync before/after graph

Fix approach:

1. Make the crash deterministic with PYTORCH_NO_CUDA_MEMORY_CACHING=1 CUDA_LAUNCH_BLOCKING=1
2. Check if it's an input mismatch (shapes, devices, dtypes)
3. Inspect the generated kernel code with TORCH_LOGS="output_code"
4. Use TORCHINDUCTOR_NAN_ASSERTS=1 to find the first kernel producing bad values
5. Check for dynamic shapes issues (historically a common source of IMA)

Triton Kernel Failures

Triton assertion failures or index-out-of-bounds in generated kernels.

Diagnosis:

TORCH_LOGS="output_code,schedule" python your_script.py

Key files:

• torch/_inductor/codegen/triton.py -- Triton codegen
• torch/_inductor/scheduler.py -- kernel fusion decisions

Fix approach:

1. Get the generated Triton kernel from output_code logs
2. Check index computations for off-by-one or wrong stride calculations
3. Look at the IR (TORCH_COMPILE_DEBUG=1) to trace back to the FX op
4. Check if fusion decisions created invalid index combinations

Key Source Files

Using the Minifier

The minifier reduces a failing graph to the smallest reproduction:

# Step 1: Generate the minifier launcher
TORCHDYNAMO_REPRO_AFTER=aot TORCHDYNAMO_REPRO_LEVEL=2 python your_script.py

# Step 2: Run the minifier
python minifier_launcher.py minify

# Step 3: Run the minimized repro
python minifier_launcher.py run

For accuracy issues, use level 4:

TORCHDYNAMO_REPRO_AFTER=aot TORCHDYNAMO_REPRO_LEVEL=4 python your_script.py