Reward Penalty Engineering

This skill does NOT contain reward component examples or scale tables. Those live in their respective locations:

What	Where
Component reference & scale ranges	starter_kit_schedule/templates/reward_config_template.yaml
Archived reward/penalty instances	starter_kit_schedule/reward_library/
Terrain strategies & reward code	quadruped-competition-tutor skill
Stage-specific reward overrides	curriculum-learning skill
Reward weight search spaces	hyperparameter-optimization skill
Visual reward debugging	subagent-copilot-cli skill

When to Use This Skill

The Exploration Cycle

Reward engineering is iterative. Every change follows this cycle:

    ┌──────────────┐
    │   DIAGNOSE   │ ← What behavior is wrong?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │  HYPOTHESIZE  │ ← What reward signal could fix it?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   IMPLEMENT   │ ← Minimal change, one variable at a time
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │     TEST      │ ← Short run (1-2M steps), multiple seeds
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   EVALUATE    │ ← Did the hypothesis hold?
    └──────┬───────┘
           ▼
    ┌──────────────┐
    │   ARCHIVE     │ ← Record result in reward library
    └──────┬───────┘
           │
           ▼
      Next cycle

Rule: Never change more than one reward dimension per cycle. If you change both the termination penalty AND add a new gait reward, you cannot attribute outcomes.

Phase 1: Diagnose

Behavioral Signals

Before touching rewards, identify what behavior is wrong. Not "the reward is too low" but a concrete observable:

Diagnostic Commands

# 1. Watch the policy — ALWAYS start here before looking at numbers
uv run scripts/play.py --env <env-name>

# 2. Train with rendering to see behavior in real time
uv run scripts/train.py --env <env-name> --render

# 3. TensorBoard for reward curves
uv run tensorboard --logdir runs/<env-name>

Visual Diagnosis

Use subagent-copilot-cli to analyze simulation frames and training curves:

# Describe what you see, ask what reward signal is missing
copilot --model gpt-4.1 --allow-all -p "Watch this simulation frame. The robot is <describe behavior>. What reward signal might cause this?" -s

Key insight: A reward signal is "missing" if the agent has no gradient pointing toward the desired behavior in its current state. The fix may be a new reward, a penalty, or reshaping an existing one.

Phase 2: Hypothesize

Formulating a Good Hypothesis

A testable reward hypothesis has three parts:

1. Behavior target: What the robot should do differently
2. Signal mechanism: What mathematical signal encodes that behavior
3. Expected side effect: What might go wrong

Template:

"If I add/modify <signal> with weight <w>, the robot should <desired behavior>, but might also <risk>."

Discovery Strategies

When you don't know what to try, use these strategies to generate candidates:

Strategy 1: Inversion

Take the undesired behavior and directly penalize it.

Robot bouncing → penalize vertical velocity Robot spinning → penalize angular velocity Robot retreating → penalize backward displacement

Strategy 2: Shaping the Gradient

If the robot is stuck, the reward surface is flat in its current region. Add a signal that creates local gradient:

Robot stuck far from goal → Add distance-based shaping (sigmoid, exponential) Robot stuck near goal → Add fine-grained proximity bonus Robot stuck on terrain edge → Add progress checkpoints

Strategy 3: Proxy Decomposition

Break the competition score into component sub-goals and create a signal for each:

Final score = traversal + bonus zones + time bonus → Create separate signals for: forward progress, zone proximity, speed

Strategy 4: Biomimetic Analogy

What would a real quadruped "want" in this situation?

Stairs → lift knees higher Uneven ground → keep center of mass low Obstacles → slow down, increase awareness

Strategy 5: Ablation Discovery

Temporarily remove one existing reward and see what degrades:

# Remove component to see its effect
python scripts/train.py --env <env> --seed 42 --cfg-override "reward_config.scales.<component>=0.0"

If removing a component doesn't change behavior, it was irrelevant. If behavior collapses, it was critical.

Strategy 6: Competition-Score Alignment

Compare training reward to competition scoring rules. Gaps indicate missing signals:

Competition awards points for stopping in smiley zones → but training reward only rewards forward velocity → mismatch: need a "stop in zone" signal

Refer to quadruped-competition-tutor skill for competition scoring rules.

Strategy 7: Browse the Library

Check previously tried components in the reward library before inventing new ones:

# Browse archived reward components
Get-ChildItem starter_kit_schedule/reward_library/components/ | Select-Object Name
# Read a specific component's notes
Get-Content starter_kit_schedule/reward_library/components/<name>.yaml

Phase 3: Implement

Principles

1. One variable at a time — Change a single reward component per experiment
2. Minimal change — Prefer adjusting a weight before adding new code
3. Use existing infrastructure — Check reward_config_template.yaml for components that can be enabled/disabled before writing new code

Where to Make Changes

Change Magnitude Guidelines

When adjusting weights, use multiplicative steps not additive:

• Small adjustment: ×0.5 or ×2 (halve or double)
• Medium adjustment: ×0.1 or ×10
• Large adjustment: ×0.01 or ×100

For new components, start with a weight that produces reward magnitude comparable to existing dominant terms (check reward_breakdown logs).

Phase 4: Test

🔴 AutoML-First Testing (MANDATORY)

NEVER iterate manually with train.py, changing one reward weight, running, reading TensorBoard, killing, repeating. This is manual one-at-a-time search — slow, error-prone, and wasteful. ALWAYS use automl.py for batch reward hypothesis testing.

The correct workflow:

1. Add your reward hypothesis as a search range in REWARD_SEARCH_SPACE (in automl.py)
2. Run automl.py --hp-trials 8+ to test multiple configurations in one batch
3. Read the structured comparison in starter_kit_log/automl_*/report.md
4. Archive results in the reward library

Example: Testing near_target_speed activation radius

# In automl.py REWARD_SEARCH_SPACE:
"near_target_speed": {"type": "uniform", "low": -2.0, "high": -0.1},
"near_target_activation": {"type": "choice", "values": [0.3, 0.5, 1.0, 2.0]},

Then run: uv run starter_kit_schedule/scripts/automl.py --mode stage --hp-trials 8

Why AutoML is better:

• Tests N configurations in parallel with identical conditions
• Produces structured comparison table (reward, reached%, distance, ep_len)
• Bayesian suggestion improves with each trial
• No manual TensorBoard reading or experiment-killing needed

Exception: train.py is acceptable for testing ONLY when:

• Smoke test — --max-env-steps 200000 to verify new reward code compiles
• Visual debugging — --render to watch behavior qualitatively
• The reward requires NEW CODE in _compute_reward() (test compilation first, then use automl)

Experiment Protocol

# Record this BEFORE running the experiment