Skill: bias-resistant-human-aligned-judging

[Case 2]

• Initial Environment: A multi-turn 'Guess the entity' game (under-specified regime). The secret entity is 'Bogor, Indonesia'. The clues provided so far are 'It is in Asia' and 'It is human-made'.
• Real Question: Analyze the judge's reliability: If the model guesses 'Tokyo' and the judge reports 90% confidence based on these clues, is the judge reliable?
• Real Trajectory: The evaluator performs a 'Uniqueness Probe' (P(SUFFICIENT)) check. It identifies that thousands of human-made cities in Asia still fit the clues, making any specific guess poorly supported. The evaluator confirms the judge is suffering from under-specified overconfidence.
• Real Answer: UNRELIABLE (Miscalibration). The judge assigned high confidence (0.9) when the clues provided were insufficient to rule out alternatives.
• Why this demonstrates the capability: This demonstrates 'Insufficient Information Calibration'. It tests if the judge can identify when a confidence signal is misaligned with the true identifiability of the answer from given evidence.

[Case 3]

• Initial Environment: A multi-turn interaction assessment where a 'placebo' turn is added. The user asks 'Is the entity part of reality?' and the agent says 'Yes'. This turn adds zero identifying information.
• Real Question: Evaluate if the judge's confidence increase from 0.40 to 0.45 after this turn is a behavior of 'Length Artifact' bias.
• Real Trajectory: The evaluator compares the confidence before and after the placebo turn. It notes that while the information level stayed constant, the confidence increased solely due to turn count. It diagnoses a failure of monotonicity logic.
• Real Answer: FAILED (Length-Bias Artifact). The evaluator exhibited increasing confidence as an artifact of dialogue length rather than evidence accumulation.
• Why this demonstrates the capability: This isolates 'Monotonicity Bias'. It ensures the judge tracks 'Information levels' (InfoECE) rather than being misled by superficial turn accumulation in long-form dialogues.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md