Use when assessing the reliability of behavioral measurement through interobserver agreement methods, calculating IOA, and troubleshooting low agreement.
Interobserver agreement quantifies the extent to which two or more independent observers agree on the occurrence of a target behavior when simultaneously observing the same events. IOA is a necessary condition—though not a sufficient one—for establishing measurement reliability.
The simplest method. Compare the total counts from both observers for the entire session.
Formula: (Smaller count / Larger count) × 100
When to use: Quick check when the behavior occurs at a low-to-moderate rate and the session is brief.
Limitation: Can inflate agreement when behavior rates are high—observers may record the same total but disagree on when specific instances occurred.
Divide the session into intervals. Compare the counts within each interval.
Formula: Calculate agreement for each interval (smaller/larger × 100), then average across all intervals.
When to use: When the behavior occurs at moderate-to-high rates and you need more precision than total count IOA.
Compare observer records on each discrete trial. Score each trial as "agree" or "disagree."
Formula: (Agreements / [Agreements + Disagreements]) × 100
When to use: Discrete trial training, structured tasks with clear trial boundaries.
For interval recording data. Compare each interval and score as agree or disagree.
Formula: (Intervals with agreement / Total intervals) × 100
When to use: Whole-interval, partial-interval, or momentary time sampling data.
Limitation: Can inflate agreement when the behavior occurs at very high or very low rates (base rate problem).
Only consider intervals where at least one observer scored the behavior as occurring.
Formula: (Intervals both scored occurrence / Intervals at least one scored occurrence) × 100
When to use: When the behavior is infrequent (low base rate). Standard interval IOA inflates agreement by counting all the "no-no" agreements.
Only consider intervals where at least one observer scored the behavior as not occurring.
Formula: (Intervals both scored non-occurrence / Intervals at least one scored non-occurrence) × 100
When to use: When the behavior is very frequent (high base rate). Standard interval IOA inflates agreement by counting all the "yes-yes" agreements.
Synonym for scored-interval IOA in some texts. The more conservative measure when behavior is infrequent.
Synonym for unscored-interval IOA in some texts. The more conservative measure when behavior is frequent.
| Behavior Characteristic | Recommended IOA Method |
|---|---|
| Discrete trials with clear boundaries | Trial-by-trial |
| Low-rate behavior with interval recording | Scored-interval |
| High-rate behavior with interval recording | Unscored-interval |
| Moderate-rate behavior with interval recording | Interval-by-interval |
| Event recording, low-to-moderate rate | Total count or exact count per interval |
| Duration recording | Total duration or mean duration per occurrence |
When using duration recording, IOA can be calculated by:
Minimum reporting elements:
Example: "Interobserver agreement was assessed on 30% of sessions across all phases using trial-by-trial IOA. Mean agreement was 94% (range: 87%–100%)."