Psychology Research Guide

Power Analysis Before You Collect Data

from statsmodels.stats.power import TTestIndPower, FTestAnovaPower
import numpy as np

# Two-sample t-test power analysis
analysis = TTestIndPower()

# Question: "How many participants per group for d=0.5, power=0.80?"
n_per_group = analysis.solve_power(
    effect_size=0.5,     # Cohen's d (medium effect)
    alpha=0.05,
    power=0.80,
    alternative="two-sided",
)
print(f"Required N per group: {int(np.ceil(n_per_group))}")  # 64

# For small effects (d=0.2), which are common after replication
n_small = analysis.solve_power(effect_size=0.2, alpha=0.05, power=0.80)
print(f"Required N per group for d=0.2: {int(np.ceil(n_small))}")  # 394

# One-way ANOVA (3 groups)
anova_analysis = FTestAnovaPower()
n_anova = anova_analysis.solve_power(
    effect_size=0.25,    # Cohen's f (medium)
    alpha=0.05,
    power=0.80,
    k_groups=3,
)
print(f"Required N per group (ANOVA): {int(np.ceil(n_anova))}")  # 53

Effect Sizes That Reviewers Expect

Important: Post-replication-crisis psychology finds that most real effects are small (d = 0.2-0.4). Design for small effects unless you have strong prior evidence for larger ones.

Pre-Registration

What to Pre-Register

Pre-registration template (AsPredicted.org format):

1. HYPOTHESES
   H1: Participants in the gratitude condition will report higher
   life satisfaction (SWLS scores) than those in the control
   condition (d >= 0.3).

2. DESIGN
   - 2 (gratitude vs. control) between-subjects
   - Random assignment via Qualtrics randomizer

3. PLANNED SAMPLE
   - N = 200 per condition (400 total)
   - Power: 0.90 for d = 0.3 at alpha = 0.05
   - Recruitment: Prolific, US residents, 18-65

4. EXCLUSION CRITERIA (stated before data collection)
   - Failed attention check (embedded in survey)
   - Completion time < 3 minutes or > 30 minutes
   - Duplicate IP addresses

5. MEASURED VARIABLES
   - DV: Satisfaction With Life Scale (SWLS; Diener et al., 1985)
   - Manipulation check: "How grateful do you feel right now?" (1-7)
   - Covariates: Age, gender, baseline mood (PANAS)

6. ANALYSIS PLAN
   - Primary: Independent samples t-test on SWLS scores
   - Secondary: ANCOVA controlling for baseline PANAS-PA
   - Exploratory: Moderation by trait gratitude (GQ-6)

7. ANYTHING ELSE
   - All deviations from this plan will be labeled as exploratory
   - We will report all conditions and all measures

Pre-Registration Platforms

Statistical Analysis

The Modern Analysis Workflow

import pandas as pd
import pingouin as pg
from scipy import stats

# Load data
df = pd.read_csv("experiment_data.csv")

# Step 1: Descriptive statistics by condition
descriptives = df.groupby("condition").agg(
    n=("dv", "count"),
    mean=("dv", "mean"),
    sd=("dv", "std"),
    median=("dv", "median"),
).round(3)

# Step 2: Check assumptions
# Normality
for condition in df["condition"].unique():
    subset = df[df["condition"] == condition]["dv"]
    stat, p = stats.shapiro(subset)
    print(f"{condition}: Shapiro-Wilk W={stat:.3f}, p={p:.3f}")

# Homogeneity of variance
levene_stat, levene_p = stats.levene(
    df[df["condition"] == "treatment"]["dv"],
    df[df["condition"] == "control"]["dv"],
)

# Step 3: Primary analysis with effect size and CI
result = pg.ttest(
    df[df["condition"] == "treatment"]["dv"],
    df[df["condition"] == "control"]["dv"],
    paired=False,
    alternative="two-sided",
)
print(result[["T", "dof", "p-val", "cohen-d", "CI95%", "BF10"]])

# Step 4: Bayesian analysis (increasingly expected)
bf10 = float(result["BF10"].values[0])
print(f"Bayes Factor BF10 = {bf10:.2f}")
if bf10 > 10:
    print("Strong evidence for H1")
elif bf10 > 3:
    print("Moderate evidence for H1")
elif bf10 > 1:
    print("Anecdotal evidence for H1")