Hypothesis Testing Guide

Common Errors

Test Selection Guide

Running Tests in Python

Independent Samples t-Test

from scipy import stats
import numpy as np
import pingouin as pg

# Generate example data
control = np.random.normal(50, 10, n=30)
treatment = np.random.normal(55, 10, n=30)

# Check normality assumption
stat_c, p_c = stats.shapiro(control)
stat_t, p_t = stats.shapiro(treatment)
print(f"Normality p-values: control={p_c:.3f}, treatment={p_t:.3f}")

# Check homogeneity of variance
stat_l, p_l = stats.levene(control, treatment)
print(f"Levene's test p={p_l:.3f}")

# Run t-test
t_stat, p_val = stats.ttest_ind(control, treatment, equal_var=(p_l > 0.05))

# Effect size (Cohen's d)
cohens_d = (treatment.mean() - control.mean()) / np.sqrt(
    ((len(control)-1)*control.var() + (len(treatment)-1)*treatment.var())
    / (len(control) + len(treatment) - 2)
)

print(f"t={t_stat:.3f}, p={p_val:.4f}, Cohen's d={cohens_d:.3f}")

One-Way ANOVA with Post-Hoc Tests

import pandas as pd

df = pd.DataFrame({
    'score': np.concatenate([
        np.random.normal(50, 10, 30),
        np.random.normal(55, 10, 30),
        np.random.normal(60, 10, 30)
    ]),
    'group': np.repeat(['A', 'B', 'C'], 30)
})

# ANOVA
aov = pg.anova(data=df, dv='score', between='group', detailed=True)
print(aov)

# Post-hoc pairwise comparisons (Tukey HSD)
posthoc = pg.pairwise_tukey(data=df, dv='score', between='group')
print(posthoc[['A', 'B', 'diff', 'p-tukey', 'hedges']])

Chi-Square Test of Independence

# Contingency table
observed = pd.DataFrame(
    [[45, 30], [25, 50]],
    index=['Method A', 'Method B'],
    columns=['Success', 'Failure']
)

chi2, p, dof, expected = stats.chi2_contingency(observed)
cramers_v = np.sqrt(chi2 / (observed.values.sum() * (min(observed.shape) - 1)))

print(f"chi2={chi2:.3f}, p={p:.4f}, Cramer's V={cramers_v:.3f}")

Power Analysis and Sample Size

Power analysis answers: "How many participants do I need?"

from statsmodels.stats.power import TTestIndPower, FTestAnovaPower

# For a two-sample t-test
analysis = TTestIndPower()

# Calculate required sample size
n = analysis.solve_power(
    effect_size=0.5,   # Cohen's d (medium effect)
    alpha=0.05,
    power=0.80,
    ratio=1.0,         # Equal group sizes
    alternative='two-sided'
)
print(f"Required n per group: {int(np.ceil(n))}")

# Power curve
import matplotlib.pyplot as plt

sample_sizes = np.arange(10, 200, 5)
powers = [analysis.power(effect_size=0.5, nobs1=n, ratio=1.0, alpha=0.05)
          for n in sample_sizes]

fig, ax = plt.subplots()
ax.plot(sample_sizes, powers)
ax.axhline(0.8, color='red', linestyle='--', label='Power = 0.80')
ax.set_xlabel('Sample Size per Group')
ax.set_ylabel('Statistical Power')
ax.legend()
fig.savefig('power_curve.pdf')

Effect Size Reference Table

Multiple Comparison Corrections

When running multiple tests, the family-wise error rate inflates. Use corrections:

from statsmodels.stats.multitest import multipletests

p_values = [0.01, 0.04, 0.03, 0.08, 0.002]

# Bonferroni (conservative)
reject_bonf, pvals_bonf, _, _ = multipletests(p_values, method='bonferroni')

# Benjamini-Hochberg FDR (less conservative)
reject_bh, pvals_bh, _, _ = multipletests(p_values, method='fdr_bh')

for i, p in enumerate(p_values):
    print(f"p={p:.3f} | Bonferroni: {pvals_bonf[i]:.3f} ({reject_bonf[i]}) "
          f"| BH-FDR: {pvals_bh[i]:.3f} ({reject_bh[i]})")

Best Practices

• Always report effect sizes alongside p-values. A significant p-value with a tiny effect size is rarely meaningful.
• Pre-register your analysis plan. This prevents p-hacking and HARKing (Hypothesizing After Results are Known).
• Check assumptions before running parametric tests. Use non-parametric alternatives when assumptions are violated.
• Use confidence intervals. They convey both effect magnitude and precision.
• Report exact p-values (p = 0.032), not thresholds (p < 0.05). Except when p < 0.001.
• Consider Bayesian alternatives. Bayes factors provide evidence for H0, not just against it.
• Plan sample sizes a priori. Power analysis should be done before data collection, not after.

References

• scipy.stats Documentation
• pingouin Documentation -- Friendly statistics in Python
• statsmodels Documentation
• APA 7th Edition Statistics Reporting Guide
• Statistical Tests Cheat Sheet