Care Program Effectiveness

Methodology

Step 1 — Define Program Cohort and Measurement Periods

Establish clear cohort boundaries:

• Enrollment criteria: Minimum program engagement threshold (e.g., ≥ 2 care manager contacts)
• Pre-period: 12 months before program enrollment (baseline measurement)
• Post-period: 12 months after program enrollment (outcome measurement)
• Wash-out period: Exclude first 30-60 days post-enrollment to allow intervention ramp-up
• Continuous enrollment: Require continuous eligibility through pre and post periods
• Exclusion criteria: Deceased during measurement, disenrolled, < minimum engagement

Step 2 — Construct Comparison Group

Build a matched comparison cohort using propensity score methods:

• Matching variables: Age, sex, baseline RAF score, prior-year cost, prior-year utilization (IP admits, ED visits), chronic condition count, dual-eligible status
• Matching method: 1:1 nearest-neighbor propensity score matching with caliper ≤ 0.2 SD
• Balance diagnostics: Verify standardized mean differences < 0.1 on all matching variables
• Sensitivity analysis: Test robustness with alternative methods (exact matching, inverse probability weighting)

If a formal comparison group is unavailable, use pre-post with trend adjustment as a secondary approach, acknowledging weaker causal inference.

Step 3 — Measure Outcome Domains

Evaluate across four outcome domains:

Clinical Outcomes:

• Disease-specific metrics (HbA1c change for diabetes, BP control for hypertension)
• Composite quality scores (percentage meeting all HEDIS targets)
• Medication adherence (PDC ≥ 80%)

Utilization Outcomes:

• Inpatient admissions per 1,000 (all-cause and condition-specific)
• ED visits per 1,000 (all-cause and avoidable per NYU algorithm)
• 30-day readmission rate
• PCP and specialist visit frequency

Cost Outcomes:

• Total cost of care PMPM (allowed amount, including all service categories)
• Category-specific cost (IP, ED, outpatient, pharmacy, ancillary)
• Cost trend (program vs. comparison growth rate)

Patient Experience Outcomes (if available):

• Patient satisfaction scores, self-reported health status, functional assessments

Step 4 — Calculate Program Impact

Apply difference-in-differences (DID) methodology:

Impact = (Post_program - Pre_program) - (Post_comparison - Pre_comparison)

• Calculate DID for each outcome metric
• Test statistical significance using regression with clustered standard errors
• Report 95% confidence intervals for all impact estimates
• Calculate effect sizes (Cohen's d) for clinical outcomes

Step 5 — Assess Dose-Response Relationship

Evaluate whether greater program engagement produces larger effects:

• Stratify by engagement level (low: 1-2 contacts, medium: 3-6, high: 7+)
• Test for dose-response gradient across engagement tiers
• Identify minimum effective dose (engagement threshold for measurable impact)
• Analyze engagement predictors (demographics, condition type, referral source)

Step 6 — Calculate Cost-Effectiveness

Determine program financial performance:

• Gross savings: Total cost avoidance in intervention group vs. comparison
• Program cost: Staffing, technology, overhead allocated to program
• Net savings: Gross savings minus program cost
• ROI: (Net savings / Program cost) × 100
• Cost per quality-adjusted outcome: Program cost / number of members achieving target

Step 7 — Synthesize and Recommend

Produce an integrated assessment:

• Summarize findings across all four outcome domains
• Rate program effectiveness: highly effective, moderately effective, inconclusive, or ineffective
• Identify sub-populations with strongest and weakest response
• Recommend: expand, maintain, modify (specify modifications), or sunset
• Define metrics and timeline for next evaluation cycle

Output Specification

Program Effectiveness Report:
├── Executive Summary (program description, key findings, recommendation)
├── Methodology Overview (study design, matching, measurement periods)
├── Cohort Description (enrollment, demographics, baseline equivalence)
├── Clinical Outcomes (pre/post, DID, significance, effect size)
├── Utilization Outcomes (admits, ED, readmissions — DID with CI)
├── Cost Outcomes (PMPM, total cost, cost avoidance)
├── Dose-Response Analysis (engagement tiers, response gradient)
├── ROI Calculation (gross savings, net savings, cost per outcome)
├── Sub-Group Analysis (by condition, acuity, demographics)
├── Limitations and Sensitivity Analyses
└── Recommendations and Next Steps

Analysis Framework

Effectiveness Rating Criteria

Common Program Benchmarks

• Disease management programs: 8-15% IP reduction, ROI 1.5:1-3:1
• Transitional care: 20-30% readmission reduction in first 90 days
• Remote patient monitoring: 15-25% ED reduction for enrolled CHF patients
• Care coordination: 5-10% total cost reduction for high-risk members

Examples

Example 1 — CHF Care Management Program Evaluate a CHF disease management program with 1,200 enrolled members over 18 months. Match 1:1 against 1,200 non-enrolled CHF patients. DID analysis shows: IP admissions −22% (p=0.003), ED visits −18% (p=0.01), 30-day readmissions −31% (p=0.001), total cost PMPM −$142 (p=0.008). Program cost $1.8M, gross savings $3.1M, net savings $1.3M, ROI 1.72:1. Recommend expansion to all CHF patients with EF < 40%.

Example 2 — Diabetes Prevention Program Assess a DPP-modeled lifestyle intervention for 800 pre-diabetic members. After 12 months, 34% of participants achieved ≥ 5% weight loss vs. 12% in comparison (p < 0.001). Diabetes conversion rate 4.2% vs. 8.8% in comparison (p=0.02). Program cost per prevented diabetes case: $3,200 vs. estimated 5-year diabetes care cost of $48,000.

Guidelines

• Propensity score matching requires sufficient overlap in covariate distributions; report overlap statistics
• Pre-post analysis without a comparison group is insufficient for causal claims — label clearly as associational
• Account for regression to the mean, particularly for programs targeting high-cost members
• Report all outcomes including null or negative findings to avoid publication bias
• Use intent-to-treat analysis as primary; per-protocol as sensitivity analysis

Validation Checklist

• Comparison group is well-balanced on all matching variables (SMD < 0.1)
• Pre-period trends are parallel between groups (parallel trends assumption)
• Continuous enrollment requirement applied consistently to both groups
• Program costs include all direct and allocated indirect costs
• Statistical tests are appropriate for outcome data types
• Sensitivity analyses confirm robustness of primary findings
• Limitations are clearly documented

HIPAA Compliance

This skill processes Protected Health Information (PHI) for program evaluation, which constitutes health care operations under HIPAA (45 CFR §164.506). All outputs must comply with HIPAA Privacy and Security Rules. Apply minimum necessary standards to data access. De-identify comparison group outputs. Individual-level program data must be maintained in access-controlled systems. Report aggregate outcomes only in external communications, applying minimum cell-size suppression (n ≥ 11) for sub-group analyses.