Validate Daily

Use the correct date for each query:

• Game data (box scores, stats, predictions): Use GAME_DATE
• Processing status (scraper runs, Phase 3 completion): Use PROCESSING_DATE

Interactive Mode (User Preference Gathering)

If the user invoked the skill without specific parameters, ask them what they want to check:

Use the AskUserQuestion tool to gather preferences:

Question 1: "What would you like to validate?"
Options:
  - "Today's pipeline (pre-game check)" - Check if today's data is ready before games start
  - "Yesterday's results (post-game check)" - Verify yesterday's games processed correctly
  - "Specific date" - Validate a custom date
  - "Quick health check only" - Just run health check script, no deep investigation

Question 2: "How thorough should the validation be?"
Options:
  - "Standard (Recommended)" - Priority 1 + Priority 2 checks
  - "Quick" - Priority 1 only (critical checks)
  - "Comprehensive" - All priorities including spot checks

Based on their answers, determine scope:

• Today pre-game: Run standard workflow, note predictions may not exist yet
• Yesterday post-game: Run Yesterday's Results Validation Workflow
• Specific date: Ask for date, then run validation for that date
• Quick health check: Run Phase 1 only (health check script)

If the user already provided parameters (e.g., specific date in their message), skip the questions and proceed with those parameters.

Date Determination

After determining what to validate, set the target dates:

If "Today's pipeline (pre-game check)":

• GAME_DATE = TODAY (games scheduled for tonight)
• PROCESSING_DATE = TODAY (data should be ready now)

If "Yesterday's results (post-game check)":

• GAME_DATE = YESTERDAY (games that were played)
• PROCESSING_DATE = TODAY (scrapers ran after midnight)

If "Specific date":

• GAME_DATE = USER_PROVIDED_DATE
• PROCESSING_DATE = DAY_AFTER(USER_PROVIDED_DATE)

# Set dates in bash for queries
GAME_DATE=$(date -d "yesterday" +%Y-%m-%d)      # For yesterday's results
PROCESSING_DATE=$(date +%Y-%m-%d)               # Today (when processing ran)

# Or for pre-game check
GAME_DATE=$(date +%Y-%m-%d)                     # Today's games
PROCESSING_DATE=$(date +%Y-%m-%d)               # Today

Critical: Use GAME_DATE for game data queries, PROCESSING_DATE for processing status queries.

Current Context & Timing Awareness

First: Determine current time and game schedule context

• What time is it now? (Pre-game ~5 PM ET vs Post-game ~6 AM ET next day)
• Are there games today? Check the schedule
• What data should exist by now? (Timing affects expectations)

Key Timing Rules:

• Pre-game (5 PM ET): Betting data, game context, ML features should exist. Predictions may not exist yet (games haven't happened).
• Post-game (6 AM ET): Everything including predictions should exist for yesterday's games.
• Off-day: No games scheduled is normal, not an error.

Standard Validation Workflow

Phase 0: Proactive Quota Check (NEW)

IMPORTANT: Check BigQuery quotas FIRST to prevent cascading failures.

# Check current quota usage for partition modifications
bq show --format=prettyjson nba-props-platform | grep -A 10 "quotaUsed"

# Or check recent quota errors in logs
gcloud logging read "resource.type=bigquery_resource AND protoPayload.status.message:quota" \
  --limit=10 --format="table(timestamp,protoPayload.status.message)"

What to look for:

• Partition modification quota: Should be < 5000/day (limit varies by project)
• Recent "Quota exceeded" errors in logs
• If quota issues detected → Mark as P1 CRITICAL and recommend batching writes

Common cause: pipeline_logger writing too many events to partitioned run_history table

Phase 0.1: Deployment Drift Check (Session 81 - CRITICAL)

IMPORTANT: Verify all critical services are deployed with latest code.

Why this matters: Bug fixes committed but not deployed cause issues to persist. Sessions 82, 81, and 64 had critical bugs that were fixed in code but not deployed for hours/days.

Real Examples:

• Session 82: Coordinator fix committed but not deployed
• Session 81: Worker env vars fix committed but not deployed
• Session 64: Backfill ran with OLD code 12 hours after fix was committed (50.4% hit rate vs 58%+)

What to check:

# Run deployment drift check
./bin/check-deployment-drift.sh --verbose

Expected Result:

• All services show ✓ Up to date or have acceptable drift (<24 hours)
• No services showing STALE DEPLOYMENT

If STALE DEPLOYMENT detected:

Critical Services (must be up-to-date):

• prediction-worker - Generates predictions
• prediction-coordinator - Orchestrates predictions
• nba-grading-service - Grades predictions
• nba-phase3-analytics-processors - Analytics processing
• nba-scrapers - Data collection

Investigation if drift detected:

# See what changed since deployment
SERVICE="prediction-worker"
DEPLOYED_SHA=$(gcloud run services describe $SERVICE --region=us-west2 \
  --format="value(metadata.labels.commit-sha)")

git log --oneline $DEPLOYED_SHA..HEAD -- predictions/worker/

# If critical fixes found, deploy immediately
./bin/deploy-service.sh $SERVICE

Reference: Sessions 82, 81, 64 handoffs

Phase 0.12: Live-Export Env Var Drift Check (Session 302 - CRITICAL)

Purpose: Verify the live-export Cloud Function has its BDL_API_KEY env var intact. On Feb 22, a --set-env-vars deployment wiped BDL_API_KEY, causing live-grading to silently regenerate stale data for an entire evening.

Why this matters: live-export is a Cloud Function (not Cloud Run), so standard deployment drift checks don't cover it. Without BDL_API_KEY, BDL live box score lookups fail silently and the live-grading JSON gets regenerated every 3 minutes with all-pending, zero-actual data.

What to check:

# Check BDL_API_KEY is present in live-export env vars
gcloud functions describe live-export \
  --region=us-west2 \
  --project=nba-props-platform \
  --format="json" | jq '.environmentVariables | keys'

# Run the full env var drift detector (supports Cloud Functions)
./bin/monitoring/verify-env-vars-preserved.sh live-export

Expected result:

• BDL_API_KEY present and non-empty in env vars
• GCP_PROJECT present
• verify-env-vars-preserved.sh shows ALL REQUIRED VARIABLES PRESENT

Alert thresholds:

• BDL_API_KEY missing: CRITICAL — live scoring is completely broken
• GCP_PROJECT missing: CRITICAL — function may not find resources

If BDL_API_KEY missing:

# Immediate fix: re-add BDL_API_KEY from Secret Manager
BDL_API_KEY=$(gcloud secrets versions access latest --secret=BDL_API_KEY --project=nba-props-platform)
gcloud functions deploy live-export \
  --region=us-west2 \
  --project=nba-props-platform \
  --update-env-vars="BDL_API_KEY=$BDL_API_KEY"

Also check live-freshness-monitor:

./bin/monitoring/verify-env-vars-preserved.sh live-freshness-monitor

Reference: Session 302 (BDL_API_KEY wiped by --set-env-vars deployment)

Phase 0.2: Heartbeat System Health (NEW)

IMPORTANT: Check Firestore heartbeat collection for document proliferation.

Why this matters: Heartbeat documents should be ONE per processor (one doc gets updated). If processors create NEW documents for each run, the collection grows unbounded (100k+ docs), causing performance degradation and Firestore costs.

What to check:

# Check Firestore heartbeat document count
python3 -c "
from google.cloud import firestore
db = firestore.Client(project='nba-props-platform')
docs = list(db.collection('processor_heartbeats').stream())
bad = [d for d in docs if '_None_' in d.id or '_202' in d.id]
total = len(docs)
bad_count = len(bad)

print(f'Total heartbeat documents: {total}')
print(f'Bad format (old pattern): {bad_count}')
print(f'Expected: ~30-50 documents')

if bad_count > 0:
    print(f'\n⚠️  WARNING: {bad_count} old format documents detected!')
    print('Sample bad documents:')
    for doc in bad[:5]:
        print(f'  {doc.id}')
    print('\nAction: Run bin/cleanup-heartbeat-docs.py')

if total > 100:
    print(f'\n⚠️  WARNING: Too many documents ({total})!')
    print('Expected: ~30-50 (one per active processor)')
    print('Possible cause: Heartbeat code creating new docs instead of updating')
    print('Action: Check shared/monitoring/processor_heartbeat.py')
"

Expected result:

• Total documents: 30-50
• Bad format documents: 0

If issues detected:

Cleanup command:

# Preview cleanup
python bin/cleanup-heartbeat-docs.py --dry-run

# Execute cleanup
python bin/cleanup-heartbeat-docs.py

Investigation command (if proliferation detected):

# Find which processors created docs in last hour
python3 -c "
from google.cloud import firestore
from datetime import datetime, timedelta
db = firestore.Client(project='nba-props-platform')
docs = list(db.collection('processor_heartbeats').stream())
cutoff = datetime.now() - timedelta(hours=1)

recent_bad = []
for doc in docs:
    data = doc.to_dict()
    last_hb = data.get('last_heartbeat')
    if '_None_' in doc.id or '_202' in doc.id:
        if last_hb and hasattr(last_hb, 'replace') and last_hb.replace(tzinfo=None) > cutoff:
            recent_bad.append(doc.id)

if recent_bad:
    print(f'⚠️  {len(recent_bad)} bad documents created in last hour')
    print('Offending processors:')
    for doc_id in set([d.split('_None_')[0].split('_202')[0] for d in recent_bad]):
        print(f'  {doc_id}')