Run the data pipeline step by step — execute each notebook, validate output, and fix issues before proceeding. Use for weekly data refreshes.
Run the TNCasino data pipeline interactively. Execute each notebook in order, validate the output after each step, and investigate/fix any failures before moving on.
Important: Scrapers are fragile and break often. When a scraper fails, inspect the error, check if the source site changed, and adapt selectors or logic as needed. Do not skip a failed step — fix it first.
Use $ARGUMENTS to control which steps to run (e.g., /run-pipeline 01-04 or /run-pipeline 07 for a single step). Default: run all 9 steps.
For each notebook, run it with jupyter nbconvert --to notebook --execute backend/notebooks/<notebook>.ipynb --output <notebook>.ipynb and then validate.
01_league_controlleague.db has current week's matchups and all roster data. Confirm row counts look reasonable.02_projections_controlprojections.db for new projection rows. Verify each source contributed data (some sources may fail independently). Report which sources succeeded/failed.03_post_scraping_processing04_match_projections_to_sleeper05_compute_player_week_stats06_team_lineup_optimizer07_monte_carlo_simulationsmontecarlo.db was created/updated. Verify simulation counts and that odds look reasonable (no 100-0 blowouts unless warranted).08_database_validation09_playoff_oddsReport a summary: which steps succeeded, which needed fixes, and any data quality concerns.