Run retail store scrapers with common configurations
Run the retail store scraper with specified configuration.
| Command | Description |
|---|---|
/run-scraper verizon | Single retailer with defaults |
/run-scraper all --test | All retailers, test mode (10 stores each) |
/run-scraper target --limit 50 | Target with 50-store limit |
/run-scraper all --incremental | Incremental update (new/changed only) |
/run-scraper verizon --proxy residential | Single retailer with proxy |
/run-scraper all --cloud | All retailers with GCS sync |
| Flag | Description |
|---|---|
--retailer <name> |
| Run specific retailer (verizon, att, target, tmobile, walmart, bestbuy, telus, cricket) |
--all | Run all enabled retailers concurrently |
--test | Test mode - 10 stores per retailer |
--limit <n> | Limit stores per retailer |
--resume | Resume from checkpoints |
--incremental | Only scrape new/changed stores |
--proxy <mode> | Proxy mode: direct, residential, web_scraper_api |
--validate-proxy | Validate proxy credentials before running |
--format <types> | Export formats: json,csv,excel,geojson |
--cloud | Sync to GCS after scraping |
--gcs-history | Save timestamped history copies |
--states <codes> | Target specific states (Verizon only) |
--exclude <retailers> | Exclude specific retailers from --all |
--status | Check status without running |
source venv/bin/activatepython run.py {parsed_args}# Quick test run
python run.py --retailer verizon --test
# Production run with proxy and cloud sync
python run.py --all --proxy residential --cloud
# Incremental update for specific retailers
python run.py --retailer target --retailer walmart --incremental
# Full run excluding problematic retailers
python run.py --all --exclude bestbuy --proxy residential
--proxy residential--validate-proxy firstdata/{retailer}/output/