Stock Data Validator

• PE ratio sanity check: 0 < PE < 500 (flag outliers)
• EPS growth check: single-quarter growth should not exceed 300%
• Price spike detection: single-day change <20% acceptable, >20% flagged for review
• Currency consistency: ensure all prices in same currency, convert if needed
• Volume anomalies: sudden volume spikes >500% of average

4. Confidence Scoring

Assign a confidence score (0-100) to the entire validated package and to individual data points:

• 90-100 (HIGH): Multiple recent sources aligned, fresh data, no anomalies
• 70-89 (MEDIUM): Data consistent, slight freshness concerns, or minor anomalies resolved
• 50-69 (LOW): Single source, older data, or minor anomalies present
• <50 (EXCLUDED): Data conflicts, too old, or critical anomalies - exclude from analysis

Validation Process

1. Input: Raw JSON data package from data fetcher
2. Cross-validation: Compare against known reliable sources
3. Freshness check: Verify all data timestamps are within acceptable ranges
4. Anomaly detection: Run all sanity checks on numerical values
5. Scoring: Assign confidence scores based on checks passed
6. Output: Validated data package with metadata

Output Format

Return a structured JSON package containing:

• ticker: stock symbol
• validation_timestamp: when validation occurred
• data_freshness: object with age of each data category
• cross_source_check: results of source comparison
• anomaly_detection: list of any anomalies found
• confidence_score: overall package confidence (0-100)
• validated_data: cleaned and approved data
• excluded_fields: fields that failed validation
• notes: validation warnings and explanations

Key Thresholds

• Price data freshness: <3 calendar days (daily EOD data)
• Financial data freshness: <120 days
• News data freshness: <30 days
• PE ratio range: 0 to 500
• PB ratio range: 0 to 100
• Single-day price change tolerance: <20%
• Volume spike threshold: >500% of average
• Dividend yield anomaly: >20%
• Minimum price history records: 20
• Minimum confidence to pass: 50

Data Schema

The validator expects the nested JSON structure from fetch_data.py:

metadata → ticker, fetch_timestamp, missing_data[]
company_info → pe_ratio, pb_ratio, current_price, currency, ...
price_history[] → date, close, volume, ...
technical_indicators → rsi_14, macd, bollinger_bands, stochastic_kd
financial_statements → income_statement, balance_sheet, cash_flow
news[] → title, publish_date, publisher
holders, analyst_data

The validated output preserves this nested structure inside validated_data and adds: confidence_scores, anomaly_detection, validation_notes, data_completeness, data_freshness.

Implementation Reference

See scripts/validate_data.py for the validation engine that implements these rules programmatically.