Dream Interpreter

Scope

Dream Interpreter operates in two modes:

Commands (execution):

# Analyze current system state with 24h prediction window
dream-interpreter analyze --window=24 --confidence=0.85

# Deep dive on specific service with custom data sources
dream-interpreter deep --service=payment-gateway --sources="logs,traces,metrics" --hours=168

# Generate correlation matrix for all services
dream-interpreter correlate --output-format=html --filter="http_5xx_count>10"

# Predict specific failure scenarios
dream-interpreter predict --scenario="cache-miss-cascade" --stress-test

# Validate prediction model against known incidents
dream-interpreter validate --incident-id=INC-2024-0847 --backtest

# Export insights to Grafana dashboard
dream-interpreter export --format=grafana-json --panel-group="System Health"
dream-interpreter export --format=pagerduty --severity=warning+

# Compare today's pattern against last Monday baseline
dream-interpreter compare --baseline="last-monday" --deviation-threshold=0.3

# Watch mode - continuous analysis every 15 minutes
dream-interpreter watch --interval=900 --alert-webhook="https://hooks.slack.com/..."

# Analyze with specific model (timeseries, anomaly, hybrid)
dream-interpreter analyze --model=lstm-hybrid --retrain

# Query historical predictions
dream-interpreter query --date="2024-01-15" --min-score=0.75

Subcommands:

• analyze: Single comprehensive analysis pass
• deep: Multi-dimensional deep dive
• correlate: Cross-service correlation analysis
• predict: Scenario-based prediction
• validate: Model validation against known data
• export: Format and output results
• compare: Baseline comparison
• watch: Continuous monitoring mode
• query: Historical data query

Detailed Work Process

Phase 1: Data Ingestion (0-5 minutes)

1. Read environment variables: SYSTEM_LOG_PATH, METRICS_DB_URL, JAEGER_ENDPOINT
2. Connect to Loki/Prometheus/Elasticsearch instances using configured queries
3. Fetch last ANALYSIS_WINDOW_HOURS (default 72) of structured logs
4. Pull metrics: CPU, memory, network, disk IO, application-specific (from METRICS_DB_URL)
5. Extract distributed traces from Jaeger/Tempo for services with error rate > 2%
6. Parse and normalize all timestamps to UTC
7. Cache raw data to /tmp/dream-raw-${TIMESTAMP}.parquet for reproducibility

Phase 2: Pattern Extraction (5-15 minutes)

1. Log Pattern Recognition:

   • Use DBSCAN clustering on log message embeddings (TF-IDF + sentence-transformers)
   • Identify "noisy" log streams (more than 50 unique templates per service)
   • Detect log volume spikes (>3x baseline) with Mann-Kendall test
   • Flag services with repeated error patterns (same error code > 10 times in 5 min)

2. Metric Anomaly Detection:

   • Apply Seasonal-Trend decomposition using LOESS (STL)
   • Calculate z-scores for each metric time series
   • Mark anomalies: |z-score| > 3.0 and duration > 2 consecutive points
   • Cross-reference anomalies across correlated metrics (Pearson r > 0.7)

3. Trace Analysis:

   • Build service dependency graph from trace spans
   • Calculate per-service p99 latency, error rate, and span count
   • Detect "slow cascade": service A latency ↑ → service B timeout ↑ within 30s
   • Identify "circuit breaker thrash": repeated open/close cycles in 5m window

Phase 3: Correlation Engine (15-25 minutes)

1. Construct multi-dimensional feature matrix:
   • Rows: 5-minute time bins
   • Columns: [log_error_rate_X, metric_cpu_Y, trace_latency_Z, event_count_W...]
2. Calculate Granger causality between metric pairs (max lag = 3 bins = 15min)
3. Run Random Forest feature importance to identify top predictors of errors
4. Apply Community Detection (Louvain algorithm) on the correlation graph
5. Generate correlation heatmap and export to PNG (default 1920x1080)

Phase 4: Prediction Generation (25-35 minutes)

1. For each identified anomaly cluster:
   • Build LSTM or Prophet model (auto-select based on data size)
   • Forecast next 6 hours with 95% confidence intervals
   • Calculate probability of exceeding thresholds:
     • Error rate > 5%
     • Latency p99 > 2s
     • Memory usage > 85%
2. Apply SHAP (SHapley Additive exPlanations) to each prediction
   • Generate "why this happened" explanation for top 3 features
3. Score prediction confidence:
   • 0.0-0.4: Noise (discard)
   • 0.4-0.7: Weak signal (store only)
   • 0.7-0.9: Medium risk (alert at severity=warning)
   • 0.9-1.0: High risk (alert at severity=critical)

Phase 5: Report Generation (35-40 minutes)

1. Compile Markdown report with sections:
   • Executive Summary (3 bullet points)
   • Critical Findings (with confidence scores)
   • Correlation Matrix (top 10 relationships)
   • Time Series Forecasts (4 charts)
   • Chain of Evidence (causal links)
   • Recommended Actions (2-3 prioritized)
2. Export JSON to /var/lib/dream-interpreter/insights/latest.json
3. If WATCH_MODE=true, push alerts to configured webhook with payload:

{
  "text": "Dream Interpreter Alert",
  "attachments": [
    {
      "title": "High Risk Prediction",
      "fields": [
        {"title": "Service", "value": "checkout-service"},
        {"title": "Issue", "value": "Database connection exhaustion"},
        {"title": "Confidence", "value": "94%"},
        {"title": "ETA", "value": "2024-01-20T14:30:00Z"}
      ]
    }
  ]
}

4. Write audit log: /var/log/dream-interpreter/audit.log with SHA256 of input data

Golden Rules

1. Never trust single metric: An isolated CPU spike means nothing. Must have at least 2 correlated signals before raising warning.

2. Baseline must be recent: No comparing to data older than 30 days. Weekly seasonality is OK, monthly is not. Use compare --baseline="last-week-same-dow" for day-of-week normalization.

3. Silence is golden: If prediction confidence < 0.7, DO NOT send alerts. Log quietly and move on. False positives destroy credibility.

4. Chain of evidence required: Every prediction must show at least 3 evidence points forming a temporal chain (A → B → C). Without chain, mark as "speculative" and downgrade to notice.

5. Respect data freshness: If metrics are > 5 minutes stale or logs > 15 minutes stale, abort analysis and raise system alert. We're analyzing the past, not the present.

6. No changes, only observation: Dream Interpreter is diagnostic only. Never auto-scale, restart services, or modify infrastructure. That would corrupt the dataset for next run.

7. Report only actionable insights: If prediction says "something might fail" but doesn't specify what, how, and where, it's useless. Filter out non-specific findings.

8. Preserve forensic data: Keep raw input Parquet files for 7 days. If an incident occurs, query --incident-reference=XXX must be able to reconstruct exactly what we saw.

9. Account for maintenance windows: If a change event (deployment, config push, backup) occurred within 2 hours of anomaly start, downgrade confidence by 0.2. Humans caused it, not systems.

10. Never interpret outside domain: If system is a payment gateway, don't make UX recommendations. Stick to infrastructure/state analysis. Specialization > generalization.

Examples

Example 1: Predictive Database Exhaustion

$ dream-interpreter deep --service=orders-db --sources="metrics,logs" --hours=72

[PHASE 1] Fetched: 2.1M log entries, 15.3K metric points, 847 traces
[PHASE 2] Detected:
  • Connection pool wait time ↑ (z-score: +4.2) for 45 minutes
  • "too many connections" errors: 1,247 instances (last hour)
  • Transaction commit latency p99: 2.1s → 8.7s
[PHASE 3] Correlation: connection_wait_time ⟷ error_count (r=0.89, p<0.001)
[PHASE 4] LSTM prediction (conf=0.88):
  • In 3.5 hours: connections will exceed pool limit
  • Contributing factors: long-running queries (+0.31 SHAP), connection leak (+0.28)
[PHASE 5] Report: /var/lib/dream-interpreter/reports/2024-01-19-143022.md

=== CRITICAL FINDING ===
Service: orders-db
Issue: Connection pool exhaustion in 3.5h
Confidence: 88%
Evidence Chain:
  14:02: Slow query detected (duration > 30s)
  14:15: Connection wait time began rising
  14:22: Error rate spiked to 12%
  14:30: Current prediction window started

Recommended Action:
  1. Check application `MAX_POOL_SIZE` setting (likely 20, should be 50)
  2. Investigate long-running query: SELECT * FROM orders WHERE status='pending' (executing 4m)
  3. Deploy query index: CREATE INDEX idx_orders_pending ON orders(status)

Example 2: False Positive Filtering

$ dream-interpreter analyze --window=168

[PHASE 2] Detected anomaly: CPU usage spike on worker-12 (z-score: +3.8)
[PHASE 3] Correlation: CPU ⟷ git-pull events (r=0.92, lag=0)
[PHASE 9] Change event detected: git pull at 02:15 UTC (2 min before spike)
[APPLYING RULE 9] Downgrading confidence: 0.82 → 0.62 (below threshold)
[PHASE 5] Not sending alert. Logging as "maintenance artifact".

Example 3: Watch Mode Operational Output

$ dream-interpreter watch --interval=900 --alert-webhook="https://hooks.slack.com/services/..."

[RUN 2024-01-19T14:00:00Z] Analysis complete in 8m 42s
  New findings: 0 (medium: 0, high: 0)
  Active predictions: 3 (2 expiring in <2h)
  Slack notification sent: false

[RUN 2024-01-19T14:15:00Z] Analysis complete in 9m 11s
  New findings: 1 (medium: 1, high: 0)
  Active predictions: 4
  Slack notification sent: true
  Alert ID: DREAM-20240119-1415-7f3a8b
  Preview: "API Gatewaylatencyp99isexpectedtoexceed2sat...

Rollback Commands

If Dream Interpreter causes issues (misconfiguration, bad alert storms), immediately:

# 1. Stop watch mode if running
pkill -f "dream-interpreter watch"
systemctl stop dream-interpreter-watch.timer  # if using systemd timer

# 2. Flush recent alerts (last 24h)
curl -X POST "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=${PD_TOKEN}" \
  -d '{"incident":{" DeweyDecimalClassification":"remediation"}}'

# OR if using webhook-based alerts, send reconciliation:
dream-interpreter export --format=json-lines | \
  jq -r '.alert_id' | \
  xargs -I {} curl -X DELETE "https://alerts.example.com/{}"

# 3. Revert configuration to last known good
cp /var/backups/dream-interpreter/config-2024-01-18.json \
   ~/.dream-interpreter/config.json
systemctl restart dream-interpreter

# 4. If database corruption occurred:
pg_restore -h localhost -U dream -d dream_analytics \
  /var/backups/dream-interpreter/db-$(date -d yesterday +%Y%m%d).dump

# 5. If alert fatigue happened, temporarily raise threshold:
echo 'export PREDICTION_THRESHOLD=0.95' >> ~/.bashrc
# Let old alerts expire naturally (max TTL is 24h by default)

# 6. Full disable (emergency only):
systemctl mask dream-interpreter-watch.timer
# Remove from crontab:
crontab -l | grep -v dream-interpreter | crontab -

Verification Steps

After deployment or update:

# 1. Check skill loads correctly
dream-interpreter --version
# Expected: Dream Interpreter 1.2.0

# 2. Test data access (dry-run)
dream-interpreter analyze --dry-run --window=1
# Expected: "HYDRATED: fetched 0 logs (test mode), 0 metrics"
# Should NOT error on connection strings

# 3. Verify dependencies
python3 -c "import dream_interpreter; print(dream_interpreter.__version__)"
# If ImportError: pip3 install -r requirements-extra.txt

# 4. Check output permissions
touch /var/lib/dream-interpreter/reports/test.md
# If PermissionError: chown -R $(whoami) /var/lib/dream-interpreter

# 5. Validate env vars
dream-interpreter validate-config
# Expected: All required environment variables set

# 6. Run with single known dataset
dream-interpreter analyze --window=24 --test-dataset=sample-incident-0847
# Expected: Should predict "database connection exhaustion" with conf > 0.85

# 7. If watch mode: test alert delivery
dream-interpreter watch --once --alert-webhook="https://webhook.site/your-test-id"
# Check webhook.site for received payload

# 8. Resource limits: monitor first run
time dream-interpreter analyze --window=72
# Expected: < 15 minutes, < 1.5GB RAM

Dependencies & Requirements

Python Packages (pip install -r requirements.txt):

psutil>=5.9.0        # System metrics
numpy>=1.21.0        # Numerical arrays
pandas>=1.3.0        # Data manipulation
scikit-learn>=1.0.0  # ML algorithms
tensorflow>=2.10.0   # LSTM models (CPU-only)
prophet>=1.1.0       # Time series forecasting
statsmodels>=0.14.0  # Statistical tests
loki-client>=1.8.0   # Log queries
prometheus-api-client>=0.5.0  # Metrics
jaeger-client>=1.6.0 # Trace parsing
shap>=0.41.0         # Model explainability
seaborn>=0.11.0      # Visualizations
matplotlib>=3.4.0    # Plotting

System Services (must be accessible):

• Prometheus: http://localhost:9090 (metrics)
• Grafana Loki: http://localhost:3100 (logs)
• Jaeger: http://localhost:16686 (traces)
• PostgreSQL: localhost:5432 (insights storage)

Required Environment Variables:

export SYSTEM_LOG_PATH="loki://localhost:3100"
export METRICS_DB_URL="prometheus://localhost:9090"
export TRACE_ENDPOINT="jaeger://localhost:16686"
export ANALYSIS_DB="postgresql://dream:password@localhost/dream_analytics"
export PREDICTION_THRESHOLD="0.75"
export ANALYSIS_WINDOW_HOURS="72"
export SLACK_WEBHOOK_URL="https://hooks.slack.com/..."

File Structure Created:

~/.dream-interpreter/
├── config.json              # User configuration
├── models/                  # Trained LSTM models (auto-created)
│   ├── lstm_service_x.h5
│   └── prophet_service_y.pkl
├── reports/                 # Generated analyses (JSON + Markdown)
├── insights/               # Latest prediction aggregates
└── cache/                  # Downloaded raw data (auto-cleaned after 7d)

Troubleshooting

Issue: "ImportError: No module named 'tensorflow'"

Fix: Dream Interpreter requires TensorFlow for LSTM predictions. Install minimal CPU version:

pip3 install --no-cache-dir tensorflow-cpu==2.10.0

Or run without LSTM: dream-interpreter analyze --model=prophet-only

Issue: "MemoryError during correlation phase"

Fix: The feature matrix is too large. Limit analysis window or number of services:

dream-interpreter deep --service=api-gateway  # single service only
# OR increase swap:
sudo fallocate -l 4G /swapfile2 && chmod 600 /swapfile2 && mkswap /swapfile2 && swapon /swapfile2

Issue: No correlations found (all r < 0.3)

Check:

1. Are metrics actually correlated? Use dream-interpreter correlate --raw-output to inspect data.
2. Is the time window too short? Try --hours=168 (one week).
3. Are timestamps aligned? Ensure all sources use UTC. Set TZ=UTC before running.
4. Could be that your system is well-designed with no hidden couplings. That's good.

Issue: Alert storm (>50 alerts in one run)

Immediate fix: Raise threshold:

export PREDICTION_THRESHOLD=0.90
# Restart watch mode

Root cause: Check SHAP explanations. If top features are "log count" or "metric count", you're correlating volume with volume. Filter high-cardinality services in config.

Issue: Stale data warning ("Metrics are 20 minutes old")

Cause: Your metrics collector (Prometheus node exporter) is down or scrape interval is too long. Fix:

1. Check Prometheus targets: curl http://localhost:9090/api/v1/targets
2. If targets down, restart exporters.
3. Increase --stale-tolerance=1800 to allow older data (not recommended for critical systems).

Issue: Watch mode doesn't send Slack alerts

Debug:

dream-interpreter watch --once --debug-webhook
# This prints: POST https://hooks.slack.com/... payload size: 2048
# Then curl -v output

If 403/401: Slack webhook URL is invalid or revoked. Regenerate at api.slack.com. If timeout: Outbound HTTPS blocked. Check proxy settings or firewall.

Issue: Model predict fails with "ValueError: Input contains NaN"

Cause: Gaps in time series metrics. Clean with interpolation:

dream-interpreter analyze --preprocess=interpolate-linear

Or use --preprocess=drop-gaps (reduces data but removes NaNs).

Issue: Performance degraded after adding new service

Symptom: Analysis time doubled from 8min to 16min. Cause: New service generates 500k logs/hour, increasing matrix width. Fix:

1. Filter low-traffic services: add "exclude_services": ["new-service"] to config.json
2. Increase feature aggregation: --metrics-aggregation=5m (coarser)
3. Upgrade resources: CPU cores ≥ 4, RAM ≥ 2GB.

Issue: SHAP explanations all zeros

Cause: Model not trained or features not standardized. Fix:

dream-interpreter analyze --retrain --full-retrain
# This rebuilds models with proper feature scaling

If persists: Check that target variable (errors) has variance. All-zero target → no learning possible.

dream-interpreter query --date=today | jq '.findings[].error_count' | sort -u
# Should have values in [0, 100]. If all 0 → your system is error-free (or logging is broken).

Advanced

Custom Data Source Integration

Dream Interpreter can ingest from non-standard sources via Python plugins:

# ~/.dream-interpreter/plugins/custom_source.py
from dream_interpreter import DataSource

class CloudWatchSource(DataSource):
    def fetch(self, start, end):
        import boto3
        logs = boto3.client('logs').filter_log_events(
            logGroupName='/aws/lambda/my-func',
            startTime=int(start.timestamp()*1000),
            endTime=int(end.timestamp()*1000)
        )
        return self.normalize(logs)

Register plugin in config:

{
  "plugins": ["~/.dream-interpreter/plugins/custom_source.py"],
  "sources": ["cloudwatch", "loki", "prometheus"]
}

Fine-tuning Prediction Horizon

The default 6-hour prediction is optimized for batch jobs. For web traffic, shorten:

dream-interpreter predict --horizon=2 --zoom-lookback=30m
# Validates patterns only from last 30 minutes for fast-changing metrics

CI/CD Integration Block

Add to your .gitlab-ci.yml to prevent deployments when system is unstable:

predictive_health_check:
  script:
    - dream-interpreter analyze --window=24 --ci-mode
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  allow_failure: false
  artifacts:
    paths: ["/var/lib/dream-interpreter/reports/latest.json"]

The job fails (non-zero exit) if "critical" predictions exist with conf > 0.85.

© 2024 SMOUJBOT. Skill validated against production systems: 2024-01-15.