Research Deep Dive

Key Files

Detector Base Class Pattern

New detectors MUST follow this pattern:

from qcml_geometry.observables import BaseRegimeDetector
import numpy as np

class MyNewDetector(BaseRegimeDetector):
    """One-line description.

    Detailed description of the observable and its theoretical basis.

    Args:
        window: Rolling window size for computation.
        hilbert_dim: Dimension of Hilbert space (default 4).
    """

    def __init__(self, window: int = 60, hilbert_dim: int = 4, **kwargs):
        self.window = window
        self.hilbert_dim = hilbert_dim

    @property
    def name(self) -> str:
        return "My New Observable"

    def fit(self, X: np.ndarray, **kwargs) -> 'MyNewDetector':
        """Fit on pre-crisis data only (causal)."""
        # Store training statistics for normalization
        self._train_mean = np.mean(X, axis=0)
        self._train_std = np.std(X, axis=0) + 1e-8
        return self

    def compute_regime_scores(self, X: np.ndarray) -> np.ndarray:
        """Produce 1-D regime score time series.

        CRITICAL: Must use expanding windows only — no future data.
        Higher scores = more anomalous / crisis-like.
        """
        scores = np.zeros(len(X))
        for t in range(self.window, len(X)):
            window_data = X[t - self.window:t]  # past only
            scores[t] = self._compute_observable(window_data)
        return scores

    def _compute_observable(self, window_data: np.ndarray) -> float:
        """Compute the observable value for a single window."""
        # Implementation here
        pass

Data Pipeline

from experiments.data_loader import fetch_data, create_feature_matrix, ALL_CRISES

# Fetch market data
df = fetch_data(['SPY', 'DIA'], '2005-01-01', '2025-12-31')

# Create feature matrix (returns, realized vol, etc.)
X = create_feature_matrix(df, symbols=['SPY', 'DIA'])

# Crisis definitions
crisis = ALL_CRISES['2008_gfc']
# {'start': '2008-09-01', 'end': '2009-03-31', 'label': 'GFC 2008'}

Evaluation

from experiments.evaluation import compute_cohens_d_with_ci, cliffs_delta

# Split scores into crisis vs normal periods
crisis_mask = (dates >= crisis['start']) & (dates <= crisis['end'])
crisis_scores = scores[crisis_mask]
normal_scores = scores[~crisis_mask]

# Cohen's d with bootstrap CI
d, ci_lo, ci_hi = compute_cohens_d_with_ci(
    crisis_scores, normal_scores,
    n_bootstrap=1000,  # use 1000 for smoke test speed
    seed=42
)

Smoke Test Template

Workers running empirical tests should follow this pattern:

import json
import numpy as np
from pathlib import Path

# Setup
SMOKE_CRISES = ['2008_gfc', '2020_covid', '2022_rates', '2023_svb']
KEEP_THRESHOLD = 0.3

from experiments.data_loader import fetch_data, create_feature_matrix, ALL_CRISES
from experiments.evaluation import compute_cohens_d_with_ci

# Fetch data
df = fetch_data(['SPY', 'DIA'], '2005-01-01', '2025-12-31')
X = create_feature_matrix(df, symbols=['SPY', 'DIA'])
dates = X.index if hasattr(X, 'index') else None

# Instantiate detector
detector = MyNewDetector(window=60)

# For each crisis, fit on pre-crisis data, score full series
results = {}
for crisis_key in SMOKE_CRISES:
    crisis = ALL_CRISES[crisis_key]
    # Fit on data before crisis
    pre_crisis_mask = dates < crisis['start']
    detector.fit(X[pre_crisis_mask].values)
    # Score full series
    scores = detector.compute_regime_scores(X.values)
    # Evaluate
    crisis_mask = (dates >= crisis['start']) & (dates <= crisis['end'])
    d, ci_lo, ci_hi = compute_cohens_d_with_ci(
        scores[crisis_mask], scores[~crisis_mask],
        n_bootstrap=1000, seed=42
    )
    results[crisis_key] = {'d': float(d), 'ci_lo': float(ci_lo), 'ci_hi': float(ci_hi)}

# Verdict
d_values = [r['d'] for r in results.values()]
median_d = float(np.median(d_values))
max_d = float(np.max(d_values))
passes = max_d > KEEP_THRESHOLD  # keep if ANY crisis shows signal

# Save
output = {
    'detector': detector.name,
    'crises_tested': SMOKE_CRISES,
    'results': results,
    'median_d': median_d,
    'max_d': max_d,
    'passes_threshold': passes,
}
Path('smoke_results.json').write_text(json.dumps(output, indent=2))

Question Themes

The 100 questions span 10 themes:

1. Novel Observables (Q1-Q15): New quantum/geometric observables
2. Embedding Architecture (Q16-Q25): Hilbert space construction
3. Dead Signal Resurrection (Q26-Q35): Fix low-performing observables
4. Fusion & Combination (Q36-Q45): Multi-observable combination methods
5. Statistical & Evaluation (Q46-Q55): Evaluation methodology
6. Cross-Asset & Data (Q56-Q65): New asset classes and data sources
7. Theoretical Foundations (Q66-Q75): Mathematical rigor
8. Practical Applications (Q76-Q85): Real-world utility
9. Competitive Landscape (Q86-Q95): Comparison with alternatives
10. Wild Cards (Q96-Q100): Unconventional ideas

Existing Results Context

Current top performers (from Paper 1, 36 methods x 17 crises):

• Reduced Purity: d=0.834 (#1)
• Hamilton MS: d=0.713 (#2)
• CUSUM: d=0.625 (#3)
• Berry Phase Rate: d=0.608 (#4)
• Spectral Entropy: d=0.530

Dead signals (d < 0.05):

• QGT Phase Rigidity: d=0.013
• Berry Velocity Coupling: d=0.012
• Curvature Rate: d=0.013

Key insight: QCML methods have |rho|=0.132 vs baselines (highly orthogonal), making fusion valuable.

Anti-Overfitting Rules

1. No future data: All detectors use expanding windows
2. No crisis-label leakage: Scores computed without knowledge of crisis dates
3. Smoke test only: Quick subset results are directional, not final
4. Walk-forward awareness: Promising signals flagged for walk-forward validation
5. Orthogonality: New signals must show |rho| < 0.7 with existing top signals
6. Complexity penalty: Prefer simpler detectors at equal d