Od Expert

from pyod.utils.ad_engine import ADEngine
engine = ADEngine()
state = engine.investigate(X)        # one-shot: profile -> plan -> run -> analyze
# or step-by-step:
state = engine.start(X)              # profile data
state = engine.plan(state)           # select top-N detectors
state = engine.run(state)            # execute, compute consensus
state = engine.analyze(state)        # quality assessment, best detector
state = engine.iterate(state, fb)    # iterate based on feedback
report = engine.report(state)        # final report

state.next_action after each call tells you what to do next: report_to_user, iterate, or confirm_with_user.

For knowledge-only queries (no execution), the legacy methods engine.profile_data, engine.list_detectors, engine.explain_detector, engine.compare_detectors, engine.get_benchmarks all still work.

Master decision tree

When the user provides data, walk this tree to pick the starting detector(s) before calling engine.start. The tree is your default; ADEngine's planner may refine it, but knowing the right starting point reduces wasted iterations.

Is the data sequential (timestamps, ordered events)?
├── Yes → time series. See references/time_series.md.
│         Default starters: `TimeSeriesOD` bridge over `ECOD`,
│         `MatrixProfile`, `SpectralResidual`.
└── No  → Is the data a graph (nodes + edges)?
          ├── Yes → graph. See references/graph.md.
          │         Default starters: `DOMINANT`, `CoLA`, `AnomalyDAE`.
          │         Requires: pip install pyod[graph]
          └── No  → Is the data text or image?
                    ├── Yes → embedding. See references/text_image.md.
                    │         Default: `EmbeddingOD` with sentence-transformers
                    │         (text) or HuggingFace ViT (image), wrapped over
                    │         `LOF` / `KNN`.
                    └── No  → tabular. See references/tabular.md.
                              Default starters by row count and contamination:
                              - n < 1k:           `ECOD` or `HBOS`
                              - 1k ≤ n ≤ 100k:    `IForest` + `ECOD` + `LOF`
                              - n > 100k:         `IForest` + `HBOS`
                              - high-D (D > 50):  `COPOD` or `SUOD`

If the data has multiple modalities (e.g., tabular + text columns), see Trigger 9 in the escalation section below.

Top-10 critical pitfalls

These are pitfalls that silently produce wrong results if ignored. The agent must check for each on every session before reporting.

1. Unscaled features for distance-based detectors. LOF, KNN, OCSVM, CBLOF require scaled features. If engine.profile_data reports any feature with std > 10 or range > 100, scale (StandardScaler or RobustScaler) before running. The default engine.start flow does NOT auto-scale.
2. Contamination assumed instead of estimated. The default contamination is 0.1, but real datasets vary widely. ADEngine's profiler does NOT estimate contamination. After engine.run + engine.analyze, check state.analysis['consensus_analysis']['anomaly_ratio'] — if that ratio is far from the domain's true rate, re-plan with an explicit contamination via engine.iterate(state, {"action": "adjust_contamination", "value": <rate>}). A contamination mismatch silently shifts every threshold.
3. Deep learning detector on tiny data. Do not run AutoEncoder, VAE, DeepSVDD, or AnoGAN on datasets with fewer than 1000 rows. They overfit immediately. Trigger 6 (escalation) catches this; recommend ECOD / IForest / HBOS instead.
4. Graph detector without PyG installed. DOMINANT, CoLA, CONAD, AnomalyDAE, GUIDE, Radar, ANOMALOUS require pyod[graph]. Check with importlib.util.find_spec("torch_geometric") before recommending. Trigger 7 catches this.
5. Mixing categorical and numerical without encoding. PyOD detectors expect numeric input. Categorical columns must be one-hot or label encoded first. engine.profile_data will fail or produce nonsense if string columns are present.
6. Ignoring state.quality.separation. Separation < 0.1 means the consensus is essentially noise. Do NOT report "found anomalies" with high confidence in that case. Trigger 4 catches this.
7. Single-detector runs. Never report from a single detector. Always run the top-3 from engine.plan and use consensus. The exception is when the user explicitly requested a specific detector via the detectors= argument.
8. Time series treated as tabular. If the data has a timestamp column AND row order matters, it is time series, not tabular. Tabular detectors will report most boundary points as anomalies. Trigger 1 catches modality ambiguity.
9. Reporting raw scores instead of percentiles or labels. Raw decision_function scores are not interpretable across detectors. Always report decision_scores_ ranks, percentiles, or labels_ (binary). The result interpretation patterns in references/workflow.md show the right phrasings.
10. Missing the requires-extra check. Some detectors require optional extras (pyod[xgboost] for XGBOD, pyod[suod] for SUOD, pyod[combo] for FeatureBagging). Check engine.explain_detector(name) before recommending; if the extra is missing, suggest the install command and pick a substitute.

Adaptive escalation triggers

Run autonomously by default. Pause and ask the user only when one of these triggers fires. Full detail with example phrasings in references/workflow.md.

1. Modality ambiguity — data has timestamps but also feature columns
2. Contamination uncertainty — heuristic range > 5x (e.g., 1%-25%)
3. Detector disagreement — state.quality.agreement < 0.4 after running
4. Quality assessment weak — state.quality.separation < 0.1 OR state.quality.stability < 0.5
5. Labels mentioned but not provided — user said "I have known fraud cases" but did not pass labels
6. Heavy detector + small data — DL detector requested, n < 1000
7. Missing optional extra — graph requested but pyod[graph] not installed
8. High-stakes domain hint — medical, fraud, security, safety mentioned
9. Cross-modality ambiguity — mixed tabular + text columns
10. Result feels too confident — > 90% detector agreement (suspiciously clean)
11. Iteration loop deadlock — 2 rounds of engine.iterate with no improvement

If none of these triggers fire, proceed to engine.report without asking.

References for depth

Load these on demand based on the modality and phase:

• references/workflow.md — autonomous loop pattern, full escalation triggers with phrasing, cardio canonical worked example, result interpretation patterns
• references/pitfalls.md — 20 more pitfalls beyond the top-10, by phase, severity-tagged
• references/tabular.md — decision table, top detectors, worked snippets, tabular-specific pitfalls
• references/time_series.md — same structure for time series
• references/graph.md — same structure for graph (includes PyG install detection)
• references/text_image.md — EmbeddingOD-based detection for text and image

Always cite your reasoning

When you report a result, include a short "what I assumed and why" section. The user is non-expert; they need to know what decisions you made on their behalf so they can sanity check or correct if needed. Format::

**What I assumed**:
- Data type: <type> (auto-detected from <heuristic>)
- Contamination: <value> (<source: estimated / domain-supplied / default>)
- Detectors: <list> (selected by <reason>)
- Primary detector: <name> (chosen because <metric>)

If any of these assumptions look wrong to the user, they say so and we iterate. Without this section, the user has no way to sanity check the agent's choices.