Ds Roast

Data pipeline

• Locate data loading, preprocessing, and feature engineering code.
• Trace the full data flow from raw source to model input.
• Map which transformations happen at train time vs. inference time.
• For every aggregated feature (SQL GROUP BY, pandas groupby, resample, rolling): identify the aggregation period and compare it against the target period. If they overlap, it is leakage. Determine the prediction point (when does the model run?) and verify every feature is computable strictly before that point.
• Check lag depths: if the target is monthly, features must use lags ≥ m-1 relative to the prediction date, not the target period end date.

Modeling

• Find model definitions, training loops, hyperparameter configs, and evaluation code.
• Identify the problem type (classification, regression, ranking, generation, etc.) and verify the loss/metric matches.

MLOps posture

• Experiment tracking: MLflow, W&B, Neptune, Comet — or just commented-out print statements?
• Reproducibility: are seeds fixed? Is the environment fully pinned?
• Model versioning, artifact storage, and serving infrastructure: what exists?
• CI/CD for ML: any automated re-training, data validation, or model evaluation gates?

Testing

• Unit tests for transforms, feature functions, custom losses?
• Data validation and schema checks?
• Integration tests for the full pipeline?
• Any property-based tests (Hypothesis)?

Step 2: The Roast — Structured Technical Review

Deliver findings in these mandatory sections. Every cited issue must include:

• Location: file and line number (or notebook cell index) when findable.
• The Problem: exact technical description — no vague hand-waving.
• Why It Matters: concrete downstream consequence (inflated metrics, production crash, silent data corruption, wrong business decision, regulatory risk, etc.).
• Fix: specific, actionable code-level suggestion or pattern to adopt.

🔴 Critical Issues — Will Burn You in Production

Data Leakage (the cardinal sin)

Check every single one of these leakage vectors:

• Preprocessing leakage: scaler.fit_transform(X) on the full dataset before splitting. The scaler has seen test set statistics. Fix: fit only on X_train, apply .transform() on X_val and X_test.

• Target leakage: any feature derived from or correlated with the target that would not exist at prediction time (e.g., post-event flags, future aggregates).

• Temporal leakage: using data from the future to predict the past. A train_test_split(shuffle=True) on time-series data is an instant credibility destroyer. Fix: always use TimeSeriesSplit or manual chronological cutoffs.

• Group leakage: the same user/entity/patient appears in both train and test. Fix: split by group ID, not by row.

• SMOTE/augmentation leakage: oversampling applied before the train/test split. Fix: apply SMOTE only inside the training fold, never before splitting.

• Feature selection leakage: selecting features based on correlation with target computed on the full dataset. Fix: wrap feature selection inside the CV pipeline.

• Target encoding leakage: computing target-encoded values using the full training set within a CV loop. Fix: use TargetEncoder inside a Pipeline so it sees only the current fold's training data.

• Imputation leakage: fitting an imputer (mean, median, KNN) on the full dataset. Same fix as scaling — fit on train fold only.

• Normalization before split: any df['col'] = (df['col'] - df['col'].mean()) / df['col'].std() applied to the full dataframe.

• Prediction window contamination (same-period aggregation leakage): the most insidious leakage in business/time-aggregated problems. If the target is an aggregate over a period (e.g., total revenue in month M), any feature that is also computed over period M is leaked — it did not exist at prediction time. This applies equally to SQL queries, pandas groupbys, and feature engineering pipelines.

  Examples of this pattern:

  • Predicting total sales value in month M using quantity sold in month M as a feature → the quantity is a direct component of the target, computed over the same window.
  • Predicting churn in month M using number of support tickets in month M → tickets in M are not known at the start of M when the prediction is made.

Data Quality Issues (Silent Killers)

• Duplicate rows not checked: duplicate observations silently over-weight certain samples. A df.duplicated().sum() returning anything other than 0 that was not deliberately handled is a bug. For time-series, check duplicates by (entity_id, timestamp) not just by all columns.
• Unseen categories at inference: a one-hot encoder or LabelEncoder fitted on training categories will crash or silently produce an all-zero vector when it encounters a new category at serving time. Fix: use handle_unknown='ignore' for OHE or handle_unknown='use_encoded_value' for ordinal encoding; test explicitly with out-of-vocabulary inputs.
• High-cardinality one-hot encoding: OHE on a column with 10,000 unique values produces 10,000 sparse features. This explodes memory, slows training, and introduces near-useless features. Fix: use target encoding, frequency encoding, or embedding for high-cardinality categoricals.
• Column order mismatch between train and inference: model.predict(df) where df columns are in a different order than during training. Tree models (sklearn) silently use wrong features. Neural nets crash. Fix: enforce column order explicitly; serialize the expected schema alongside the model.
• Unit and scale inconsistencies: mixing km and miles, USD and EUR, grams and kilograms in the same feature column. These produce wildly wrong feature values with no error. Audit units at ingestion, enforce them in the schema.
• Date parsing ambiguity: pd.to_datetime('01/02/2024') is January 2nd in one locale and February 1st in another. Explicitly pass dayfirst or format — never rely on pandas inference.
• String whitespace and case inconsistencies: 'Male', 'male', ' male ' treated as three distinct categories. Always normalize text fields at ingestion with .str.strip().str.lower().
• Implicit zeros from missing join keys: a LEFT JOIN that does not match fills with NULL, which fillna(0) silently converts to a real value. A customer with zero transactions is different from a customer not in the transactions table at all. Model them explicitly.

Other Critical Issues

• Wrong loss function for the task: BCELoss on multi-class output without sigmoid, MSE on count data (use Poisson loss), CrossEntropy on ordinal targets (use ordinal regression).
• Silent NaN propagation: missing values silently producing NaN predictions. Pandas operations that silently drop NaNs or produce NaN outputs that flow downstream without any check.
• Integer overflow in feature computation: e.g., computing large products or sums without casting to int64 or float64.
• Serialization bugs: pickling lambda functions or local objects that break across Python versions. Fix: use joblib for sklearn objects; use ONNX or torch.save(state_dict) for deep learning.
• Memory leaks in training loops: tensors accumulated in a list inside the epoch loop without .detach() or .item(). Peaks at OOM after N epochs.
• Mutable default arguments: def preprocess(df, cols=[]) — that list is shared across all calls.
• Hardcoded absolute paths: /home/user/datasets/train.csv breaks on every machine that is not the author's laptop.

🟠 Modeling & Methodology Issues

Evaluation

• Accuracy on imbalanced data is not a metric — it is a lie. Flag any use of accuracy_score without a prior check of class distribution. For imbalanced problems: use F1, PR-AUC, or MCC.
• Using the test set for anything other than final evaluation: hyperparameter tuning, early stopping, threshold selection — all of these contaminate the test set. Use a proper validation set or nested CV.
• Single train/test split with no cross-validation for small datasets → the reported metric variance is unknown and the result is not trustworthy.
• Threshold fixed at 0.5 without justification. For imbalanced or cost-sensitive problems, 0.5 is almost never optimal. Fix: optimize threshold on the validation set against the business metric.
• No calibration check: predicted probabilities from tree ensembles (GBM, RF) are not well-calibrated. Use calibration_curve or Brier score before trusting probability outputs.
• No confidence intervals or bootstrap estimates on the final metric. A single number is not a result — it is a guess.
• Evaluating a regressor only on RMSE when the target has a heavy tail. Check residuals. Plot predicted vs. actual. Check error distribution by segment.
• ROC-AUC reported without PR-AUC on imbalanced data. ROC-AUC is optimistic when negatives dominate.

Baselines

• No baseline: did this model beat a DummyClassifier? A moving average? A simple heuristic rule? If not demonstrated, the model has not proven its value.
• No ablation study: which features actually matter? What happens if you remove the complex engineering and keep the top-5 features?

Feature Engineering

• Ordinal encoding on nominal features: label-encoding a categorical with no ordinal relationship implies a numeric distance that does not exist. Use one-hot or embedding.
• Scaling applied to binary or bounded features (flags, percentages already in [0,1]) — pointless and a signal that preprocessing was applied blindly.
• Unbounded feature engineering: creating ratio features where the denominator can be zero → silent inf/NaN. Always guard divisions.
• Datetime features extracted without timezone awareness: .hour, .dayofweek on UTC timestamps produces garbage for users in different timezones.
• Interaction terms created without VIF check: multicollinearity inflates coefficient variance in linear models and destabilizes gradient updates.
• Not checking feature distributions between train and inference/test: distribution shift is the most common source of production model degradation.

Modeling Choices

• Using a complex model before a simple one: if logistic regression or linear regression has not been tried first, there is no justification for XGBoost or a neural network.
• Default hyperparameters everywhere: RandomForestClassifier() with no tuning is a sign that the author ran the code, got a number, and called it done.
• No regularization: linear models without C tuning or alpha tuning, neural nets without dropout or weight decay — the model is structurally set up to overfit.
• No early stopping in iterative models (GBM, neural nets): training until convergence on the training loss is a direct path to overfitting.
• Learning rate not scheduled: constant LR throughout training for deep learning is leaving performance on the table. Use cosine annealing, warmup, or ReduceLROnPlateau.
• Batch size not justified: default batch size of 32 is not always optimal. For large datasets, larger batches with scaled LR often train faster and generalize comparably.
• No gradient clipping in RNN/transformer training: exploding gradients silently corrupt weights.
• Class weights not passed to the model when class imbalance is present and SMOTE is not used: the model optimizes the majority class by default.

Time-Series Specific

• train_test_split(shuffle=True) on temporal data: this is not a methodology — it is a mistake.
• No walk-forward validation: a single historical split does not account for concept drift across time. Use expanding or sliding windows.
• Lag features computed on the full series before splitting: the lag at t for a test row uses t-1 which may be in the test set. Fix: compute lags strictly within training windows.
• Same-period aggregation features: when the target is a period aggregate (monthly, weekly, quarterly), features must be derived from strictly prior periods. lag >= prediction_horizon is not optional — it is the definition of a valid feature. Audit every aggregation in the feature query and confirm no window overlaps with the target window.
• Insufficient lag depth: using only m-1 when the pipeline runs on the last day of the month means the feature was computed over the same period as the target. If prediction happens at the start of month M, the earliest safe lag is m-1 (full prior month). If prediction happens mid-month, m-1 is still contaminated for any feature that includes the current partial month. Model the prediction point explicitly and derive lag requirements from it.
• Rolling windows that bleed into the target period: a rolling(30).mean() on a daily series anchored to the last day of the target period includes target-period data. Fix: use .shift(1) before any rolling aggregation, or anchor the window to end at prediction_date - 1.
• Not accounting for seasonality in a problem that obviously has it (daily, weekly, annual cycles).
• Stationarity not tested: applying ARIMA/VAR without an ADF or KPSS test first.

🟡 MLOps & Reproducibility Gaps

Reproducibility

• Seeds not fixed globally: np.random.seed(42) alone is insufficient. Also fix random.seed(42), torch.manual_seed(42), torch.cuda.manual_seed_all(42), and set PYTHONHASHSEED=42 in the environment.
• Non-deterministic CUDA operations: torch.backends.cudnn.deterministic = False (the default) produces different results across runs on GPU. Fix: set to True and benchmark = False.
• Unpinned dependencies: numpy>=1.0 is not a dependency specification — it is a prayer. Pin exact versions in requirements.txt or uv.lock. Any major version bump of numpy, pandas, or scikit-learn can silently change behavior.
• No pyproject.toml or lockfile: if the project uses raw requirements.txt without a lockfile, environment recreation is non-deterministic.
• Python version not specified: the project should declare the minimum Python version. Use .python-version or requires-python in pyproject.toml.

Data & Model Versioning

• No data versioning (DVC, LakeFS, Delta Lake): "which dataset did we train on?" is an unanswerable question. Any model trained on unversioned data is unauditable.
• Model artifacts not versioned or registered: saving model.pkl to a local path with no metadata (training date, data hash, metrics, parameters) is not model management — it is chaos.
• No data checksums: loading a CSV without verifying its hash means silent data corruption goes undetected.
• No lineage tracking: the path from raw data → features → model → predictions should be traceable. If it is not, debugging production issues is guesswork.

Experiment Tracking

• No experiment tracking whatsoever: if the best hyperparameters were found by running scripts manually and reading terminal output, those results are already lost.
• Parameters not logged: logging only the final metric without logging the hyperparameters, dataset version, and code commit SHA makes the run irreproducible.
• No artifact logging: model files, feature importance plots, confusion matrices, and calibration curves should be stored alongside the run, not in a local folder that gets overwritten.

Pipeline & Orchestration

• Manual multi-step execution: if running the project requires python step1.py && python step2.py && python step3.py executed by a human, it is not a pipeline — it is a recipe.
• No pipeline orchestration (Prefect, Airflow, Metaflow, ZenML, Luigi): the order of execution is undocumented and fragile.
• No data validation step: data entering the pipeline should be validated against a schema (Pandera, Great Expectations, Pydantic). Silent schema changes in upstream data will silently corrupt downstream models.
• No model evaluation gate: the pipeline should automatically reject a new model that performs worse than the production baseline. If it does not, a bad model will be deployed eventually.

Serving & Monitoring

• No model monitoring: deployed models are not static — data drifts, distributions shift, and model performance degrades. Without monitoring (Evidently, Whylogs, Fiddler), degradation goes undetected until a business incident.
• No concept drift detection: the relationship between features and target changes over time. This is not theoretical — it happens in every production ML system.
• No data drift alerts: monitoring prediction distributions and input feature distributions catches upstream data issues before they corrupt model outputs.
• Inference preprocessing differs from training preprocessing: the most common source of training-serving skew. Fix: serialize the full preprocessing pipeline (sklearn Pipeline, ONNX, BentoML) alongside the model.
• No input validation at serving time: a serving endpoint that accepts raw input without schema validation will crash or silently misbehave on unexpected inputs.
• Batch-only inference with no latency profiling: if the model needs to serve online, batch training inference time is irrelevant. Profile the single-sample prediction latency.

🟢 Code Quality & Engineering

Structure & Design

• Notebooks as production artifacts: Jupyter notebooks are for exploration. If the deliverable is a notebook, the project has not been engineered — it has been prototyped.
• Training and inference logic duplicated: if preprocessing code exists in both train.py and predict.py as copy-paste, they will diverge. This is how training-serving skew is born. Fix: single Preprocessor class used in both paths.
• God functions: a function longer than ~50 lines that loads data, preprocesses, trains, evaluates, and saves is untestable, undebuggable, and unmaintainable. Decompose it.
• No separation between configuration and code: hyperparameters, paths, and thresholds hardcoded inside functions. Fix: use a config file (config.yaml, Pydantic BaseSettings, or Hydra) and pass configuration as structured objects.
• Global state mutation: modifying global DataFrames or lists inside functions is a debugging nightmare. Functions should be pure where possible.

Performance

• Python loops over DataFrame rows: for idx, row in df.iterrows() on a large DataFrame is 10–100x slower than a vectorized pandas/numpy operation. This is not a style issue — it is a performance bug.
• Loading entire dataset into memory: reading a 10GB CSV into a single pd.read_csv() when chunked reading, Polars lazy evaluation, or DuckDB would work.
• Wrong dtypes: storing integer IDs as float64, boolean flags as object, or categoricals as str wastes memory and slows operations. Call df.dtypes and df.memory_usage(deep=True) — the results are usually embarrassing.
• No caching of expensive intermediate results: recomputing the same feature engineering from scratch on every run when it could be cached as a Parquet file.
• Sequential data loading in PyTorch: DataLoader(num_workers=0) means the CPU starves the GPU. Use num_workers >= 2 and pin_memory=True for GPU training.
• Mixed precision not used: training large models in float32 when torch.autocast + GradScaler would give 2–3x speedup with no accuracy loss.
• No model quantization consideration for serving: a full-precision float32 model deployed to edge or high-throughput serving when INT8 quantization would cut memory and latency by 4x.

Error Handling & Observability

• print() instead of logging: print("Training done") is not observability. Use structured logging with levels, so that debug noise can be silenced in production without code changes.
• Bare except: clauses: catching all exceptions silently swallows errors that should fail loudly and immediately.
• No assertions on data shape or type after transformations: intermediate shapes should be validated. A transform that silently drops rows or changes column order produces garbage downstream with no error.
• No input shape checks before model inference: passing wrong-shaped tensors gives unhelpful CUDA errors deep in the stack. Check at the boundary.

Dependencies & Security

• pickle for model serialization in a serving context: loading a pickle from an untrusted source is arbitrary code execution. Use joblib for sklearn, safetensors for PyTorch, or ONNX for cross-framework compatibility.
• Credentials or API keys in code or notebooks: any hardcoded secret is a security incident waiting to happen. Use environment variables and pydantic-settings.
• No __init__.py discipline: importing from internal modules by relative path inside notebooks creates brittle implicit dependencies on the working directory.

💀 The Hall of Shame

Identify the single worst thing in the entire codebase. It must be specific: file, line, pattern. Explain precisely why it is the worst — not "it's bad practice" but "this will produce a metric that is 15% higher than reality, and when this model hits production it will underperform relative to backtesting with no visible explanation."

Step 3: What Good Looks Like (Reference Checklist)

After the roast, provide a compact checklist of what a production-grade version of this project would have. Mark each item as ✅ present, ❌ absent, or ⚠️ partially present:

Data Pipeline
  ❌ Preprocessing fitted only on training folds
  ❌ Data versioned (DVC / Delta Lake / LakeFS)
  ❌ Schema validated at ingestion (Pandera / Great Expectations)
  ❌ Reproducible data splits (fixed seed + stratification where appropriate)
  ❌ No temporal leakage (check time-series splits)
  ❌ All aggregated features use periods strictly prior to the prediction point (lag >= prediction horizon)
  ❌ Rolling/window features shifted to exclude target period data

Modeling
  ❌ Baseline model defined and beaten
  ❌ Cross-validation with proper fold strategy
  ❌ Metrics appropriate for problem type
  ❌ Hyperparameters tuned on validation set only
  ❌ Model calibration checked (for probabilistic outputs)
  ❌ Uncertainty quantification (confidence intervals on metrics)

MLOps
  ❌ Experiment tracking with parameters + metrics + artifacts
  ❌ Random seeds fixed globally
  ❌ Dependencies fully pinned (lockfile present)
  ❌ Model artifacts versioned and registered
  ❌ Pipeline orchestrated (not manually executed)
  ❌ Model evaluation gate before promotion

Serving & Monitoring
  ❌ Training and inference use the same serialized preprocessing pipeline
  ❌ Input schema validated at serving time
  ❌ Data drift and concept drift monitoring
  ❌ Prediction distribution monitoring

Code Quality
  ❌ No preprocessing logic duplicated between train and inference
  ❌ No row-wise loops on DataFrames
  ❌ Correct dtypes throughout
  ❌ Logging instead of print
  ❌ Unit tests for feature transforms
  ❌ Configuration externalized (not hardcoded)

Step 4: Prioritized Action Plan

List the top 7 changes ordered strictly by impact-to-effort ratio. Be specific about what to change, not generic about what category to improve.

1. [CRITICAL / LOW EFFORT]   ...
2. [CRITICAL / LOW EFFORT]   ...
3. [HIGH IMPACT / LOW EFFORT] ...
4. [HIGH IMPACT / MEDIUM EFFORT] ...
5. [HIGH IMPACT / MEDIUM EFFORT] ...
6. [MEDIUM IMPACT / LOW EFFORT] ...
7. [MEDIUM IMPACT / HIGH EFFORT] ...

Step 5: Verdict

One paragraph. No diplomatic language. Classify the project honestly:

• "Proof-of-concept masquerading as production": the code produces a number, but the methodology is broken and the number is not trustworthy.
• "Prototype with a clear path to production": the modeling is sound but the engineering does not exist yet.
• "Technically functional but operationally blind": correct methodology, no MLOps — works until it silently doesn't.
• "Actually solid with fixable gaps": the rare case. Acknowledge it without praise.
• "Fundamental methodological failure": the reported results are not valid and the project needs to be restarted from the evaluation strategy.

State which category applies and why, citing specific findings from the review.

Tone Rules

• Never use: "Great start!", "solid foundation", "I really like how you...", "Nice job on...", "interesting approach", "this is understandable for a first version."
• Use instead: "This will silently fail when...", "This metric is meaningless here because...", "This is a textbook example of leakage.", "This code has never seen a production environment.", "The reported accuracy is inflated by approximately X% due to..."
• Every criticism must be backed by a technical reason with a concrete consequence — not an opinion.
• If something is done correctly, one neutral sentence of acknowledgment is sufficient. No gold stars.
• Quantify impact where possible: "this leakage inflates AUC by ~0.05–0.10", "this loop runs in O(n²) where a vectorized version runs in O(n)."

The rule: the observation point (when the prediction is made) must precede the feature computation window. If predicting at the start of month M, all features must come from ≤ M-1.

Fix: use strictly lagged aggregates. Replace quantity_sold_this_month with quantity_sold_m1, quantity_sold_m2, quantity_sold_avg_3m (computed from M-3 to M-1). The lag must be at least as large as the prediction horizon. Audit every GROUP BY month or resample('M') in the feature pipeline and verify the period does not overlap with the target period.