ML Competition — Features & Validation Strategy

OOF accumulation

oof = np.zeros(len(train_df))           # allocate once, full training length
for fold, (tr_idx, va_idx) in enumerate(splits):
    model.fit(X[tr_idx], y[tr_idx])
    oof[va_idx] = model.predict_proba(X[va_idx])[:, 1]  # assign by index, not stack

• Always collect full arrays, never fold-by-fold averages — oof[val] = preds, not oof_list.append(preds.mean())
• OOF arrays are train-rows only — even if combined (train + auxiliary) data drives fold splits, only score oof[va_idx[va_idx < n_train]]

Target encoding and fold-local statistics

# ✅ CORRECT — computed inside fold loop, only on training portion
for fold, (tr_idx, va_idx) in enumerate(splits):
    te = train_df.iloc[tr_idx].groupby("cat_col")["target"].mean()
    train_df.loc[va_idx, "cat_te"] = train_df.loc[va_idx, "cat_col"].map(te)

# ❌ WRONG — computed before split, leaks validation targets into training features
train_df["cat_te"] = train_df.groupby("cat_col")["target"].transform("mean")

Tuner vs final training folds

• Tuner folds: 3-fold for speed
• Final training: 5-fold × 5 seeds
• Both must use identical CV strategy and same group column

Leakage checklist — before finalizing any feature

1. Is this feature computed using future information relative to the prediction time?
2. Does this feature use any information from the validation/test fold?
3. Is this a fold-local statistic computed after the split or before it?
4. Does OOF score dramatically exceed LB? (>+0.01 in AUC = suspect leakage)

Feature Cache Discipline

FEAT_CACHE = cfg.feat_cache   # e.g. "cache/features_v3.pkl"

def load_or_build_features(df):
    if os.path.exists(FEAT_CACHE):
        return pd.read_pickle(FEAT_CACHE)
    feats = engineer_features(df)
    feats.to_pickle(FEAT_CACHE)
    return feats

Hard rules:

• Bump the version number in config.yaml (feat_cache: cache/features_v3.pkl) on every change to engineer_features()
• Never patch a cached file in-place — delete old cache or bump version
• build_model_matrices() is called once per process and cached in a module-level variable; do not call it inside the fold loop

Reference Files

See Also