Data Quality Auditor

Use when a specific column, metric, or pipeline stage is suspected.

1. Ask: What broke, when did it start, and what changed upstream?
2. Run the relevant script against the suspect columns only
3. Compare distributions against a known-good baseline if available
4. Trace issues to root cause (source system, ETL transform, ingestion lag)

Mode 3 — Ongoing Monitoring Setup

Use when the user wants recurring quality checks on a live pipeline.

1. Identify the 5–8 critical columns driving key metrics
2. Define thresholds: acceptable null %, outlier rate, value domain
3. Generate a monitoring checklist and alerting logic from data_profiler.py --monitor
4. Schedule checks at ingestion cadence

Tools

scripts/data_profiler.py

Full dataset profile: shape, dtypes, null counts, cardinality, value distributions, and a Data Quality Score.

Features:

• Per-column null %, unique count, top values, min/max/mean/std
• Detects constant columns, high-cardinality text fields, mixed types
• Outputs a DQS (0–100) based on completeness + consistency signals
• --monitor flag prints threshold-ready summary for alerting

# Profile from CSV
python3 scripts/data_profiler.py --file data.csv

# Profile specific columns
python3 scripts/data_profiler.py --file data.csv --columns col1,col2,col3

# Output JSON for downstream use
python3 scripts/data_profiler.py --file data.csv --format json

# Generate monitoring thresholds
python3 scripts/data_profiler.py --file data.csv --monitor

scripts/missing_value_analyzer.py

Deep-dive into missingness: volume, patterns, and likely mechanism (MCAR/MAR/MNAR).

Features:

• Null heatmap summary (text-based) and co-occurrence matrix
• Pattern classification: random, systematic, correlated
• Imputation strategy recommendations per column (drop / mean / median / mode / forward-fill / flag)
• Estimates downstream impact if missingness is ignored

# Analyze all missing values
python3 scripts/missing_value_analyzer.py --file data.csv

# Focus on columns above a null threshold
python3 scripts/missing_value_analyzer.py --file data.csv --threshold 0.05

# Output JSON
python3 scripts/missing_value_analyzer.py --file data.csv --format json

scripts/outlier_detector.py

Multi-method outlier detection with business-impact context.

Features:

• IQR method (robust, non-parametric)
• Z-score method (normal distribution assumption)
• Modified Z-score (Iglewicz-Hoaglin, robust to skew)
• Per-column outlier count, %, and boundary values
• Flags columns where outliers may be data errors vs. legitimate extremes

# Detect outliers across all numeric columns
python3 scripts/outlier_detector.py --file data.csv

# Use specific method
python3 scripts/outlier_detector.py --file data.csv --method iqr

# Set custom Z-score threshold
python3 scripts/outlier_detector.py --file data.csv --method zscore --threshold 2.5

# Output JSON
python3 scripts/outlier_detector.py --file data.csv --format json

Data Quality Score (DQS)

The DQS is a 0–100 composite score across five dimensions. Report it at the top of every audit.

Scoring thresholds:

• 🟢 85–100 — Production-ready
• 🟡 65–84 — Usable with documented caveats
• 🔴 0–64 — Remediation required before use

Proactive Risk Triggers

Surface these unprompted whenever you spot the signals:

• Silent nulls — Nulls encoded as 0, "", "N/A", "null" strings. Completeness metrics lie until these are caught.
• Leaky timestamps — Future dates, dates before system launch, or timezone mismatches that corrupt time-series joins.
• Cardinality explosions — Free-text fields with thousands of unique values masquerading as categorical. Will break one-hot encoding silently.
• Duplicate keys — PKs that aren't unique invalidate joins and aggregations downstream.
• Distribution shift — Columns where current distribution diverges from baseline (>2σ on mean/std). Signals upstream pipeline changes.
• Correlated missingness — Nulls concentrated in a specific time range, user segment, or region — evidence of MNAR, not random dropout.

Output Artifacts

Remediation Playbook

Missing Values

Outliers

• Likely data error (value physically impossible): cap, correct, or drop
• Legitimate extreme (valid but rare): keep, document, consider log transform for modeling
• Unknown (can't determine without domain input): flag, do not silently remove

Duplicates

1. Confirm uniqueness key with data owner before deduplication
2. Prefer keep='last' for event data (most recent state wins)
3. Prefer keep='first' for slowly-changing-dimension tables

Quality Loop

Tag every finding with a confidence level:

• 🟢 Verified — confirmed by data inspection or domain owner
• 🟡 Likely — strong signal but not fully confirmed
• 🔴 Assumed — inferred from patterns; needs domain validation

Never auto-remediate 🔴 findings without human confirmation.

Communication Standard

Structure all audit reports as:

Bottom Line — DQS score and one-sentence verdict (e.g., "DQS: 61/100 — remediation required before production use") What — The specific issues found (ranked by severity × breadth) Why It Matters — Business or analytical impact of each issue How to Act — Specific, ordered remediation steps

Related Skills

When NOT to use this skill:

• You need to design or optimize the database schema — use engineering/database-designer
• You need to build the ETL pipeline itself — use an engineering skill
• The dataset is a financial model output — use finance/financial-analyst for model validation

References

• references/data-quality-concepts.md — MCAR/MAR/MNAR theory, DQS methodology, outlier detection methods