Data Quality

Intro

Data quality is measurable across six dimensions and enforced with explicit validation rules at every pipeline boundary. Codify producer and consumer expectations as data contracts and run them as automated checks rather than relying on manual review.

Overview

The six dimensions

Evaluate data against:

• Completeness — null rate per column, row count vs expected
• Accuracy — spot checks against a source of truth or reference
• Consistency — cross-field validation (end >= start, city matches zip)
• Timeliness — freshness since last update, SLA adherence
• Uniqueness — distinct count of natural keys vs total rows
• Validity — regex matches, range checks, enum membership

Validation rules

Schema validation enforces column names, types, and nullability. Range checks bound numerics (age 0–150, price >= 0). Format checks enforce ISO 8601 dates, email regexes, normalized phone numbers. Referential integrity confirms foreign keys resolve. Business rules express domain truths (order total = sum of line items). Freshness checks confirm the latest record is within the expected window. Categorize each rule by