Databricks SA Knowledge Base

Architecture Patterns

The 5-Layer Template (Use Every Time)

SOURCES → INGESTION → MEDALLION (Bronze/Silver/Gold) → GOVERNANCE → CONSUMPTION

For each layer, always state:

1. What — technology/pattern
2. Why — the trade-off rationale
3. What could go wrong — failure mode + mitigation

Medallion Layer Guidance

Bronze

• Append-only Delta tables (no deletes)
• Raw schema — store exactly what came from the source
• Audit trail: _source_system, _ingestion_ts, _file_path metadata columns
• No PII masking (raw fidelity for regulatory audit)
• Liquid Cluster on (ingestion_date, source_system) for time-windowed queries

Silver

• Data integration layer — one record per business entity
• SCD Type 2 for history: use AUTO CDC with STORED AS SCD TYPE 2 TRACK HISTORY ON
• Data Vault for complex multi-source integration: Hubs (business keys), Links (relationships), Satellites (context + history)
• Apply column masks for PII at this layer (access Unity Catalog row filters)
• Liquid Cluster on business keys (customer_id, account_id)
• Validate data quality rules here — reject to quarantine tables, don't block Bronze

Gold

• Consumption-optimized: star schemas (Kimball) or OBTs for specific use cases
• Materialized Views for heavy aggregations refreshed on schedule
• Liquid Cluster on most-filtered fact columns (e.g., (transaction_date, product_id))
• Photon-optimized for all joins (up to 18x on star joins)
• Separate logical clusters by consumer type: BI, risk, regulatory, API

Star Schema on Databricks

• Fact tables: Liquid Clustered on date + FK dimensions most filtered
• Dimension tables: small dims → broadcast join; large dims → Liquid Cluster on PK
• Photon delivers up to 18x for star schema workloads specifically
• Recommended pattern: dimensional marts in Gold, Data Vault raw vault in Silver

Data Vault on Databricks

Maps to Medallion:

• Bronze → Staging Area (raw ingest)
• Silver → Raw Vault (Hubs, Links, Satellites with _LoadDate, _RecordSource)
• Silver/Gold boundary → Business Vault (derived satellite calculations)
• Gold → Data Marts (Kimball star schemas for consumption)
• Identity columns for surrogate key generation
• PIT (Point-in-Time) and Bridge tables in Gold to pre-join satellites

Migration Framework (Teradata/Netezza → Databricks)

Steve's IBM background is a key differentiator here. Walk-through:

1. Assess — Lakebridge Analyzer scans legacy environment

   • Classifies tables, views, ETL, stored procedures by complexity
   • Produces automated report in hours, not months

2. Strategy — Three migration approaches:

   • Federation-First (lowest risk): Lakehouse Federation queries legacy DW in place; migrate incrementally domain by domain
   • Parallel Run: Both systems live; Lakebridge reconciler validates parity row-by-row
   • Medallion Refactor: Re-architect into Bronze/Silver/Gold during migration (highest value, highest effort)

3. Convert — Lakebridge transpilers:

   • BladeBridge (mature, Teradata BTEQ)
   • Morpheus (next-gen, T-SQL, PL/SQL)
   • Switch (LLM-powered, experimental)
   • Automates up to 80% of SQL conversion

4. Optimize — Liquid Clustering + Predictive Optimization + Photon

   • Reported results: 2.7x faster, 12x cost efficiency post-migration

5. Govern — Unity Catalog replaces legacy DW security model

Steve's personal angle: "At IBM I spent 8 years helping customers manage Netezza environments. I've seen the pain of proprietary platforms firsthand — the scaling limitations, the vendor lock-in, the forklift upgrade cycles. That's exactly why the open lakehouse approach resonates with me deeply."

Competitive Intelligence

Databricks vs Snowflake

Diplomatic framing (never say Snowflake is bad):

"Both platforms have strengths. Databricks differentiates on the unified platform story — data engineering, DW, and ML on one platform with open formats. For a customer doing more than just BI, the TCO advantage is significant because you eliminate the integration tax between separate systems."

Databricks vs Synapse / Fabric

"Azure Synapse was a warehouse bolted onto Spark as an afterthought. Databricks was built Spark-native from day one — every feature decision is optimized for the lakehouse pattern. Microsoft Fabric is newer and interesting but less mature. For a customer with complex data engineering + ML workloads alongside BI, Databricks is the proven platform."

Reference Files

Load these on demand for deeper detail:

Scenario Files

Practice scenarios with pre-populated discovery answers:

Scripts