Pointline Data Lake — Architecture & Design

Full rationale with trade-off analysis: references/design-rationale.md

Module Dependency Graph

schemas (leaf)
   ↑
protocols ← depends on schemas for type hints
   ↑
   ├── ingestion ← schemas, protocols, dim_symbol
   ├── storage   ← schemas, protocols
   └── research  ← schemas, dim_symbol

vendors ← schemas only (no other internal deps)

Rules for extending:

• schemas/ has zero internal dependencies. Keep it leaf.
• vendors/ depends only on schemas/. Never import from ingestion/, storage/, or research/.
• research/ never imports from ingestion/ or storage/ (except via protocols).
• New modules must fit the DAG. Circular dependencies = design smell.

Change Risk Assessment

Before any PR, classify the risk level:

L2 triggers — any change to:

• pointline/schemas/ (table definitions)
• pointline/dim_symbol.py (SCD2 logic)
• pointline/storage/contracts.py (storage protocols)
• Tie-break keys or partition_by in any spec
• PIT join semantics in ingestion/pit.py or research/primitives.py
• Encoding rules (PRICE_SCALE, QTY_SCALE, timestamp format)

Design Review Checklist

Apply when reviewing any L1+ change:

• Deterministic? Same inputs → same outputs? No wall-clock, no random?
• Idempotent? Re-running safe? No duplicate rows?
• No lookahead? Symbol resolution via as-of join? Forward-only alignment?
• ts_local_us preserved? Replay fidelity maintained?
• As-of joins, not exact? Validity windows, not equality?
• Fixed-point intact? No decode mid-pipeline? Decode only at research output?
• Quarantine, not drop? Invalid rows captured with reason?
• Tie-break keys correct? Full deterministic ordering?
• Module boundaries respected? No dependency cycle introduced?
• Schema-as-code? Spec read before touching table definitions?

Planning & Proposal Standards

ExecPlans (for features/refactors) — follow .agent/PLANS.md:

• Self-contained: a novice can implement end-to-end
• Living document with Progress, Surprises, Decision Log, Outcomes
• Concrete steps with exact file paths and expected output
• Validation and acceptance criteria
• Idempotence and recovery plan

Research Proposals — follow .agent/PROPOSALS.md:

• Testable hypothesis in plain language
• Data sources with PIT constraints
• Experiment design: splits, baselines, ablations
• Evaluation metrics with pass/fail thresholds
• 1-5 clarifying questions (decision-oriented, with defaults)

Full planning reference: references/planning.md

Design Patterns for Extension

Adding a New Event Table

1. Ask: Does it belong in the existing schema hierarchy? Could it be a column on an existing table instead?
2. Define spec in pointline/schemas/ — set columns, tie-break keys, partition_by, scaled columns
3. Determine validation rules — what constitutes an invalid row?
4. Ensure PIT coverage — does the new table need dim_symbol resolution?
5. Update pipeline — add table alias, validation dispatch, any canonicalization
6. Write research API — does load_events() need special handling?

Adding a New Vendor

1. Map vendor schema to canonical pointline tables (which existing tables? new tables needed?)
2. Create parser module at pointline/vendors/<vendor>/ — depends only on schemas
3. Handle exchange mapping — add to EXCHANGE_TIMEZONE_MAP if new exchanges
4. Plan canonicalization — how do vendor column names/encodings map to canonical form?
5. Define Bronze layout — archive format, file naming convention, extraction pipeline
6. Test with real data — sample files in tests/fixtures/

Adding a New Dimension

1. Evaluate SCD type — Type 1 (overwrite) vs Type 2 (versioned) vs Type 3 (limited history)
2. Define natural key and tracked columns (which changes trigger new version?)
3. Design validity semantics — valid_from_ts_us <= event_ts < valid_until_ts_us?
4. Plan bootstrap path — initial load from vendor snapshot
5. Plan incremental path — upsert logic, delisting/removal handling
6. Add storage contract — new protocol or extend DimensionStore?

Modifying Schema Encoding

This is always L2. Schema changes = rebuild. Checklist:

1. Is the change necessary or can the goal be achieved differently?
2. What's the rebuild cost? (time, compute, Bronze still available?)
3. Does the change affect downstream consumers (research API, feature pipelines)?
4. Update spec, rebuild, validate, update research API

Full extension patterns: references/extension-patterns.md

Data Modeling Decisions

Why Int64 Microseconds for Timestamps

• Microsecond resolution matches exchange feed precision
• Int64 arithmetic is fast and exact (no float rounding)
• UTC avoids timezone ambiguity in storage
• Exchange-local conversion only for trading_date derivation

Why Fixed-Point 10^9 Scale

• 9 decimal places covers all exchange price/quantity precision
• Single scale factor simplifies code (no per-column scale lookup at runtime)
• Int64 range: ±9.2 × 10^18 → max representable value ~9.2 × 10^9 in decoded units
• Sufficient for crypto (BTC ~$100k = 100,000 × 10^9 fits) and equities

Why (exchange, trading_date) Partitioning

• Query pattern: researchers load single exchange + date range
• Partition pruning eliminates 99%+ of data on typical queries
• One partition per exchange-day keeps file counts manageable
• trading_date derived from exchange-local time (not UTC date) matches researcher mental model

Why Composite Tie-Break Keys

• Exchange timestamps have ties (multiple events at same microsecond)
• file_id + file_seq provide total ordering within ties
• Deterministic sort = reproducible research results
• Different tables need different keys (orderbook adds book_seq)

Non-Goals (Explicit Boundaries)

• Not real-time. Batch ingestion only. No streaming, no pub/sub.
• Not multi-user. Single-machine, single-writer. No concurrent write conflict resolution beyond OCC on dim_symbol.
• Not distributed. No Spark, no cluster. Polars on one machine.
• Not a trading system. Research only. No order management, no execution.
• Not backward compatible. Schema change = rebuild from Bronze. Explicitly chosen trade-off.