Australian Bank CDR API Expert

Where configuration lives

See references/cdr-config-and-coverage.md for schema, gap reasons, and a no-gaps checklist. See references/all-banks-and-rba.md for the full bank list, CDR protocol references, and RBA data requirements.

Workflow

1. Adding or fixing a bank

   • Ensure lenders.json has correct products_endpoint (and additional_products_endpoints if the bank exposes multiple hosts). Use CDR register URLs or known-good public product URLs (/cds-au/v1/banking/products).
   • Set canonical_bank_name and register_brand_name to match CDR register branding so discovery fallback (when config is missing) resolves the right brand. Check discovery.ts brand matching and extractBrands / selectBestMatchingBrand.
   • Add or tune the playbook in lender-playbooks.ts: cdrVersions (try [3,4,5,6,2,1]), minRatePercent/maxRatePercent, dailyMinConfidence/historicalMinConfidence, includeKeywords/excludeKeywords so desired products are not dropped. Avoid over-narrow keywords that cause zero accepted rows.

2. Avoiding datafeed gaps

   • Do not truncate product lists or page counts: see docs/hard-limits-registry.json (e.g. api.ingest.max_products_per_lender_clamp, api.ingest.cdr_page_limit_runtime). Pagination must continue until exhaustion or resume from saved state.
   • Ensure discovery has a fallback: either a configured products_endpoint in lenders.json or a register match. Use configuredProductEndpoints and candidateProductEndpoints in product-endpoints.ts so the pipeline tries configured then discovered endpoints.
   • For savings/term deposits, same endpoints serve product index; product category filtering is in cdr-savings.ts (isSavingsAccount, isTermDeposit). Ensure no over-filtering by category/name that would drop valid products.

3. Diagnosing gaps

   • Coverage gap audit: runCoverageGapAudit / listCoverageGapRows (see coverage-gap-audit.ts, lender-dataset-status.ts). Gap reasons come from lender-dataset-invariants.ts (e.g. index_fetch_not_succeeded, detail_processing_incomplete, failed_detail_fetches_present, detail_fetch_events_missing). Use admin coverage-gap report and remediation if enabled.
   • CDR audit: runCdrPipelineAudit (see cdr-audit.ts) checks retrieved/processed/stored/archived/tracked stages (e.g. missing raw objects, missing fetch_event links, missing series_key, presence tracking). Fix failures in order: discovery/index → detail fetch → parse → write → lineage.
   • Admin and logs: Status and actionable logs (/admin/logs/system/actionable), CDR audit run, live CDR repair routes. Check production logfiles when debugging (see AGENTS.md / debug-use-logfiles rule).

4. After changes

   • Run npm run test:api and npm run typecheck:api from repo root. If you changed ingest or pipeline code, run the relevant tests (e.g. discovery, cdr-http, normalize-quality).
   • For deploy-related changes, follow the fix-commit-verify loop and run npm run test:homepage and npm run test:api (and npm run test:archive if relevant).

Datasets and data flow

• Home loans: CDR product index → filter to residential mortgage product IDs → fetch product detail (mortgage-specific) → parse to normalized rows → historical_loan_rates / latest series. Playbooks and filters in lender-playbooks.ts, mortgage-fetch.ts, mortgage-parse.ts.
• Savings / term deposits: Same CDR product index and detail; classification by productCategory/name in cdr-savings.ts; normalized to savings and term-deposit tables.
• Data quality: Rows carry data_quality_flag (e.g. cdr_live, parsed_from_wayback_cdr). Public and analytics paths prefer cdr_live% for “live” CDR. Ensure CDR path sets this so downstream views see live data.

RBA data (same frequency as banks, reliable storage)

• Cash rate: Implemented. Source: RBA F1 CSV (https://www.rba.gov.au/statistics/tables/csv/f1-data.csv). Ingest: workers/api/src/ingest/rba.ts (collectRbaCashRateForDate). Stored in rba_cash_rates (collection_date, cash_rate, effective_date, source_url, fetched_at). Called at start of each daily run in bootstrap-jobs.ts. Backfill: backfillRbaCashRatesForDateRange; admin POST /admin/rba/backfill.
• Headline inflation and other key inflation metric: Required to be obtained at the same frequency as bank data and stored reliably. Official sources include RBA statistics tables and ABS (e.g. CPI). If not yet in the DB, add tables, ingest from RBA/ABS, and schedule with the daily run so RBA and bank data share the same collection cadence. See references/all-banks-and-rba.md.

Key invariants

• product_key is the longitudinal identity: bank_name|product_id|security_purpose|repayment_type|lvr_tier|rate_structure. Charts and exports must group/filter by product_key for one series per product over time.
• No mock data: Tests use real data or fixtures from real ingest; no mock D1 or stubbed API responses for business data (see .cursor/rules/no-mock-test-data.mdc).
• No silent truncation: Hard limits registry forbids dropping products or pages without continuation; any throttle must resume from saved state.

When in doubt, read the reference files and the source files above; then fix configuration or pipeline so expected_detail_count is set correctly, index and detail fetches succeed, written_row_count reflects accepted CDR data with no unintended gaps, and RBA data (cash rate and inflation metrics) is collected and stored at the same frequency as bank data.