RFC: Data Lake for Fintech
Status: Draft
Author: Banking Challenges Team
Date: 2024-01-15
Version: v0.1
Problem Statement
Context
Financial institutions process millions of transactions daily. The generated data needs to be stored, processed, and analyzed for reports, compliance, fraud detection, and business insights. Transactional systems (OLTP) are not suitable for large-scale analytics.
This RFC proposes a Data Lake for analytical purposes, capable of ingesting data from multiple sources (SPI, DICT, Ledger, Open Finance) and making it available for queries and reports.
Goals
- Centralize data from all Banking Challenge services
- Support analytical queries without impacting OLTP systems
- Enable historical reporting and trend analysis
- Provide raw data access for data science teams
- Support audit and compliance requirements
Proposed Solution
Architecture Overview
┌─────────────────────────────────────────────────────────────────────┐
│ Data Lake Architecture │
│ │
│ Data Sources │
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │
│ │Ledger│ │ SPI │ │ DICT │ │ OF │ │ NFS-e│ │
│ └───┬──┘ └──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘ │
│ │ │ │ │ │ │
│ ┌───▼───────▼────────▼────────▼────────▼──────────────────────┐ │
│ │ Ingestion Layer │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Kafka │ │ CDC (DMS)│ │ Batch │ │ │
│ │ │ (Stream) │ │(Debezium)│ │ (S3 sync)│ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────── ▼ ──────────────────────────────────┐ │
│ │ Storage Layer │ │
│ │ │ │
│ │ Bronze (Raw) Silver (Cleaned) Gold (Aggregated) │ │
│ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ │
│ │ │ JSON/Parquet │ │ Parquet (SQL) │ │ Parquet (SQL) │ │ │
│ │ │ /raw/ledger/ │ │ /silver/ledger/│ │ /gold/reports/ │ │ │
│ │ │ /raw/spi/ │ │ /silver/spi/ │ │ /gold/kpis/ │ │ │
│ │ └────────────────┘ └────────────────┘ └────────────────┘ │ │
│ │ │ │
│ │ MinIO / S3 Compatible │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────── ▼ ──────────────────────────────────┐ │
│ │ Query Layer │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │Presto/ │ │ Spark │ │ Athena │ │ Trino │ │ │
│ │ │Trino │ │ │ │ │ │ │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────── ▼ ──────────────────────────────────┐ │
│ │ Consumption Layer │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Metabase │ │ Superset│ │ Python │ │ API │ │ │
│ │ │(BI) │ │(Dashboard)│ │(Notebook)│ │(REST) │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘Data Layers
Bronze (Raw)
- Data as-is from sources
- Immutable, append-only
- Format: JSON, raw Parquet
- Partitioned by date
parquet
/raw/ledger/year=2024/month=01/day=15/transactions_0001.parquet
/raw/spi/year=2024/month=01/day=15/payments_0001.parquetSilver (Cleaned)
- Cleaned, validated, deduplicated
- Schema enforced
- Format: Parquet (optimized for query)
- Partitioned by date + type
parquet
/silver/ledger/transaction_type=PIX/year=2024/month=01/day=15/
/silver/spi/status=ACCEPTED/year=2024/month=01/day=15/Gold (Aggregated)
- Business-level aggregations
- KPIs, metrics, reports
- Pre-computed for performance
- Format: Parquet, tables
parquet
/gold/kpis/daily_volume/year=2024/month=01/day=15/
/gold/reports/monthly_settlement/year=2024/month=01/Database Schema (Mermaid ERD)
API Design
Query Data Lake
http
POST /api/v1/datalake/query
Content-Type: application/json
{
"query": "SELECT date, COUNT(*) as tx_count, SUM(amount) as volume
FROM silver.ledger_transactions
WHERE date BETWEEN '2024-01-01' AND '2024-01-31'
AND type = 'PIX'
GROUP BY date
ORDER BY date",
"format": "json"
}List Available Tables
http
GET /api/v1/datalake/tablesTrigger Data Export
http
POST /api/v1/datalake/export
Content-Type: application/json
{
"table": "gold.daily_volume",
"dateRange": { "start": "2024-01-01", "end": "2024-01-31" },
"format": "parquet"
}Trade-offs and Alternatives
| Alternative | Pros | Cons |
|---|---|---|
| Traditional Data Warehouse | Strong consistency, SQL-native | Expensive, schema-on-write, less flexible |
| Data Lake (chosen) | Cheap storage, schema-on-read, flexible | More complex governance, consistency challenges |
| Hybrid (Lakehouse) | Best of both worlds | Higher complexity, newer tech |
| PostgreSQL Analytics | Simple, no new infra | Performance degradation at scale |
Chosen: Data Lake with MinIO (S3) + Medallion Architecture (Bronze/Silver/Gold)
Security Considerations
- Encryption at Rest: All data encrypted via MinIO SSE-S3
- Encryption in Transit: TLS for all data transfers
- Access Control: IAM policies per layer (Bronze/Silver/Gold)
- Data Masking: PII columns masked in Silver/Gold layers
- Audit Trail: All queries logged
- Retention Policy: Bronze (90 days), Silver (1 year), Gold (indefinite)
- Compliance: LGPD data deletion capability
Open Questions
- Should we use Apache Iceberg or Delta Lake for table format?
- Real-time streaming vs micro-batch for ingestion?
- How to handle schema evolution in Silver layer?
- Data catalog tool: Apache Atlas or Amundsen?