Analytical Data Patterns: Data Warehouse, Data Lake, Data Mesh
Contents
The Operational vs Analytical Divide
Operational data (OLTP) and analytical data serve fundamentally different purposes:
- Operational data: transactional, drives daily business operations, consistency and availability critical, OLTP access patterns.
- Analytical data: business intelligence, trend analysis, ML model training, supports long-term strategic decisions; staleness more acceptable; large aggregations; OLAP access patterns.
These access patterns are incompatible at scale — heavy analytical queries on an OLTP database impede transactional throughput. Every architecture must decide how to manage this separation.
Pattern 1: Data Warehouse
Approach: Extract-Transform-Load (ETL) — operational data is extracted from source databases, transformed into a unified schema (typically Star Schema), and loaded into a centralised analytical store. Analysis happens on the warehouse.
Star Schema: denormalised design separating "facts" (measurable data: counts, amounts, durations) from "dimensions" (descriptive context: names, locations, categories). Optimised for aggregation queries; not for transactional workloads.
Trade-offs:
| Advantage | Disadvantage |
|---|---|
| Centralised consolidation of data | Integration brittleness (transformation logic tightly coupled to operational schemas) |
| Dedicated analytics silo | Extreme partitioning of domain knowledge (requires both domain and analytics expertise) |
| Powerful query and aggregation capabilities | High complexity and maintenance cost |
| — | Synchronisation bottlenecks impact operational systems |
| — | Rarely delivered business value proportional to investment |
Why it fails in distributed architectures: a data warehouse centralises analytically-transformed data in a technically-partitioned schema, losing domain context. Schema changes in any operational service cascade through transformation pipelines. Domain expertise must reside both in operational systems and in the warehouse team. Not compatible with microservices' bounded context model.
Pattern 2: Data Lake
Approach: Load-and-Transform — operational data is dumped into the lake in raw (or minimally-transformed) form. Transformation and analysis happen on demand by data scientists querying the lake directly.
Rationale: data warehouse pre-built schemas frequently didn't match the query patterns actually needed, wasting the enormous transformation effort. By storing data "closer to raw", the burden of transformation becomes reactive rather than proactive.
Trade-offs:
| Advantage | Disadvantage |
|---|---|
| Less up-front transformation | Domain relationships still hard to discover in the unstructured lake |
| Better suited to ML (raw data often preferred) | PII and sensitive data risk (domain context stripped, recombination possible) |
| Less integration brittleness | Still technically partitioned — loses domain separation |
| — | Data staleness from batch-style ingestion |
Why it still fails: both Data Warehouse and Data Lake are technically partitioned approaches. Modern architecture trends favour domain partitioning (microservices, bounded contexts). Both approaches extract data from its domain context and centralise it, losing the domain perspective required to make the analytical data meaningful. They treat data as a separate concern from the services that produce it.
Pattern 3: Data Mesh
Data Mesh (Zhamak Dehghani) applies the domain ownership principle from microservices to analytical data, rather than centralising it (→ Software Architecture The Hard Parts ch. 14; see also Dehghani's book Data Mesh).
Definition: a sociotechnical approach to sharing, accessing, and managing analytical data in a decentralised fashion, aligning data ownership with business domains and enabling peer-to-peer consumption.
Four principles:
Domain ownership of data: the domain team most familiar with the data — those who produce it or are its primary consumers — owns and shares it. No central data team or warehouse intermediary.
Data as a product: data is treated as a product with defined consumers, service-level objectives, quality metrics, and domain team ownership. The architectural mechanism is the data product quantum (DPQ).
Self-serve data platform: a shared platform layer supports discovery, creation, and monitoring of data products without requiring each domain team to build the infrastructure from scratch.
Computational federated governance: organisation-wide compliance, security, and privacy policies are embedded as code in each DPQ via a governance sidecar. Policy is federated — agreed by domain owners — but enforced automatically at the point of data access.
Trade-offs:
| Advantage | Disadvantage |
|---|---|
| Domain teams own their analytical data (aligned with microservices model) | Requires contract coordination with DPQ |
| Excellent decoupling between analytical and operational data | Requires asynchronous communication and eventual consistency |
| Follows domain partitioning (not technical) | Higher complexity to set up across many domains |
| Federates governance without a central bottleneck | — |
Data Product Quantum (DPQ)
The DPQ is the core architectural unit of the data mesh. It is deployed adjacent to but operationally separate from its cooperating service (→ Software Architecture The Hard Parts ch. 14).
Three DPQ types:
- Source-aligned (native) DPQ: provides analytical data on behalf of its cooperating service; acts as the analytical "twin" of a microservice.
- Aggregate DPQ: aggregates data from multiple inputs, either synchronously (for low-latency requirements) or asynchronously (for eventual analytical views).
- Fit-for-purpose DPQ: custom DPQ for a specific analytical requirement (ML, specific BI report, knowledge graph).
Cooperative quantum: the DPQ and its cooperating service form a cooperative quantum pair:
- Operationally independent — the DPQ can fail without taking down the operational service.
- Async + eventual — communication between the DPQ and its service always uses an async/eventual saga type (Parallel Saga or Anthology Saga). Never synchronous or transactional — that would defeat the purpose of operational/analytical separation.
- Tight contract coupling between the DPQ and its cooperating service; looser coupling to the analytics/reporting quantum.
Static coupling note: the DPQ is part of the architecture quantum's static coupling — its availability is required for the service to provide analytical capabilities, just as a message broker must be available if messaging is used.
Governance sidecar: each DPQ includes an embedded sidecar (supplied by the platform team) that executes policy at the point of data access — handling access control, compliance, schema validation, and PII protection. This is the "computational" in computational federated governance.
Relationship to Service Mesh and Sidecar
The data mesh pattern draws explicitly from the service mesh pattern (Sidecar Service Mesh): just as the service mesh uses sidecars to manage operational coupling (logging, monitoring, authentication) orthogonal to business logic, the DPQ sidecar manages analytical governance orthogonal to the service's operational concerns. Dehghani applied the same "orthogonal coupling via a sidecar" principle to the analytical data domain.
How Different Sources Treat It
| Source | Perspective |
|---|---|
| Software Architecture The Hard Parts | Ch 14 covers the Data Warehouse, Data Lake, and Data Mesh patterns as the architectural evolution of analytical data management; introduces DPQ and cooperative quantum; co-authored with Zhamak Dehghani who invented Data Mesh |
| Learning Domain Driven Design | DDD-centric treatment in ch. 16. Analytical model context: OLTP (entity-centric, real-time) vs OLAP (fact/dimension tables, append-only, flexible queries). Fact tables (business activities, similar to domain events; star schema = one dimension level; snowflake schema = multi-level normalised dimensions). Data warehouse: centralised ETL-based model; coupling to operational DB implementation detail; cross-team friction; doesn't scale to big data. Data lake: store raw operational data; transform later; becomes a data swamp at scale (no schema enforcement → chaos). Data mesh as "DDD for analytical data": Principle 1 — Decompose around domains (align analytical model ownership with bounded contexts; same team owns OLTP + OLAP); Principle 2 — Data as a product (well-defined output ports, SLAs, versioning, polyglot storage); Principle 3 — Enable autonomy (platform team provides interoperability infrastructure); Principle 4 — Build an ecosystem (federated governance body). DDD/Data Mesh alignment: (a) ubiquitous language drives analytical model design; (b) OHS = analytical model is a published language for analytical consumers; (c) CQRS generates analytical projections alongside operational model; (d) bounded context integration patterns apply to analytical models too |
Open question: Data Mesh requires significant organisational maturity — domain teams must take responsibility for their analytical data as a product, which is a substantial culture shift. The book does not deeply address the transition path from a centralised data warehouse to a data mesh.
Related Concepts
- Architecture Quantum — DPQ as cooperative quantum; static/dynamic coupling for analytical data
- Sidecar Service Mesh — the organisational precedent for the DPQ governance sidecar; orthogonal coupling
- Reuse Patterns — DPQ governance sidecar is an operational-concerns sidecar applied to analytical data
- Saga — Parallel Saga and Anthology Saga are the only appropriate saga types for DPQ-to-service communication
- Data Decomposition — operational data decomposition; data mesh handles the analytical data layer on top
- Zhamak Dehghani — inventor of Data Mesh; co-author of SATH