Scalability

Definition

Scalability is the ability of a system to handle growth in some operational dimension — typically throughput (requests per unit time), data volume, stable response time under increasing load, or the capacity for analytics over growing datasets. A scalable system can accommodate growth by adding resources, and ideally can also scale down to reduce costs during low-demand periods.

Why It Matters

Systems that are not architecturally designed to scale face exponential retrofit costs when scale is eventually needed. The foundations of scalability — stateless services, replication-friendly data designs, async decoupling — cannot easily be bolted on. Early decisions (framework choice, state management, data model) create strong path dependencies (→ Foundations Of Scalable Systems ch. 1).

Two Universal Scaling Strategies

Both strategies recur at every layer: service tier, caching tier, data tier, network (→ Foundations Of Scalable Systems ch. 1).

Scale Up vs Scale Out

Scale up (vertical scaling): replace a resource with a more powerful one. Simple, no code changes, but has a hard upper bound and single point of failure. Good first step; carries a system surprisingly far.

Scale out (horizontal scaling): run multiple replicas of a resource. Requires stateless designs so any replica can serve any request. No theoretical upper bound on capacity additions. Failure of one replica degrades capacity but not availability (→ Load Balancing, Foundations Of Scalable Systems ch. 2).

Stateless Services as an Enabling Condition

Scale-out only works if services are stateless: no per-session state lives in the service process. Any session state (shopping cart, auth token, user preference) must be externalised to a shared store (e.g., Redis). Without this, a load balancer must route all requests in a session to the same replica — sticky sessions — which eliminates the availability benefits of replication and complicates failure handling.

Amdahl's Law

Serial code fractions set a hard ceiling on how much parallelism can help:

• 50% serial code → adding beyond ~8 CPU cores yields no further speedup
• 5% serial code → adding beyond ~2,048 CPU cores yields no further speedup

Implication: multithreaded, parallel code is not a performance optimisation but a prerequisite for scalability. Poorly threaded code cannot be made scalable by hardware alone (→ Foundations Of Scalable Systems ch. 2).

The same logic applies at the code level: synchronised critical sections are the "serial fraction." Keeping them as short as possible directly improves scalable throughput. This means: hold locks only for the minimum time necessary; prefer concurrent data structures (e.g., ConcurrentHashMap with per-shard locking) over coarse-grained locks that serialise the entire collection (→ Foundations Of Scalable Systems ch. 4).

Thread pools are the server-side mechanism for controlling parallelism: a fixed-size pool bounds the number of concurrent requests, preventing resource exhaustion (memory/CPU) while maximising utilization. Tuning pool size is a scalability lever. Underthreading leaves CPU idle; overthreading causes memory exhaustion and context-switch overhead (→ Foundations Of Scalable Systems ch. 4).

Hyperscale

Hyperscale systems achieve exponential growth in computational and storage capabilities while costs grow only linearly. The canonical property is elastic scale-down as well as scale-up: Netflix scales down compute at 5am and up at 9pm. Physical infrastructure cannot do this; cloud software can (→ Foundations Of Scalable Systems ch. 1).

Scalability Trade-Offs with Other Characteristics

Scalability cannot be optimised in isolation — it requires deliberate trade-offs with other quality attributes:

Open question: When should scalability be introduced? Gorton cautions that introducing distributed infrastructure before there is a clear requirement can create development inertia. This is consistent with evolutionary architecture — treat scalability as a just-in-time concern where fitness functions flag the tipping point before it becomes a crisis (→ Building Evolutionary Architectures, Fitness Functions).

Architecture Evolution Pattern

Systems typically scale in a predictable sequence (→ Foundations Of Scalable Systems ch. 2):

1. Monolith — single app + DB; fine under light load
2. Scale up — larger hardware; simple, limited ceiling
3. Scale out — stateless replicas + load balancer + external session store
4. Caching layer — distributed cache (Redis/memcached) absorbs read traffic (→ Caching)
5. Distributed database — data tier splits horizontally (→ Partitioning, Replication)
6. Multiple independent tiers — each service scaled independently; BFF pattern for client-specific frontends
7. Async decoupling — queue-based write path wherever immediate persistence is not required (→ Messaging)

How Different Sources Treat It

Related Concepts

• Load Balancing — mechanism for distributing load across replicas
• Caching — primary optimisation strategy for read-heavy workloads
• Replication — data-tier scalability; the consistency trade-off
• Partitioning — data-tier scalability; sharding to distribute load
• Queueing Theory — Little's Law, utilisation curves, and tail latency: the formal physics of scalable design
• Backpressure — flow control that prevents producers from overrunning consumers in a scaled-out system
• Architecture Characteristics — scalability in the quality attribute taxonomy
• Consistency Models — what replication for scalability costs
• Messaging — async queueing as a scalability pattern
• Cost As Architectural Force — scaling decisions as cost decisions
• Performance And Capacity — capacity planning workflow and diagnosis
• Cost Vs Availability — scale-out investment in the broader availability/cost trade-off
• Observability — required to identify bottlenecks and guide scaling decisions
• Serverless — managed auto-scaling model where the cloud provider handles scale-out