Intermediate System design concept · Handling Scale & Bottlenecks · 43 mins read

Load Balancing Systems

A load balancer turns many servers into one usable service entry point.

Open the interactive lesson See it in Design Messaging App System

Round Robin

Round robin sends each new request to the next backend in order, which is simple and fast but blind to actual work in flight.

Intuition

A load balancer needs a default rule for splitting requests across identical-looking servers. Round robin solves that by rotating through the backend list one request at a time: server A, then B, then C, then back to A. The algorithm is popular because it is simple, predictable, and cheap to execute, but that same simplicity means it assumes each request costs roughly the same. If one request is a 2 ms cache hit and the next is a 20 second report export, pure round robin still treats them as equal.

Mental Model

Think of the balancer as holding a pointer into a circular array of servers. Each new request advances the pointer by one slot and hands the request to that server, regardless of current CPU, queue depth, or open connections. Weighted round robin extends this idea by placing more copies of a stronger server into the rotation so it appears more often. Think of it like: It is like dealing cards around a poker table: player 1 gets the first card, player 2 gets the next, player 3 gets the next, then the cycle repeats. That is fair if everyone needs the same number of cards, but not if one player is already overloaded with side tasks.

Building Blocks

An ordered list of healthy backends. The algorithm needs a stable pool it can cycle through one slot at a time.
A pointer or index that advances on each routing decision. That tiny bit of state is what makes the algorithm simple and cheap.
Health checks and draining. Dead or retiring nodes must leave the rotation cleanly so the sequence does not send fresh traffic into failure.
Optional weights for mixed-capacity fleets. Bigger servers can get more turns, but the algorithm still stays blind to live load.

Definitions

Round Robin

A policy that sends each new request to the next backend in a repeating fixed order.

It balances turn-taking, not actual resource usage.
Its value is simplicity and very low routing overhead.

Weighted Round Robin

A round-robin variant where stronger servers appear more often in the rotation.

It models known capacity differences between nodes.
It still does not react to temporary hot spots.

Request Cost Variance

The fact that one request may consume much more time or CPU than another.

High variance is where round robin starts looking unfair in practice.
Equal request counts do not mean equal backend load.

Connection Draining

Stopping new traffic to a server while allowing in-flight work to finish.

It matters during deploys and autoscaling events.
Without it, removal from the pool can interrupt user requests.

Patterns

Use plain round robin for stateless services where most requests are short and similar in cost.
Use weighted round robin when the fleet is intentionally heterogeneous and capacity differences are known ahead of time.
Replace it with a more adaptive policy when long-lived or highly uneven requests dominate.

Strategies

Start with round robin as a baseline only if the workload is already mostly uniform and stateless.
Measure request-duration spread before trusting equal turns as equal work.
Use weights to reflect stable machine-size differences, not as a substitute for real-time load feedback.

What round robin actually balances

Round robin balances request count, not effort. If one node gets ten cheap cache hits while another gets ten report exports, the algorithm still considers the distribution fair because each node received the same number of turns.

That makes round robin a good fit only when request cost is fairly uniform and any backend can handle any request. Once duration or CPU cost varies widely, a deterministic sequence can pile slow work onto one unlucky node while others stay underused.

Weighted round robin helps only with known structural differences such as bigger instances. It does not solve short-term imbalance caused by what requests are happening right now.

Tradeoffs

Round robin is trivial to implement and cheap to run, but it is blind to live backend load.
Equal turn-taking is easy to reason about, but equal request counts do not guarantee equal CPU, memory, or queue usage.
Weights improve mixed fleets, but they still encode expected capacity rather than current saturation.

Real World

NGINX commonly uses round robin as the default upstream policy because it works well for stateless web tiers with similar request cost.
HAProxy supports both round robin and weighted round robin, making it a classic example for homogeneous versus mixed-capacity backend pools.

Interview

Questions interviewers ask

How does round robin work, and why is it such a common default?
When is weighted round robin useful?
Why can equal request counts still produce unequal server load?
What workloads make round robin a poor choice?

What a strong answer covers

Candidates should describe the rotating pointer model, explain the good-fit case of stateless uniform requests, and explicitly say that round robin balances turns rather than true backend load.

Common traps

Saying round robin balances load perfectly instead of balancing only request turns.
Treating weights as live feedback rather than a static capacity hint.
Recommending round robin for long-lived or highly uneven workloads without qualification.

Quiz

What does a basic round-robin load balancer optimize directly?

Lowest server CPU at every moment
Equal turn-taking across healthy servers
Fewest open database transactions
Shortest application queue depth

Round robin rotates requests in order, so it equalizes request turns rather than measuring live resource usage. That is why it is simple but not load-aware.

Which situation is the best fit for plain round robin?

A chat system dominated by long-lived WebSocket connections
A report service where some requests run for minutes
A stateless CRUD API where requests have similar cost
A legacy app that requires session affinity

Round robin works best when any server can handle any request and request cost is fairly uniform. Stateless web APIs with similar request shapes match that assumption.

Why might round robin overload one server even when all servers receive the same number of requests?

Because request duration and resource cost can differ a lot
Because TCP always reorders packets
Because health checks disable caching
Because DNS picks only one backend

Equal request counts are not the same as equal work. A few expensive or long-running requests can keep one node much busier than others.

What does weighted round robin add on top of plain round robin?

It pins each user to one backend with a cookie
It routes strictly by shortest response time
It ignores unhealthy nodes without health checks
It gives higher-capacity servers more turns in the rotation

Weights represent expected backend capacity, so stronger nodes receive traffic more often. The algorithm is still not reacting to live instantaneous load.

Which statement about round robin is most accurate?

It is usually the best algorithm for highly variable streaming workloads
It is simple and fast, but blind to real-time backend load
It requires all backends to maintain user session state locally
It removes the need for health checks during deploys

That is the core tradeoff: operational simplicity in exchange for weak awareness of true backend utilization. You still need health checks and often a more adaptive policy for uneven work.

Least Connections

Least connections routes each new request to the backend with the fewest active connections so it adapts better when request duration is uneven.

This section is part of the full PRISM roadmap, with worked examples, trade-off tables, interview questions and a quiz.

Unlock the full lesson

Sticky Sessions

Sticky sessions keep a client pinned to the same backend, which can preserve local session state but fights even load distribution and graceful failover.