Key takeaway: A load balancer does more than distribute traffic across servers — it is the mechanism for handling failures and for managing state, and both of these are where load balancing gets genuinely hard, because distributing traffic is easy and doing it correctly under failure and with state is not.
Why Distribution Is the Easy Part
The basic job of a load balancer — spreading incoming traffic across a set of servers so no single server is overwhelmed — is straightforward. The load balancer receives requests and routes them to servers, using a strategy such as round-robin or least-connections, and this part is well understood and easy to implement.
The hard parts are what happen around the distribution. When a server fails, the load balancer has to detect the failure and stop sending traffic to it, and when the application has state, the load balancer has to ensure that requests that depend on state go to the right server. Both of these are where load balancing gets genuinely difficult.
The Failure Problem
| Failure aspect | What it involves |
|---|---|
| Detection | How the load balancer knows a server has failed |
| Removal | How traffic is stopped going to the failed server |
| Recovery | How the server is brought back into rotation |
| Health checks | How the load balancer verifies a server is healthy |
When a server fails, the load balancer has to detect the failure — through health checks that probe the server — and stop sending traffic to it, so that requests are not routed to a server that cannot handle them. This is essential for availability, because without it, a failed server would keep receiving traffic and failing requests.
The failure handling is only as good as the health check — a health check that does not accurately reflect whether the server can actually serve requests will route traffic to servers that are not really healthy, or remove servers that are.
The State Problem
The state problem is that many applications have state — a user’s session, a cache, an in-memory value — that lives on a particular server. If the load balancer routes a request to a different server than the one holding the state, the request cannot be served correctly.
The common solutions are sticky sessions, where the load balancer routes a user’s requests to the same server, and moving the state out of the servers — into a shared store such as a database or cache — so that any server can serve any request. The right solution depends on the application and its state.
The Bottom Line
Treat load balancing as more than traffic distribution, since the load balancer is the mechanism for handling failures and for managing state, and both are where it gets genuinely hard. Ensure the load balancer detects and removes failed servers through accurate health checks, and address the state problem deliberately — through sticky sessions or by moving state out of the servers — since routing a request to the wrong server for its state is a failure that distribution alone does not solve.
