What Is a Load Balancer, and How Does It Actually Work?

A load balancer sits in front of your application's servers and decides, for every incoming request, which server actually handles it — spreading traffic across multiple machines instead of sending it all to one. It's one of the first pieces of infrastructure that shows up the moment a single server stops being enough, whether that's because of traffic volume, a need for zero-downtime deployments, or simply not wanting one machine's failure to take the whole application down.

The problem a load balancer actually solves

A single server has a hard ceiling: a fixed amount of CPU, memory and network capacity, and if it goes down, everything behind it goes down with it. Running the same application on multiple servers solves both problems in principle — more total capacity, and no single point of failure — but only if something in front of those servers actually knows how to split incoming traffic between them, detect when one has failed, and stop sending it requests until it recovers. That's the specific, narrow job a load balancer does. Everything else people associate with load balancers (SSL termination, health checks, sticky sessions) is built around that one core responsibility.

How a load balancer actually decides where a request goes

Round robin

The simplest strategy: requests are handed to each backend server in turn, cycling through the list. It's easy to reason about and works well when every server has roughly equal capacity and every request is roughly equally expensive to handle.

Least connections

Instead of blindly cycling through servers, the load balancer tracks how many active connections each backend currently has and sends the next request to whichever has the fewest. This handles uneven request costs better than round robin — if one request happens to be much slower than others, least-connections naturally avoids piling more work onto the server still busy with it.

IP hash / consistent hashing

The client's IP address (or another identifying value) is hashed to consistently pick the same backend server for that client's requests. This is one way to achieve session affinity — the same user keeps landing on the same server — without a load balancer needing to inspect application-level session data.

Weighted algorithms

Any of the above strategies can be weighted, so a beefier server (more CPU/memory) receives proportionally more traffic than a smaller one in the same pool — useful when a fleet isn't made up of identical machines, such as during a gradual hardware upgrade.

Health checks — how it knows a server is actually down

A load balancer periodically sends a request to each backend server — often a simple HTTP request to a dedicated health-check endpoint — and expects a specific response within a specific time. Miss enough consecutive checks, and the load balancer marks that server unhealthy and stops routing new traffic to it, automatically, without anyone needing to intervene. Once the server starts responding to health checks correctly again, it's added back into rotation. This is the mechanism that actually delivers on "no single point of failure" — it's not that a server never fails, it's that failure gets detected and routed around within seconds, rather than serving errors to users until a human notices.

Layer 4 vs Layer 7 load balancing

A Layer 4 (transport-layer) load balancer works at the TCP/UDP level — it sees IP addresses and ports, but not the actual HTTP request content, and simply forwards packets to a chosen backend. It's fast and protocol-agnostic, but it can't make routing decisions based on, say, the URL path or a header. A Layer 7 (application-layer) load balancer understands HTTP itself — it can route /api/* requests to one set of backends and /images/* to another, inspect headers, terminate SSL/TLS before forwarding plaintext internally, and make much more granular routing decisions. Most modern web applications use Layer 7 load balancing (or a Layer 7 component in front of a Layer 4 one) specifically because that content-aware routing is so useful in practice.

Session affinity (sticky sessions) — and why it's often avoided

Some applications store session state (a shopping cart, a logged-in session) in memory on whichever server first handled that user, meaning every subsequent request from that user needs to land on the same server — this is what sticky sessions (via IP hash or a cookie the load balancer sets) provide. The tradeoff: it undermines even load distribution (a server with many active "stuck" sessions can't shed load to a less busy one) and makes that server's failure worse for its stuck users (their session is simply gone). The common alternative — storing session state in a shared store like Redis instead of in-memory on one server — lets any backend serve any request statelessly, which is why "stateless application servers" is such a common design goal in practice, not just a theoretical nicety.

Where load balancers actually run

• Managed cloud load balancers — AWS's Application/Network Load Balancer, Google Cloud Load Balancing, Azure Load Balancer — handle health checks, scaling and SSL termination as a managed service, without you operating the load-balancing software yourself.
• Self-hosted software load balancers — NGINX, HAProxy, Envoy — run as their own process, configured via a config file or, in Envoy's case, often driven dynamically by a service mesh control plane.
• DNS-based load balancing — returning different IP addresses for the same hostname to different clients (round robin DNS) — a cruder, less responsive form of distribution that doesn't handle server failure as gracefully as an actual load balancer with health checks.
• The load balancer built into an orchestration platform — Kubernetes' own Service resource load-balances across a Deployment's Pods internally, often with an external cloud load balancer sitting in front of the cluster's entry point.

SSL/TLS termination

Encrypting and decrypting HTTPS traffic costs CPU, and doing it separately on every single backend server is both wasteful and operationally awkward — every server would need the certificate installed and kept up to date. SSL/TLS termination moves that work to the load balancer: the client's encrypted connection ends there, and the load balancer forwards the request to backend servers over plain HTTP on the internal network (which is typically private and trusted). This centralises certificate management to one place, offloads the encryption overhead from application servers, and is one of the most common reasons a Layer 7 load balancer sits in front of an application even when traffic distribution alone wouldn't have required one. Some setups re-encrypt traffic between the load balancer and backends for defense-in-depth on genuinely sensitive internal networks, trading some of that simplicity for stronger internal encryption.

Load balancing and autoscaling working together

A load balancer distributes traffic across whatever backend servers currently exist; autoscaling changes how many backend servers exist, based on load. The two are designed to work together: as traffic climbs and existing servers approach their capacity, an autoscaling group spins up new instances, and the load balancer's health checks detect them as soon as they pass their first successful check and start routing traffic to them — with no manual reconfiguration. When traffic drops, instances are scaled back down, and the load balancer simply stops sending them requests once they're deregistered. Neither piece does much on its own without the other: a load balancer with a fixed server count can't handle a genuine capacity crunch, and autoscaling without a load balancer in front has no coordinated way to actually distribute traffic across the instances it creates.

Global server load balancing (GSLB)

Everything described so far load-balances traffic across servers within one data center or region. Global server load balancing operates one level up — routing a user's request to the nearest or healthiest entire region or data center, typically via DNS that resolves differently depending on the requester's location, current regional health, or latency measurements. A user in Mumbai gets routed to a Mumbai (or nearest healthy) data center; a user in Frankfurt gets routed to a European one. This is what lets a genuinely global application survive an entire region going down — GSLB detects the region's health check failing and stops directing new users there, routing them to the next-nearest healthy region instead, while each individual region still runs its own local load balancer underneath handling server-level distribution.

Load balancer vs API gateway vs reverse proxy — terms that get blurred

A reverse proxy is the broader category — anything that sits in front of one or more backend servers and forwards requests on their behalf. A load balancer is a reverse proxy whose specific job is distributing traffic across multiple equivalent backends. An API gateway is also commonly built on reverse-proxy technology, but its job is broader still — authentication, rate limiting, request/response transformation, routing across many different backend services, not just distributing load across replicas of one service. In practice, a single piece of software (NGINX, Envoy) can be configured to do any of these jobs, or several at once, which is exactly why the terms get used loosely — the distinction is about the job being done, not the specific tool doing it.

Connection draining

When a server is about to be taken out of rotation — during a deploy, a scale-down, or manual maintenance — abruptly cutting its connections mid-request would fail whatever requests happened to be in flight at that moment. Connection draining (sometimes called graceful deregistration) tells the load balancer to stop sending that server new requests immediately, while letting its already-in-progress requests finish normally within a configured grace period, before it's fully removed from the pool. This is what actually makes a rolling deploy or a scale-down event invisible to users — without it, some fraction of requests would simply fail every time a server is cycled out, which quickly adds up across frequent deploys.

Managed vs self-hosted — the practical cost/control tradeoff

A managed cloud load balancer (an AWS ALB, for instance) charges based on usage and requires no server of your own to run the load-balancing software, patch it, or scale it — the tradeoff is less granular control over exact routing logic and being tied to that cloud provider's specific feature set and pricing model. A self-hosted load balancer (NGINX, HAProxy, Envoy) gives full control over configuration and can run anywhere, including on-premises, but you're responsible for its own uptime, scaling and patching — the load balancer itself becomes one more piece of infrastructure that could fail if not set up with redundancy of its own. Most teams on a major cloud provider default to the managed option specifically to avoid operating that extra piece of infrastructure themselves, reaching for a self-hosted option mainly when they need routing behaviour the managed product doesn't support, or when running outside a single cloud provider's ecosystem entirely.

Common mistakes when introducing a load balancer

• Storing session state in server memory and then relying on sticky sessions to paper over it, instead of moving to a shared session store that lets any backend handle any request.
• Setting health-check intervals too infrequently, so a genuinely failed server keeps receiving traffic for far longer than necessary before being marked unhealthy.
• Forgetting to update application logging/rate-limiting logic to read the real client IP from a forwarded-for header, since every request now arrives from the load balancer's own IP rather than the original client's.
• Load-balancing across replicas that aren't actually interchangeable (different code versions, different config) during a rolling deploy without accounting for the brief window where both versions serve traffic simultaneously.
• Treating a load balancer as a substitute for actually having more than one backend server — it adds no redundancy on its own if every request still ultimately lands on the same single machine.
• Skipping connection draining during deploys, so in-flight requests get abruptly cut off the moment a server is taken out of rotation instead of finishing gracefully.
• Running the load balancer itself as a single instance with no redundancy of its own, quietly turning it into the exact single point of failure it was introduced to eliminate.