Hide outline
Feedback
Guide
Was this lesson helpful?
Dive Deeper
Generate a follow-up sub-lesson on any aspect of this topic
Guide complete
WhatsApp’s Erlang Gateway at Massive Scale
Generate a follow-up sub-lesson on any aspect of this topic
WhatsApp’s Erlang Gateway at Massive Scale
TLDR;
- A gateway can hold millions of idle mobile TCP connections if it treats each connection as cheap and isolated.
- Most scaling pain past a few hundred thousand sockets comes from contention, not raw CPU.
- Clustering only works when user to node placement is deliberate and reconnect storms are planned for.
WhatsApp’s Erlang gateway is a stateful edge tier whose main job is to keep persistent TCP connections open for mobile chat clients, route messages to the right online socket, and fall back to store and forward when the recipient is offline. The historically documented target was to move from an initial state around 200k connections per server to 1M and then beyond 2M per server, because the product depends on always-on reachability even when users are mostly idle.
One big server fails at around 2M sockets when it treats connections as heavyweight, shares locks too broadly, or couples unrelated work onto a single hot path. At that point the bottleneck is usually contention in the runtime, kernel, or application level shared structures rather than a lack of cores, and the failure mode is a brittle cluster that cannot absorb overloads or disruptions without cascading disconnects. Here is the baseline shape of the gateway tier in front of its state stores:
In this baseline, clients hold long-lived TCP sockets through a TCP load balancer to a pool of Erlang nodes, and those nodes talk to an auth and roster store on the back end. The stress point is the combination of connection count and background traffic such as keepalives, because it turns small per-connection costs into system-wide lock contention and scheduler pressure.
Connection management works at scale when the gateway treats a TCP connection as a bounded resource with an explicit memory and message rate budget, not as an unbounded session that can accumulate state and queues. The mental model is an accept loop split into acceptor pools which are multiple concurrent acceptors that pull new sockets off the listen queue, then hand each accepted socket to a dedicated lightweight process that owns reads, writes, and per-connection timers. This shape matters because it lets the node accept bursts without serializing on one acceptor, while keeping per-connection work local so it scales across schedulers.
Once you attach a budget, you can check whether the machine fails by RAM, by packets per second, or by message handling rate. A simple heuristic is where is the bytes of process state, socket buffers, and mailbox slack you allow per connection, and you separately estimate keepalive load as . Plugging your own numbers into that budget is what keeps 2M sockets from turning into a surprise out of memory or a slow spiral where queues grow faster than they drain. Let’s run those estimates with the same connection scale:
The estimate ties three pressures together, total connections, bytes per connection, and keepalive interval, into two observable limits, RAM consumption and background messages per second. When either estimate is close to the host’s real ceiling, the acceptor pool stops being a gateway and becomes a crash amplifier because new connects and reconnects arrive exactly when you can least afford extra memory or queue growth.
Design rule Keep per-connection state small enough that doubling connections does not change garbage collection behavior.
Process isolation works when each connection is handled by a separate lightweight process whose failure does not take down the listener, the acceptors, or unrelated connections. In Erlang systems, a supervision tree is a parent child structure where parents restart failed children according to a policy, which turns crashes into localized restarts instead of whole-node outages. For a gateway, that means a bad packet decode, a bug in one user’s state machine, or a single socket error should only kill one connection process, while the acceptor pool keeps taking new sockets.
Restarts still have user-visible meaning. When a per-connection process crashes, the TCP socket typically drops, the client reconnects, and the gateway must treat the reconnect as a new session that re-authenticates and rehydrates minimal routing state. Keeping durable state out of the connection process is what prevents restart loops from corrupting global tables, and it is also what makes overload survivable because the system can choose to refuse new sessions while preserving existing ones. This supervision slice shows where the restart boundaries sit:
In this tree, the listener stays up, acceptors can be restarted as a group, and each connection process is its own fault domain. The key behavioral change is that crashes become disconnects for one user at a time, instead of turning into a global lockup where every connection starts timing out together.
Message routing inside the gateway scales when the connection process does minimal parsing and hands off to a router that can find the destination without a single global lock or a single central bottleneck. The usual trick is to partition routing state, for example by hashing user identifiers to a set of router processes or ETS tables, so that lookups and updates spread across schedulers. This matters because once you have millions of connections, even small amounts of shared bookkeeping can dominate, which is exactly what WhatsApp engineers observed when scaling from hundreds of thousands to millions and chasing contention fixes.
The gateway also needs two delivery modes. If the recipient is online, it routes to the current connection process and writes to the socket. If the recipient is offline, it routes to a spool or store and forward component and returns an acknowledgement path that does not block the sender’s connection loop. This single-message flow keeps the critical path short and makes offline handling an explicit branch, not an accidental timeout path:
The flow highlights five actors, the sender connection process, the router, the recipient connection process, and the offline spool as the alternative sink. The important change is that routing and storage decisions happen in a partitionable layer, so one hot user or one busy shard does not serialize every other user’s messages.
Fault tolerance under overload works when the gateway intentionally drops or slows the right work, instead of letting queues grow until the runtime spends all its time managing backlogs. At massive connection counts, overload often arrives as correlated bursts, for example an outage recovery or a world event spike, and the first symptom is long message queues and rising latency that trigger more retransmits and reconnects. A gateway that tries to be fair to all clients by queueing everything usually fails harder because it preserves load long enough to knock over critical subsystems like routing tables or back end stores.
Three mechanisms interact here. Backpressure caps per-connection outbound queue length so slow clients cannot accumulate unbounded mailboxes. Load shedding refuses new sessions or drops non-critical traffic when the node crosses a threshold such as longest message queue. Per-user rate limits prevent a single user or chatty group from consuming router capacity during a spike. The tradeoff is user experience, where aggressive shedding causes disconnects, while conservative queueing causes high latency and eventually a node-wide collapse. Tune those tradeoffs to see the regime changes:
The tuning surface connects queue limits and rate caps to two outcomes, latency and disconnect rate. The main behavioral shift happens when queues stop being an elastic buffer and start being a work multiplier, because every queued message consumes memory, scheduling time, and garbage collection, which reduces effective throughput further.
Caveat If reconnects are cheap for clients, load shedding can create a reconnect storm.
Clustering for millions works when the system has a stable way to decide which node should own a user’s online session, and it keeps that mapping stable enough to avoid constant cross-node forwarding. A consistent hashing scheme is a method that maps keys to nodes so that when membership changes, only a fraction of keys move, which reduces churn-induced reroutes. In an Erlang gateway cluster, the mapping decision controls three costs at once, cross-node message volume, cache locality for per-user state, and the blast radius when a node disappears.
There are a few common ways to do user placement. A sticky load balancer can pin a user to a node based on IP or a token, which is simple but can break under NAT churn or uneven load. A shared routing table can track user to node presence, which makes forwarding explicit but adds update traffic and consistency questions. Rendezvous hashing can assign a user to a node without a centralized table, but it still needs a membership view that does not flap. Compare these options on the dimensions that matter at millions of sockets:
The comparison is really about where state lives and how often it changes. When membership is stable, hashing keeps cross-node traffic low. When membership churns or when clients reconnect frequently, a routing table can become a write hot spot and sticky load balancing can produce uneven distribution that shows up as one node hitting queue limits long before the rest.
The historically documented gateway design starts to break down when multi-device and richer metadata force more fanout and more cross-node coordination than a per-user single-socket model assumed. Presence and typing indicators create high fanout patterns, and when those fanouts cross node boundaries they multiply routing messages inside the cluster. End-to-end encryption also shifts what the gateway can inspect and optimize, and it can move load from content processing to metadata routing and key lookup, which makes back end dependencies more visible during spikes.
Another break point is operational. When a network partition heals or a data center event forces mass reconnects, the gateway tier experiences a thundering herd of TCP handshakes, auth lookups, and state rebuilds, and the earlier overload mechanics decide whether the cluster stabilizes or oscillates. WhatsApp’s later migration notes also show that even inside the company, the gateway boundary moved, with more work pushed into other stacks when tuning a pure Erlang forwarding layer became difficult at scale. Reveal the hidden costs that tend to dominate these moments:
The three costs highlight where the gateway stops being just a socket holder. Presence fanout increases per-event amplification, cross-node routing increases internal traffic relative to user traffic, and reconnect storms turn a transient event into sustained overload because each reconnect forces work in multiple subsystems at once.
An Erlang-inspired gateway still holds up when it is explicit about what the gateway owns. The gateway owns socket lifetimes, per-connection backpressure, and fast routing of small messages, and it treats everything else as a dependency with a cost. WhatsApp’s scaling story is mostly a story about chasing contention and partitioning work so that background behaviors do not turn into global locks at connections.
The redesign triggers are equally concrete. If multi-device semantics require frequent fanout updates, if your back end calls cannot tolerate eventual consistency or caching, or if reconnect storms are part of normal operations such as mobile network churn, then the gateway needs stricter admission control and likely a split where heavy work moves out of the connection owning processes. The pattern to keep is the same one the gateway started with, make each connection cheap, isolate failures, and keep routing decisions partitionable, because those are the properties that let millions of persistent sockets behave like manageable units instead of a single fragile megaprocess.