Hide outline
Feedback
Guide
Was this lesson helpful?
Dive Deeper
Generate a follow-up sub-lesson on any aspect of this topic
Guide complete
WhatsApp’s Erlang Gateway At 2M+ Connections
Generate a follow-up sub-lesson on any aspect of this topic
WhatsApp’s Erlang Gateway At 2M+ Connections
TLDR;
- Two million persistent TCP sessions turns file descriptors and socket memory into first-class capacity limits.
- At scale, contention in schedulers, ports, and kernel locks shows up before CPU saturation.
- Heartbeat cadence plus jitter decides whether a network flap is a blip or a reconnect storm.
A gateway that keeps one long lived TCP socket per phone looks simple until the connection count is the constraint. At WhatsApp, the publicly reported goal was reaching on the order of 1M connections per server and then pushing to 2M and beyond on FreeBSD with Erlang on the BEAM VM, which forces every design choice to answer the same question. What is the per connection cost in kernel state, BEAM state, and background traffic, and what happens when the network misbehaves.
At this scale, a naive design breaks in boring places. The process hits file descriptor ceilings, the kernel spends time in global locks, the runtime burns cycles scheduling tiny bits of work across many runnable entities, and routine health checks become packet floods if they synchronize. A single server that was stable at 200k connections can become brittle under overload or failure if it couples these costs together instead of partitioning them.
The core pattern is one TCP connection per device, accepted by a listener and handed to a per connection Erlang process that owns the socket through a BEAM port and the FreeBSD TCP stack. That mapping gives isolation, simple failure containment, and cheap concurrency in the runtime, but it means the system has to hold 2M socket control blocks, 2M file descriptors, and 2M mostly idle processes without letting background work dominate. The high level layout looks like this:
In that diagram, the key components are the client population, the TCP acceptor path, the per connection handler process, the BEAM VM, and the FreeBSD TCP stack. The important number is the per server concurrent connection count, which WhatsApp reported pushing from early bottlenecks around ~425k to 1M and then to 2M and even 2.8M at peak, which is why the gateway becomes an exercise in removing contention rather than adding more CPU.
Two million established TCP sessions implies a hard floor on kernel memory, descriptor limits, and packet processing headroom even when almost no one is sending messages. The operational trick is treating sockets like a capacity planned resource, not an implementation detail. The server configuration WhatsApp described had dual NICs, many cores, and large RAM, which helps, but it does not remove the need to budget per socket buffers and ensure the system can service periodic keepalives and retransmits during loss. The arithmetic is where the real constraints show up:
The estimator ties together three ceilings that otherwise get debugged too late. File descriptors cap out unless kern.maxfiles and kern.maxfilesperproc are raised, socket memory balloons if default buffers are left large, and packet rate can spike during synchronized heartbeats or reconnects. WhatsApp reported peaks such as 571k packets per second at 2.8M connections, which is a reminder that the gateway is often packet bound during pathological events even when application throughput seems fine.
At 2M connections, the design leans on Erlang lightweight processes so each connection can be represented as an isolated state machine with its own mailbox. A lightweight process is BEAM managed concurrency with small per process heaps and reduction based preemptive scheduling rather than one OS thread per connection, which is what keeps the naive thread per socket approach from exploding. The question becomes where shared structures appear, because WhatsApp’s scaling work from 200k to 2M was largely contention fixes, including BEAM internal bottlenecks and app level partitioning. Here are the moving parts that determine when the system stops scaling:
The building blocks map to distinct failure signatures. Schedulers and their run queues show imbalance as runnable queues grow while CPU is still available, ports show lock contention when many sockets are active at once, and message queues show backpressure before you see CPU saturation. WhatsApp called out that initial bottlenecks appeared around 425k connections, and the path to 1M and then 2M was repeated contention removal, which is why monitoring statistics(message_queues) and scheduler utilization is more predictive than watching average CPU.
Rule of thumb If CPU is not pegged but latency and backlog rise, assume contention or queueing, not lack of cores.
The gateway spends much of its time moving bytes between the NIC, the TCP stack, and the runtime’s socket drivers, so the scalability question is whether event notification and socket ownership changes stay local or trigger global locks. FreeBSD uses kqueue for scalable eventing, and in this architecture the BEAM inet driver registers interest, wakes up on readability or writability, then hands work to the owning Erlang process. The point is less epoll versus kqueue and more how many lock acquisitions, time reads, and setsockopt calls occur per packet or per accept under heavy churn. This is the hot path at a glance:
In that flow, the lock and wakeup points that matter are the NIC interrupt and driver queues, the TCP hash and socket lookup, the kqueue filter list, the BEAM port lock, and any runtime wide locks taken for time or timers. WhatsApp specifically reported fixes such as backporting a TSC based timecounter to make gettimeofday(2) cheaper and reducing setopts calls in prim_inet:accept and inet:tcp_controlling_process, which is exactly the kind of change that removes a small per connection constant that becomes dominant at 2M.
Reaching 2M connections requires pushing kernel defaults out of the way so the system can even represent the workload. A sysctl is a runtime tunable kernel parameter on FreeBSD, and the important ones here are the limits that bound open files, socket objects, and lookup structures. WhatsApp called out obvious ceilings like kern.ipc.maxsockets, file descriptor limits, and explicitly sized the TCP hash with net.inet.tcp.tcphashsize=524288 to reduce collisions at high connection counts. These are the knobs that change the shape of the problem:
The scan is mainly about what each knob unlocks. Raising kern.maxfiles and kern.maxfilesperproc prevents descriptor exhaustion, raising kern.ipc.maxsockets prevents socket allocation failure, and increasing net.inet.tcp.tcphashsize reduces contention and CPU in connection lookup. WhatsApp also mentioned driver quirks such as issues with MSI X queue stalls and backporting the igb driver, which matters because a gateway that is packet rate sensitive can look like an application regression when it is really a driver queueing collapse.
Once the kernel can hold 2M sockets, the next budget is BEAM memory tied up in 2M connection processes. Each process carries a small heap and stack, plus a mailbox that can grow during bursts, and binaries used for network payloads are managed differently than small terms. A binary in Erlang is a byte container that can be reference counted and shared, which is good for avoiding copies but can pin memory if references linger across processes. WhatsApp emphasized allocator tuning such as preferring mseg over malloc and configuring allocator instances per scheduler with flags like +Mut 24, which are tactics for keeping allocation scalable and superpage friendly under many small allocations. The tradeoff looks like this under bursty traffic:
The tuning widget is really balancing two bad outcomes. Smaller per process heaps reduce the idle footprint across millions of processes, but they increase garbage collection frequency when a client sends a burst and the connection process has to allocate terms quickly. Larger heaps reduce GC frequency and tail latency during bursts, but they waste memory across idle sessions. WhatsApp’s allocator and GC related patches and flags exist because at 2M connections, even tiny inefficiencies in allocation paths and per scheduler memory management become visible as contention and latency spikes.
Garbage collection is normally a per process local event, which is why BEAM can scale, but at 2M connections the dangerous case is correlated activity. A network flap is a brief loss and recovery that makes many clients retransmit, reconnect, or resend heartbeats in the same window, which turns millions of mostly sleeping processes into runnable ones that allocate and enqueue messages at once. WhatsApp explicitly patched in GC throttling when the message queue is long, which is a recognition that mailbox growth is an early warning signal and that unbounded per process work during overload accelerates collapse. The failure mode looks like this when correlations line up:
The reveal is that the first symptom is not a single slow process, it is system wide scheduler pressure paired with rising longest queue and a growing population of non empty mailboxes. GC work rises because each process tries to keep up with its mailbox, and scheduler contention rises because all those processes become runnable together. WhatsApp’s broader lesson that scaling from 200k to 2M was about contention fixes fits this pattern, because the collapse is often a lock and queueing problem triggered by correlated client behavior, not a raw compute limit.
Design rule Any periodic client behavior must be jittered or it becomes a synchronized load generator.
A heartbeat is liveness traffic that keeps NAT mappings warm and detects dead peers, but at millions of clients it is also a packet rate generator and a reconnect trigger. A heartbeat here means an application level ping or keepalive exchange distinct from TCP keepalive, which the server uses to decide when to close state and when to prompt a reconnect. The resilient design goal is a cadence that meets detection requirements while ensuring that a single event like a router glitch does not cause everyone to reconnect at once, which WhatsApp experienced in the form of mass disconnect reconnect events that produced unstable cluster states. The options are easier to compare on the axes that matter:
In the comparison, aggressive heartbeats improve detection time but raise steady state packets per second and battery cost and they increase the risk that reconnection attempts synchronize. Relaxed heartbeats plus TCP keepalive reduce background traffic but lengthen failure detection and can let dead sockets linger. Adaptive heartbeats with jitter reduce synchronization risk and spread reconnect load, which directly targets the mass reconnect failure WhatsApp described after a backend router glitch, where the event itself was short but the induced behavior destabilized the system.
When the gateway approaches its limits, the fastest debugging loop is determining whether time is lost in the kernel, inside BEAM locks, or in application level coupling. WhatsApp leaned heavily on BEAM scheduler metrics, message queue statistics, and lock counting, calling lock counters invaluable, alongside OS level tools like DTrace and kernel lock counting. That combination is what lets you say this is a kqueue and driver issue, or this is port lock contention, or this is an application table or gen_server that became a funnel. The checkpoint below is the kind of triage that keeps scaling work from becoming folklore:
The metrics that matter are the scheduler utilization and run queue behavior, the distribution and process message queue lengths, and lock counters broken down by location. If scheduler utilization is high and balanced, but lock wait counts spike, contention is the limiter. If longest message queues climb while CPU is still available, a small number of processes is coupling work and needs partitioning. This is consistent with WhatsApp’s report that most of the work from 200k to 2M was contention removal, with fixes ranging from runtime patches to application level partitioning of ETS and Mnesia access.
A gateway that holds millions of persistent sockets is a budgeting exercise across kernel objects, runtime memory, and background traffic, and the defensible decisions are the ones that remove shared hot spots before adding complexity. The system stays stable when it partitions work to avoid single process funnels, keeps per socket and per process memory small enough to fit in RAM with headroom, and uses heartbeat jitter and backoff so client behavior does not synchronize during flaps. The most important lesson from WhatsApp’s reports is that raising the connection count is often easier than keeping the system resilient during overload and disruptions, which is why they set resilience as a core performance goal alongside 1M connections per server.
A redesign is forced when the per connection model stops being the right abstraction, not when a single tuning knob runs out. If regulatory or product requirements demand much faster dead peer detection, heartbeat traffic can dominate and the architecture may need regional fanout or protocol changes. If mobile networks become more bursty and reconnect rates rise, the gateway may need explicit admission control and staged reconnect, not just more schedulers. The pattern still starts with one socket per phone, but the trigger for change is when correlated client behavior makes the steady state assumptions false.