Go SSE fan-out baseline: 7.5M events/s, then it collapses

Source code: ikouchiha47/gothun — src/likes/

the problem is different now

Chapters 1-12 chased raw RPS on a static JSON endpoint. The bottlenecks were all in the network path: IRQ affinity, NIC saturation, prefork, compression. The server did the same work every request.

Fan-out is a different shape. One POST /like/post_42 needs to reach every viewer currently watching post_42. If 10,000 viewers are watching, one write becomes 10,000 SSE events. The metric is no longer requests/second — it’s events delivered per second, and the ceiling is wherever the broadcast loop breaks.

This is the likes system: users watch live streams, hit like, and the count updates in real time for everyone watching.

what we built

POST /like/:postID
  → increment counter (sharded atomic, LRU dedup)
  → hub.Publish(postID, newCount)
  → topic goroutine reads from broadcast channel
  → iterates subscriber slice, sends to each sub.Ch
  → ServeSSE drains sub.Ch → writes SSE event → flush

Each postID is a topic. Each topic has:

• One buffered broadcast channel (64 slots)
• One run() goroutine reading from it
• A []*Subscriber slice — one entry per open SSE connection

type topic struct {
    broadcast chan int64
    mu        sync.RWMutex
    subs      []*Subscriber
}

func (t *topic) run() {
    for val := range t.broadcast {
        t.mu.RLock()
        targets := t.subs
        t.mu.RUnlock()
        for _, sub := range targets {
            select {
            case sub.Ch <- val:
            case <-sub.Done:
            default: // slow reader — drop, not block
            }
        }
    }
}

Slow reader policy: drop, not block. A viewer with a slow connection cannot stall the broadcast loop for everyone else. Like counts are not a reliable stream — the client only needs the latest value, not every intermediate count.

Each Subscriber is owned entirely by its ServeSSE goroutine — it creates the channels, reads from Ch, and closes Done when the connection ends. The hub only stores the reference. This avoids the classic “send on closed channel” panic: the hub never touches channel lifecycle.

setup

• Server: c6in.8xlarge (32 vCPU, 64GB, 50 Gbps) — single process, prefork disabled (write path and SSE hub must share memory)
• Writers: like-writer — 500 goroutines, 10 posts round-robin, fire-and-forget POST
• SSE clients: sse-load-client — split across 2× c8gn.large (Graviton4 arm64, 2 vCPU each). Each client machine is limited to ~32K outbound ports, so two clients cover 64K connections
• Delivery%: events_received / (writes × subs_per_post) × 100 — measured client-side by comparing received SSE events against server’s increments_total

Ramp: 2K → 10K → 20K → 64K SSE connections, 30s measurement window per phase.

results

what this means

Events/s peaks at 20K connections (~7.5M/s) then collapses. At 20K connections across 10 posts, each topic has 2K subscribers. The run() goroutine can iterate 2K subscriber channels in well under 1ms — it keeps up. At 64K (6.4K subs/topic), iteration takes longer than the inter-event gap. The broadcast channel fills. Events back up. Delivery falls to 5%.

Write throughput collapses under SSE load. With no SSE readers, the write path peaks at ~80K inc/s. At 2K SSE connections it’s 24K/s. At 64K it’s 8K/s. The fan-out goroutines and the write goroutines are competing for CPU cores on the same non-prefork process. The OS scheduler doesn’t know that fan-out is less important than writes — it gives them equal time.

~47KB per SSE connection. (3128 - 626) MB / 62K ≈ 40KB. Each connection holds: a net/http goroutine (~8KB stack minimum), a Subscriber struct with two buffered channels, and TCP read/write buffers. This matches expectation.

Delivery% and events/s are different bottlenecks. Even at 20K connections where events/s peaks, delivery is only 33%. The 7.5M events/s figure means the fan-out loop is working hard — but a lot of those sends are dropping into the default case because subscriber channels are full. The subscriber channel (8 slots) fills when the SSE write can’t keep up with the broadcast rate.

Next: shard each topic’s fan-out across N goroutines. If delivery% improves, iteration was the limit. If it doesn’t, it was CPU contention all along.