How do I cut GC pressure in Go with sync.Pool and escape analysis under load?

Short answer: GC pauses are a symptom; the real enemy is allocation rate. Cut allocations first and GC relaxes on its own.

Short answer

Your spikes aren’t because GC is “bad” — 50k packets per second means hundreds of thousands of short-lived objects per second, so GC runs constantly and aggressively to collect that garbage, and the Stop-the-World pauses bleed into your response times. The fix isn’t to disable GC, it’s to give it less work. I covered the concurrency side of this path in concurrency in Go: goroutines and channels in practice.

Why

1. Pause frequency follows allocation frequency. The more garbage you produce, the more often GC runs; changing its settings doesn’t change your production rate.

2. The cheapest allocation is the one you never make. If a value can stay on the stack, GC never sees the object at all — cheaper than pooling it.

3. Pooling can backfire. Pooling objects that hold pointers to other heap data keeps that large graph alive too, and memory swells more than you expect.

What to do

1. Don’t fly blind: measure before/after with pprof. The alloc_space profile tells you the line allocating the most; GODEBUG=gctrace=1 shows GC frequency and pause duration. Measure under the same load before and after.

2. Cut allocations at the root. Pre-size slices/maps with make([]T, 0, n) when you know the count; reuse bytes.Buffer and the JSON decoder; avoid needless []byte↔string copies. A single copy multiplied 50k times becomes a mountain.

3. Run escape analysis, keep it on the stack. The go build -gcflags=-m output tells you which values escape to the heap. Returning a pointer out of a function or boxing a value into an interface causes most escapes.

4. Reuse on the hot path with sync.Pool. Take short-lived buffers, structs and decoders from a pool and return them. Reset the object before you Put it so stale data doesn’t leak. Keep only flat, self-contained buffers/structs in the pool.

5. Leave GOGC/GOMEMLIMIT for last. They make GC less frequent but won’t fix a path allocating 50k times a second; final tuning, not the first move.

Bottom line: the order is clear — profile → kill the top allocator → pool the survivors → then tune GOGC/GOMEMLIMIT. The real win comes from producing no garbage on the hottest path at all.