How do I detect and prevent goroutine leaks in a long-running Go service?

Short answer: a steadily climbing count means goroutines blocked forever on a channel/lock/IO with no cancellation path. Measure and dump first, then fix the spawn site’s contract.

The instinct is to blame the growing count on load, but a leak has a signature that a spike doesn’t: it never comes back down. Separate those two before you touch any code, because they need completely different fixes.

1. Confirm the count first. Export runtime.NumGoroutine() as a metric. If it grows monotonically with load and never recedes even at idle, it’s a leak, not a spike. That distinction is the foundation of the whole diagnosis.
2. The pprof goroutine dump is the smoking gun. Wire up net/http/pprof and grab /debug/pprof/goroutine?debug=2 (raw stacks) or debug=1 (aggregated). The stack with thousands of copies, all parked on the same channel recv/send or select, is exactly the leak site.
3. The root cause is almost always missing cancellation. A goroutine writing to a channel whose reader is gone, a for range ch that never closes, or an HTTP call with no context/timeout. Every goroutine needs an exit path: ctx.Done(), a closed channel, or bounded work.
4. Propagate context everywhere. Pass context.Context down, honor it in select, and put timeouts on every network call (http.Client.Timeout, a query context for the DB). A goroutine parked on a stuck network read leaks until the process dies.
5. Test for leaks. Use go.uber.org/goleak in TestMain so a test fails if goroutines outlive it. That’s the cheapest way to catch new leaks per spawn site, before they reach production.
6. Bound your concurrency. Instead of an unbounded go func() per event, use a worker pool / semaphore. A leak in an unbounded spawn model has no ceiling; a pool caps the damage from the start. It also makes the pprof dump far easier to read, because the parked goroutines cluster at a handful of named workers instead of thousands of anonymous closures.

func worker(ctx context.Context, jobs <-chan Job) {
    for {
        select {
        case <-ctx.Done(): // an exit path for every goroutine
            return
        case j, ok := <-jobs:
            if !ok {
                return
            }
            process(ctx, j)
        }
    }
}

# Find the parked stack at peak in production:
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' | less

Bottom line: personally I’d wire NumGoroutine() onto a dashboard and attach the pprof endpoint right away; take a goroutine?debug=2 dump at peak, find the parked stack with thousands of copies, and make that spawn site honor ctx.Done(). Then I’d add goleak so it never regresses. A leak isn’t a “mystery” — the dump points you at the exact line.