How do I detect and prevent goroutine leaks in a long-running Go service?

Short answer: a steadily climbing count means goroutines blocked forever on a channel/lock/IO with no cancellation path.

Short answer

Measure and dump first, then fix the spawn site’s contract. The instinct is to blame the growing count on load, but a leak has a signature that a spike doesn’t: it never comes back down. Separate those two before you touch any code, because they need completely different fixes. I laid out the channel and select contract underneath all of this in the Go concurrency post; a leak is what shows up where that contract isn’t held.

Why

1. The root cause is almost always missing cancellation. A goroutine writing to a channel whose reader is gone, a for range ch that never closes, or an HTTP call with no context/timeout. Every goroutine needs an exit path: ctx.Done(), a closed channel, or bounded work.
2. A stuck network read holds a goroutine until the process dies. When context isn’t propagated down and network calls carry no timeout, the select never returns; the goroutine parks and stays there.
3. An unbounded spawn model gives the leak no ceiling. A service that fires an unbounded go func() per event hits no upper limit once a leak starts — the count just keeps climbing with load.

What to do

1. Confirm the count first. Export runtime.NumGoroutine() as a metric. If it grows monotonically with load and never recedes even at idle, it’s a leak, not a spike. That distinction is the foundation of the whole diagnosis.

2. The pprof goroutine dump is the smoking gun. Wire up net/http/pprof and grab /debug/pprof/goroutine?debug=2 (raw stacks) or debug=1 (aggregated). The stack with thousands of copies, all parked on the same channel recv/send or select, is exactly the leak site.

   # Find the parked stack at peak in production:
   curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' | less

3. Propagate context everywhere. Pass context.Context down, honor it in select, and put timeouts on every network call (http.Client.Timeout, a query context for the DB). A goroutine parked on a stuck network read leaks until the process dies.

   func worker(ctx context.Context, jobs <-chan Job) {
       for {
           select {
           case <-ctx.Done(): // an exit path for every goroutine
               return
           case j, ok := <-jobs:
               if !ok {
                   return
               }
               process(ctx, j)
           }
       }
   }

4. Test for leaks. Use go.uber.org/goleak in TestMain so a test fails if goroutines outlive it. That’s the cheapest way to catch new leaks per spawn site, before they reach production.

5. Bound your concurrency. Instead of an unbounded go func() per event, use a worker pool / semaphore. A leak in an unbounded spawn model has no ceiling; a pool caps the damage from the start. It also makes the pprof dump far easier to read, because the parked goroutines cluster at a handful of named workers instead of thousands of anonymous closures.

Bottom line: personally I’d wire NumGoroutine() onto a dashboard and attach the pprof endpoint right away; take a goroutine?debug=2 dump at peak, find the parked stack with thousands of copies, and make that spawn site honor ctx.Done(). Then I’d add goleak so it never regresses. A leak isn’t a “mystery” — the dump points you at the exact line.