All articles

Go's stuck goroutine problem (and why to let it fail)

Go can't kill a stuck goroutine, but River v0.40's new stuck-job handler can expose a goroutine stuck in an indefinite wait, preserve worker capacity, and trigger a restart before stuck jobs exhaust a process.

Go’s goroutines and channels are a true delight. In about a minute, with two or three lines of code and no package imports, you can have a demo running with concurrent goroutines passing values back and forth. There’s no more better word for it than “fun”, and certainly a far cry from ecosystems where getting started means first choosing which third-party packages to bring in or trying to figure out which async runtime you should be using.

But even though they’re well designed, they don’t protect programmers from every possible form of misuse. Take the most basic channel receive:

func receive(resultChan <-chan string) string {
    // Blocks until a value arrives or the channel is closed.
    return <-resultChan
}

As long as a value eventually arrives or the channel is closed, the code returns. However, if the sender has disappeared (say it returned early because of a logic bug) and no other goroutine can send or close the channel, the receive waits forever. For the purposes of this article, we’ll call a goroutine in this state “stuck”.

On an open channel, the mirror image is a send on an unbuffered channel with no receiver, or on a buffered channel whose buffer is full:

func send(resultChan chan<- string, result string) {
    // On an open channel, blocks until a receiver or buffer space is available.
    resultChan <- result
}

That sounds like an obvious pitfall to avoid, but an ordering bug can create the situation indirectly. Here, a non-blocking receive uses default and returns if no value is ready at that instant:

func receive(resultChan <-chan string) {
    select {
    case result := <-resultChan:
        fmt.Println(result)
    default:
        // Return immediately when no value is ready.
        return
    }
}

With an unbuffered channel, a later sender then waits forever because its only receiver has disappeared.

There’s a few other cases that can also make a goroutine stuck that we’ll skip code samples for, but are all mistakes that are relatively easy to make:

  • Sending or receiving on a nil channel (say one accidentally left uninitialized).
  • Ranging over a channel (range resultsChan) where the source channel is never closed, never allowing the loop to return.
  • Waiting to lock a mutex that’s never unlocked.

Stuck vs. non-cancellable

A closely related problem is non-cancellable code, which often leaves goroutines stuck. Consider the receive example again:

func receive(resultChan <-chan string) string {
    return <-resultChan
}

Suppose a goroutine is waiting here when its context is cancelled. A cancelled context doesn’t interrupt Go code automatically or raise an exception that changes control flow. Instead, anywhere that code might otherwise block, it must react explicitly by observing the context. In the example above, even if a value will eventually arrive on resultChan, the goroutine can’t return promptly in response to cancellation because the context is ignored.

Keeping code cancellable generally involves using a select in places that would otherwise perform a blocking send or receive:

func receive(ctx context.Context, resultChan <-chan string) (string, error) {
    select {
    case result := <-resultChan:
        return result, nil
    case <-ctx.Done():
        return "", ctx.Err()
    }
}

If no value arrives, cancelling the context makes the ctx.Done() case ready and lets the function return.

Cancellation is important for background job libraries like River. When a job times out, or when a hard shutdown cancels active work, River cancels the job’s context and expects worker code to return in a timely manner.

Production impact

Goroutines in Go are cheap — starting with a stack of 2 kB and another 600 bytes or so of bookkeeping — but leaving enough of them permanently stuck is still going to have a detrimental effect on an application. And although system allocation is minimal, user code is more than likely to have allocated considerably more resources that a stuck goroutine will retain.

A particular danger is code that starts a goroutine that will get stuck in a loop, or a type of background job that gets regularly stuck and eventually comes to saturate its queue. This can bring an entire application grinding to a halt as there are no more free slots for non-stuck jobs to occupy.

Uninterruptible by design

Go exposes no API for forcibly terminating another goroutine. Careful cancellation design can prevent most problems, but if a goroutine has no cooperative path out, removing it requires terminating the process.

This language design seems annoying at first glance, but there are good reasons for it. A goroutine might hold locks or shared state that would make it unsafe to kill. Forced termination could also skip its defer calls, leaving resources unreleased.

This is why River cancels work contexts when jobs time out or active work must stop. Go doesn’t let you interrupt a goroutine, but it does provide cooperative cancellation, and contexts are the conventional way to carry that signal.

River v0.40’s stuck handler

Bugs still happen. River assumes that jobs will normally return when their contexts are cancelled, but it provides tools for the cases that don’t.

Before v0.40, River detected jobs that exceeded their timeout and hadn’t returned within a configurable margin after cancellation. Its response was minimal: it logged the anomaly and continued as usual. This was better than nothing, but users had to comb their logs to notice it, making it easy to miss.

River v0.40 adds a new capability to help: JobStuckHandler.

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    JobStuckHandler: func(ctx context.Context, params river.JobStuckHandlerParams) river.JobStuckHandlerResult {
        slog.ErrorContext(ctx, "River job is stuck",
            "job_id", params.ID,
            "job_kind", params.Kind,
            "queue", params.Queue,
            "total_stuck_jobs", params.TotalStuckJobs,
        )
        return river.JobStuckHandlerResult{}
    },
})

The first question is why the job got stuck. The handler’s a good place to capture a complete runtime stack dump:

riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    JobStuckHandler: func(ctx context.Context, params river.JobStuckHandlerParams) river.JobStuckHandlerResult {
        slog.ErrorContext(ctx, "River job is stuck",
            "job_id", params.ID,
            "job_kind", params.Kind,
            "queue", params.Queue,
            "goroutine_dump", captureGoroutineDump(),
        )


        return river.JobStuckHandlerResult{}
    },
})


func captureGoroutineDump() string {
    buf := make([]byte, 1 << 20) // 1 MiB
    for {
        n := runtime.Stack(buf, true)
        if n < len(buf) {
            return string(buf[:n])
        }


        buf = make([]byte, 2 * len(buf))
    }
}

An operator can inspect the logs later to see where the worker was stuck and use that as a clue to the cause. Go has no supported API for requesting only one specific other goroutine’s stack by identity, so we have no choice here but to use runtime.Stack with to capture the stacks of every goroutine.

Because the process can’t forcibly remove these goroutines, we recommend using the handler’s TotalStuckJobs field (which counts all jobs currently considered stuck across the client) in conjunction with a “let it crash” philosophy:

const maxStuckJobsBeforeRestart = 10


riverClient, err := river.NewClient(riverpgxv5.New(dbPool), &river.Config{
    JobStuckHandler: func(_ context.Context, params river.JobStuckHandlerParams) river.JobStuckHandlerResult {
        if params.TotalStuckJobs > maxStuckJobsBeforeRestart {
            panic(fmt.Sprintf("too many stuck River jobs: %d", params.TotalStuckJobs))
        }


        return river.JobStuckHandlerResult{}
    },
})

The “let it crash” philosophy, popularized by Erlang, favors failing fast when a process reaches an unexpected state and relying on supervision to restore a pristine version. The philosophy calls for terminating and restarting a misbehaving process rather than layering rarely run and poorly vetted recovery logic into it. Go doesn’t have Erlang/OTP’s expansive facilities with built-in supervisors and process trees, but we get a poor man’s version using OS-level processes and a service manager like systemd or Kubernetes to keep them alive.

In River’s case, an unrecovered panic terminates the process, and the supervisor brings it back without the stuck goroutines. This is a blunt instrument. Healthy in-flight work is interrupted too, so it works, but should be avoided if possible. Jobs left in the running state are eventually found by River’s job rescuer and requeued for execution.

Replacing worker slots

Each running River job occupies a worker slot and executes in its own goroutine. MaxWorkers bounds the concurrent jobs for a queue within one client process:

Queues: map[string]river.QueueConfig{
    river.QueueDefault: {MaxWorkers: 100},
},

A stuck job continues to occupy its worker slot, reducing the process’s concurrency. If enough jobs stop returning, the queue can be left with few or no slots available for new work.

To counter this loss of capacity, JobStuckHandlerResult provides an AddWorkerSlot field. Setting it to true tells River to treat the stuck execution as no longer occupying its slot, letting another job start:

JobStuckHandler: func(_ context.Context, params river.JobStuckHandlerParams) river.JobStuckHandlerResult {
    return river.JobStuckHandlerResult{
        // Tell River to replace the slot occupied by this stuck job.
        AddWorkerSlot: true,
    }
},

Of course, the replacement isn’t free. The original goroutine still consumes memory and may hold other resources, so AddWorkerSlot needs to be used sparingly and with care. We recommend combining both policies — replace worker slots up to a limit, then crash once it’s exceeded:

const maxStuckJobsBeforeRestart = 10


JobStuckHandler: func(_ context.Context, params river.JobStuckHandlerParams) river.JobStuckHandlerResult {
    if params.TotalStuckJobs > maxStuckJobsBeforeRestart {
        panic(fmt.Sprintf("too many stuck River jobs: %d", params.TotalStuckJobs))
    }


    return river.JobStuckHandlerResult{
        // Tell River to replace the slot occupied by this stuck job.
        AddWorkerSlot: true,
    }
},

Detect, preserve, restart

Three takeaways:

  • Go can’t forcibly terminate a goroutine, so make potentially blocking channel operations context-aware.
  • River v0.40’s JobStuckHandler can log job details and goroutine stacks when work ignores cancellation.
  • AddWorkerSlot preserves capacity temporarily; pair it with a stuck-job limit and a supervised process restart.

A well-behaved Go program should never experience a stuck job. Even so, implementing JobStuckHandler takes only a minute or two and provides a cheap hedge against rare bugs that could otherwise degrade production.