Normally, errored jobs are retried until their maximum number of allowed attempts, after which they become discarded and after 7 days are reaped by the job cleaner. Removing discarded jobs like this isn't always desirable because a job that failed until it was discarded may represent a fundamental problem that developers may want to address, and losing it might mean losing state. However, keeping them in the jobs table indefinitely is also problematic because having them accumulate forever can cause trouble for live operations as discarded jobs bloat the table.
The dead letter queue is a compromise. Instead of deleting discarded jobs, they're moved to a separate table (river_job_dead_letter) for long term retention. The dead letter table is smaller on a per-job basis because it has fewer default indexes, and it keeps the database healthier because it has little operational load.
The dead letter queue is a feature of River Pro ✨. If you haven't yet, install River Pro and run the pro migration line.
Added in River Pro v0.16.0.
Basic usage
The dead letter queue is off by default. With the dead letter queue off, discarded jobs are retained in river_job for river.Config.DiscardedJobRetentionPeriod (default 7 days) before they're deleted permanently.
It's enabled by setting riverpro.Config.DeadLetter.Enabled to true. With the dead letter queue on, discarded jobs are still retained in river_job for river.Config.DiscardedJobRetentionPeriod (default 7 days), but are then moved to the dead letter table river_job_dead_letter intead of being deleted. Dead letter jobs remain in river_job_dead_letter indefinitely until an operator removes them manually.
An example of a Pro client with the dead letter queue on:
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ // The standard client's DiscardedJobRetentionPeriod setting // dictates time before discarded jobs are made dead letter. // If omitted, defaults to 7 days. DiscardedJobRetentionPeriod: 7 * 24 * time.Hour,
Queues: map[string]river.QueueConfig{ river.QueueDefault: {MaxWorkers: 100}, }, Workers: workers, } DeadLetter: riverpro.DeadLetterConfig{ Enabled: true, },})if err != nil { // handle error}Retrying dead letter jobs
Dead letter jobs are retried with the Pro client's Go API:
deadLetterJobID := 123
insertRes, err := riverClient.DeadLetterJobRetry(ctx, deadLetterJobID)if err != nil { // handle error}The job is moved from river_job_dead_letter back to river_job, retaining its kind, args, maximum attempts, priority, and queue, but with its state set back to available and its attempt number and errors reset so that it's ready to be worked again.
Like with most River functions, there's a transactional variant of the same function:
insertRes, err := riverClient.DeadLetterJobRetryTx(ctx, tx, deadLetterJobID)if err != nil { // handle error}