Normally, errored jobs are retried until their maximum number of allowed attempts, after which they become discarded and after 7 days are reaped by the job cleaner. Removing discarded jobs like this isn’t always desirable because a job that failed until it was discarded may represent a fundamental problem that developers may want to address, and losing it might mean losing state. However, keeping them in the jobs table indefinitely is also problematic because having them accumulate forever can cause trouble for live operations as discarded jobs bloat the table.
The dead letter queue is a compromise. Instead of deleting discarded jobs, they’re moved to a separate table (river_job_dead_letter) for long term retention. The dead letter table is smaller on a per-job basis because it has fewer default indexes, and it keeps the database healthier because it has little operational load.
The dead letter queue is a feature of River Pro ✨. If you haven’t yet, install River Pro and run the pro migration line.
Basic usage
Added in River Pro for Go v0.16.0.
The dead letter queue is off by default. With the dead letter queue off, discarded jobs are retained in river_job for river.Config.DiscardedJobRetentionPeriod (default 7 days) before they’re deleted permanently.
It’s enabled by setting riverpro.Config.DeadLetter.Enabled to true. With the dead letter queue on, discarded jobs are still retained in river_job for river.Config.DiscardedJobRetentionPeriod (default 7 days), but are then moved to the dead letter table river_job_dead_letter instead of being deleted. Dead letter jobs remain in river_job_dead_letter indefinitely until an operator removes them manually.
An example of a Pro client with the dead letter queue on:
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ // The standard client's DiscardedJobRetentionPeriod setting // dictates time before discarded jobs are made dead letter. // If omitted, defaults to 7 days. DiscardedJobRetentionPeriod: 7 * 24 * time.Hour,
Queues: map[string]river.QueueConfig{ river.QueueDefault: {MaxWorkers: 100}, }, Workers: workers, }, DeadLetter: riverpro.DeadLetterConfig{ Enabled: true, },})if err != nil { // handle error}The dead letter queue is off by default. Enable it with dead_letter: true on River::Pro::Config. Discarded jobs move to river_job_dead_letter after the core discarded_job_retention_period, which defaults to seven days:
client = River::Pro::Client.new(driver, config: River::Pro::Config.new( core: River::Config.new( discarded_job_retention_period: 7 * 24 * 3600, queues: {default: 100}, workers: workers ), dead_letter: true ))Durations are in seconds. With the dead letter queue enabled, the core retention setting applies to discarded jobs in every queue, including queues with a per-queue override. Dead letter jobs remain indefinitely until retried or removed by an operator.
Enable the dead letter queue with .dead_letter() on the Pro client builder. The core maintenance settings determine how long discarded jobs stay in the live table:
use riverqueue::{MaintenanceConfig, QueueConfig, Retention, database::PostgresDatabase};use riverqueue_pro::ProClient;use std::time::Duration;
let client = ProClient::builder(PostgresDatabase::new(pool)) .dead_letter() .maintenance(MaintenanceConfig::default().with_discarded_job_retention( Retention::DeleteAfter(Duration::from_secs(7 * 24 * 60 * 60)), )) .queue("default", QueueConfig::new(100)) .workers(workers) .build()?;The default discarded-job retention is seven days. With dead lettering enabled, it applies to every queue regardless of per-queue discarded retention overrides. Dead letter rows remain until retried or removed.
Retrying dead letter jobs
Dead letter jobs are retried with the Pro client’s Go API:
deadLetterJobID := 123
insertRes, err := riverClient.DeadLetterJobRetry(ctx, deadLetterJobID)if err != nil { // handle error}The job is moved from river_job_dead_letter back to river_job, retaining its kind, args, maximum attempts, priority, and queue, but with its state set back to available and its attempt number and errors reset so that it’s ready to be worked again.
Like with most River functions, there’s a transactional variant of the same function:
insertRes, err := riverClient.DeadLetterJobRetryTx(ctx, tx, deadLetterJobID)if err != nil { // handle error}Retry a dead letter job with job_dead_letter_retry:
dead_letter_job_id = 123job = client.job_dead_letter_retry(dead_letter_job_id)The operation atomically removes the dead letter record and inserts a replacement through insertion plugins, with a new ID and reset attempts and errors. It preserves the job’s kind, args, maximum attempts, priority, queue, and grouping metadata. The replacement is available, including for sequence and workflow jobs; retry these deliberately because this bypasses their normal pending state.
To include the retry in an application transaction, use the driver’s transaction block. With ActiveRecord:
ActiveRecord::Base.transaction do job = client.job_dead_letter_retry(dead_letter_job_id)endWith Sequel, use DB.transaction on the database passed to the driver. The retry raises River::NotFoundError for an unknown ID; a failed insertion leaves the dead letter intact.
Use client.dead_letter().retry(id) to atomically remove the dead letter row and reinsert the job:
let retried = client.dead_letter().retry(123).await?;println!("Replacement job: {}", retried.job.id);The replacement has a new ID, reset attempts and errors, and preserves the original arguments, kind, queue, priority, maximum attempts, and grouping metadata. Insert hooks and middleware run again. Retrying sequence or workflow jobs makes them available immediately, bypassing their normal pending state.
Unique-job constraints still apply. If a matching live job prevents insertion, retried.unique_skipped_as_duplicate is true and retried.job is that existing job; the dead letter row is still removed.
Attach a transaction to include the retry in an application transaction:
let mut tx = pool.begin().await?;let retried = client.dead_letter().retry(123).tx(&mut tx).await?;tx.commit().await?;An unknown ID returns riverqueue_pro::Error::NotFound. A failed standalone retry rolls back and leaves the dead letter row intact; roll back your own transaction on failure. Inspect retained jobs with client.dead_letter().get(id).await? or page through them with client.dead_letter().list(DeadLetterListParams::new().max(100)).await?.