Per-queue job retention

Normally River’s job cleaner uses global settings that determine how long it should retain cancelled, completed, and discarded jobs, but additional retention granularity is also available on a per-queue basis. This is useful in cases like where a hyper-frequent high-volume job should only be retained for a minimal amount of time to help keep the database’s size down, but a more important job related to account billing should be retained for an extended period for auditability.

Per-queue job retention is a feature of River Pro ✨. If you haven’t yet, install River Pro.

Added in River Pro for Go v0.16.0.

River’s standard (i.e. non-pro) behavior is that all jobs are cleaned according to global settings configurable through properties on River.Config:

Cancelled: CancelledJobRetentionPeriod, defaults to 24 hours.
Completed: CompletedJobRetentionPeriod, defaults to 24 hours.
Discarded: DiscardedJobRetentionPeriod, defaults to 7 days.

An example of how to configure these retention periods:

riverClient, err = river.NewClient(riverpropgxv5.New(dbPool), &river.Config{
    CancelledJobRetentionPeriod: 24 * time.Hour,
    CompletedJobRetentionPeriod: 24 * time.Hour,
    DiscardedJobRetentionPeriod: 7 * 24 * time.Hour,
    Queues: map[string]river.QueueConfig{
        river.QueueDefault: {
            MaxWorkers: 100,
        },
    },
    Workers:  workers,
})
if err != nil {
    // handle error
}

The core River::Config accepts retention periods in seconds. Cancelled and completed jobs default to 24 hours; discarded jobs default to seven days:

client = River::Client.new(driver,
  config: River::Config.new(
    cancelled_job_retention_period: 24 * 3600,
    completed_job_retention_period: 24 * 3600,
    discarded_job_retention_period: 7 * 24 * 3600,
    queues: {default: 100},
    workers: workers
  ))

Configuring per-queue retention

Go
Ruby

Per-queue retention requires a River Pro client. Retention settings have the same names as the global configuration, and are settable on each entry in the riverpro.Config.ProQueues map:

riverClient, err = riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{
    Config: river.Config{
        Workers:  workers,
    },
    ProQueues: map[string]riverpro.QueueConfig{
        "queue-hyper-frequent": { // retain a short time
            CancelledJobRetentionPeriod: 10 * time.Minute,
            CompletedJobRetentionPeriod: 10 * time.Minute,
            DiscardedJobRetentionPeriod: 24 * time.Hour,
            MaxWorkers:                  100,
        },
        "queue-long-term-retention": { // retain a long time
            CancelledJobRetentionPeriod: 7 * 24 * time.Hour,
            CompletedJobRetentionPeriod: 7 * 24 * time.Hour,
            DiscardedJobRetentionPeriod: 30 * 24 * time.Hour,
            MaxWorkers:                  100,
        },
    },
})
if err != nil {
    // handle error
}

Queues that aren’t present in ProQueues, or which don’t have custom retention settings (i.e. left unset and default to their zero values), will be cleaned according to global configuration the same way as any other job.

Set retention periods on entries in River::Pro::Config’s pro_queues:

client = River::Pro::Client.new(driver,
  config: River::Pro::Config.new(
    core: River::Config.new(workers: workers),
    pro_queues: {
      "queue-hyper-frequent" => River::Pro::QueueConfig.new(
        cancelled_job_retention_period: 10 * 60,
        completed_job_retention_period: 10 * 60,
        discarded_job_retention_period: 24 * 3600,
        max_workers: 100
      ),
      "queue-long-term-retention" => River::Pro::QueueConfig.new(
        cancelled_job_retention_period: 7 * 24 * 3600,
        completed_job_retention_period: 7 * 24 * 3600,
        discarded_job_retention_period: 30 * 24 * 3600,
        max_workers: 100
      )
    }
  ))

An unset (nil) queue retention period inherits the core setting. Set -1 to disable ordinary cleanup for a state. With dead_letter: true, discarded jobs move to the dead letter queue after the core retention period regardless of per-queue overrides. With workflow_aware_retention: true, workflow jobs are retained and cleaned as a whole workflow instead.