Ephemeral jobs are jobs which are removed from the database immediately upon completion. Instead of being updated to a completed state and left to be eventually reaped by the job cleaner, they’re purged post-haste with a DELETE operation. This trades the observability that would’ve been available from a completed job row for improved operational robustness stemming from jobs being cycled out of the database more quickly.
Use of this feature is recommended to be judicious, reserving it for select, high-volume jobs which will benefit particularly from being removed expediently, while leaving most jobs as non-ephemeral so they follow their normal job lifecycle.
Ephemeral jobs are a feature of River Pro ✨. If you haven’t yet, install River Pro.
Basic usage
Added in River Pro for Go v0.16.0.
Make a job ephemeral by adding an implementation for EphemeralOpts() that returns riverpro.EphemeralOpts:
type MyEphemeralJobArgs struct{ Message string `json:"message"`}
func (a MyEphemeralJobArgs) Kind() string { return "my_ephemeral_job"}
func (a MyEphemeralJobArgs) EphemeralOpts() riverpro.EphemeralOpts { return riverpro.EphemeralOpts{}}Currently, EphemeralOpts has no properties, but is reserved for future options.
Implement ephemeral_opts on the argument class and return River::Pro::EphemeralOpts.new. Insert the job through a River::Pro::Client:
MyEphemeralJobArgs = Struct.new(:message, keyword_init: true) do def kind = "my_ephemeral_job" def to_json = JSON.generate(to_h) def ephemeral_opts = River::Pro::EphemeralOpts.newend
client.insert(MyEphemeralJobArgs.new(message: "still here"))EphemeralOpts currently has no settings. To make all jobs in a queue ephemeral, set ephemeral: true on its River::Pro::QueueConfig. Ephemeral jobs cannot be workflow tasks. Transactional completion with client.job_complete_tx(job) also deletes an ephemeral job when the transaction commits.
Set with_ephemeral(true) on the job type’s default insert options:
use riverqueue::{InsertOpts, JobArgs};use riverqueue_pro::ProInsertOptsExt as _;use serde::{Deserialize, Serialize};
#[derive(Deserialize, JobArgs, Serialize)]#[river(kind = "my_ephemeral_job", insert_opts = Self::insert_opts)]struct MyEphemeralJobArgs { message: String,}
impl MyEphemeralJobArgs { fn insert_opts() -> InsertOpts { InsertOpts::default().with_ephemeral(true) }}
client .insert(MyEphemeralJobArgs { message: "still here".into(), }) .await?;Use a ProClient for insertion and work. For one insertion, pass .opts(InsertOpts::default().with_ephemeral(true)) instead. To make an entire queue ephemeral, configure it with QueueProConfig::new(QueueConfig::new(100)).with_ephemeral(true).
Successful ephemeral jobs are deleted in the transaction that completes them. Cancelled and discarded jobs follow normal retention. Insertion rejects workflow tasks marked ephemeral or assigned to an ephemeral queue. Existing workflow tasks are retained with their workflow if a queue later becomes ephemeral.
Transitions to retryable and discarded
Ephemeral jobs are deleted immediately where they’d normally transition from running to completed, but other states behave normally. When an ephemeral job fails, it transitions to either retryable or discarded (depending on whether it’s exhausted its retry policy) like any non-ephemeral job would.
Operational advantages
In most cases use of ephemeral jobs won’t provide a huge advantage over non-ephemeral jobs, but they can be useful in high throughput situations:
Pages in Postgres B-tree indexes may split as new records are added, but when records are removed, they don’t recombine without a
REINDEX. Removing high volume job rows immediately leaves room in indexes for new jobs to be added, which may avoid page splits.River’s normal job removal involves doing work twice: once to complete a row from
runningtocompleted, and then with another pass to deletecompletedrows. This is a nominal amount of effort for most workloads, but it might matter in the presence of huge numbers of jobs.