Durable periodic jobs are the same as River’s normal periodic jobs, except that they have their next run times persisted to the database to provide more robust, predictable scheduling. Unlike non-durable periodic jobs, run times are guaranteed (or close to guaranteed) even across restarts, crashes, or leader elections.
Durable periodic jobs are a feature of River Pro ✨. If you haven’t yet, install River Pro and run its migrations.
Usage
Added in River Pro for Go v0.15.0.
Durable periodic jobs largely use the same API as standard periodic jobs, but are activated with a pair of extra options:
- In a Pro client’s
riverpro.Config,DurablePeriodicJobs.Enabledshould be set to true. - Periodic jobs intended to be durable should be given an
IDproperty throughPeriodicJobOptsto uniquely identify them.
For example:
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ PeriodicJobs: []*river.PeriodicJob{ river.NewPeriodicJob( river.PeriodicInterval(15*time.Minute), func() (river.JobArgs, *river.InsertOpts) { return PeriodicJobArgs{}, nil }, &river.PeriodicJobOpts{ // (2) jobs intended to be durable are assigned a unique ID ID: "my_periodic_job", }, ), }, ... }, DurablePeriodicJobs: riverpro.DurablePeriodicJobsConfig{ // (1) `DurablePeriodicJobs.Enabled` must be set to true for any // periodic jobs to be activated Enabled: true, },})Notably, the API enables mixing of durable and non-durable periodic jobs. Even when DurablePeriodicJobs.Enabled is true for the entire client, only periodic jobs with an ID are made durable (those without behave like normal periodic jobs).
Use the core River::PeriodicJob API, give each durable schedule a stable id, and set durable_periodic_jobs: true on River::Pro::Config:
periodic = River::PeriodicJob.new( id: "my_periodic_job", schedule: River::PeriodicInterval.new(15 * 60), constructor: -> { PeriodicJobArgs.new })
client = River::Pro::Client.new(driver, config: River::Pro::Config.new( core: River::Config.new( periodic_jobs: [periodic], queues: {default: 100}, workers: workers ), durable_periodic_jobs: true ))Durations are in seconds. Schedules without an id keep the normal in-memory behavior, so durable and non-durable schedules can coexist. A constructor can return args, [args, insert_opts], or nil to skip an occurrence.
Register a DurablePeriodicJob with a stable ID and enable persistence on the Pro client:
use riverqueue::{IntervalSchedule, QueueConfig, database::PostgresDatabase};use riverqueue_pro::{DurablePeriodicConfig, DurablePeriodicJob, ProClient};use std::time::Duration;
let periodic = DurablePeriodicJob::new( "my_periodic_job", IntervalSchedule::new(Duration::from_secs(15 * 60))?, || PeriodicJobArgs {},)?;let client = ProClient::builder(PostgresDatabase::new(pool)) .durable_periodic_config(DurablePeriodicConfig::default().with_enabled(true)) .durable_periodic_job(periodic) .queue("default", QueueConfig::new(100)) .workers(workers) .build()?;Define PeriodicJobArgs and register its worker as for ordinary periodic jobs. Every possible leader must register the same ID and schedule. Without with_enabled(true), these registrations run as ordinary in-memory schedules. Registrations made with the core .periodic_job(...) method remain non-durable.
Use client.durable_periodic_jobs().add(periodic)? and .remove_by_id("my_periodic_job") to change this client’s registrations while it runs.
ID required to make jobs durable
Every durable schedule needs a stable ID that uniquely identifies it in the database. Use the same ID and schedule on every client that may become leader.
Implementation notes
Noteworthy details on the implementation of durable periodic jobs:
Each job gets a record tracking its next run time in the
river_periodic_jobtable, which is how state persists across restarts. This table is raised as part of River Pro’spromigration line.Durable periodic jobs are assigned IDs to track them in
river_periodic_jobuniquely. These IDs are orthogonal to the “kind” of individual jobs (which is also a form of identifier) so that it’s possible to configure multiple periodic jobs that insert the same job kind. This might be useful in case different periods should insert with different job parameters.
Running on start
Periodic jobs have a PeriodicJobOpts.RunOnStart option that largely exists so that non-durable periodic jobs that are on long run schedules get some opportunity to run occasionally. Without it, they’d otherwise be frozen out if a client restarts more frequently than a job’s run interval.
Use of RunOnStart generally isn’t necessary for durable periodic jobs because their schedules are tracked across restarts. However, RunOnStart still works for them and will cause those jobs to run on client start or leader election despite their next persisted run time. Users may want to take care to remove its use for any durable periodic jobs they configure.
River::PeriodicJob accepts run_on_start: true to insert immediately on startup or leader election. This is useful for non-durable schedules that might otherwise restart before their interval elapses.
Durable jobs retain their next run time across restarts, so they generally do not need run_on_start. Leave it at its default false unless an extra run on startup is intentional.
DurablePeriodicJob::with_options accepts DurablePeriodicJobOpts::default().with_run_on_start(true) to insert on startup or leader election. The default is false.
A durable schedule already preserves its next run across restarts, so leave this disabled unless the extra run is intentional.
Running on start is usually unnecessary
Durable periodic jobs retain their next run time across restarts. Leave running on start disabled unless an extra run on startup or leader election is intentional.
Changing the run schedule of an existing job
Unlike non-durable jobs, changing a periodic job’s run interval will have no immediate effect because a next run time was already stored in the database from the original schedule. For example, if a job was initially scheduled to run once a year but is then rescheduled to run once an hour, an initial next run that’s a year from now is still in the database, and still authoritative.
A new schedule can be assigned by setting a new periodic job ID. Start with the initial schedule and ID:
PeriodicJobs: []*river.PeriodicJob{ river.NewPeriodicJob( river.PeriodicInterval(365*24*time.Hour), ... &river.PeriodicJobOpts{ ID: "my_periodic_job", }, ),},periodic = River::PeriodicJob.new( id: "my_periodic_job", schedule: River::PeriodicInterval.new(365 * 24 * 3600), constructor: -> { PeriodicJobArgs.new })use riverqueue::IntervalSchedule;use riverqueue_pro::DurablePeriodicJob;use std::time::Duration;
let periodic = DurablePeriodicJob::new( "my_periodic_job", IntervalSchedule::new(Duration::from_secs(365 * 24 * 60 * 60))?, || PeriodicJobArgs {},)?;Change the schedule, and while doing so put in a new unique ID for the periodic job. Any convention works, but an easy one is to append a version suffix like *_v2:
PeriodicJobs: []*river.PeriodicJob{ river.NewPeriodicJob( river.PeriodicInterval(1*time.Hour), ... &river.PeriodicJobOpts{ ID: "my_periodic_job_v2", }, ),},periodic = River::PeriodicJob.new( id: "my_periodic_job_v2", schedule: River::PeriodicInterval.new(3600), constructor: -> { PeriodicJobArgs.new })use riverqueue::IntervalSchedule;use riverqueue_pro::DurablePeriodicJob;use std::time::Duration;
let periodic = DurablePeriodicJob::new( "my_periodic_job_v2", IntervalSchedule::new(Duration::from_secs(60 * 60))?, || PeriodicJobArgs {},)?;With a periodic job no longer configured in Go code to run it, the original non-v2 periodic job record becomes orphaned, and that’s okay. After being unused by the periodic job service for a preset span of time (default 24 hours, see DurablePeriodicJobsConfig.StaleThreshold), the Pro client will clean it up automatically.
When the original ID is no longer configured, its database record becomes orphaned and is cleaned up after 24 hours without a keepalive. Customize this with River::Pro::DurablePeriodicJobsConfig.new(stale_threshold: ...), passed as the durable_periodic_jobs: value. The threshold is in seconds.
When no client keeps the original ID alive, its persisted record is removed after 24 hours by default. Customize that interval with DurablePeriodicConfig::default().with_enabled(true).with_stale_threshold(Duration::from_secs(...)), passed to .durable_periodic_config(...) on the builder.
If the old and new schedules are similar enough, this step isn’t necessary because the old schedule will only ever have any effect on a single future run time. As soon as that time is reached, the new schedule takes effect, so as long as the original run time wasn’t too far off in the future, the differing schedules won’t make a big difference. For example, if a job that was originally scheduled to run once every 30 minutes is rescheduled to once every 15 minutes, the next run will still be 30 minutes away, but as soon as that’s done, the new 15 minute schedule takes effect.