Durable periodic jobs

Durable periodic jobs are the same as River's normal periodic jobs, except that they have their next run times persisted to the database to provide more robust, predictable sheduling. Unlike non-durable periodic jobs, run times are guaranteed (or close to guaranteed) even across restarts, crashes, or leader elections.

Durable periodic jobs are a feature of River Pro ✨. If you haven't yet, install River Pro and run its migrations.

Added in River Pro v0.15.

Usage

Durable periodic jobs largely use the same API as standard periodic jobs, but are activated with a pair of extra options:

1. In a Pro client's riverpro.Config, DurablePeriodicJobs.Enabled should be set to true.
2. Periodic jobs intended to be durable should be given an ID property through PeriodicJobOpts to uniquely identify them.

For example:

riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{
    Config: river.Config{
        PeriodicJobs: []*river.PeriodicJob{
            river.NewPeriodicJob(
                river.PeriodicInterval(15*time.Minute),
                func() (river.JobArgs, *river.InsertOpts) {
                    return PeriodicJobArgs{}, nil
                },
                &river.PeriodicJobOpts{
                    // (2) jobs intended to be durable are assigned a unique ID
                    ID: "my_periodic_job",
                },
            ),
        },
        ...
    },
    DurablePeriodicJobs: riverpro.DurablePeriodicJobsConfig{
        // (1) `DurablePeriodicJobs.Enabled` must be set to true for any
        // periodic jobs to be activated
        Enabled: true,
    },
})

Notably, the API enables mixing of durable and non-durable periodic jobs. Even when DurablePeriodicJobs.Enabled is true for the entire client, only periodic jobs with an ID are made durable (those without behave like normal periodic jobs).

Implementation notes

Noteworthy details on the implementation of durable periodic jobs:

• Each job gets a record tracking its next run time in the river_periodic_job table, which is how state persists across restarts. This table is raised as part of River Pro's pro migration line.

• Durable periodic jobs are assigned IDs to track them in river_periodic_job uniquely. These IDs are orthogonal to the "kind" of individual jobs (which is also a form of identifier) so that it's possible to configure multiple periodic jobs that insert the same job kind. This might be useful in case different periods should insert with different job parameters.

Interaction with RunOnStart

Periodic jobs have a PeriodicJobOpts.RunOnStart option that largely exists so that non-durable periodic jobs that are on long run schedules get some opportunity to run occasionally. Without it, they'd otherwise be frozen out if a client restarts more frequently than a job's run interval.

Use of RunOnStart generally isn't necessary for durable periodic jobs because their schedules are tracked across restarts. However, RunOnStart still works for them and will cause those jobs to run on client start or leader election despite their next persisted run time. Users may want to take care to remove its use for any durable periodic jobs they configure.

Workflows are created with a workflow builder struct using riverpro.NewWorkflow(), and tasks are added to the workflow until it is prepared for insertion. Jobs and args are defined like any other River job.

Changing the run schedule of an existing job

Unlike non-durable jobs, changing a periodic job's run interval will have no immediate effect because a next run time was already stored in the database from the original schedule. For example, if a job was initially scheduled to run once a year but is then rescheduled to run once an hour, an initial next run that's a year from now is still in the database, and still authoritative.

A new schedule can be assigned by setting a new periodic job ID. Start with the initial schedule and ID:

PeriodicJobs: []*river.PeriodicJob{
    river.NewPeriodicJob(
        river.PeriodicInterval(365*24*time.Hour),
        ...
        &river.PeriodicJobOpts{
            ID: "my_periodic_job",
        },
    ),
},

Change the schedule, and while doing so put in a new unique ID for the periodic job. Any convention works, but an easy one is to append a version suffix like *_v2:

PeriodicJobs: []*river.PeriodicJob{
    river.NewPeriodicJob(
        river.PeriodicInterval(1*time.Hour),
        ...
        &river.PeriodicJobOpts{
            ID: "my_periodic_job_v2",
        },
    ),
},

With a periodic job no longer configured in Go code to run it, the original non-v2 periodic job record becomes orphaned, and that's okay. After being unused by the periodic job service for a preset span of time (default 24 hours, see DurablePeriodicJobsConfig.StaleThreshold), the Pro client will clean it up automatically.

If the old and new schedules are similar enough, this step isn't necessary because the old schedule will only ever have any effect on a single future run time. As soon as that time is reached, the new schedule takes effect, so as long as the original run time wasn't too far off in the future, the differing schedules won't make a big difference. For example, if a job that was originally scheduled to run once every 30 minutes is rescheduled to once every 15 minutes, the next run will still be 30 minutes away, but as soon as that's done, the new 15 minute schedule takes effect.