Brandur Leach Today, we're shipping River Pro v0.26 with a brand-new Pro feature: active job rescue.
Every job queue needs to handle the case where a process running jobs dies catastrophically. It may crash so hard and fast that the queue doesn't have an opportunity to give those jobs a soft landing by setting them back to available for another worker to pick up.
This is where job rescue comes into play. A queue detects jobs that are marked as running but aren't making progress (because, for example, the client that was running them is gone), and makes them eligible to run again. River has included a basic form job rescue since v0, with a maintenance service that periodically checks for this condition, resets their state to available, and notifies clients that there's new work to pick up.
The current implementation works based off job start times and expected duration. If a job has been running longer than job.Timeout + Config.RescueStuckJobsAfter (a safety margin in case jobs improperly continue to run past their timeout, default: 1 hour), it's assumed to be in need of rescue. It works fine, but often produces undesirable time-to-work, especially for long timeouts and rescue-after margins. A job with a 30-minute timeout will be rescued, but only after an hour and a half:
timeToRescue := job.Timeout + RescueStuckJobsAfter = 30m + 1h = 1h30m(Notably, you can get this number down by changing job timeouts and RescueStuckJobsAfter configuration, but the point is that standard job rescue is a passive affair.)
An even more worse case is when a job's timeout has been explicit set to -1 (no timeout). The rescuer can't determine whether the job is still running, so even if its client has crashed, the job is left is.
From passive timeouts to active recovery
River Pro's new active job rescue solves both problems. To implement concurrency limits, River Pro already tracks heartbeats for each active queue across all clients. Combining the timestamps for when each job started and its queue began heartbeating is enough information for River to determine definitively whether a job marked as running is indeed still doing so.
The worst-case delay of job.Timeout (default: 1 minute) + RescueStuckJobsAfter (default: 1 hour) becomes independent of the job's timeout, consisting instead of the heartbeat interval (default: 30 seconds) plus a short safety margin (in case a client is temporarily unable to report heartbeats, default: 60 seconds), clearing jobs orphaned by a crash much sooner.
| Configuration | Old mean | New mean | New range | Mean speedup | Delay reduction |
|---|---|---|---|---|---|
| Default configuration | 59m 45s | 1m 30s | 1m-2m | 39.8x | 97.5% |
| 5m timeout | 1h 2m 45s | 1m 30s | 1m-2m | 41.8x | 97.6% |
| 30m timeout | 1h 15m 15s | 1m 30s | 1m-2m | 50.2x | 98.0% |
| 1h timeout | 1h 30m 15s | 1m 30s | 1m-2m | 60.2x | 98.3% |
| 6h timeout | 4h 0m 15s | 1m 30s | 1m-2m | 160.2x | 99.4% |
(The table above is a little contrived, but shows how dramatic the time-to-work reduction can be when RescueStuckJobsAfter is left at its default or for jobs with longer timeouts.)
Even better, active job rescue comes with few tradeoffs. River Pro already includes a table that tracks each client and queue, so no new migrations or additional database operations are necessary. As before, the table is UNLOGGED, so even with frequent writes, it doesn't put any pressure on WAL. Because all liveness determinations can be made with existing data, active job rescue avoids writes to river_job, producing no new dead tuples or extra pressure on operational health.
No configuration required
If you're on River Pro, just upgrade to v0.26 and active job rescue will activate automatically. Otherwise, try River Pro today to get it along with a growing slate of features including workflows V2, concurrency limits, batch execution, and much more.