Encrypted jobs encrypt the args column of each database job row, providing an extra layer of security that may prevent total compromise in case of database breach. River Pro provides a built-in encryptor using NaCl Secretbox, but it’s easy to customize for any desired cryptography by implementing an encryptor.
Encrypted jobs are a feature of River Pro ✨. If you haven’t yet, install River Pro.
Basic usage
Added in River Pro for Go v0.12.0.
Encryption is enabled by installing riverencrypt.EncryptHook (see plugins and hooks) on a River client:
import ( "encoding/base64"
"riverqueue.com/riverpro/riverencrypt" "riverqueue.com/riverpro/riverencrypt/riversecretbox")
decodedBytes, err := base64.StdEncoding.DecodeString( "iRmwTuVGl2BAwTUPRTJbP/iA2EKpTrzXpEcNIXG2BI0=",)if err != nil { panic(err)}
var key [32]byteif copy(key[:], decodedBytes) != 32 { panic("expected to copy exactly 32 bytes")}
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ Plugins: []rivertype.Plugin{ riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)), }, },})EncryptHook tolerates jobs coming out of the queue not being encrypted, so it’s safe to enable encryption without migration. Turning it off again takes more attention.
Install libsodium and add the optional RbNaCl dependency to your application’s Gemfile:
gem "rbnacl", "~> 7.1"Add River::Pro::EncryptPlugin to the core configuration of every client that inserts or works encrypted jobs. Load a persistent Base64-encoded key from your secret manager:
require "base64"require "riverqueue-pro"
key = Base64.strict_decode64(ENV.fetch("RIVER_ENCRYPTION_KEY"))
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(key))
client = River::Pro::Client.new(driver, config: River::Pro::Config.new( core: River::Config.new( plugins: [encrypt_plugin], queues: {default: 100}, workers: workers ) ))The encryptor requires a raw 32-byte string. The plugin accepts existing unencrypted jobs, so encryption can be enabled without a data migration. Turning it off requires draining encrypted jobs first.
Enable the optional secretbox feature on riverqueue-pro in Cargo.toml, and add base64 = "0.23" to decode a key from your secret manager. The encrypt feature alone provides the hook and custom-encryptor interface; secretbox adds the built-in cipher.
Install an EncryptHook on every client that inserts or works encrypted jobs:
use base64::{Engine as _, engine::general_purpose::STANDARD};use riverqueue::{QueueConfig, database::PostgresDatabase};use riverqueue_pro::ProClient;use riverqueue_pro::encrypt::EncryptHook;use riverqueue_pro::encrypt::secretbox::{SecretKey, SecretboxEncryptor};
let encoded_key = std::env::var("RIVER_ENCRYPTION_KEY")?;let key = SecretKey::try_from(STANDARD.decode(&encoded_key)?.as_slice())?;let client = ProClient::builder(PostgresDatabase::new(pool)) .hook(EncryptHook::new(SecretboxEncryptor::new([key])?)) .queue("default", QueueConfig::new(100)) .workers(workers) .build()?;SecretKey validates the decoded key’s 32-byte length. Existing plaintext jobs still work, so enabling encryption requires no data migration. Workers receive decoded, typed arguments. The hook can also be installed on the core Client builder.
NaCl Secretbox
River Pro bundles in a default encryptor using NaCl Secretbox. NaCl is high-performance, public domain encryption designed by the same person responsible for Curve25519, a secure elliptic curve commonly found in modern keys generated by OpenSSH. It’s suitable for most use cases, and recommended unless specific cryptography is required for compliance reasons. Implement an encryptor to use alternate cryptography.
Use riverencrypt/riversecretbox.Encryptor along with EncryptHook:
import ( "riverqueue.com/riverpro/riverencrypt" "riverqueue.com/riverpro/riverencrypt/riversecretbox")
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ Plugins: []rivertype.Plugin{ riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)), }, },})Use River::Pro::SecretBoxEncryptor with River::Pro::EncryptPlugin:
encryptor = River::Pro::SecretBoxEncryptor.new(key)encrypt_plugin = River::Pro::EncryptPlugin.new(encryptor)core = River::Config.new(plugins: [encrypt_plugin], workers: workers)Its NaCl Secretbox ciphertext format is compatible with River Pro for Go. Both clients can encrypt and decrypt each other’s job arguments when configured with the same keys.
Wrap SecretboxEncryptor in an EncryptHook and pass it to the client builder’s .hook(...) method:
use riverqueue_pro::encrypt::EncryptHook;use riverqueue_pro::encrypt::secretbox::SecretboxEncryptor;
let hook = EncryptHook::new(SecretboxEncryptor::new([key])?);With the same keys, the ciphertext and job envelope are compatible with Go and Ruby clients. The cipher uses a fresh random nonce for every encryption. SecretKey and the encryptor erase their key material when dropped and omit it from debug output.
Keys are 32 random bytes. By convention they’re encoded to something like base64 so they can be stored safely in an env var or other form of vault:
import "encoding/base64"
var key [32]byteif _, err := rand.Reader.Read(key[:]); err != nil { panic(err)}encodedKey := base64.StdEncoding.EncodeToString(key[:])fmt.Printf("encoded key: %s\n", encodedKey)require "base64"require "securerandom"
key = SecureRandom.random_bytes(32)encoded_key = Base64.strict_encode64(key)puts "encoded key: #{encoded_key}"Generate the key once and store it securely. Load that same key on every client and across restarts.
Generate a key once using the operating system random source. With getrandom = "0.4" and base64 = "0.23" in your application’s dependencies:
use base64::{Engine as _, engine::general_purpose::STANDARD};
let mut key = [0_u8; 32];getrandom::fill(&mut key)?;let encoded_key = STANDARD.encode(key);Save encoded_key in your secret manager and load that same key on every client and across restarts. Key generation belongs in a separate provisioning step, not client startup.
encoded key: iRmwTuVGl2BAwTUPRTJbP/iA2EKpTrzXpEcNIXG2BI0=And decoded again with:
decodedBytes, err := base64.StdEncoding.DecodeString(encodedKey)if err != nil { panic(err)}
var decodedKey [32]byteif copy(decodedKey[:], decodedBytes) != 32 { panic("expected to copy exactly 32 bytes")}decoded_key = Base64.strict_decode64(encoded_key)raise ArgumentError, "expected 32 bytes" unless decoded_key.bytesize == 32use base64::{Engine as _, engine::general_purpose::STANDARD};use riverqueue_pro::encrypt::secretbox::SecretKey;
let decoded_key = SecretKey::try_from(STANDARD.decode(&encoded_key)?.as_slice())?;SecretKey::try_from rejects decoded values that are not exactly 32 bytes.
Encrypting specific jobs
You can forego installing the plugin globally to install it only on specific jobs by implementing JobArgsWithPlugins:
var encryptHook = riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key))
type JobEncryptedArgs struct{}
func (JobEncryptedArgs) Kind() string { return "job_encrypted" }
func (JobEncryptedArgs) Plugins() []rivertype.Plugin { return []rivertype.Plugin{ encryptHook, }}To encrypt a single job kind, install a plugin with a job_kinds_include filter on the client:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(key), job_kinds_include: ["job_encrypted"])core = River::Config.new(plugins: [encrypt_plugin], workers: workers)Ruby configures encryption through client plugins. The filter applies to both encryption and decryption, so configure it consistently on inserters and workers.
Use JobKindSelection::only to select a single kind:
use riverqueue_pro::encrypt::secretbox::SecretboxEncryptor;use riverqueue_pro::encrypt::{EncryptHook, EncryptHookConfig, JobKindSelection};
let hook = EncryptHook::with_config( SecretboxEncryptor::new([key])?, EncryptHookConfig::new().with_selection(JobKindSelection::only(["job_encrypted"])),);Install it with .hook(hook). The filter controls both encryption and decryption, so configure inserters and workers consistently.
Alternatively, EncryptHookConfig takes a JobKindsInclude option to indicate that only a subset of jobs should be encrypted. This is functionally identical, but may be more convenient because it’s used in a non-static context where an error can be returned (e.g. in case an encryptor failed to initialize):
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Plugins: []rivertype.Plugin{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ Encryptor: riversecretbox.NewEncryptor(key),
// Only encrypt/decrypt job args included in this list. JobKindsInclude: []string{ (ThisJobWillBeEncryptedArgs{}).Kind(), }, }), }, },})List multiple kinds in job_kinds_include to encrypt a subset of your jobs:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(key), job_kinds_include: ["charge_card", "send_private_message"])core = River::Config.new(plugins: [encrypt_plugin], workers: workers)Select multiple kinds with the same filter:
use riverqueue_pro::encrypt::secretbox::SecretboxEncryptor;use riverqueue_pro::encrypt::{EncryptHook, EncryptHookConfig, JobKindSelection};
let hook = EncryptHook::with_config( SecretboxEncryptor::new([key])?, EncryptHookConfig::new().with_selection(JobKindSelection::only([ "charge_card", "send_private_message", ])),);JobKindsExclude is also available to indicate that all jobs except some kinds should be encrypted:
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Plugins: []rivertype.Plugin{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ Encryptor: riversecretbox.NewEncryptor(key),
// Encrypt/decrypt all job args except those in this list. JobKindsExclude: []string{ (ThisJobWill_NOT_BeEncryptedArgs{}).Kind(), }, }), }, },})Use job_kinds_exclude to encrypt all kinds except those listed:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(key), job_kinds_exclude: ["public_announcement"])core = River::Config.new(plugins: [encrypt_plugin], workers: workers)Configure either an include list or an exclude list. With neither filter, the plugin encrypts all job kinds.
Use JobKindSelection::except to encrypt every kind except those listed:
use riverqueue_pro::encrypt::secretbox::SecretboxEncryptor;use riverqueue_pro::encrypt::{EncryptHook, EncryptHookConfig, JobKindSelection};
let hook = EncryptHook::with_config( SecretboxEncryptor::new([key])?, EncryptHookConfig::new().with_selection(JobKindSelection::except(["public_announcement"])),);The default selection is JobKindSelection::All. Install one encryption hook per selected kind; avoid overlapping hooks that would wrap the same arguments twice.
Key rotation
Key rotation is carried out in phases to make sure existing jobs don’t become unworkable because they were encrypted using a key that’s no longer available.
Step 0: Start out with the original key configured:
keyOld := mustDecodeBase64EncodedKey("fdnQ7+v/5Pb28rYqpynRSdzWfqs1gD6/J/0I9IUh65s=")_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Plugins: []rivertype.Plugin{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyOld,)),},},})Step 1: Add a new key in the first encryptor position, leaving the original key in the second position. The new key encrypts new jobs, but in case it can’t decrypt a job,
EncryptHookfalls back to the original key. Deploy.keyNew := mustDecodeBase64EncodedKey("T8sUAPOQNSDDMAiMyfrK8EaLOlY/cJ21PPNn1InCqIQ=")_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Plugins: []rivertype.Plugin{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyNew,keyOld,)),},},})Step 2: After all jobs using the original key have been drained, remove the original key. Consider that jobs may be queued for future retries in case of error, so this may take some time (up to three weeks after first run using the default retry policy).
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Plugins: []rivertype.Plugin{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyNew,)),},},})
Start with the old key:
old_key = Base64.strict_decode64(ENV.fetch("RIVER_ENCRYPTION_KEY_OLD"))encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(old_key))First deploy every reader with both keys, keeping the old key first so writers still encrypt with it:
new_key = Base64.strict_decode64(ENV.fetch("RIVER_ENCRYPTION_KEY_NEW"))encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(old_key, new_key))After all readers can decrypt the new key, deploy with the new key first. The first key encrypts new jobs; decryption tries every configured key:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(new_key, old_key))Remove the old key only after all jobs that need it have drained, including scheduled retries and any retained dead letter jobs you may retry:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(new_key))At each step, put the plugin in the core configuration’s plugins array and deploy it to inserters and workers.
The first key passed to SecretboxEncryptor::new encrypts new jobs, and all configured keys are tried for decryption. Load keys from your secret manager at each deployment:
use base64::{Engine as _, engine::general_purpose::STANDARD};use riverqueue_pro::encrypt::EncryptHook;use riverqueue_pro::encrypt::secretbox::{SecretKey, SecretboxEncryptor};
fn load_key(name: &str) -> Result<SecretKey, Box<dyn std::error::Error>> { let encoded = std::env::var(name)?; Ok(SecretKey::try_from(STANDARD.decode(encoded)?.as_slice())?)}
let old_key = load_key("RIVER_ENCRYPTION_KEY_OLD")?;let new_key = load_key("RIVER_ENCRYPTION_KEY_NEW")?;
// First deploy every reader with both keys, while writers still use the old key.let hook = EncryptHook::new(SecretboxEncryptor::new([old_key, new_key])?);After every reader has both keys, deploy with the new key first:
let old_key = load_key("RIVER_ENCRYPTION_KEY_OLD")?;let new_key = load_key("RIVER_ENCRYPTION_KEY_NEW")?;let hook = EncryptHook::new(SecretboxEncryptor::new([new_key, old_key])?);Keep the old key until all jobs that need it have drained, including scheduled jobs, retries, and retained dead letter jobs you may retry. Then deploy with just the new key:
let new_key = load_key("RIVER_ENCRYPTION_KEY_NEW")?;let hook = EncryptHook::new(SecretboxEncryptor::new([new_key])?);At each stage, install hook on inserters and workers. SecretKey is moved into the encryptor; reload the keys when constructing a new configuration.
Record when all writers switched to the new key. As an initial check on Postgres, count unfinished jobs created before that cutoff (replace the timestamp parameter for your SQL client). This is a conservative check based on creation time, not an identification of the encryption key; also account for dead letter jobs or other retained jobs that may be retried:
SELECT state, count(*)FROM river_jobWHERE created_at <= @new_key_deployed_time AND STATE NOT IN ('cancelled', 'completed', 'discarded')GROUP BY 1ORDER BY 2 DESC;Turning encryption on and off
Enabling encrypted jobs needs no additional work beyond adding the encryption plugin to clients, even if there are already existing unencrypted jobs in the database. In case a job is dequeued that’s not encrypted, it’ll be worked without attempting to decrypt it.
Disabling encrypted jobs needs more consideration because removing the encryption plugin might leave orphaned jobs in the database that can no longer be decrypted.
To disable encryption safely, start by activating the DecryptOnly option instead of removing EncryptHooks completely:
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Plugins: []rivertype.Plugin{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ DecryptOnly: true, Encryptor: riversecretbox.NewEncryptor(key), }), }, },})Set decrypt_only: true to stop encrypting new jobs while continuing to decrypt existing ones:
encrypt_plugin = River::Pro::EncryptPlugin.new( River::Pro::SecretBoxEncryptor.new(key), decrypt_only: true)core = River::Config.new(plugins: [encrypt_plugin], workers: workers)Use EncryptionMode::DecryptOnly to stop encrypting new inserts while preserving decryption:
use riverqueue_pro::encrypt::secretbox::SecretboxEncryptor;use riverqueue_pro::encrypt::{EncryptHook, EncryptHookConfig, EncryptionMode};
let hook = EncryptHook::with_config( SecretboxEncryptor::new([key])?, EncryptHookConfig::new().with_mode(EncryptionMode::DecryptOnly),);Install the hook with .hook(hook) and keep all required read keys until encrypted jobs have drained, including any retained jobs you may retry.
Jobs will continue to be decrypted, but new ones won’t be encrypted.
The encryption plugin can be removed completely after all encrypted jobs have drained out of the database (this may take up to three weeks after first run using the default retry policy). Query for encrypted jobs that may still be worked using the river_encrypt field added by the encryption plugin (Postgres):
SELECT state, count(*)FROM river_jobWHERE args ? 'river_encrypt' AND STATE NOT IN ('cancelled', 'completed', 'discarded')GROUP BY 1ORDER BY 2 DESC;Implementing an encryptor
Because River ships with limited built-in cryptography options, it might be necessary to add your own by implementing the Encryptor interface:
type Encryptor interface { Decrypt(cipher []byte) ([]byte, error) Encrypt(plain []byte) []byte}Decryptdecrypts cipher text to plain text. It should try all available keys in case keys are being rotated. In case no encryption key matches, it should returnErrNoKeyDecrypted.Encryptencrypts plain text to cipher text.
A custom encryptor must respond to encrypt(encoded_json) and decrypt(ciphertext). Both take and return byte strings; decryption must return the original JSON string and raise if authentication or decryption fails.
For example, adapt your application’s existing authenticated encryption service (the service methods here are illustrative):
class AppEncryptor def initialize(service) @service = service end
def encrypt(plain) @service.encrypt(plain) end
def decrypt(cipher) @service.decrypt(cipher) endend
encrypt_plugin = River::Pro::EncryptPlugin.new(AppEncryptor.new(encryption_service))The plugin handles the JSON and Base64 envelope around the ciphertext. A custom encryptor should try every retained key during rotation, and must never return plaintext if authentication fails.
Implement riverqueue_pro::encrypt::Encryptor for your application’s authenticated cipher. Its interface is:
use riverqueue_pro::encrypt::EncryptionError;
trait Encryptor: Send + Sync + 'static { fn decrypt(&self, cipher: &[u8]) -> Result<Vec<u8>, EncryptionError>; fn encrypt(&self, plain: &[u8]) -> Result<Vec<u8>, EncryptionError>;}The trait above shows the required signatures; implement the crate’s Encryptor, then pass your implementation to EncryptHook::new. The hook handles JSON and the Base64 envelope. These methods are synchronous and should not perform network I/O.
Encryption must return fresh authenticated ciphertext. Decryption must reject malformed or unauthenticated data and try retained read keys during rotation. Use EncryptionError::operation("encrypt", error) or EncryptionError::operation("decrypt", error) to preserve an underlying error. A decryption failure prevents the worker from running.
Considerations before use
Why not to use encrypted jobs
Encrypted jobs have disadvantages that should be weighed before enabling them. Encryption inherently makes job args opaque, making them illegible to human operators examining them through tools like psql or River UI.
Many hosted database providers already provide encryption at rest, which is good enough to provide reasonable security and meet compliance objectives. If that’s the case for you, it might be advisable to skip encrypting job args, keeping jobs more introspectable and production easier to work with.
Why to use encrypted jobs
Encryption at rest provides a meaningful security benefit, but doesn’t protect against all breaches. An attacker that’s able to attack your database at the application layer may be able to siphon out its contents (as they’ve done to many huge companies over the years like Equifax, Nordstrom, or T-Mobile). On disk data might be encrypted, but that’d be bypassed as data is breached at a much higher level.
Encrypted jobs add another defensive layer. In case of data exfiltration, the attacker only obtains job metadata and job args secured with strong enough crypto that they won’t practically be able to attack it.
However, even this approach isn’t bulletproof. If an attacker is able to gain access to an application’s database and its runtime including key secrets used in encryption, it’s back to a total breach as there’s nothing stopping them from reversing the encryption on the stolen data set using the stolen keys.
Applications should protect against this as much as possible by keeping secrets stored separately from code, which is itself stored separately from data. The harder each component is to access, copy, and combine into a working chain to get back to plain text, the better.