Encrypted jobs encrypt the args column of each database job row, providing an extra layer of security that may prevent total compromise in case of database breach. River Pro provides a built-in encryptor using NaCL Secretbox, but it's easy to customize for any desired cryptography by implementing an encryptor.
Encrypted jobs are a feature of River Pro ✨. If you haven't yet, install River Pro.
Added in River Pro v0.12.0.
Basic usage
Encryption is enabled by installing riverencrypt.EncryptHook (see function hooks) on a River client:
import ( "encoding/base64"
"riverqueue.com/riverpro/riverencrypt" "riverqueue.com/riverpro/riverencrypt/riversecretbox")
decodedBytes, err := base64.StdEncoding.DecodeString( "iRmwTuVGl2BAwTUPRTJbP/iA2EKpTrzXpEcNIXG2BI0=",)if err != nil { panic(err)}
var key [32]byteif copy(key[:], decodedBytes) != 32 { panic("expected to copy exactly 32 bytes")}
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ Hooks: []rivertype.Hook{ riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)), }, },})EncryptHook tolerate jobs coming out of the queue not being encrypted, so it's safe to enable encryption without migration. Turning it off again takes more attention.
NaCl Secretbox
River Pro bundles in a default encryptor using NaCl Secretbox. NaCl is high-performance, public domain encryption designed by the same person responsible for Curve25519, a secure elliptic curve commonly found in modern keys generated by OpenSSH. It's suitable for most use cases, and recommended unless specific cryptography is required for compliance reasons. Implement an encryptor to use EncryptHook with alternate cryptography.
Use riverencrypt/riversecretbox.Encryptor along with EncryptHook:
import ( "riverqueue.com/riverpro/riverencrypt" "riverqueue.com/riverpro/riverencrypt/riversecretbox")
riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{ Config: river.Config{ Hooks: []rivertype.Hook{ riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)), }, },})Keys are 32 random bytes. By convention they're encoded to something like base64 so they can be stored safely in an env var or other form of vault:
import "encoding/base64"
var key [32]byteif _, err := rand.Reader.Read(key[:]); err != nil { panic(err)}encodedKey := base64.StdEncoding.EncodeToString(key[:])fmt.Printf("encoded key: %s\n", encodedKey)encoded key: iRmwTuVGl2BAwTUPRTJbP/iA2EKpTrzXpEcNIXG2BI0=And decoded again with:
decodedBytes, err := base64.StdEncoding.DecodeString(encodedKey)if err != nil { panic(err)}
var decodedKey [32]byteif copy(decodedKey[:], decodedBytes) != 32 { panic("expected to copy exactly 32 bytes")}Encrypting specific jobs
You can forego installing hooks globally to install them only on specific jobs by implementing JobArgsWithHooks:
var encryptHook = riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key))
type JobEncryptedArgs struct{}
func (JobEncryptedArgs) Kind() string { return "job_encrypted" }
func (JobEncryptedArgs) Hooks() []rivertype.Hook { return []rivertype.Hook{ encryptHook, }}Alternatively, EncryptHookConfig takes a JobKindsInclude option to indicate that only a subset of jobs should be encrypted. This is functionally identical, but may be more convenient because it's used in a non-static context where an error can be returned (e.g. in case an encryptor failed to initialize):
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Hooks: []rivertype.Hook{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ Encryptor: riversecretbox.NewEncryptor(key),
// Only encrypt/decrypt job args included in this list. JobKindsInclude: []string{ (ThisJobWillBeEncryptedArgs{}).Kind(), }, }), }, },})JobKindsExclude is also available to indicate that all jobs except some kinds should encrypted:
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Hooks: []rivertype.Hook{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ Encryptor: riversecretbox.NewEncryptor(key),
// Encrypt/decrypt all job args except those in this list. JobKindsExclude: []string{ (ThisJobWill_NOT_BeEncryptedArgs{}).Kind(), }, }), }, },})Key rotation
Key rotation is carried out in phases to make sure existing jobs don't become unworkable because they were encrypted using a key that's no longer available.
Step 0: Start out with the original key configured:
keyOld := mustDecodeBase64EncodedKey("fdnQ7+v/5Pb28rYqpynRSdzWfqs1gD6/J/0I9IUh65s=")_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Hooks: []rivertype.Hook{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyOld,)),},},})Step 1: Add a new key in the first encryptor position, leaving the original key in the second position. The new key encrypts new jobs, but in case it can't decrypt a job,
EncryptHookfalls back to the original key. Deploy.keyNew := mustDecodeBase64EncodedKey("T8sUAPOQNSDDMAiMyfrK8EaLOlY/cJ21PPNn1InCqIQ=")_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Hooks: []rivertype.Hook{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyNew,keyOld,)),},},})Step 2: After all jobs using the original key have been drained, remove the original key. Consider that jobs may be queued for future retries in case of error, so this may take some time (up to three weeks after first run using the default retry policy).
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{Config: river.Config{Hooks: []rivertype.Hook{riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(keyNew,)),},},})
Record the time when the new encryption key was deployed, then query the database for how many jobs using the original key are still eligible for work:
SELECT state, count(*)FROM river_jobWHERE created_at > @new_key_deployed_time AND STATE NOT IN ('cancelled', 'completed', 'discarded')GROUP BY 1ORDER BY 2 DESC;Turning encryption on and off
Enabling encrypted jobs needs no additional work beyond adding the function hooks to clients, even if there are already existing unencrypted jobs in the database. In case a job is dequeued that's not encrypted, it'll be worked without attempting to decrypt it.
Disabling encrypted jobs needs more consideration because removing the encryption hooks might leave orphaned jobs in the database that can no longer be decrypted.
To disable encryption safely, start by activating the DecryptOnly option instead of removing EncryptHooks completely:
_, err := riverpro.NewClient(riverpropgxv5.New(nil), &riverpro.Config{ Config: river.Config{ Hooks: []rivertype.Hook{ riverencrypt.NewEncryptHookConfig(&riverencrypt.EncryptHookConfig{ DecryptOnly: true, Encryptor: riversecretbox.NewEncryptor(key), }), }, },})Jobs will continue to be decrypted, but new ones won't be encrypted.
The encrypt hook can be removed completely after all encrypted jobs have drained out of the database (this may take up to three weeks after first run using the default retry policy). Query for encrypted jobs that may still be worked using the special river_encrypt field added by encrypt hooks:
SELECT state, count(*)FROM river_jobWHERE args ? 'river_encrypt' AND STATE NOT IN ('cancelled', 'completed', 'discarded')GROUP BY 1ORDER BY 2 DESC;Implementing an encryptor
Because River ships with limited built-in cryptography options, it might be necessary to add your own by implementing the Encryptor interface:
type Encryptor interface { Decrypt(cipher []byte) ([]byte, error) Encrypt(plain []byte) []byte}Decryptdecrypts cipher text to plain text. It should try all available keys in case keys are being rotated. In case no encryption key matches, it should returnErrNoKeyDecrypted.Encryptencrypts plain text to cipher text.
Considerations before use
Why not to use encrypted jobs
Encrypted jobs have disadvantages that should be weighed before enabling them. Encryption inherently makes job args opaque, making them illegible to human operators examining them through tools like psql or River UI.
Many hosted database providers already provide encryption at rest, which is good enough to provide reasonable security and meet compliance objectives. If that's the case for you, it might be advisable to skip encrypting job args, keeping jobs more introspectable and production easier to work with.
Why to use encrypted jobs
Encryption at rest provides a meaningful security benefit, but doesn't protect against all breaches. An attacker that's able to attack your database at the application layer may be able to siphon out its contents (as they've done to many huge companies over the years like Equifax, Nordstrom, or T-Mobile). On disk data might be encrypted, but that'd be bypassed as data is breached at a much higher level.
Encrypted jobs add another defensive layer. In case of data exfiltration, the attacker only obtains job metadata and job args secured with strong enough crypto that they won't practically be able to attack it.
However, even this approach isn't bulletproof. If an attacker is able to gain access to an application's database and its runtime including key secrets used in encryption, it's back to a total breach as there's nothing stopping them from reversing the encryption on the stolen data set using the stolen keys.
Applications should protect against this as much as possible by keeping secrets stored separately from code, which is itself stored separately from data. The harder each component is to access, copy, and combine into a working chain to get back to plain text, the better.