Defense in depth with encrypted jobs

Along with concurrency limits, River Pro v0.12 ships with support for encrypted jobs, a feature that securely encrypts database job args (the variable part of the record containing job-specific information), while leaving allowances for flexible cryptography, selective encryption, and key rotation.

We don't expect encrypted jobs to be needed by all River installations, but in cases where data is especially sensitive, they add a layer of security that makes exfiltration more difficult in case of an application level vulnerability. Using in-database encryption might've helped contain the blast radius in infamous cases like Equifax, Nordstrom, or T-Mobile for example, all of which resulted in leaks of massive user databases after attackers successfully compromised the application attached to them.

Encryption is enabled through the use of a client hook:

import (
    "encoding/base64"

    "riverqueue.com/riverpro/riverencrypt"
    "riverqueue.com/riverpro/riverencrypt/riversecretbox"
)

// Key is 32 random bytes, but will generally be encoded to something like
// base64 to store in an env var or other vault.
decodedBytes, err := base64.StdEncoding.DecodeString(
    "iRmwTuVGl2BAwTUPRTJbP/iA2EKpTrzXpEcNIXG2BI0=",
)
if err != nil {
    panic(err)
}

var key [32]byte
if copy(key[:], decodedBytes) != 32 {
    panic("expected to copy exactly 32 bytes")
}

riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{
    Config: river.Config{
        Hooks: []rivertype.Hook{
            riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)),
        },
    },
})

The hook takes an Encryptor, a small interface providing an implementation for specific cryptography:

type Encryptor interface {
    // Decrypt decrypts plaintext to ciphertext. It should try all available
    // keys before failing, and return the specific ErrNoKeyDecrypted error in
    // case of failure due to no suitable decryption keys.
    Decrypt(cipher []byte) ([]byte, error)

    // Encrypt encrypts plaintext to ciphertext.
    Encrypt(plain []byte) []byte
}

Encryptor makes implementing a shim between River and a crypto package easy for cases where prescribed crypto algorithms are necessary for compliance. It also keeps the interface adaptable enough so that future, stronger cryptography can make its way into River in the future as appropriate.

Securebox and DJB-ware

Encrypted jobs ship with one encryptor out of the box: riversecretbox.Encryptor, an Encryptor implementation for NaCL Secretbox, strong, fast encryption that should be a suitable default for most users.

In 2013, Edward Snowden disclosed worldwide mass surveillance on the part of the NSA, and his revelations led to the discovery of a back door in Dual EC DRBG algorithm, which had been blessed by NIST and RSA to encourage worldwide use. The security community became more skeptical of cryptography with opaque roots overnight, leading to more scrutiny on elliptic curve parameters proposed by the NSA/NIST. There was now a plausible reason to fear that weak fundamentals were selected on purpose to give the US government an edge in cracking emergent crypto.

The corollary was a predictable surge in interest for open, patent-free, independently verifiable elliptic curves. Curve25519, created by cryptographer Daniel J. Bernstein (DJB) was originally published in 2005, but didn't see major interest until the events of 2013, after which it surged. Starting in 2014, OpenSSH default to Curve25519-based ECDH, and in fact your SSH key is probably using it right now. Support for Curve25519 has been added to GnuPG, DNSSEC, DKIM, and TLS 1.3.

Curve25519 isn't DJB's only production, and he's been responsible for a number of other pieces of software and security-adjacent projects. One of those is NaCl Securebox, an encryption scheme built for speed, security, and correctness, all using a simple API that's difficult to accidentally misuse by design. Not to mention that happily for us, Go's x/crypto makes a package implementing it easily available.

Based on its well-documented benefits and pristine pedigree in the form of a spotless vulnerability track record, we chose Securebox as River's first encryptor implementation. Use is straightforward, involving creating an encryptor and defining a key:

import (
    "riverqueue.com/riverpro/riverencrypt"
    "riverqueue.com/riverpro/riverencrypt/riversecretbox"
)

riverClient, err := riverpro.NewClient(riverpropgxv5.New(dbPool), &riverpro.Config{
    Config: river.Config{
        Hooks: []rivertype.Hook{
            riverencrypt.NewEncryptHook(riversecretbox.NewEncryptor(key)),
        },
    },
})

Secretbox should be appropriate for most uses, but in cases where alternative cryptography is preferred, implementing Encryptor is easy. Here's the one for Secretbox at only 35 lines of code:

type Encryptor struct {
    keys [][keySize]byte
}

func (e *Encryptor) Decrypt(cipher []byte) ([]byte, error) {
    if len(cipher) <= nonceSize {
        return nil, fmt.Errorf("encrypted args should be at least %d bytes long", nonceSize)
    }

    var nonce [nonceSize]byte
    copy(nonce[:], cipher[0:nonceSize])
    withoutNonce := cipher[nonceSize:]

    for _, key := range e.keys {
        var opened []byte
        opened, ok := secretbox.Open(opened[:0], withoutNonce, &nonce, &key)
        if ok {
            return opened, nil
        }
    }

    return nil, riverencrypt.ErrNoKeyDecrypted
}

func (e *Encryptor) Encrypt(plain []byte) []byte {
    var nonce [nonceSize]byte
    if _, err := rand.Read(nonce[:]); err != nil {
        panic(err) // panic because this will ~never happen in real life
    }

    var box []byte
    box = secretbox.Seal(box[:0], plain, &nonce, &e.keys[0])

    return append(nonce[:], box...)
}

When not to encrypt jobs

It might seem tempting to pop encryption onto every stack as an extra precauation, but we're compelled to point out that encryption is not always the best compromise between security and operability.

Encrypted jobs do offer an added layer of security, but in many cases a database will already be encrypted at rest, which offers good security and ticks many compliance boxes.

An encrypted disk offers good protection in cases where a disk image or physical server is stolen, ensuring it's useless without a corresponding key. But it doesn't protect against a successful attack against a running application which has access to the online, unencrypted database, and this is where encrypted jobs might be useful. In case an attacker gains access to an application and siphons out its database piece by piece, they'll find the sensitive parts of River jobs inaccessible behind a layer of strong encryption.

And even that's not the end of the story. If an attacker finds their way into an online application and its memory containing River encrypted job keys, the app is backed to be being compromised, as by titrating both jobs table and keys, they can reverse the encryption at leisure.

Like any layer of security, encrypted jobs are not bulletproof, but rather meant to add one more layer of tightening to make an attacker's life that much harder. For best results, keys should be stored orthogonally to data, which are stored orthogonally to code, making it that much harder to bring all the pieces of the puzzle together.

A downside of encryption is that it doesn't just make life harder for bad actors, but for the good guys too. Instead of being able to view naked job args with a tool like psql or River UI, they're obfuscated behind a layer of ciphertext that'd have to be reversed to get back to something legible by a human. Another inconvenience is that in case a key is leaked (or theoretically whenever an employee with access to one leaves the company), keys should be rotated, which is a bit of a project.

Encrypt freely, but deliberately

Encrypted jobs aren't an all or nothing proposition. As a happy compromise, jobs can be encrypted on a per-kind basis. We'd recommend encrypting the most sensitive jobs, and leaving the rest of less sensitive ones to use encryption at rest so they stay easily instrospectable.

See River Pro and for more details, the full documentation for encrypted jobs.