In-memory queue

River can be run as a fully in-memory queue by combining the SQLite driver with SQLite’s :memory: mode. This is useful for development, testing, and ephemeral workloads where persistence isn’t needed.

An in-memory queue behaves identically to a persistent one — jobs are inserted, worked, retried, and are even transactional — but all state is lost when the process exits. There’s no disk I/O, no cleanup, and no external dependencies.

Sample code

An in-memory database starts empty, so run migrations on the same database before starting workers. A separate CLI process would migrate a different in-memory database.

Open a database/sql connection with the :memory: DSN and pass it to River’s SQLite driver:

import (
    "database/sql"

    _ "modernc.org/sqlite"

    "github.com/riverqueue/river"
    "github.com/riverqueue/river/riverdriver/riversqlite"
    "github.com/riverqueue/river/rivermigrate"
)

dbPool, err := sql.Open("sqlite", ":memory:")
if err != nil {
    panic(err)
}
defer dbPool.Close()

dbPool.SetMaxOpenConns(1)

migrator, err := rivermigrate.New(riversqlite.New(dbPool), nil)
if err != nil {
    panic(err)
}

_, err = migrator.Migrate(ctx, rivermigrate.DirectionUp, nil)
if err != nil {
    panic(err)
}

workers := river.NewWorkers()
river.AddWorker(workers, &MyWorker{})

riverClient, err := river.NewClient(riversqlite.New(dbPool), &river.Config{
    Queues: map[string]river.QueueConfig{
        river.QueueDefault: {MaxWorkers: 100},
    },
    Workers: workers,
})
if err != nil {
    panic(err)
}

Since an in-memory database starts empty every time, it’s necessary to run migrations before starting the client. The CLI can’t be used in this instance because it would open its own connection to a separate in-memory database.

No cleanup is necessary after this code runs. The in-memory database is discarded when the connection closes.

Open an in-memory Sequel database with a single connection, then use the same driver for migrations and the client:

require "riverqueue-sequel"

DB = Sequel.sqlite(max_connections: 1)
driver = River::Driver::Sequel.new(DB)
River::Migrator.new(driver).migrate

client = River::Client.new(driver,
  config: River::Config.new(queues: {default: 10}, workers: workers))

Keep the single connection open while the client is in use, and disconnect it only after stopping workers.

Create a single-connection SQLite pool for sqlite::memory: and migrate it before building the client:

let pool = riverqueue::sqlx::sqlite::SqlitePoolOptions::new()
    .max_connections(1)
    .connect("sqlite::memory:")
    .await?;

riverqueue::migrate::SqliteMigrator::new(pool.clone()).migrate_up().await?;

let client = riverqueue::Client::builder(pool)
    .workers(workers)
    .queue("default", riverqueue::QueueConfig::new(10))
    .build()?;

Enable the sqlite Cargo feature. A private in-memory database belongs to one connection, so use a single-connection pool.

Create an in-memory SQLite driver, migrate it, and start a client with your registered workers. The using declarations clean up when the scope exits:

import { SqliteDriver } from "@riverqueue/driver-sqlite";
import { createMigrator } from "@riverqueue/migrate";

using driver = SqliteDriver.memory();
await createMigrator(driver).migrateUp();

const client = new Client(driver, {
  workers,
  queues: { default: { maxWorkers: 10 } },
});

await using run = await client.start();

// Insert and work jobs while this scope is active.

Use SqliteDriver.memory() rather than passing a private :memory: handle to the constructor. Application handles can be opened with driver.connect() and must also be closed.

SQLite only

In-memory queues are SQLite only, and as such the same concurrency considerations as file-backed SQLite apply. Follow the connection model for your language’s driver and keep transactions short to avoid blocking.