Skip to main content

Swap a Storage Backend per Application

Most applications reach a point where one operation needs a different backend in different places. A deployment reads its files from disk, for example, while its tests would rather not touch the disk at all. Context-Generic Programming (CGP) is a library for that kind of choice. It enables pluggable trait implementations in Rust, so each application type can choose, at compile time, which implementation it uses.

In this tutorial, we will build a working profile-picture lookup and link out to the details as we go. By the end, one lookup function will read pictures from a folder in the application and from memory in a test, and the compiler will check that each choice is complete. The tutorial assumes you know Rust, including async functions, and can add a dependency. It assumes no CGP, though the Hello World tutorial is a shorter first look at the same starting point.

Set up the project​

Start a new binary project with cargo new profile-pictures, then add three dependencies to its Cargo.toml: cgp itself, anyhow for errors, and tokio, because the lookup is async and reads files through Tokio:

Cargo.toml
[dependencies]
cgp = "0.8.0"
anyhow = "1"
tokio = { version = "1", features = ["fs", "macros", "rt-multi-thread"] }

Every module that uses CGP imports its prelude. The program fits in one file, so these imports go at the top of src/main.rs:

src/main.rs
use std::collections::HashMap;
use std::path::Path;

use anyhow::anyhow;
use cgp::prelude::*;

The application context​

The lookup needs two pieces of data: a table of users, and somewhere to keep their pictures. Each user may have a picture, stored under an object ID:

#[derive(Clone, PartialEq, Eq, Hash)]
pub struct UserId(pub u64);

#[derive(Clone)]
pub struct User {
pub name: String,
pub profile_picture_object_id: Option<String>,
}

The application itself is a struct called App. CGP calls a type like this a context: a type that stands for this application, which is where its choices live. A context can be an empty struct, but this one has fields that hold the values its operations need. Its database is a HashMap standing in for a real database, so the tutorial runs without a server, and its local_dir names the folder that holds the pictures:

#[derive(HasField)]
pub struct App {
pub database: HashMap<UserId, User>,
pub local_dir: String,
}

The #[derive(HasField)] line lets CGP code read those fields by name, and the next step relies on it.

Read the application's data with functions​

The lookup has two steps: find the user, then load the picture. We write each one as a function marked #[cgp_fn], and mark the arguments that should come from the application's fields with #[implicit]:

#[cgp_fn]
#[async_trait]
pub async fn get_user(
&self,
#[implicit] database: &HashMap<UserId, User>,
user_id: &UserId,
) -> anyhow::Result<User> {
database
.get(user_id)
.cloned()
.ok_or_else(|| anyhow!("no user with ID {}", user_id.0))
}

#[cgp_fn]
#[async_trait]
pub async fn fetch_storage_object(
&self,
#[implicit] local_dir: &str,
object_id: &str,
) -> anyhow::Result<Vec<u8>> {
let data = tokio::fs::read(Path::new(local_dir).join(object_id)).await?;
Ok(data)
}

Notice that callers never pass database or local_dir. #[cgp_fn] turns each function into a method on any type whose fields match the implicit arguments by name and type, so App gains get_user and fetch_storage_object without an impl block of its own. The match allows a borrowed form, too: the local_dir argument is a &str, and it reads App's String field.

Both functions are async, so each also carries #[async_trait]. Without it, Rust would warn about the async fn in the public trait that #[cgp_fn] generates.

To try them, main writes a picture into a pictures folder, builds an App with one user, and calls both methods:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
tokio::fs::create_dir_all("pictures").await?;
tokio::fs::write("pictures/alice.png", b"not really a PNG").await?;

let app = App {
database: HashMap::from([(
UserId(1),
User {
name: "Alice".to_owned(),
profile_picture_object_id: Some("alice.png".to_owned()),
},
)]),
local_dir: "pictures".to_owned(),
};

let user = app.get_user(&UserId(1)).await?;
let picture = app.fetch_storage_object("alice.png").await?;
println!("{} has a {}-byte picture", user.name, picture.len());

Ok(())
}

Running cargo run prints Alice has a 16-byte picture. The bytes are not a real image, but the program never decodes them, so any bytes will do.

Combine the steps​

The two steps belong together, so the next function calls both. Because get_user and fetch_storage_object are now methods, the new function calls them on self, and declares that it needs them with #[uses]:

#[cgp_fn]
#[async_trait]
#[uses(GetUser, FetchStorageObject)]
pub async fn get_user_profile_picture(
&self,
user_id: &UserId,
) -> anyhow::Result<Option<Vec<u8>>> {
let user = self.get_user(user_id).await?;

match user.profile_picture_object_id {
Some(object_id) => Ok(Some(self.fetch_storage_object(&object_id).await?)),
None => Ok(None),
}
}

The names in #[uses(GetUser, FetchStorageObject)] are the traits #[cgp_fn] generated from the two functions: a function called get_user becomes a trait called GetUser. Declaring them there, rather than as bounds in the signature, keeps the function's interface to a user ID alone. It also means that get_user_profile_picture exists only on a type that has every field both steps read. A context with a database but no local_dir would still have get_user, but a call to get_user_profile_picture on it would fail to compile.

Replace the last lines of main, from let user onwards, with a call to the combined function:

match app.get_user_profile_picture(&UserId(1)).await? {
Some(picture) => println!("Found a {}-byte picture", picture.len()),
None => println!("No picture"),
}

Ok(())
}

cargo run now prints Found a 16-byte picture.

Add a backend for tests​

So far there is one way to fetch a picture, and it reads the disk. A test of get_user_profile_picture should keep its pictures in memory instead, which needs a second way to fetch them, chosen by the test's own context. But a #[cgp_fn] function has exactly one implementation, so the storage step needs something that can have several.

That something is a CGP component. We replace the fetch_storage_object function with a trait marked #[cgp_component]:

#[async_trait]
#[cgp_component(StorageObjectFetcher)]
pub trait CanFetchStorageObject {
async fn fetch_storage_object(&self, object_id: &str) -> anyhow::Result<Vec<u8>>;
}

Code calls CanFetchStorageObject as an ordinary trait, while StorageObjectFetcher names the trait each implementation targets. An implementation is called a provider, and we write one with #[cgp_impl], giving it a name. The disk-reading provider, FetchLocalObject, takes over the old function's body and still reads local_dir as an implicit argument:

#[cgp_impl(new FetchLocalObject)]
impl StorageObjectFetcher {
async fn fetch_storage_object(
&self,
#[implicit] local_dir: &str,
object_id: &str,
) -> anyhow::Result<Vec<u8>> {
let data = tokio::fs::read(Path::new(local_dir).join(object_id)).await?;
Ok(data)
}
}

The combined function now depends on the component rather than on the removed function, so its #[uses] names the trait that callers use:

#[cgp_fn]
#[async_trait]
#[uses(GetUser, CanFetchStorageObject)]
pub async fn get_user_profile_picture(
&self,
user_id: &UserId,
) -> anyhow::Result<Option<Vec<u8>>> {
let user = self.get_user(user_id).await?;

match user.profile_picture_object_id {
Some(object_id) => Ok(Some(self.fetch_storage_object(&object_id).await?)),
None => Ok(None),
}
}

A provider does nothing until a context chooses it. App makes its choice with delegate_components!, which works like a lookup table from each component to the provider that implements it for this context:

delegate_components! {
App {
StorageObjectFetcherComponent: FetchLocalObject,
}
}

The key, StorageObjectFetcherComponent, is a name #[cgp_component] generates for the component. Rust resolves the table at compile time, so app.fetch_storage_object(...) becomes a direct call to FetchLocalObject, without a runtime lookup. cargo run prints the same line as before.

Now for the test. Its context, TestApp, has the same database field and keeps its pictures in an objects map instead of a folder. A second provider, FetchMemoryObject, reads that map, and TestApp chooses it in its own table. All of this goes in a test module at the end of src/main.rs:

#[cfg(test)]
mod tests {
use super::*;

#[derive(HasField)]
pub struct TestApp {
pub database: HashMap<UserId, User>,
pub objects: HashMap<String, Vec<u8>>,
}

#[cgp_impl(new FetchMemoryObject)]
impl StorageObjectFetcher {
async fn fetch_storage_object(
&self,
#[implicit] objects: &HashMap<String, Vec<u8>>,
object_id: &str,
) -> anyhow::Result<Vec<u8>> {
objects
.get(object_id)
.cloned()
.ok_or_else(|| anyhow!("no object named {object_id}"))
}
}

delegate_components! {
TestApp {
StorageObjectFetcherComponent: FetchMemoryObject,
}
}

#[tokio::test]
async fn fetches_the_picture_from_memory() -> anyhow::Result<()> {
let app = TestApp {
database: HashMap::from([(
UserId(1),
User {
name: "Alice".to_owned(),
profile_picture_object_id: Some("alice.png".to_owned()),
},
)]),
objects: HashMap::from([("alice.png".to_owned(), b"fake image".to_vec())]),
};

let picture = app.get_user_profile_picture(&UserId(1)).await?;
assert_eq!(picture, Some(b"fake image".to_vec()));

Ok(())
}
}

Run cargo test, and the test passes without touching the disk. Neither get_user_profile_picture nor get_user changed to make that possible. Each context chose its storage backend in one line of its table, and the same lookup ran against a folder in the application and against memory in the test. Adding a third backend, such as a cloud bucket, would be one more provider and one more table entry. The Overview shows providers for Amazon S3 and Google Cloud Storage, written against a synchronous version of this component.

Check the wiring​

Wiring is lazy: adding a table entry does not check that the context has what the provider needs. The check happens when the component is used, and if something is missing, the error points at the place it was used, which may be far from the table that caused it. CGP's check_components! moves that check next to the table, so add one for each context, App's after its table and TestApp's inside the test module:

check_components! {
App {
StorageObjectFetcherComponent,
}
}
check_components! {
TestApp {
StorageObjectFetcherComponent,
}
}

To see what the check catches, delete the local_dir field from App, along with the line in main that sets it. FetchLocalObject still needs the field, so cargo check now fails at the check. The compiler's message is long, and it spells the missing field's name as a nested type. Shortened, it begins like this:

error[E0277]: the trait bound `App: CanUseComponent<StorageObjectFetcherComponent>` is not satisfied
...
help: the trait `HasField<Symbol<9, Chars<'l', Chars<'o', Chars<'c', Chars<'a', Chars<'l', Chars<'_', Chars<'d', Chars<'i', cgp::macro_prelude::Chars<'r', Nil>>>>>>>>>>>` is not implemented for `App`

cargo-cgp, CGP's error toolchain, explains the same failure more directly. After installing it, run cargo cgp check in place of cargo check. It names the missing field first, then the chain of requirements that led to it. The compiler reports the mistake twice, at the check and again at the call to get_user_profile_picture in main, and cargo cgp check shows it once, keeping the check's report. Here it is, without the source excerpt that points at the check:

error[E0277]: [CGP-E001] missing field `local_dir` on `App`
= note: root cause: [CGP-E110] missing field `local_dir` on `App`
this is required through the dependency chain:
[CGP-E101] consumer trait impl `CanFetchStorageObject` for context `App`
└─ [CGP-E102] provider trait impl `StorageObjectFetcher` with context `App` for provider `FetchLocalObject`
└─ [CGP-E110] missing field `local_dir` on `App`
= help: add the field `local_dir` to `App`

cargo cgp check leads with the root cause for the classes it recognizes, and the tool does not yet reshape every class. Put the field back, and both checks pass.

Where to go next​

You now have a lookup whose storage backend each application chooses, checked at compile time. To go further:


An AI agent wrote this page using the CGP knowledge base, and its examples were verified by compiling them. See How AI is used in this project.