Published in Technology & Work on A Place For My Head. Tagged , , & .

The backend for A Viral World is written in Rust using the warp web server, which I chose for how lightweight, feature-filled, and well-maintained it is. All the server routes are contained in a single module. Here’s an example of what they used to look like: 

pub fn make_categories_list_route<'a, O: Clone + Send + Sync + 'a>(
    environment: Environment<O>,
) -> impl warp::Filter<Extract = (impl Reply,), Error = reject::Rejection> + Clone + 'a {
    let recordings_path = environment.urls.recordings_path.clone();

    let handler = move || -> BoxFuture<Result<Json, reject::Rejection>> {
        let environment = environment.clone();

        async move {
            let categories =
                environment
                    .db
                    .retrieve_categories()
                    .await
                    .map_err(|e: BackendError| {
                        rejection::Rejection::new(rejection::Context::categories(), e)
                    })?;

            Ok(json(&categories))
        }
        .boxed()
    };

    warp::path(recordings_path)
        .and(warp::path("categories"))
        .and(warp::path::end())
        .and(warp::get())
        .and_then(handler)
}

pub fn make_genders_list_route<'a, O: Clone + Send + Sync + 'a>(
    environment: Environment<O>,
) -> impl warp::Filter<Extract = (impl Reply,), Error = reject::Rejection> + Clone + 'a {
    let recordings_path = environment.urls.recordings_path.clone();

    let handler = move || -> BoxFuture<Result<Json, reject::Rejection>> {
        let environment = environment.clone();

        async move {
            let genders = environment
                .db
                .retrieve_genders()
                .await
                .map_err(|e: BackendError| {
                    rejection::Rejection::new(rejection::Context::genders(), e)
                })?;

            Ok(json(&genders))
        }
        .boxed()
    };

    warp::path(recordings_path)
        .and(warp::path("genders"))
        .and(warp::path::end())
        .and(warp::get())
        .and_then(handler)
}

There is much to dislike:

The only unique parts of each route are the handler, the path, and the HTTP method. There was no need for the repetition, and none of it was warp’s fault. It was all convoluted code born out of inexperience. Now that I was more familiar with Rust in general and warp in particular, I could mold it into something more pleasing.

Put it in a Box

Before the routes themselves, I wanted to address some repetition in the definition of the server:

let categories_list_route = routes::make_categories_list_route(environment.clone());
let genders_list_route = routes::make_genders_list_route(environment.clone());
// …around 10 more routes…

let routes = categories_list_route
    .or(genders_list_route)
    // …the previously-mentioned 10 routes…
    .recover(move |r| routes::format_rejection(logger.clone(), r));

This could be better written as a collection (a Vec, in this case) that could be combined using Iterator::fold. The naïve approach wouldn’t work, however:[1] 

// invalid, compiler error
let routes = vec![
  routes::make_categories_list_route(environment.clone()),
  routes::make_genders_list_route(environment.clone()),
  // …more routes…
];

A Vec can only store objects of a single type, but with impl Trait, each individual object has a different concrete type. Instead, I needed a trait object like Box<dyn Filter>. warp already provides BoxedFilter, so I created a trivial type alias for brevity and updated the functions:

type Route = BoxedFilter<(Box<dyn Reply>,)>;

pub fn make_categories_list_route<'a, O: Clone + Send + Sync + 'a>(
    environment: Environment<O>,
) -> Route {
  // same as before
}

With the updated return type, I would later be able to store the route in a Vec<BoxedFilter<(Box<dyn Reply>,)>, removing another source of repetition.

Back to the routes.

Small improvements, big dividends

I moved the handlers into standalone functions, with another type alias for brevity, and switched to Filter::map to pass environment: 

type Route = BoxedFilter<(Box<dyn Reply>,)>;
type RouteResult = Result<Box<dyn Reply>, reject::Rejection>;

pub fn make_categories_list_route<O: Clone + Send + Sync + 'static>(
    environment: Environment<O>,
) -> Route {
    let recordings_path = environment.urls.recordings_path.clone();

    warp::path(recordings_path)
        .and(warp::path("categories"))
        .and(warp::path::end())
        .and(warp::get())
        .map(move || environment.clone())
        .and_then(categories_handler)
        .boxed()
}

That let me get rid of the async blocks and directly write async functions with simpler signatures:

async fn categories_handler<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
    let categories = environment
        .db
        .retrieve_categories()
        .await
        .map_err(|e: BackendError| {
            rejection::Rejection::new(rejection::Context::categories(), e)
        })?;

    Ok(Box::new(json(&categories)) as Box<dyn Reply>)
}

I was happier with this formulation, but repeating Box::new(val) as Box<dyn Reply> for every handler would be ever so tedious. Laziness inspired me to write my first ever macro:

macro_rules! as_box {
    ($expression:expr) => {
        Ok(Box::new($expression) as Box<dyn Reply>)
    };
}

So I could shorten it to:

async fn categories_handler<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
    // same logic as before

    as_box!(json(&categories))
}

Much better. There’s that big dividend I was referring to. After that, I moved all the handlers into their own module and used a couple of imports to shorten them. My updated code looked like this:

// routes.rs
pub fn make_categories_list_route<O: Clone + Send + Sync + 'static>(
    environment: Environment<O>,
) -> Route {
    let recordings_path = environment.urls.recordings_path.clone();

    warp::path(recordings_path)
        .and(warp::path("categories"))
        .and(warp::path::end())
        .and(warp::get())
        .map(move || environment.clone())
        .and_then(handlers::categories_list)
        .boxed()
}

// handlers.rs
pub async fn categories_list<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
    let categories = environment
        .db
        .retrieve_categories()
        .await
        .map_err(|e: BackendError| Rejection::new(Context::categories(), e))?;

    as_box!(json(&categories))
}

A big improvement in structure and clarity. The code was starting to shape up, but I hadn’t yet got to the major changes.

A useful macro appears

Having tried my hand at macros once, and knowing only three things changed in each route, I felt I might be able to remove most of the repetition with another macro. This was what I tried (with a route_filters! helper macro yet to be written): 

// compiler error
macro_rules! route {
    ($name:ident => $handler:ident; $($filters:expr),+) => (
        pub fn $name<O: Clone + Send + Sync + 'static>(environment: Environment<O>) -> Route {
            let r = environment.urls.recordings_path.clone();

            warp::any()
                .map(move || environment.clone())
                .and(warp::path(r))
                route_filters!($($filters,),+) // invalid syntax
                .and_then(handlers::$handler)
                .boxed()
        }
    );
}

route!(make_categories_list_route; warp::path!("categories"), warp::get());

Sadly, however, macros can’t be inserted in the middle of a chain of method calls. I might have been able to do it with procedural macros, which wouldn’t require two separate macros, but they seem complicated. Instead, I used an intermediate variable, whose name must be specified at the call site due to the macro hygiene rules:

macro_rules! route_filters {
    ($route_variable:ident; $first:expr) =>
        (let $route_variable = $route_variable.and($first););
    ($route_variable:ident; $first:expr, $($rest:expr),+) => (
        let $route_variable = $route_variable.and($first);
        route_filters!($route_variable; $($rest),+);
    )
}

macro_rules! route {
    ($name:ident => $handler:ident, $route_variable:ident; $($filters:expr),+) => (
        pub fn $name<O: Clone + Send + Sync + 'static>(environment: Environment<O>) -> Route {
            let r = environment.urls.recordings_path.clone();

            let $route_variable = warp::any()
                .map(move || environment.clone())
                .and(warp::path(r));

            route_filters!($route_variable; $($filters),+);

            $route_variable.and_then(handlers::$handler)
                .boxed()
        }
    );
}

(Thanks to Glenn Pierce for spotting a typo.)

Marginally less elegant, but the result of all this effort was that each route could be expressed in a single, succint line:

route!(make_categories_list_route => categories_list, rt; warp::path!("categories"), warp::get());
route!(make_genders_list_route => genders_list, rt; warp::path!("genders"), warp::get());

And judicious imports could shorten them even further:

use warp::{get, path};

route!(make_categories_list_route => categories_list, rt; path!("categories"), get());
route!(make_genders_list_route => genders_list, rt; path!("genders"), get());

Folding it all together

Finally, I combined the routes using fold (along with Filter::unify to flatten the Either types created by Filter::or, since all the routes have the same return type):

let mut routes = vec![
    r::make_categories_list_route(environment.clone()),
    r::make_genders_list_route(environment.clone()),
    // …more routes…
];

let first = routes.pop().expect("get first route");

let routes = routes
    .into_iter()
    .fold(first, |e, r| e.or(r).unify().boxed())
    .recover(move |r| routes::format_rejection(logger.clone(), r));

As a finishing touch, I could remove the common prefix from the route! macro, instead setting it once here:

let prefix = warp::path(environment.urls.recordings_path.clone());

// (same code as above)

let server = warp::serve(prefix.and(routes));

And there it was. A simpler, more attentive server that didn’t expect me to repeat myself quite as much.

One further improvement I planned to make was automatically adding routes to a module-level list at the point of definition, obviating the need for the routes variable. Unfortunately, this would require running code next to the function definition, which is not allowed. malaire on the Rust Discord mentioned some web framework crates do something like that with attribute proc macros, but I have no idea how those work internally. That might be worth investigating, eventually.

  1. I’m not sure whether impl_trait_in_bindings would allow this.↩