Rust Macros Rule: DRY warp Routes
The backend for A Viral World is written in Rust using the warp web server, which I chose for how lightweight, feature-filled, and well-maintained it is. All the server routes are contained in a single module. Here’s an example of what they used to look like:
Rustpub fn make_categories_list_route<'a, O: Clone + Send + Sync + 'a>(
environment: Environment<O>,
) -> impl warp::Filter<Extract = (impl Reply,), Error = reject::Rejection> + Clone + 'a {
let recordings_path = environment.urls.recordings_path.clone();
let handler = move || -> BoxFuture<Result<Json, reject::Rejection>> {
let environment = environment.clone();
async move {
let categories =
environment
.db
.retrieve_categories()
.await
.map_err(|e: BackendError| {
rejection::Rejection::new(rejection::Context::categories(), e)
})?;
Ok(json(&categories))
}
.boxed()
};
warp::path(recordings_path)
.and(warp::path("categories"))
.and(warp::path::end())
.and(warp::get())
.and_then(handler)
}
pub fn make_genders_list_route<'a, O: Clone + Send + Sync + 'a>(
environment: Environment<O>,
) -> impl warp::Filter<Extract = (impl Reply,), Error = reject::Rejection> + Clone + 'a {
let recordings_path = environment.urls.recordings_path.clone();
let handler = move || -> BoxFuture<Result<Json, reject::Rejection>> {
let environment = environment.clone();
async move {
let genders = environment
.db
.retrieve_genders()
.await
.map_err(|e: BackendError| {
rejection::Rejection::new(rejection::Context::genders(), e)
})?;
Ok(json(&genders))
}
.boxed()
};
warp::path(recordings_path)
.and(warp::path("genders"))
.and(warp::path::end())
.and(warp::get())
.and_then(handler)
}
There is much to dislike:
- The type constraints, (slightly long) return type, and opening path segment (
recordings_path
) are repeated for every function. - The handler itself is a closure, but because async closures aren’t yet
stable, the logic is contained in an
async move
block within the closure and returned as aBoxFuture
. - Defining the path requires many small, similar function calls.
The only unique parts of each route are the handler, the path, and the HTTP method. There was no need for the repetition, and none of it was warp’s fault. It was all convoluted code born out of inexperience. Now that I was more familiar with Rust in general and warp in particular, I could mold it into something more pleasing.
Put it in a Box
Before the routes themselves, I wanted to address some repetition in the definition of the server:
Rustlet categories_list_route = routes::make_categories_list_route(environment.clone());
let genders_list_route = routes::make_genders_list_route(environment.clone());
// …around 10 more routes…
let routes = categories_list_route
.or(genders_list_route)
// …the previously-mentioned 10 routes…
.recover(move |r| routes::format_rejection(logger.clone(), r));
This could be better written as a collection (a Vec
, in this case) that could be combined using
Iterator::fold
. The naïve
approach wouldn’t work, however:[1]
Rust// invalid, compiler error
let routes = vec![
routes::make_categories_list_route(environment.clone()),
routes::make_genders_list_route(environment.clone()),
// …more routes…
];
A Vec
can only store objects of a single type, but with impl Trait
, each individual object has a
different concrete type. Instead, I needed a trait
object
like Box<dyn Filter>
. warp already provides
BoxedFilter
, so I created a
trivial type alias for brevity and updated the functions:
Rusttype Route = BoxedFilter<(Box<dyn Reply>,)>;
pub fn make_categories_list_route<'a, O: Clone + Send + Sync + 'a>(
environment: Environment<O>,
) -> Route {
// same as before
}
With the updated return type, I would later be able to store the route in a
Vec<BoxedFilter<(Box<dyn Reply>,)>
, removing another source of repetition.
Back to the routes.
Small improvements, big dividends
I moved the handlers into standalone functions, with another type alias for brevity, and switched to
Filter::map
to pass environment
:
Rusttype Route = BoxedFilter<(Box<dyn Reply>,)>;
type RouteResult = Result<Box<dyn Reply>, reject::Rejection>;
pub fn make_categories_list_route<O: Clone + Send + Sync + 'static>(
environment: Environment<O>,
) -> Route {
let recordings_path = environment.urls.recordings_path.clone();
warp::path(recordings_path)
.and(warp::path("categories"))
.and(warp::path::end())
.and(warp::get())
.map(move || environment.clone())
.and_then(categories_handler)
.boxed()
}
That let me get rid of the async
blocks and directly write async
functions with simpler
signatures:
Rustasync fn categories_handler<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
let categories = environment
.db
.retrieve_categories()
.await
.map_err(|e: BackendError| {
rejection::Rejection::new(rejection::Context::categories(), e)
})?;
Ok(Box::new(json(&categories)) as Box<dyn Reply>)
}
I was happier with this formulation, but repeating Box::new(val) as Box<dyn
Reply>
for every handler would be ever so tedious. Laziness inspired me to write my first
ever macro:
Rustmacro_rules! as_box {
($expression:expr) => {
Ok(Box::new($expression) as Box<dyn Reply>)
};
}
So I could shorten it to:
Rustasync fn categories_handler<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
// same logic as before
as_box!(json(&categories))
}
Much better. There’s that big dividend I was referring to. After that, I moved all the handlers into their own module and used a couple of imports to shorten them. My updated code looked like this:
Rust// routes.rs
pub fn make_categories_list_route<O: Clone + Send + Sync + 'static>(
environment: Environment<O>,
) -> Route {
let recordings_path = environment.urls.recordings_path.clone();
warp::path(recordings_path)
.and(warp::path("categories"))
.and(warp::path::end())
.and(warp::get())
.map(move || environment.clone())
.and_then(handlers::categories_list)
.boxed()
}
// handlers.rs
pub async fn categories_list<O: Clone + Send + Sync>(environment: Environment<O>) -> RouteResult {
let categories = environment
.db
.retrieve_categories()
.await
.map_err(|e: BackendError| Rejection::new(Context::categories(), e))?;
as_box!(json(&categories))
}
A big improvement in structure and clarity. The code was starting to shape up, but I hadn’t yet got to the major changes.
A useful macro appears
Having tried my hand at macros once, and knowing only three things changed in each route, I felt I
might be able to remove most of the repetition with another macro. This was what I tried (with a
route_filters!
helper macro yet to be written):
Rust// compiler error
macro_rules! route {
($name:ident => $handler:ident; $($filters:expr),+) => (
pub fn $name<O: Clone + Send + Sync + 'static>(environment: Environment<O>) -> Route {
let r = environment.urls.recordings_path.clone();
warp::any()
.map(move || environment.clone())
.and(warp::path(r))
route_filters!($($filters,),+) // invalid syntax
.and_then(handlers::$handler)
.boxed()
}
);
}
route!(make_categories_list_route; warp::path!("categories"), warp::get());
Sadly, however, macros can’t be inserted in the middle of a chain of method calls. I might have been able to do it with procedural macros, which wouldn’t require two separate macros, but they seem complicated. Instead, I used an intermediate variable, whose name must be specified at the call site due to the macro hygiene rules:
Rustmacro_rules! route_filters {
($route_variable:ident; $first:expr) =>
(let $route_variable = $route_variable.and($first););
($route_variable:ident; $first:expr, $($rest:expr),+) => (
let $route_variable = $route_variable.and($first);
route_filters!($route_variable; $($rest),+);
)
}
macro_rules! route {
($name:ident => $handler:ident, $route_variable:ident; $($filters:expr),+) => (
pub fn $name<O: Clone + Send + Sync + 'static>(environment: Environment<O>) -> Route {
let r = environment.urls.recordings_path.clone();
let $route_variable = warp::any()
.map(move || environment.clone())
.and(warp::path(r));
route_filters!($route_variable; $($filters),+);
$route_variable.and_then(handlers::$handler)
.boxed()
}
);
}
(Thanks to Glenn Pierce for spotting a typo.)
Marginally less elegant, but the result of all this effort was that each route could be expressed in a single, succint line:
Rustroute!(make_categories_list_route => categories_list, rt; warp::path!("categories"), warp::get());
route!(make_genders_list_route => genders_list, rt; warp::path!("genders"), warp::get());
And judicious imports could shorten them even further:
Rustuse warp::{get, path};
route!(make_categories_list_route => categories_list, rt; path!("categories"), get());
route!(make_genders_list_route => genders_list, rt; path!("genders"), get());
Folding it all together
Finally, I combined the routes using fold
(along with Filter::unify
to flatten the Either
types created by Filter::or
,
since all the routes have the same return type):
Rustlet mut routes = vec![
r::make_categories_list_route(environment.clone()),
r::make_genders_list_route(environment.clone()),
// …more routes…
];
let first = routes.pop().expect("get first route");
let routes = routes
.into_iter()
.fold(first, |e, r| e.or(r).unify().boxed())
.recover(move |r| routes::format_rejection(logger.clone(), r));
As a finishing touch, I could remove the common prefix from the route!
macro, instead setting it
once here:
Rustlet prefix = warp::path(environment.urls.recordings_path.clone());
// (same code as above)
let server = warp::serve(prefix.and(routes));
And there it was. A simpler, more attentive server that didn’t expect me to repeat myself quite as much.
One further improvement I planned to make was automatically adding routes to a module-level list at
the point of definition, obviating the need for the routes
variable. Unfortunately, this would
require running code next to the function definition, which is not allowed. malaire on the Rust
Discord mentioned some web framework crates do something like that with attribute proc
macros, but I have no idea how those work internally
. That might be worth investigating,
eventually.
- I’m not sure whether
impl_trait_in_bindings
would allow this.↩