Today I’d like to talk a bit about how I like to organize my Rust projects. In particular, I’d like to talk about modules (you cannot talk about project organization without talking about modules).
First, I’ll define what modules are, and give a brief explanation for how they work. After that, I’ll explain my recommended approach for organizing modules in Rust.
While many people only think of modules as a way to split rust crates into multiple files, the way that I like to describe modules is by telling people that they control 3 separate things at the same time.
- code organization (which files to build and where they are)
- symbol access (how to access the things we’ve built)
- visibility boundaries (which things can access which other things)
I believe good Rust code is well-encapsulated, sensibly exported in a meaningful way, and easy to traverse and explore (without the assistance of an IDE). Before we get into how to accomplish that, let’s take a closer look at each of these properties to make sure they’re understood well enough.
The first use is the most obvious of modules - splitting code into multiple files. The reason we do this is ease of maintenance, as well as making it easier to understand and traverse the code.
The Rust Book documents how file splitting works here.
But the TL;DR is that when the compiler encounters this statement in a file:
mod fizz_buzz;
It will look for a file to compile under the following paths:
fizz_buzz.rs(same directory)fizz_buzz/mod.rs(mod.rsfile in a subdirectory offizz_buzz)
If you don’t like the default search locations, you can explicitly say where a module is located using the path attribute:
// Now it searches explicitly for `other_location.rs` (same directory)
#[path = "other_location.rs"]
mod fizz_buzz;
By default, using modules as detailed above creates a 1:1 relationship between two features which I claimed were separate (code organization and symbol exporting). This is because now, when I want to access something from fizz_buzz, I have to recurse into the path made by compiling it (fizz_buzz::some_function()).
However, there’s a powerful keyword called use which is can be used to re-export some symbol into the current module. Technically speaking, this keyword creates a path binding or import, but that ends up being a powerful concept when talking about code organization.
Usually, use is used privately within a module to allow us to be more concise when there’s no chance of being confused by ambiguity.
use std::collections::HashMap;
pub struct SomeType {
// It's not confusing to just call this a `HashMap` type.
internal_map: HashMap<u32, String>,
}
However, the use keyword takes an optional visibility modifier.
When used this way, the use keyword acts a lot like re-exporting a symbol someplace. It’s not defining a new type, but it’s saying that someone else could use a type by this name, from this path (and even allows for renaming symbols).
For instance, let’s say you had a FizzBuzz type defined within the fizz_buzz module, but you didn’t want to have to access it via fizz_buzz::FizzBuzz - you could re-export it.
mod fizz_buzz;
pub use fizz_buzz::FizzBuzz;
With the above file, any sibling or parent modules will not be able to access the fizz_buzz submodule of this module directly, but they would be able to access FizzBuzz (because this module publicly re-exports it).
So, in this way, modules also control access to symbols when used in conjunction with the use keyword and visibility controls (for mod and use). When combined with strict usage of visibility keywords, you can control where the user can access the symbol from (note the private fizz_buzz module).
Encapsulation is the method of limiting access to certain fields or functions to control code paths and ensure internal data consistency.
A good example of a type that requires good encapsulation is the NonZero type.
You could imagine, if we were to create our own NonZero type - if we weren’t careful about how we encapsulated the data, it could be possible to construct it with invalid data.
// Bad because we can trivially set `inner` to whatever (including 0).
pub struct NonZeroU32 {
pub inner: u32,
}
The correct solution here is to make inner private, and then create a constructor which is public so that we can control construction:
// Better, because we can not trivially set/change `inner`.
pub struct NonZeroU32 {
inner: u32,
}
impl NonZeroU32 {
pub fn new(n: u32) -> Option<Self> {
match n {
0 => None,
n => Some(Self {
inner: n,
})
}
}
}
This seems appropriate, but it becomes insufficient if there’s any other unrelated types that depend on the NonZero invariants defined in the same module.
pub struct NonZeroU32 {
inner: u32,
}
impl NonZeroU32 {
pub fn new(n: u32) -> Option<Self> {
match n {
0 => None,
n => Some(Self {
inner: n,
})
}
}
}
// NO LONGER OKAY! - We can invalidate invariants too easily!
pub struct DependsOnNonZeroU32 {
some_value: Option<NonZeroU32>,
}
impl DependsOnNonZeroU32 {
pub fn new(some_value: u32) -> Self {
Self {
some_value: Some(NonZeroU32 {
inner: some_value,
})
}
}
}
If you want to fix this and harden DependsOnNonZeroU32 to only be able to use the explicitly-stated public API of NonZeroU32, then you have to put them in separate sibling modules:
mod non_zero_u32 {
pub struct NonZeroU32 {
inner: u32,
}
impl NonZeroU32 {
pub fn new(n: u32) -> Option<Self> {
match n {
0 => None,
n => Some(Self {
inner: n,
})
}
}
}
}
pub use non_zero_u32::NonZeroU32;
mod depends_on_non_zero_u32 {
use super::NonZeroU32;
pub struct DependsOnNonZeroU32 {
some_value: Option<NonZeroU32>,
}
impl DependsOnNonZeroU32 {
pub fn new(some_value: u32) -> Self {
Self {
some_value: NonZeroU32::new(some_value),
}
// This no longer compiles:
//
// ```
// Self {
// some_value: Some(NonZeroU32 {
// inner: some_value,
// })
// }
// ```
}
}
}
pub use depends_on_non_zero_u32::DependsOnNonZeroU32;
This is a trivial example, but it’s what I mean when I say that modules create visibility boundaries, and helps enforce encapsulation.
Whenever you use some visibility keyword, it doesn’t mean squat within the modules (everything has private access within the same module, or from the perspective of all descendants in the tree). However, if we’re talking about non-descendants modules (such as siblings), then that’s where the checks on visibility are put into play.
NOTE: Technically, the visibility is more complicated than this, because it’s all path-based. This is where we get some visibility modifiers like pub(in some::path). Those are usually hard to reason about and maintain, so I only recommend them as an absolute last resort.
NOTE: pub(crate) is much easier to reason about. So if you have something that you don’t want to publicly export, but you do want to have access internally within the crate, you can optionally use that.
Given all the above, I think the best way to organize modules is usually like so (note emphasis on “usually”; I do sometimes find myself breaking these rules in certain exceptional cases)…
Keep logic and types out of the
mod.rsfiles - use those to control the build, organization, and re-export of your logic and types.- This will help you avoid having parent logic that child logic can trivially access the private data and functions of - recall that a child module can access the private fields and functions of any parent modules. So naturally we should want to create mostly sibling modules (unless you explicitly want that access property for closely-related sub-types).
Try to break your types into sibling submodules so that you have very explicit control over visibility and encapsulation.
- This will help you ensure that two types are properly separated from each other. If you find yourself blocked due to visibility, it will force you to think of the appropriate way to define the API of the type.
If a module doesn’t exist to serve the purpose of grouping common items, then re-export so that the existence of the module is an implementation detail.
- Just because something needs to be in a module doesn’t mean that you need to publicly export the module as a path that someone must recurse into. I find that you often don’t want that, and usually want a private module and a re-export.
You should be using some
mod.rsmodules (old style), some named modules (new style), and some explicitly-pathed modules (#[path = "..."]).- There are times for each of these - I dislike that the documentation calls one of these styles “new” and “old”. For instance, Rust programmers still tend to use
mod.rsfor simple, organizing modules.
- There are times for each of these - I dislike that the documentation calls one of these styles “new” and “old”. For instance, Rust programmers still tend to use
Probably the most surprising of these statements is 4. Let’s explore an example codebase to see where these rules are put to use, and where they are broken. We will use the code for this very website - it is a static site generator written in Rust.
The code layout for the logic responsible for building this website is:
.
├── commands
│ ├── generate
│ │ ├── about.rs
│ │ ├── articles.rs
│ │ ├── devlog.rs
│ │ ├── home.rs
│ │ ├── mod.rs
│ │ └── projects.rs
│ └── mod.rs
├── common
│ ├── generator
│ │ ├── generator.rs
│ │ ├── generator.tests.rs
│ │ ├── generator_feed_config.rs
│ │ ├── generator_href_path.rs
│ │ ├── generator_href_path.tests.rs
│ │ ├── generator_path.rs
│ │ ├── generator_path.tests.rs
│ │ ├── generator_system_path.rs
│ │ ├── generator_system_path.tests.rs
│ │ ├── generator_target_path.rs
│ │ ├── generator_target_path.tests.rs
│ │ ├── generator_traversal_config.rs
│ │ └── mod.rs
│ ├── html
│ │ ├── html_link.rs
│ │ ├── html_link.tests.rs
│ │ ├── html_writer.rs
│ │ ├── html_writer.tests.rs
│ │ ├── html_writer_config.rs
│ │ └── mod.rs
│ ├── mod.rs
│ └── site
│ ├── mod.rs
│ ├── site_articles.rs
│ ├── site_day_path.rs
│ ├── site_devlog.rs
│ ├── site_devlog_entry.rs
│ ├── site_devlog_metadata.rs
│ ├── site_devlog_month.rs
│ ├── site_devlog_year.rs
│ ├── site_location.rs
│ ├── site_month_path.rs
│ ├── site_projects.rs
│ └── site_year_path.rs
└── main.rs
Let’s talk about a few points from this structure:
I often find that you want to easily be able to find your commands for a binary, so I usually put them in a separate module named commands. The helper types (any non-trivial or shared types between the commands), are usually put in some other module (in this case I’ve simply called it common).
This is one of the times that I will break the rule about not putting logic in the mod.rs files - when we organize commands like this, it becomes a really nice way to organize the logic as well.
So my files look like this:
// main.rs
#[tokio::main]
async fn main() -> Result<()> {
let args: Vec<_> = std::env::args().collect();
commands::run(&args)
}
// commands/mod.rs
pub async fn run(args: &[String]) -> Result<()> {
match args.get(1).map(String::as_str) {
Some("generate") => generate::run().await,
Some(command) => Err(anyhow!("unknown cli command: {command}")),
None => Err(anyhow!("no command provided")),
}
}
// commands/generate/mod.rs
pub async fn run() -> Result<()> {
/* ... The actual logic for running the `generate` command ... */
}
In general, when you have some data processing that is inherently hierarchical like this, it tends to be a nice pattern to put a tiny bit of logic in the mod.rs files to not fight the hierarchical nature of the logic.
Another great example of this is when you are writing a derive macro:
.
├── my_derive_macro
│ ├── enum
│ │ ├── mod.rs
│ │ ├── named.rs
│ │ ├── unit.rs
│ │ └── unnamed.rs
│ ├── struct
│ │ ├── mod.rs
│ │ ├── named.rs
│ │ ├── unit.rs
│ │ └── unnamed.rs
│ └── mod.rs
└── lib.rs
This is because usually how we handle these macros is a bit different per type layout (at least a little bit of a difference between them).
There’s actually a lot of sharing we can do between these modules, so don’t look at this as completely separate pieces of code. The main benefit here is how easy it is to find exactly the codepath you’re interested in.
In a later article, we’ll talk about writing maintainable derive macros, don’t you worry - there’s a lot that could be said about that.
I have three different modules under common here;
generator- Logic and types relating to an abstract static site generator.
html- Logic and types relating to writing some abstract html.
site- Logic and types relating to things specific to this website we’re building.
Knowing how and when to split things into organizational modules is a bit more of an art than a science. But I’ve found that you should usually opt to have your split-points be logical places that organize higher-level concepts.
Commands are a good split-point, as mentioned for binaries. For logic, it tends to be a good idea to think about where you would split a crate into multiple sub-crates (even if you don’t end up doing that). So here, it seemed like I might want some crate that can provide general utilities for writing a static site generator, to have some crate responsible for helping write html - and then lastly - some crate that contains types specific to this site.
To be clear, I don’t currently plan on having a separate sigseis-generator or sigseis-html crate - it’s just a good mechanism for identifying separation and grouping points.
These are organizational modules, so I don’t expect to be re-exporting the types from the submodules outside of these. As such, mod.rs here looks like this:
pub mod generator;
pub mod html;
pub mod site;
This tells me that I expect someone to path through these modules to access these types (generator::Generator, html::HtmlWriter, site::SiteDevlog).
It might seem excessive, but I like to create a separate file for each type that I expect to be publicly available. In order to observe the encapsulation principles that I mentioned earlier, this makes sense. However, what might not make sense to all is that I will even do this for types that contain only data.
NOTE: This only applies to public types, if the type is private, it’s usually used internally within a type, and that can remain defined within the same file.
The reason for this is simply organization and ease of finding a type. I never have to hunt for a type within a given file, I can navigate there pretty trivially. Where is GeneratorTraversalConfig? Why, it’s in generator_traversal_config.rs, of course.
I can see folks maybe not liking this, but it works well for me. In the past I have tried being creative and putting things together in the same file - but if the types are too distant from one-another it can be hard to recall where I put one type or another. Also, it’s common to attach logic to types, so it ends up being nicer to keep the types separated in different files so that you can more closely organize the functions with the types (instead of them polluting one big file).
So if I have a Generator type, and I want to define it in the generator module, you can see that I have a separate sub-module explicitly for defining the Generator type:
--- snip ---
│ │
│ ├── generator
│ │ ├── generator.rs
│ │ │
--- snip ---
│ │ │
│ │ └── mod.rs
│ │
--- snip ---
This allows me to easily always know where the type is (generator.rs), and to know that the generator module is merely an organizational module.
For these types, I usually don’t require that folks manually recurse into these organizational modules. As such, the <module>/mod.rs file looks like this:
mod type_a;
mod type_b;
mod type_c;
pub use type_a::TypeA;
pub use type_b::TypeB;
pub use type_c::TypeC;
Here’s a cool tip too: I find that this is a really nice thing to do for declarative macros. Usually I have a few macros that I’m defining in a module, and in those cases, I like to organize the definitions of the macros in submodules.
.
├── macros
│ ├── mod.rs
│ ├── my_macro_a.rs
│ ├── my_macro_b.rs
│ └── my_macro_c.rs
└── lib.rs
I find that this organizational strategy helps when I start having more than one macro, as I can easily and quickly find a macro - since they’re simply organized by name in the module tree.
If there’s only a single macro, I usually don’t do this, though.
Personally, I dislike defining tests within the implementation.
I accept that it’s subjective, but to me it makes little to no sense. I’d rather easily be able to find the files that I want to make changes to, and I’d rather keep tests separate from implementation to make everything nice and readable.
The only reason I could have for placing them in the same file is that I don’t want to create a lone submodule for a single tests.rs module. Certainly, this would get annoying.
--- snip ---
│ │ │
│ │ ├── html_link.rs
│ │ ├── html_link
│ │ │ └── tests.rs
│ │ │
--- snip ---
Well, two things relating to this discussion might surprise you.
This is anyways what even the standard library of rust does for tests (here’s an example for the
HashMaptests).- I only mention this because for some reason people falsely believe that defining tests inline is “the rust way” to do something, it’s very obviously not. Stop telling people this.
Even this is technically unnecessary - you can just use
#[path = "..."].- I say this because I find that finding the submodules of
foo.rsby looking atfoo/...is actually kind of annoying. Plus, usually when folks are defining a submodule, it’s for testing.
- I say this because I find that finding the submodules of
So above, when you see something like:
--- snip ---
. . .
│ │ │
│ │ ├── html_link.rs
│ │ ├── html_link.tests.rs
│ │ │
. . .
--- snip ---
What I have is something like the following:
// In html/mod.rs (the organizational module)
mod html_link;
pub use html_link::HtmlLink;
// In html/html_link.rs (the module that defines the type)
#[cfg(test)]
#[path = "html_link.tests.rs"]
mod tests;
// The definition/logic for `HtmlLink` here...
// In html/html_link.tests.rs (the module that defines tests for the type)
use super::*;
// The tests for `HtmlLink` here...
I don’t find this to be confusing, and it’s pretty clear that the *.tests.rs file is not the implementation, but rather the tests relating to *.rs.
This has all the same properties as tests being defined inline - you can still access private elements of the defined type (that’s just a general property of how visibility works for submodules is that they can access direct parent private elements).
Do note, Rust doesn’t actually care about the filename or format. So *.tests.rs isn’t special or anything. I just avoid *_tests.rs, because that can sort funny in a few cases. Take for instance:
--- snip ---
. . .
│ │ │
│ │ ├── something.rs
│ │ ├── something_another_thing.rs
│ │ ├── something_tests.rs
│ │ │
. . .
--- snip ---
But really that’s only a minor inconvenience, and something/tests.rs is already pretty distant from something.rs anyways… So it’s up to you!
Now, I’d like to talk a bit about why I don’t do certain things.
This is probably controversial - but the reason I personally avoid grouping several types in one file (especially public types) is basically:
It’s really unlikely for a type to be plain (no functions or associated impl blocks).
- A well-defined public type at the very least usually has some manual trait implementations on it - things like
AsRefand so on, which cannot usually be defined using derive macros. So it’s more common than you’d think to have some simple type grow to have someimplblocks despite your intentions.
- A well-defined public type at the very least usually has some manual trait implementations on it - things like
Even if it is plain, it’s easy to lose track of where you defined certain types.
- Will you always remember where you defined that helper type? Even if you step away from the project for a year? I find that I don’t. Sometimes this isn’t an issue (like for
Iterhelpers), but not always.
- Will you always remember where you defined that helper type? Even if you step away from the project for a year? I find that I don’t. Sometimes this isn’t an issue (like for
Just keep it simple - if it’s a public type, it goes in a separate file - the file is named the name of the type. If the type is really generically-named, you’ll probably need an organizing module anyways. Small files allow us to keep related logic together anyways (and have stricter control over visibility).
As mentioned, I try to keep all types defined in sibling modules.
This isn’t strictly a necessary thing, in fact there are times I could envision that you want a sub-type to have private access to a parent type. But you should be making the explicit decision to do that, it shouldn’t be the default way that you build your software.
Usually, even if I have some very closely related types, I’ll define them separately in a containing organizational module, and then grant pub(super) access to the things that I want to share between the related modules.
A good example of this is container types, let’s imagine our own implementation of a Vec type. Instead of one big vec.rs file, I’d instead opt to do something like this:
.
├── vec
│ ├── drain.rs
│ ├── into_iter.rs
│ ├── iter.rs
│ ├── mod.rs
│ └── vec.rs
└── lib.rs
Each of these would pub use into vec/mod.rs. But because I really only want Vec (and related types) constructing these, I’ll use pub(super) on the constructors.
This creates an environment where the types are organized into submodules, and we have very controlled construction points, and very explicit holes poked in where we allow those construction points and who can construct them.
This might seem excessive - but I’ve found that I like the organization.
That’s all for now, hopefully you learned something new! In this article, we:
- Learned a bit about how modules work and the properties they control.
- Gave examples of how I like to organize my Rust crates, and why.
- Talked about why I don’t do certain organizational strategies.
