typed-ident: A Retrospective on a Type-Safe Unicode Identifier Crate

This article is long - TL;DR

• What is it? Type-safe Unicode identifiers.
• Why use it? Correctness with Unicode, flexible conversion forms (Canonical/Decorated), and high performance.
• Key Takeaway: Better boundary handling and defaults than other crates for programming identifiers specifically.

Until I get proper table of contents support on this site, here’s a manual ToC:

• The Use Case
• Common Terminology
• Design Details
  • Design: Identifier Formats
  • Design: Chunk Boundary
  • Design: Character Profiles
  • Design: Conversion Forms
• Crate Comparison
  • Baseline: typed-ident
  • Comparison: convert_case
  • Comparison: heck
  • Comparison: Edge-cases for heck and convert_case
• Interesting Design Notes
  • Design Note: Zero-Cost Casting Feels Like Magic
  • Design Note: Settling on Boundary, Delimiter, and Profile’s API
  • Design Note: Misusing PhantomData
  • Design Note: Fragment, Chunk, and Ident are str/Path-like
  • Design Note: Because of the prior note, this crate cannot be no_unsafe
• Profiling Benchmarks
  • Benchmarks: Ident::new is slower than is_xid_ident
  • Benchmarks: Ident::segments is slower than validation, but by how much depends on the identifier
  • Benchmarks: Ident::to_* conversion is difficult to summarize
• Project Summary
  • Summary: What Went Well?
  • Summary: What Went Poorly?
  • Summary: Future Improvements
• Retrospective Summary

The main use-case for this project is validating, inspecting, and mutating programming “identifiers”.

My fear of working with something like a String is that there’s nothing ensuring that what I have is properly formatted. For better or worse, I often like to punt this kind of thing up to the type layer, and protect construction of the type to ensure I have what I expect.

On top of that, having a typed representation grants me a sensible place to put helper functions relating to identifiers. These are things that I probably wouldn’t want to slap on a string via an extension trait.

I also wanted to be able to be strict about the format. That is to say, I wanted to be able to say “not only do I expect the contents of this string to be that of a valid identifier - but also I expect the format to be that of a lower_snake identifier”.

After constructing one of these types I wanted ways to safely transform and inspect the identifier - things that most people expect like to_lower_camel, or words or something like that.

That is what I needed, and I figured along the way, I’d add whatever seemed like obvious utility. It’s often hard to add such things later than when you are first designing the abstractions.

So in summary:

• Parse and validate the input format of a string (Ident::new)
• Inspect the parts and qualities of an identifier (Ident::segments, etc.)
• Various identifier reformatting operations (core::fmt, etc.)

Before continuing with this article, I’d like to establish some common terminology that I’ll use throughout it.

I thought quite a lot about these terms, so I hope they make sense to you (typed-ident uses this terminology as well).

• Identifier
  • Something validly usable in a programming language for labeling or identifying a variable, function, module, etc. (Different languages have different rules for what a “valid” identifier is.)
• Fragment
  • A slice of an identifier. Not guaranteed to itself be a valid identifier (though, it may be).
• Delimiter
  • A special character (or set of characters) in an identifier specification that is considered useful for delimiting or separating “chunks”.
  • Another definition of this could be - an explicit chunk boundary.
• Boundary
  • A separation between two slices of in-profile characters (in which the slices may be empty).
  • Essentially, a Delimiter is a specific character that causes a split, whereas a Boundary is the logical gap itself, which may be caused by a delimiter or by a change in character case.
• Chunk
  • A string of non-delimiter characters that can be organized into one or more “words” based on a character profile.
• Word
  • The largest slice of a chunk that contains no boundaries.
  • Note: this is not a linguistic word, perhaps it’s best to consider this a “chunk word”, or “identifier word”.
• Segmentation
  • The process of breaking an identifier into structural parts (delimiters, chunks, words, etc).
• Canonical Form
  • A form of an identifier is one where all of the words in an identifier had their casing normalize, and all unnecessary delimiters (for validity or caseless chunk separation) removed.
  • For example, ___2foo__bar_Baz_1_2_ lower-camel canonicalized would be _2fooBarBaz_1_2.

The developer of convert_case published a nice website on this (thank you!): https://stringcase.org/

It presents the following case formats:

I think this is a good starting point, but it felt like it didn’t quite fit perfectly to me for identifiers.

I wanted to provide a system that was open for people to define their own formats (within reason), but also I wanted to provide some presets, so I needed to come up with some names for these presets. I wanted something a bit more uniform in terms of naming structure - and I wanted to cut the formats that I consider not very useful for identifiers (like the ones with space as a delimiter).

I realized a few things about these presets, as I set-out to define them:

1. You could broadly group these into explicit-delimiter-preferred, and case-delimiter-preferred identifiers (e.g. lower_snake versus UpperCamel, etc).
2. Languages usually supported some form of explicit delimiters in all formats (e.g. despite being UpperCamel, you could force a boundary in cases where it would not naturally form; like in Tls1_3Config).
3. There was a need for a special identifier that was kind of like a camel-cased identifier, but supported both low line (_) and hyphen-minus (-) delimiters.
4. Not all combinations of these things needed to have a preset - because the combinations weren’t always interesting.

Here’s what I came up with from these rules.

Prefers explicit delimitation:

• Though I believe (underscore / hyphen) delimited variant to be uninteresting, I did come up with a fun joke name; Zolom-case, named after the Midgar Zolom from Final Fantasy 7, the giant serpent that Sephiroth skewered (get it - Snake + Kebab?).
• A more serious, better, and realistic name would simply be “snake-kebab” or “kebab-snake”, however.

Prefers case delimitation:

• Though I believe hyphen delimited variant to be uninteresting, I did come up with a fun joke name; Horse-case, inspired by the carousel horses being “skewered” by a large pole, and just that they’re generally mammals to stick with the naming scheme.
• A more serious, better, and realistic name would simply be “kebab-camel” or “camel-kebab”, however.

This leads to a few interesting observations.

• Some cases are subsets of others - some of these subsets are less obvious, for instance lower_snake can parse as lowerCamel (it’ll parse as explicitly-delimited with underscores; lower_camel).
• Some cases would be exactly the same as others - a theoretical MixedSnake and MixedCamel are basically the same thing. The only difference is intent - this is used like a snake-cased thing, not like a camel-cased thing. Syntax-wise, they’re identical, though.
• Mixed-Hybrid is a superset of all of these cases - this, as it turns out, is a super nice property. Don’t know what you’re working with, but you know it’s some kind of identifier? Well, let’s just process it as MixedHybrid. Anything should be able to cast to that (see the Zero-Cost Casting section below for this - it’s essentially a compile-time cast!).

NOTE: You can explore the presets in the documentation for the presets module. Today, SnakeIdent and KebabIdent are implemented as Mixed, but I want to make them Uniform in the future.

For a long time, I couldn’t find the documentation I needed to send me in the right direction here. And I was trying to solve for way too many “edge-cases” for how case boundary formation worked.

I tried to handle things like version postfixes, and breaking acronyms in special places, etc.

Very thankfully, I eventually came across Unicode Technical Standard #55.

This did two things:

1. Made it really clear that my decisions were based on something reasonable that people other than just me had thought about.
2. Brought to my attention a really weird edge-case of boundary formation - titlecase characters.

Titlecase characters are weird, because for the most part we visually consider titlecase to be equivalent to uppercase - except for non-greek titlecase.

Non-greek titlecase characters look like a combination of two characters (one uppercase and one lowercase), so visually it appears like it’s the start of a capitalized word (starts visually uppercase, ends visually lowercase). Greek titlecase, on the other hand, just visually appears fully uppercase.

• Example Greek Titlecase: ᾈ, ᾨ, ῌ, ᾚ
• Example Non-Greek Titlecase: Dž, Lj, Nj, Dz

So, we consider this in typed-ident for boundary formation (source code).

The main thing that we need - what even is a “valid identifier”. What does that even mean?

Well, unfortunately, it basically depends on who’s asking - there is no one definition of a “valid identifier”. But there is something kind of close - Unicode Standard Annex #31: Unicode Identifiers and Syntax.

While this standard doesn’t account for all possible identifiers (that would be impossible), I think it does a great job of laying the ground-work for a common base to think about for identifiers in a Unicode codebase. It leaves a few things up for configuration, but there are several very nice definitions here.

The gist of the document is that a valid identifier could be formed by:

• One character from XID_Start.
• Zero or more characters from XID_Continue.

This seems like as good a starting place as any for identifier validation - I used unicode-ident for validating our Unicode character profiles.

This one I had to play a bit more by-ear - because what I want didn’t seem well-represented anywhere else.

Through the development of typed-ident, I eventually settled on calling delimiters that weren’t syntactically necessary “decorations”. I don’t know if there’s a better name for these. But for example, I would say that - for lower-snake case - foo_bar_baz has only necessary delimiters (no decoration). But something like __foo_bar_baz does have some decoration (the leading __).

The reason I needed a special name for this was - the style of conversion that everyone was implementing currently - I actually didn’t want that. So I needed to name the current form that everyone used, and come up with another form that I would also provide.

I often wanted to persist decorative delimiters. I still did want it to be possible to remove delimiters it they weren’t necessary - and I could even see me wanting to keep all delimiters even if they aren’t necessary.

So I ended up naming these conversion forms:

• Canonical Form: When you normalize all of the word casing, and remove all unnecessary delimiters (including decorative delimiters).
• Decorated Form: When you normalize all of the word casing, and remove only non-decorative delimiters (if possible for the target format).
• Delimited Form: When you normalize all of the word casing, and you persist any present delimiters from input to output.

So, for instance, here’s a string: __lower__snake_case__

• Canonical Upper-Camel Form: LowerSnakeCase
• Decorated Upper-Camel Form: __Lower__SnakeCase__
• Delimited Upper-Camel Form: __Lower__Snake_Case__

I think, especially for an identifier-conversion library, it’s important to have a way to persist decorative elements. In fact, I would argue that should be the reasonable default.

However, most current case conversion libraries shed delimiters, so I think you would be hard-pressed to make that the default without surprising at least a few people.

So while I note that these forms are useful, the way I remove confusion in the API is by having them be explicitly named in their conversion function (e.g. there is no to_lower_camel, there’s instead a to_lower_camel_canonical, to_lower_camel_decorated, and to_lower_camel_delimited to force the user to make a choice).

NOTE: Snake and kebab are interesting in that their delimited form is identical to their decorated form - so for those identifier formats I simply don’t provide a to_*_delimited function. For more details on all this, see the fmt module documentation.

The major contending crates (at least that I am aware of) that provide similar functionality are…

Well, none that meet all these specific requirements.

But to be fair, the “typed” part of typed-ident is a little obscure. If we look at case conversion specifically (a very common operation on identifiers), there are a few.

Of them, I think there’s a few worth mentioning - let’s do a quick review of them.

This is just to create a baseline for comparison of the next crates in the list.

• When to Prefer: When you want to enforce the format of some input identifier, perform more complex operations on identifiers, or you want the most-correct-for-identifiers conversions.
• Conversion Speed: Our benchmark tests (randomly-generated 12~128 character identifiers) clocks 1133~7290 ns per-conversion (varying based on input data and target output format).
• Binary Cost: Release mode produces a ~452K binary for the conversion-typed example.

Example Conversions

Note: For brevity, upper-snake and the kebab variants have been omitted, since they’re largely similar to “LowerSnake”

Analysis

This might be more API than you need - but the Rust compiler is good at cutting out things you don’t use. If all you need is conversion, just use some permissive identifier format (like CamelIdent for rust identifiers), and then convert.

• When to Prefer: If you need to convert cases of things other than identifiers, or if the strict subset of formats that we support in typed-ident aren’t flexible enough for your use-case.
• Conversion Speed: Our benchmark tests (randomly-generated 12~128 character identifiers) clocks 6972~10108 ns per-conversion (varying based on input data and target output format).
• Binary Cost: Release mode produces a ~487K binary for our conversion-ccase example.

Example Conversions

Note: For brevity, upper-snake and the kebab variants have been omitted, since they’re largely similar to “LowerSnake”

Analysis

convert_case is quite a bit generic - it’s meant to deal with a wider variety of cases. I think if you aren’t dealing specifically with identifiers, you should think about using this.

Here’s a few points of criticism for the crate:

1. It only considers the canonical-form of case conversions.

Well, kind of? It’s more configurable than the other comparison crate (heck), but I can’t think of a way I could get something like decorated or delimited form out of it. It’s mostly based around “words”.

2. There are some conversion edge-cases that either aren’t handled or I disagree with.

This is kind of a catch-all. We’ll cover these in a bit, since both convert_case and heck seems to make these similar decisions.

I’ll dive deeper into exactly why I disagree with these boundary decisions in the Edge Cases section below.

NOTE: I was originally going to call-out the higher runtimes of convert_case, but they’ve just released an update specifically focusing on this which invalidated my original benchmarks! The current article accounts for the latest version (0.12.0) - but it’s greatly improved from the prior!

• When to Prefer: When you have simpler conversion needs, and can accept some small incorrect conversion steps (e.g. improper grapheme processing, dropping accents, etc). It’s faster and by far the simplest crate of the bunch.
• Conversion Speed: Our benchmark tests (randomly-generated 12~128 character identifiers) clocks 2118~5668 ns per-conversion (varying based on input data and target output format).
• Binary Cost: Release mode produces a ~459K binary for the conversion-heck example.

I attribute most of the lower size and faster speed compared to convert_case to the way heck’s transform function works (the main conversion function). However, that’s also somewhat its detriment.

Example Conversions

Note: For brevity, upper-snake and the kebab variants have been omitted, since they’re largely similar to “LowerSnake”

Analysis

heck is a very simple, no-std, and no-unsafe case conversion library.

Here’s a few points of criticism for the crate:

1. It drops combining marks - this is a huge dealbreaker for me.

For this one, the issue is that chunk breaking rule of !c.is_alphanumeric(). Combining marks aren’t alphanumeric, so they’re treated as chunk delimiters, which means they’re dropped.

NOTE: I attempted to create an issue for this on GitHub, but it appears the issue board is locked currently. This seems like a pretty bad issue to me.

2. It only considers the canonical-form of case conversions.

Like convert_case, this is mostly based on “words”, so there’s no way to get the other forms out of it. The API is mainly limited to the conversions that it provides.

3. There are some conversion edge-cases that either aren’t handled or I disagree with.

This is kind of a catch-all. We’ll cover these in a bit, since both convert_case and heck seems to make these similar decisions.

Some of these appear intentional, others are a side-effect for the way that the transform function works. Needless to say, I find them hard to reason about, and sometimes just flat-out disagree with.

I’ll dive deeper into exactly why I disagree with these boundary decisions in the Edge Cases section below.

1. On case conversion, I want chunk boundaries to be persisted.

That is to say, if I had something like tls1_3, I want that to become Tls1_3 in camel case - not Tls13. Simply put, if two chunks are explicitly separated, I imagine there must be a reason for it. I do NOT want them to be joined.

Joining can happen if there’s a way for the boundary profile to allow the two chunks to be joined while maintaining a boundary (e.g. Tls_Config to TlsConfig), but joining something that was delimited into something that isn’t is not allowed.

2. On case conversion, I want necessary delimiters to be persisted.

This sounds similar to (1), but it’s slightly different.

Consider the identifier: _2x_multiplier

I want this to convert in a way that would persist the leading _, because dropping it would lead to an invalid start character.

This is basically made possible by the Profile trait in typed-ident.

Note that this includes not stripping away all delimiters when there are no words (e.g. ____ -> _ canonical)

3. On camel-case conversion, I don’t want to form unnatural boundaries on uppercase expansion.

This one deals with an edge-case surrounding characters like ß.

This character uppercases to two characters (SS). But if you just trivially apply that mapping, and then lowercase the rest, you can end up in a situation where you have inadvertently inserted an awkward boundary.

• ßtest -> SStest -> [S, Stest] (weird!)

I would rather the first cased character be uppercase in the expansion, and then any additional characters are re-processed to lowercase to avoid this.

• ßtest -> Sstest -> [Sstest] (less weird!)

Still weird, but I think the main metric here should be “don’t add surprising boundaries”.

If you’re curious how this is implemented, see the writer.rs source code.

4. I want to be clear on my stance on localization - we don’t do it.

One subtle surprising property of heck is that it does apply some localization elements. Specifically, it will map Σ to ς when at word-end.

I can see why one would want this, but I kind of want to draw a hard line in the sand. We’re not trying to be a localization library, so I would rather just respect the case mapping that Unicode itself provides. And if users have additional localization needs, that’s on them.

Just feels like a slippery slope that I don’t want to slide down…

5. I disagree with the defaults provided by these libraries.

Well, kind of - I disagree with convert_case, but heck I kind of just disagree with the whole boundary implementation.

For convert_case, the default boundary rules say that digits can form boundaries against other non-digit characters. This is probably reasonable for a library like convert_case, but I don’t think it’s reasonable for a library focusing specifically on identifiers.

My reason for this is: there’s just too many ways you form weird or unnatural boundaries with those rules.

The simplest example is Transform3D - this segments as such considering digits: transform_3_d

I find that separation between the 3 and d to be super awkward. There’s no perfect solution here, but I’ve become a fan of just not introducing a boundary unless we’re certain it should be added. Since there’s no casing reasons why there should be a boundary anywhere in that string - typed-ident would convert this to simply: transform3d

For heck - the way transform works is by pulling forward the case of the prior cased characters within a chunk, and using that in lieu of having actual case information for future boundary decisions.

This is clever implementation strategy, and allows us to avoid using unicode-segmentation (which is the source of the fast runtime and small binary size of heck). But it produces weird, and difficult to reason about boundaries.

Taking from the same example above provided to convert_case, Transform3D would become: transform3_d

Needless to say, I find this also to be confusing and hard to describe.

With all that out of the way - I think the next main thing to talk about is anything interesting from the design.

You can probably just look at the documentation if you want to know about any of the API surface, but I’d like to comment on why I did one thing versus the other, or places where I learned something new.

In case you missed it, there’s a really cool function on Ident: cast

This function is also used for defining AsRef implementations on the type, so you could use that from a trait perspective. But it’s really cool to be able to effectively check at compile-time if one format is a subset of another, and therefore trivially castable to it.

The way this works is with a helper marker trait: SubsetOf, which you can define on profiles and delimiters to say whether one implementation contains another.

Then, we just check via the trait system if this is satisfied and allow the cast:

pub const fn cast<B2, D2, P2>(&self) -> &Ident<B2, D2, P2>
where
    D: crate::syntax::SubsetOf<D2>,
    P: crate::syntax::SubsetOf<P2>,
{
    Ident::new_unchecked(self.as_str())
}

At first, this was just something I was playing with because it sounded interesting, but it quickly became a really nice way to provide some nice ergonomics around writing generic identifier functions.

For example, you could write a function generically over all things shaped like a camel identifier:

fn do_some_work(ident: impl AsRef<CamelIdent> + ?Sized) {
    // ...
}

do_some_work(UpperCamelIdent::new("UpperCamel")?);
do_some_work(LowerCamelIdent::new("lowerCamel")?);

It goes without saying that typed-ident cannot cover all formats of identifiers.

I originally tried making it that generic, and that’s partially why this project took so long to release. It was so generic that it was unusable.

As such, I needed to settle on some properties for how these three configurable elements worked.

I think the elements in question are pretty obvious. From the information we gathered, it’s pretty clear these are the three things under the control of a specific implementation of an identifier.

Boundary had probably the biggest amount of churn, funny enough.

The reason is the Segmentation strategy type. Originally it was defined on the boundary how grapheme segmentation worked.

pub trait Boundary {
    const CAN_FIND_BOUNDARIES: bool = true;

    // How we iterate over graphemes - this is needed so that ASCII can claim a
    // non-`unicode-segmentation` implementation is viable.
    type Segmentation: Segmentation;

    fn find_boundary(chunk: &str) -> Option<NonZero<usize>>;
    fn rfind_boundary(chunk: &str) -> Option<NonZero<usize>>;
    fn has_boundary_at(chunk: &str, idx: usize) -> bool;
}

This made sense in that boundaries are what would actually be iterating over graphemes, but it made less sense in that the Profile trait was what actually was responsible for making the call about how grapheme iteration could be done.

Very late in the design, I finally decided to have it set on Profile, and then have Profile provide it to Boundary as a generic type.

This cleaned up a lot of headache for me.

One other notable thing for Boundary, I wanted it to be a pretty generic API surface, because I wanted others to be able to implement their own boundaries if they disagreed with mine. That’s why it’s just generic stuff like find_boundary and has_boundary_at.

Delimiter on the other hand, had the least amount of churn. I pretty much knew exactly what I wanted here from the start and didn’t change it once.

Well, that’s not totally true. Later in development, it became obvious that the delimiter could be optimized based on information that only the delimiter implementation knew - so I added a find_delimiter function to the trait. This allowed us to optimize ASCII delimiter searching by avoiding any kind of UTF-8 decoding whatsoever.

Profile had two things that changed a lot - the names of the functions (naming is hard), and the APPEND_CLOSED constant.

APPEND_CLOSED is an interesting concept to have on a profile, but it’s super important for performance. Removing it does cost a lot, and many of the profiles are able to be append-closed so it seemed worth having.

The original form of this was just a bool. But the problem was, there were some operations that might benefit from further optimization or would be able to act more appropriately if they knew just how “append-closed” the profile was.

So, to future-proof it, despite the current implementation only really caring if it’s Empty or Fragment append-closed, I turned this property into a type.

NOTE: This is already documented pretty thoroughly in the crate documentation. If you want to understand APPEND_CLOSED better, I recommend reading that documentation.

So, my types for defining an identifier aren’t types that you’d construct, but they need to be a part of other types. That sounds like a job for PhantomData (and in fact it is).

I originally just used this type to hold a tuple of the types I needed to pass around (like PhantomData<(B, D, P)>). But this is wrong in a way that only lifetimes could expose.

The place I ran into this being an issue was iterator types. Consider the following iterator definition:

pub struct Words<'a, B, D, P: Profile> {
    syntax: PhantomData<(B, D, P)>,
    iter: WordStrs<'a, B, P::Segmentation>,
}

impl<'a, B: Boundary, D: Delimiter, P: Profile> Iterator for Words<'a, B, D, P> {
    type Item = &'a Chunk<B, D, P>;

    #[inline]
    fn next(&mut self) -> Option<Self::Item> {
        self.iter.next().map(Chunk::new_unchecked)
    }
}

Compiling this, you will - in some cases - run into places where the lifetimes aren’t satisfied. It will say that B, D, and P don’t live long enough.

This really confused me at first, but after re-reading how PhantomData is supposed to be used, it should be used to mark types that your type may produce.

This Chunk::new_unchecked produces a &'a Chunk<B, D, P>, and so the way to fix this is to mark the phantom type as PhantomData<&'a Chunk<B, D, P>>.

This actually means something different - suggesting that the type it produces is one that satisfies the lifetime of 'a, which makes it covariant with the lifetime. This means that I won’t have to litter 'a lifetimes in all my impl blocks (this can get very annoying, because it can also lead to this virally being required in other places where you use the iterator type).

I wanted to have this separation between typed an untyped iterators so that I could either give people &'a Chunk<B, D, P> or &'a str data if they type-erase it. This understanding of PhantomData was key to making this sanely possible within my API.

You can read more about subtyping and variance in the Rustonomicon.

When I say this, I mean that you address the types as references &Ident.

It didn’t start out that way, I originally started with a structure that contained a string slice, with some phantom data to control the identifier-specific properties.

Efficiency-wise, this is effectively the same thing, but there’s a few things that become difficult using the Ident<'a> pattern.

1. I wanted to be able to slice an Ident to get a Fragment.

I literally defined a fragment as a slice of an identifier. But look at the Index operator definition:

pub trait Index<Idx>
where
    Idx: ?Sized,
{
    type Output: ?Sized;

    // Required method
    fn index(&self, index: Idx) -> &Self::Output;
}

The main issue here is that reference on the output of index.

So if I want to support slice operations on an identifier (or fragment or chunk or whatever) - I can’t use the Ident<'a> format.

I could give this up, honestly - it’s a nice-to-have. But that’s not the only issue here.

2. I wanted to be able to Box<Ident> any of these types.

This is actually more important than slice operations.

The reason why is an Ident is weird - it is only valid if certain conditions are met, one of which being that it’s non-empty. IdentBuf protects us from forming syntactically-invalid identifiers, but it starts out empty. This makes it not a good candidate for the “dynamic/owned” version of an identifier (something that would Deref to Ident).

The original version of the code had IdentBuf disallow being empty. But this is such a pain in the ass - and kind of defeats the main purpose of the type.

What you want to do with an IdentBuf is build an identifier. It’s kind of a pain to have to provide it with a valid starting identifier “fragment” before you can even start using the thing.

Plus, it complicates all the logic for mutating the buffer (how do we deal with removal which leaves the buffer empty? anything that returns an IdentBuf now may be optional, that complicates the API surface for users, etc.)

I originally wasn’t going to support Box, but adding it gave me this really nice property - I could have a valid dynamic identifier, that maybe isn’t mutable, but it came from some prior mutation and was validly non-empty.

For example - consider the function with_prefix_str.

If this function succeeds, it can only add data to an identifier, so it’s never empty. As such - let’s return a Box<Ident>. Boom - still usable as an identifier, still guaranteed non-empty.

You could say “why not have a special type that represents a non-empty Ident, like a NonEmptyIdentBuf?”

The problem with this is - in order to make it nice and usable as an Ident, I need to either provide an as_ident function (which feels weird from an API perspective - it is after all a valid ident) or implement Deref.

Let’s look at Deref:

pub trait Deref {
    type Target: ?Sized;

    // Required method
    fn deref(&self) -> &Self::Target;
}

And crap - we’re back at requiring a reference…

Even if we were to accept the small annoyance of typing .as_ident() to cast to our ident type, we have another problem - NonEmptyIdentBuf still cannot be mutable. It’s very much trying to be a Box<Ident> in struct clothing.

So, it just feels like we’re purposefully stepping away from the proper API for no good reason. I don’t love it.

3. We want these types to play nice with Cow.

I won’t continue to bemoan the whole “uh-oh the trait requires a reference again” thing, so let me just cut to the chase.

Take a look at Borrow:

pub trait Borrow<Borrowed>
where
    Borrowed: ?Sized,
{
    // Required method
    fn borrow(&self) -> &Borrowed;
}

If I want to support ToOwned and borrowing between things that can be represented as these other types - I need them to be able to be reference-able.

• Box<Ident> is able to Borrow to an Ident.
• FragmentBuf is able to Borrow to an Fragment.

Everything works as one would expect (though admittedly it is still a tad odd that IdentBuf isn’t the owned variant of an Ident; it would be nice to be similar to how PathBuf relates to Path).

I’m going to be honest with you - I’m not scared of unsafe for this crate.

The whole point of unsafe is to control the places where we need unsafe code, so that we can closely scrutinize it, and to give us the tools we need to extend the language in a sensible way.

Now what this doesn’t mean is to just use unsafe all over the place.

My general policy on it is: come up with a sensible API that one would expect in Rust, and if parts of it cannot be expressed in safe Rust, you may need to (carefully) use unsafe.

There’s plenty of places in typed-ident I really probably could have used an unsafe function, but I didn’t deem it necessary. (For example, an Ident is never empty, but I simply unwrap in functions like Ident::first_segment, instead of unwrap_unsafe. True this is possibly a slight performance degradation, but it’s a pretty cold function. The nice thing about just unwrapping in such cases where things “shouldn’t be that way” is that you catch if the user did something bad to make them that way.)

At the end of the day, there’s exactly four patterns of unsafe in play (and three of those are basically just casts to convert string slice to our str-equivalent types):

1. T::new_unchecked to cast a &str to a &T (for Fragment, Chunk, and Ident).
2. T::new_boxed to convert a String to a Box<T> (for Fragment, Chunk, and Ident).
3. T::into_boxed_str to convert a Box<T> to a Box<str> (for Fragment, Chunk, and Ident).
4. Pass-through unsafety for get_unchecked for the user of the API.

I think (4) is pretty defensible - we’re mimicking functionality on a str but for a Fragment - one of those is an unsafe version of slicing a string (str::get_unchecked), and so it felt obvious to add an equivalent such operation to Fragment.

I like to call these operations “pass-through unsafe”, because they expose some already battle-tested API that itself has an unsafe variant. In this case it’s the string slice’s str::get_unchecked function.

For (1), (2), and (3) - these are all basically the same thing.

The only risk of these being wrong would be if any of the structs these are casting to aren’t marked as transparent, or aren’t the same alignment, size, layout, etc as a string slice. But, much like Path, at the end of any of these chains of structures is a lone str value.

It’s safe, but it’s dealing with memory transmutation, so there’s no good way to express that in safe Rust.

NOTE: The method for producing these benchmarks was to produce thousands of random, valid identifiers of length 12~128 characters, and then to process each of them 200 times and produce the final average time discarding outliers (we used Criterion for benchmarking).

This should be obvious, but saying there is a required format is slower than saying “I just want some XID-valid identifier”.

However, it doesn’t tend to be that much slower. The main differentiator between how fast something is depends on whether or not it’s ASCII data.

High-Level Points

• The more case-checks needed, the higher the cost. Mixed has no case checks, and Upper/Lower has one per character.
  • General cost: Mixed <= UpperCamel <= LowerCamel < Upper < Lower
• About case profiles…
  • Mixed tends to be the “most equivalent” to the baseline (this makes sense).
  • UpperCamel and LowerCamel formats tend to only be 22~23% more expensive than mixed.
  • Upper and Lower formats tend to be between 78~113% more expensive than mixed.
• About char profiles…
  • Ascii is significantly the cheapest of them all - if you know your data is ASCII, use it.
  • Unicode is mostly what you’ll want otherwise - 122% more expensive than ASCII.
  • Strict is the most expensive given the additional checks - 160% more expensive than ASCII.
• Comparisons across char profiles is uninteresting, so the worst cost you’ll see within the same character profile is 113% (Lower).
  • But we’re talking multiples of nanoseconds (351ns vs 707ns), and this is over test data that is usually much larger than your actual identifiers are going to be - so just do what makes sense.

Again, this should be obvious - both functions involve iterating each character.

How slow it is differs depending on the kind of identifier it is, and over which charset. But for typed-ident the major factor is whether or not the type has NoBoundaries (Lower / Upper case profiles).

High-Level Points

• When comparing this with equivalent-ish functionality in convert_case (split), typed-ident was always faster.
  • This used to be by 3x - However, after the 0.12.0 release, convert_case brought its time down to within one order of magnitude from typed-ident. We are still faster, but only by ~20% on average.
  • However, because of the heavy optimization of NoBoundaries idents, typed-ident is still much faster there.
• Strict and Unicode are identical, so only Ascii and Strict/Unicode have performance differences
  • This is because Strict is just Unicode with extra validation checks, validation doesn’t matter for segmentation.
• Identifiers with NoBoundaries are the fastest to segment, but all others boundaries take about the same time.
  • This is because when we know there’s no boundaries, optimizations can occur to only look for delimiters.
  • The only presets that use NoBoundaries by default are lower_snake, lower-kebab, UPPER_SNAKE, and UPPER-KEBAB.
• Identifiers with NoBoundaries have no performance difference between the various char profiles (Ascii, Strict, Unicode).
  • Again, this is because when we know there’s no boundaries, optimizations can occur to only look for delimiters.
  • Since we have defined delimiters to always be exactly 1 char, it only needs to do char iteration in all cases.
• The only other major time difference is in selecting Ascii vs Strict/Unicode profile for boundary-containing identifiers.
  • Strict/Unicode takes 2.7x the time that Ascii takes for segmentation of boundary-containing identifiers.
• Segmentation across all identifiers thus differs vastly depending on the identifier configuration used.
  • At the low-end (no-boundaries) about the same as new (372.84~408.6ns), all the way up to ~4683ns (boundaries, unicode).

NOTE: When SnakeIdent and KebabIdent have a Uniform chase profile, they will be able to be NoBoundary as well (today they’re Mixed as a temporary measure).

Format conversion is kind of just another way of saying “segmentation + formatting”.

I think this is going to be the main thing that people want to see a comparison over (against convert_case and heck). However, all 3 of our crates have made some different decisions with respects to defaults, and to what is considered a “valid” format conversion.

So first we have to talk about some key design differences in the reformatting process.

Design Differences

• typed-ident considers the profile’s valid “ident start” character, keeping “necessary” leading delimiters.
  • Both convert_case and heck will case-convert away leading delimiters, even when they’re actually necessary (e.g. _2_example_case -> 2ExampleCase, instead of _2ExampleCase).
• typed-ident will keep a single-delimiter for anonymous identifiers.
  • Both convert_case and heck will case-convert away all delimiters if there’s no “words” (e.g. ___ -> , instead of _).
• typed-ident will not drop “required” delimiters between chunks that force separation.
  • Both convert_case and heck will, in some cases, join two intentionally separated chunks (e.g. TLS1_3Config -> Tls13Config instead of Tls1_3Config).
• typed-ident handles “capitalization” of camel cased boundaries differently.
  • Both convert_case and heck do uppercase expansion (e.g. ßtest -> SStest).
  • typed-ident considers this invalid, because we’ve created an awkward boundary between S and Stest. It will instead produce Sstest for the same transformation.
• typed-ident has decided that the default boundary configuration SHOULD NOT include boundaries against ASCII digits.
  • convert_case enables such boundaries by default, though along the same token, you can disable it.
  • heck does not respect ASCII digit boundaries. Though it may seem to sometimes, that’s a side-effect of the way it transforms.

You could re-implement heck’s boundary policy in typed-ident, actually (leading to a similar reduction in binary size and speed improvements)! However, I don’t want to make that boundary policy the default for the crate - it’s difficult to justify as such.

Regardless, the main factors for performance are the inputs and output formats. Note that for outputs; Delimited means something like (lower_snake or UPPER-KEBAB, etc), but Bounded means something case-bounded like (lowerCamel or UpperCamel, etc.)

High-Level Points

• Like always, there’s a difference between having Ascii and Unicode data - ASCII is always faster (this makes sense).
  • Most implementations were an orders of magnitude faster in the average case.
• convert_case was always the slowest among all test inputs of the same charset - but again the 0.12.0 update brought them within a single order of magnitude of typed-ident.
  • With these changes I’m happy to report that it then mostly comes down to preference - it’s only a little slower.
• So for typed-ident…
  • ASCII data ranges from 1139~4178ns per call - (0.52~1.91x the cost of heck; sometimes cheaper, sometimes more expensive)
  • Unicode data ranges from 1133~7290ns per call - (0.20~1.29x the cost of heck; sometimes cheaper, sometimes more expensive)

I believe this demonstrates that heck and typed-ident are within a single order of magnitude from one another, so which you use depends on what you think of the boundary calculation, and whether you need the surrounding capabilities.

• Literally every bit of testing caught some bug along the way - bench testing, unit testing, doc testing, etc.
• Speaking of, I love doc testing - it’s great not only to demonstrate simple usage of an API, but also prove (via doctests) that it actually functions.
• Documentation was also important - sometimes I would start documenting things, and then I’d think “this is hard to explain” because it was named or abstracted poorly, leading to several redesign passes.
• Bench testing was really important to get an idea of how we fared against the other implementations - this is important if there are other crates that do something similar to what you do.
• The only thing I used AI for was to help me with naming and review my code. I don’t really like AI, but it’s pretty great at asking questions like “what’s a good name for…”, “can you review docs/code of…”, etc.
• Loved that I learned about PhantomData and Unicode in great detail in this project (always good to learn something new).

• I wrestled with that non-empty property of identifiers a bit too much. Settling on it only being possible for Ident was the best final design choice
  • I originally also applied it to Chunk and Fragment, that was a nightmare.
  • I suppose the learning here is “not everything needs represented in typestate”.
• I spent a lot of time trying to solve too many edge-cases for boundary formation, when keeping it simple was probably best.
  • Maybe I should have spent more time really thinking hard about the boundary calculation instead of just kitbashing it.
• I feel though I’ve implemented a lot of things, I know I didn’t need all of them right now - so I probably implemented more than I needed.
  • This is a double-edged sword - I think it’s hard to redesign stuff later, so would rather get it all out there on paper now. But also, I could easily see someone looking at this and saying “yeah, no - that’s too much code, thank you”.
• Maybe I spent a little too much time bench testing and looking at data relating to that. I was kind of just having fun there…
  • I suppose what I learned here is to focus on what you actually need to deliver and save the rest for post-release.
  • There are a few things I’ve left undone in this crate - believe it or not. So I’m trying to put that into practice now.
• I was really bummed to learn that case conversion often couldn’t move you into a definite other obviously-typed identifier format.
  • Not sure what I learned here, I guess “you can’t always get what you want”.

• I want to add a Uniform case format which will allow for SnakeIdent and KebabIdent to only be represented by uniform cases. (Today, it’s Mixed, which is surprising because it means SnakeIdent = CamelIdent, counterintuitively).
  • This will definitely be a breaking change, but hey - this is why were not in beta!
• The next most important is generating the tables for non-Greek titlecase (right now they’re manually authored).
• Adding syn support so that you can zero-cost cast between a syn::Ident and certain typed_ident::Ident configurations.
• Adding fuzz testing over the mutation methods - I think they could really benefit from them.
• Adding integration testing for more thorough checks on conversion properties.
• Mostly though, I need to just use it and see how it goes before claiming it’s a beta - and get feedback from others.

Thanks for reading all this! I hope you learned something along the way (I know I did).

I certainly haven’t listed everything interesting here, but this article is getting long… And I want to release this thing eventually. If you have any questions, don’t hesitate to reach out to ask them!

Additionally, if you like my work and want to support me to continue seeing crates and articles like this, I’d greatly appreciate any donations. Check out my “Support Me on Ko-Fi!” page if you’d like to drop me a few bucks for my effort. I’d really appreciate it!