Unhandled Expression

nom 5 is here

2019-06-17T11:00:00+02:00

nom, the Rust parser combinators library, is now available at version 5. This is the most mature version of nom. This is the one that feels “done”. This is the parser library that I wanted when I started nom 5 years ago. It’s here at last.

nom 5 is a complete rewrite of the internal architecture, to use functions instead of macros, while keeping backward compatibility with existing macros based parsers, and making the error type completely generic.

As an example, here are some elements of a JSON parser:

fn string(i: &str) -> IResult<&str, &str> {
  delimited(
    char('\"'),
    parse_str,
    char('\"')
  )(i)
}

fn array(i: &str) -> IResult<&str, Vec<JsonValue>> {
  delimited(
    char('['),
    separated_list(
      char(','),
      json_value
    ),
    char(']')
  )(i)
}

fn json_value(i: &str) -> IResult<&str, JsonValue> {
  alt((
    map(hash, JsonValue::Object),
    map(array, JsonValue::Array),
    map(string, |s| JsonValue::Str(String::from(s))),
    map(double, JsonValue::Num),
    map(boolean, JsonValue::Boolean),
  ))(i)
}

Why?

Why move away from macros? And why use macros in the first place if it could be done with plain functions?

The short answer is that it couldn’t be done until recently. The long answer is that when I started nom in 2014, Rust was different, less powerful. At that time I had an intriguing idea: what if we had parser combinators that can return a reference to the input data instead of copying it? This could fix some of the performance problems that were frequent with parser combinators.

It turned out that yes, it was possible, but making it usable required some work. I needed to add lifetimes everywhere; closures were hard to manipulate. Combining functions directly, which was my initial vision based on Haskell’s Parsec, was not possible. Still, declarative macros were already working pretty well, and gave us a powerful meta programming tool.

So nom ended up being a macros based DSL, that would generate code depending on macros calling other macros, rewriting their arguments, etc.

// named is used to declare a function and its arguments
named!(hex_primary<&str, u8>,
  map_res!(
    take_while_m_n!(2, 2, is_hex_digit),
    from_hex
  )
);

named!(hex_color<&str, Color>,
  // do_parse applies parsers in sequence
  do_parse!(
           // tag recognizes a specific string
           tag!("#")   >>
    red:   hex_primary >>
    green: hex_primary >>
    blue:  hex_primary >>
    (Color { red, green, blue })
  )
);

assert_eq!(
  hex_color("#2F14DF"),
  Ok(("", Color {
    red: 47,
    green: 20,
    blue: 223,
  }))
);

For an insight into how it works, see how the opt! combinator, that wraps a parser result in an Option (using None if there’s an error), is defined:

#[macro_export(local_inner_macros)]
macro_rules! opt(
  ($i:expr, $submac:ident!( $($args:tt)* )) => (
    {
      use $crate::lib::std::result::Result::*;
      use $crate::lib::std::option::Option::*;
      use $crate::Err;

      let i_ = $i.clone();
      match $submac!(i_, $($args)*) {
        Ok((i,o))          => Ok((i, Some(o))),
        Err(Err::Error(_)) => Ok(($i, None)),
        Err(e)             => Err(e),
      }
    }
  );
  ($i:expr, $f:expr) => (
    opt!($i, call!($f));
  );
);

The first argument $i is the input passed by the calling macro, and it is given as first argument to the child parser, so if we had opt!(input, parser), we would generate the following code:

let i_ = input.clone();
match parser(i_) {
  Ok((i,o))          => Ok((i, Some(o))),
  Err(Err::Error(_)) => Ok((input, None)),
  Err(e)             => Err(e),
}

While macros could become complex, the generated code was quite simple, mainly using nested pattern matching. This made most combinators easy to write, and the generated code was fast.

At the same time, macros had issues that plagued nom for a long time. The smallest typo when calling a macro would result into inscrutable error messages. Macros variants that were not tested were never compiled (it was harder to catch mistakes). Macro argument parsing limited the syntax.

If you were willing to learn a bit about nom and accept some of its quirks, along with the array of tools to make building and debugging parsers much easier; writing parsers were a fun, interactive process of figuring out data byte after byte.

But macros were hard to learn for many people, and I had to find another way.

Using functions

Thanks to the development of the “impl Trait” feature in Rust, I had an idea: could I make a function that accepts a function as argument, and returns another function, making everything generic?

As it turns out, yes, I can. See, for example, the pair combinator, that takes as argument 2 parsers, and returns a new parser that produces a tuple of the results of both child parsers:

pub fn pair<I, O1, O2, E, F, G>(first: F, second: G)
  -> impl Fn(I) -> IResult<I, (O1, O2), E>
  where
    F: Fn(I) -> IResult<I, O1, E>,
    G: Fn(I) -> IResult<I, O2, E>,
    E: ParseError<I>,

{
  move |input: I| {
    let (input, o1) = first(input)?;
    second(input).map(|(i, o2)| (i, (o1, o2)))
  }
}

// you can call directly the resulting parser like this:
let result = pair(parser1, parser2)(input);

(IResult is the parser result type of nom)

Not only is it possible, but the code is reasonably easy to write: I built an example parser library based on this in a few days.

The previous hexadecimal color parser could be rewritten like this:

fn hex_primary(input: &str) -> IResult<&str, u8> {
  map_res(
    take_while_m_n(2, 2, is_hex_digit),
    from_hex
  )(input)
}

fn hex_color(input: &str) -> IResult<&str, Color> {
  let (input, _) = tag("#")(input)?;
  let (input, (red, green, blue)) = tuple((hex_primary, hex_primary, hex_primary))(input)?;

  Ok((input, Color { red, green, blue }))
}

And the existing macros were rewritten to use the function combinators under the hood, so existing parsers will work the same way with nom 5, except for how errors are handled, and some details around streaming parsers you’ll see below.

Error management

nom 4 relied on an error type that would change depending on the verbose-errors cargo feature. If you used the feature, you would get more context on errors at the price of performance, if you didn’t you could only see which combinator saw an error and the input position.

Unfortunately, cargo features are additive, so if any transitive dependency was using nom with that feature, it would be activated everywhere.

Additionally, this error type supported a “custom” error variant that was generic, which resulted in painful type inference errors from time to time.

nom 5 changes this by making the error management completely generic, and the way you write your code or call the parsers will decide on the error type. The trick is in the ParserError<Input> trait you saw earlier in the pair combinator’s code.

By default, the error type in nom is a tuple: (Input, nom::error::ErrorKind). But you can use a more precise error type, like nom::error::VerboseError, that will provide more context. Or you can define your own error type, with exactly the information you need. You can learn more about it in the error management guide.

Streaming VS complete parsers

Another pain point of previous nom versions was in the management of streaming or complete input. In streaming mode, we assume that we do not have the entire data, and might get more by reading again from a file or socket. In complete mode, we know we have the entire data.

Some parsers are built differently depending on how the input data works. If you were writing an integer parser from text, in complete mode you could just read all the digits, even until the end of input, because you would have all of the data. With streaming input, if you reach the end of the data, you do not know if more digits can appear. So you have to wait until you encounter a character that is not a digit.

From the beginning, nom has made this distinction explicit, because it was designed with network protocols and big file formats in mind, through the Incomplete variant of parser results, that indicates the parser cannot decide and needs more data.

Making it usable, though, was challenging, especially when building parsers for smaller formats, like configuration files or programming languages.

The first solution was to use the complete! combinator, which transforms an Incomplete result into an error. Then having specialized versions of some combinators. But this was making parsers hard to write, and figuring out which parser was returning Incomplete was annoying.

nom 4 introduced a new idea: what if the input type decided if we were in streaming or complete mode? The CompleteByteSlice and CompleteStr were the complete input versions of &[u8] and &str that were considered as streaming input. Unfortunately, this made things worse: those types were littering the code, converting back and forth with the underlying byte slices or strings was tedious, and they inexplicably made parsers slower.

So nom 5 comes with a cleaner approach:

CompleteByteSlice and CompleteStr are gone
macros based parsers always work in streaming mode
for function combinators that would work differently in streaming or complete mode, there are different versions in submodules, like nom::bytes::streaming::tag and nom::bytes::complete::tag

This way, you can choose explicitely which version you use, and can even use a bit of both depending on the context. As an example, in a binary format, you might use nom::bytes::streaming::take to get a slice of the input, then parse that slice with parsers that work on complete input.

Various other changes

nom now uses the lexical crate for float parsing. Having a good float parser was important, but it is a complex enough topic to leave it to another crate.

This release was also a good opportunity to clean things up. Over the years, nom accumulated a lot of code, ideas, bad function names. So a lot of parsers were renamed or removed entirely, and the code was reorganized. For more information, check out the changelog.

The documentation was entirely rewritten: documentation and examples are available for every type, function and macro. The guides were updated to follow the new code. And we now have cleaner examples:

The JSON parsing example explains how to write a parser, from simple parts to more complex parsers, and shows how the new flexible error management can generate great error messages.
The S expression example demonstrates how we can parse and interpret programming languages.
The iterator example shows new patterns that are available in nom 5. It is now easy to make an iterator out of a parser and input data, parsing and producing values as needed. There is even a new iterator combinator to help with that.

Performance

Switching to functions changes the entire code layout. Now, a lot of the combinator code can be shared, instead of being generated every time it is called.

So, in most cases it will improve performance, but in a few others we might get worse results. Future versions will smooth things out: nom 4 was the peak of what was possible with the macros system, while I’m just getting started with the new design!

To reproduce the experiments, check that:

you’re using jemallocator instead of the system allocator. Test with and without it and see how it affects the results
you’re using link time optimization (LTO) when compiling, which will reduce code size

Here is what I add to Cargo.toml for benchmarks:

[profile.bench]
lto = true
codegen-units = 1

And now, some benchmarks between nom 4.2.3 and 5.0.0:

HTTP parser: 20% faster (non optimized one, runs at 500MB/s)
INI file parser: 20% faster
JSON file parser: 20% slower (work in progress, allocations affect it a lot)
float parser: 98% faster thanks to the lexical crate
recognize_float parser: 62% faster (not using the lexical crate)

No comparison between parser libraries this time, I have not yet rewritten all the nom examples in the parser benchmarks repository.

Thanks

The macros to functions refactoring and the documentation rewrite were huge and required an enormous effort. I would not have been able to do this release without all of nom’s contributors that helped me along the way, the developers that started using nom 5 while it was still in beta. Also I’m thankful for the support from my employer, Clever Cloud, which is now a large nom user :D

I am happy that after all this time, this project is still going strong, helping people write good parsers to power production software or weekend projects. Happy hacking with nom!

FOSS is free as in toilet

2018-11-27T21:00:00+01:00

I am a bit dissatisfied with the use of the Tragedy of the commons to represent issues with free and open source software development. It is not an abstract resource that can be depleted when overused. It is not magically maintained if left alone.

It is based on the work of people, and we should not erase those people.

Unfortunately (and it is by design), most of the licences and the vocabulary around it are focused on the software’s user. After all, they work by reducing the creator’s right to empower the user.

As examples of this vocabulary, we have the distinction between “free as in beer” and “free as in speech” to show that the “free” word in “free software” has more to do with freedom and people’s rights to use, study, modify and share a program, than its actual price. Although, in practice, the overwhelming majority of FOSS will not cost you anything.

This model has won, FOSS is everywhere, companies not only use it, but even heavily rely on it, millions of devices run with it.

But at which cost? Open source developers are burning out. Some core libraries, on which basically everything relies, are maintained by very small teams of people working on their free time. We still have the right to study the software, but the most interesting parts are now in the user’s data, which is jealously guarded by a few huge companies.

This is not a tragedy, this is a fucking farce.

Let’s own up to the absurdity of talking about a personal freedom that depends mainly on hidden people working for free. Let’s add more ridicule to it. Let’s start using a new expression to describe it:

FOSS IS FREE AS IN TOILET

Nobody believes that a free toilet will be magically cleaned up and maintained, somebody has to do it, and that person would better get paid for it. Sharing a toilet means that you flush, clean up after yourself, and always leave some paper, it’s basic manners. And yet, like toilets, as FOSS gets used by more and more people, it gets more likely that you will see obnoxious people that shit all over your commons and then complain about it. And nobody will want to take care of it.

Treating correctly the people who work on the software you use is just basic FOSS hygiene.

No, pest is not faster than nom

2018-10-04T15:00:00+02:00

As the main developer of nom, the Rust parser combinators library, I’m usually happy to see other parser libraries appear in Rust. The language’s strengths play well in that space, and writing parsers is a nice way to explore it.

I’m also happy to see them compete in benchmarks with nom, since it keeps me on my toes, and improves performance for everybody. Combine has been an interesting competitor here, showing how far one can go without using macros.

That said, the recent release of pest 2.0, a parsing expression grammar library, is pushing me to be petty :D

That library has been using a benchmark against nom in its readme for a long time. I wrote that benchmark to help in the comparison, but the results were slightly misleading:

The nom benchmark and the “pest (custom AST)” benchmark were converting the JSON file to Rust types (Vec, HashMap for objects, booleans, strings and floating point numbers). While the simple pest parser was validating the file and generating a list of tokens without converting them. So, yes, of course it will be faster than nom.

There was further work on the benchmarks, while nom 4 was still in preparation, and pest got interesting results:

The benchmarks were run on a late 2013 Macbook Pro, quad core 2,3 GHz Intel Core i7.

	basic	canada.json	apache_builds.json	data.json
combine	(fails)	127,775,522 ns/iter (+/- 11,140,676) = 17 MB/s	3,732,534 ns/iter (+/- 795,836) = 34 MB/s	241,407 ns/iter (+/- 40,575) = 38 MB/s
nom	1,333 ns/iter (+/- 247) = 57 MB/s	62,971,567 ns/iter (+/- 6,311,768) = 35 MB/s	1,209,550 ns/iter (+/- 323,936) = 105 MB/s	62,008 ns/iter (+/- 11,685) = 149 MB/s
pest	1,405 ns/iter (+/- 238) = 54 MB/s	27,701,820 ns/iter (+/- 3,961,221) = 81 MB/s	1,694,463 ns/iter (+/- 338,194) = 75 MB/s	131,851 ns/iter (+/- 22,667) = 70 MB/s

I could have left it at that.

But today (October 4th, 2018), the pest website featured a very misleading graph.

Yes, it’s very easy to make nom look bad if you:

conveniently remove the slowest pest parser
do not put any link to the benchmark code
avoid saying how many iterations are done, on which file, etc
start the horizontal axis at 20ms instead of 0ms, so that nom appears twice slower

this is ridiculous, and deserves all the pettiness I can muster.

The real benchmark

I took the old benchmark reused the canada.json file, added the data.json file from pest’s own JSON benchmark, applied pest’s current way to bench its code in my benchmarks.

After upgrading to nom 4 and pest 2, and fixing some old bugs in the nom parser, I could reproduce the old results (still on a late 2013 Macbook Pro, quad core 2,3 GHz Intel Core i7, yes my laptop is getting old):

nom:
- canada.json: 60,734,229 ns/iter (+/- 17,775,618)
- data.json: 23,937 ns/iter (+/- 9,992)
pest:
- canada.json: 35,041,472 ns/iter (+/- 5,454,302)
- data.json: 14,665 ns/iter (+/- 2,041)

Yes, a pest 2.0 parser that does not convert the input to Rust types is indeed faster than a nom 4.0 parser that does convert the input to Rust types.

But what happens if I write a nom 4.0 parser that does not convert its input to Rust types? It’s actually a bit easier if I don’t try to generates floats, bools, etc. It’s not exactly what pest does. Pest stores indexes along with the original slice, while this one will return sub slices for each JSON element. Still, here are the results:

nom:
- canada.json: 60,734,229 ns/iter (+/- 17,775,618)
- data.json: 23,937 ns/iter (+/- 9,992)
pest:
- canada.json: 35,041,472 ns/iter (+/- 5,454,302)
- data.json: 14,665 ns/iter (+/- 2,041)
nom_spans (returning slices instead of converting to Rust types):
- canada.json: 20,623,381 ns/iter (+/- 1,952,297)
- data.json: 10,757 ns/iter (+/- 1,462)

Or, represented as graphs that do not actually mislead us with their axis:

So, here we are, nom is actually still a lot faster than pest. I still find pest’s parser interesting, because the grammar is quite readable, but I tend to prefer parser combinators, because they make it easy to reuse parsers and combinators and allow you to write your own custom elements as you wish.

So, here we are, if you see ways to improve the benchmarks, and in the process make nom or pest faster, please do it :)

nom 4.0: faster, safer, simpler parsers

2018-05-14T13:00:00+02:00

I’m delighted to announce that nom, the extremely fast Rust parser combinators library, has reached major version 4.

TL;DR: the new nom version is simpler, faster, has a better documentation, and you can find a summary of what changed in the upgrade documentation.

side note: how fast is nom? it can reach 2GB/s when parsing HTTP requests.

Since nom is now a well established, serious project, we got a brand new logo, courtesy of Ange Albertini. The nom monster will happily eat your data byte by byte :)

It took nearly 6 months of development and the library went through nearly 5 entire rewrites. Compare that to previous major releases, which took a month at most to do. But it was worth it! This new release cleans up a lot of old bugs and unintuitive behaviours, simplifies some common patterns, is faster, uses less memory, gives better errors, but the way parsers are written stay the same. It’s like an entirely new engine under the same body work!

Moving from `IResult` to `Result`

This was a long standing request. nom used a three-legged enum as return type for the parsers:

// example parser signature
fn parser(input: I) -> IResult<I,O> { ... }

pub enum IResult<I,O,E=u32> {
  /// remaining input, result value
  Done(I,O),
  /// indicates the parser encountered an error. E is a custom error type you can redefine
  Error(Err),
  /// Incomplete contains a Needed, an enum than can represent a known quantity of input data, or unknown
  Incomplete(Needed)
}

pub enum Needed {
  /// needs more data, but we do not know how much
  Unknown,
  /// contains the required total data size
  Size(usize)
}

// if the "verbose-errors" feature is not active
pub type Err<E=u32> = ErrorKind;

// if the "verbose-errors" feature is active
pub enum Err<P,E=u32> {
  /// An error code, represented by an ErrorKind, which can contain a custom error code represented by E
  Code(ErrorKind),
  /// An error code, and the next error
  Node(ErrorKind, Vec<Err<P,E>>),
  /// An error code, and the input position
  Position(ErrorKind, P),
  /// An error code, the input position and the next error
  NodePosition(ErrorKind, P, Vec<Err<P,E>>)
}

That old IResult structure did not transform well to the commonly used Result, people did not want to see the Incomplete case (when the parser indicates it does not have enough data to decide) if they do not need it. And the different error types depending on the verbose-errors feature were confusing and causing errors when nom appeared multiple times in dependency trees.

So I replaced it with a new, Result based design:

pub type IResult<I, O, E = u32> = Result<(I, O), Err<I, E>>;

pub enum Err<I, E = u32> {
  /// There was not enough data
  Incomplete(Needed),
  /// The parser had an error (recoverable)
  Error(Context<I, E>),
  /// The parser had an unrecoverable error
  Failure(Context<I, E>),
}

pub enum Needed {
  /// needs more data, but we do not know how much
  Unknown,
  /// contains the required additional data size
  Size(usize)
}

// if the "verbose-errors" feature is inactive
pub enum Context<I, E = u32> {
  Code(I, ErrorKind),
}

// if the "verbose-errors" feature is active
pub enum Context<I, E = u32> {
  Code(I, ErrorKind),
  List(Vec<(I, ErrorKind)>),
}

Aside from being more compatible with, like, the whole Rust ecosystem, this new design has lots of interesting points:

the Context enum is now extended by the verbose-errors feature, so it is the same type
errors always store position information
Incomplete has moved to the error case so you can easily ignore it
IResult::Done(remaining, value) has been replaced with Ok((remaining, value)) so you could easily do let (remaining, value) = parser(input)?; like you would do with other Result based functions
the Err enum now contains aFailure case epresenting an unrecoverable error (combinators like alt! will not try another branch)

And we get all of these benefits while keeping the same memory footprint in “simple” errors mode, and reducing it in “verbose” errors mode:

size of `IResult`	simple errors	verbose errors
nom 3	40 bytes	64 bytes
nom 4	40 bytes	48 bytes

And it gets faster! Depending on the format, I have seen improvements between 4% and 40%!

Dealing with Incomplete usage

nom’s parsers are designed to work around streaming issues: if there is not enough data to decide, a parser will return Incomplete instead of returning a partial value that might be false.

As an example, if you want to parse alphabetic characters then digits, when you get the whole input abc123;, the parser will return abc for alphabetic characters, and 123 for the digits, and ; as remaining input.

But if you get that input in chunks, like ab then c123;, the alphabetic characters parser will return Incomplete, because it does not know if there will be more matching characters afterwards. If it returned ab directly, the digit parser would fail on the rest of the input, even though the input had the valid format.

For some users, though, the input will never be partial (everything could be loaded in memory at once), and the solution in nom 3 and before was to wrap parts of the parsers with the complete!() combinator that transforms Incomplete in Error.

nom 4 is much stricter about the behaviour with partial data, but provides better tools to deal with it. Thanks to the new AtEof trait for input types, nom now provides the CompleteByteSlice(&[u8]) and CompleteStr(str) input types, for which the at_eof() method always returns true. With these types, no need to put a complete!() combinator everywhere, you can juste apply those types like this:

named!(parser<&[u8], O>, ... );

// becomes

named!(parser<CompleteByteSlice, O>, ... );

named!(parser<&str, O>, ... );

// becomes

named!(parser<CompleteStr, O>, ... );

And as an example, for a unit test:

assert_eq!(parser("abcd123"), Ok(("123", "abcd"));

// becomes

assert_eq!(parser(CompleteStr("abcd123")), Ok((CompleteStr("123"), CompleteStr("abcd")));

These types allow you to correctly handle cases like text formats for which there might be a last empty line or not, as seen in one of the examples.

If those types feel a bit long to write everywhere in the parsers, it’s possible to alias them like this:

type Input = CompleteByteSlice;

pub fn Input(input:&'a[u8]) -> Input {
  CompleteByteSlice(input)
}

Better documentation

Previous documentation was scattered around and hard to navigate, especially when trying to find the exact combinator that would work perfectly for what we want.

So the new README is now more about what can be done instead of an incomplete reference.

The documentation homepage is an introduction to parser combinators and nom’s design, with some examples to show common combinators like do_parse.

There’s a brand new “choosing a combinator” doc to help you find what you need, arranged by categories, with example usage and expected results.

And there are new examples for a lot of parser and combinators.

Various fixes

no_std usage is now working correctly. For most of nom’s combinators, you will not need anything more than core, and a few basic combinators like many0 or separated_list require alloc.

Some parsers sometimes ended up in the middle of UTF-8 characters, now those streams are handled correctly.

Performance

Here is a comparison of nom’s internal benchmarks, between 3.2.1 and 4.0.0, in default mode:

$ cargo benchcmp 3.2.1.bench 4.0.0.bench
 name             3.2.1.bench ns/iter  4.0.0.bench ns/iter  diff ns/iter   diff % 
 arithmetic       759                  469                          -290  -38.21% 
 ini              998 (109 MB/s)       897 (122 MB/s)               -101  -10.12% 
 ini_key_value    45 (400 MB/s)        47 (382 MB/s)                   2    4.44% 
 ini_keys_values  91 (483 MB/s)        85 (529 MB/s)                  -6   -6.59% 
 ini_str          1,399 (77 MB/s)      1,396 (78 MB/s)                -3   -0.21% 
 json_bench       2,149                1,694                        -455  -21.17% 
 http_test        700                  659                           -41   -5.86% 

And here is the difference for verbose errors:

$ cargo benchcmp 3.2.1.bench 4.0.0.bench 
 name             3.2.1.bench ns/iter  4.0.0.bench ns/iter  diff ns/iter   diff % 
 arithmetic       1,731                1,523                        -208  -12.02% 
 ini              1,199 (90 MB/s)      1,061 (103 MB/s)             -138  -11.51% 
 ini_key_value    70 (257 MB/s)        60 (300 MB/s)                 -10  -14.29% 
 ini_keys_values  133 (330 MB/s)       111 (405 MB/s)                -22  -16.54% 
 ini_str          1,525 (71 MB/s)      1,621 (67 MB/s)                96    6.30% 
 json_bench       2,905                2,193                        -712  -24.51% 
 http_test        854                  941                            87   10.19% 

nom 4 is not faster everywhere, ther are still some small regressions that will be fixed soon, but overall we see great improvements.

With nom 4 and some recent work around integrating lookup tables and vectorization in nom parsers, we can get impressive results in HTTP parsing (comparing with combine, Joyent’s http-parser, httparse and picohttpparser):

the future for nom

nom 4 is a huge release, and the new design will probably take some time to settle. There’s probably a lot of low hanging fruit on the performance side, and I look forward to my next obsessive profiling sessions.

Meanwhile, nom will continue to happily munch bytes for you, as one of the fastest, most reliable parsing libraries available.

In the future, I’m interested in supporting more use cases, like zero allocation parsers or Web Assembly usage, and integrating more SIMD work, like what was done for the HTTP parser. The goal is to get nom parsers to secure data access in more and more systems, whatever the language or platform.

PoC: compiling to eBPF from Rust

2018-02-02T21:33:08+01:00

guess who has an eBPF tracer written in Rust? This guy👍 pic.twitter.com/3aE1giGWeK
— Geoffroy Couprie (@gcouprie) January 27, 2018

I have been playing with eBPF (extended Berkeley Packet Filters), a neat feature present in recent Linux versions (it evolved from the much older BPF filters). It is a virtual machine running in the kernel, to which you can send code from userland, and that code can be used to filter packets or trace parts of the kernel code.

What makes eBPF really nice is how the kernel handles it. You send a program in bytecode format to the kernel, it then checks it, verifying, for example, that there are no loops, thus guaranteeing that the program will terminate, and it will then apply JIT compilation, making the resulting code quite fast. Even better, that code can be loaded and unloaded at any time through a syscall, and you can set up shared data structures between the eBPF program and your own, to efficiently gather data.

As an example, you can use eBPF (and the XDP - eXpress Data Path - feature) to write very efficient firewalls, or employ BCC (BPF Compiler Collection) to trace a process’s IO events.

I’m looking at how we could use that to trace applications on our infrastructure at Clever Cloud. There are a few things we should know about the tooling first.

At the beginning, people wrote their program using the bytecode directly:

/* Compare IPv4 with one word instruction (32bit)*/
struct bpf_insn insn[] = {
  /* If skb->protocol != ETH_P_IP, skip this whole block. The offset will be set later. */
  BPF_JMP_IMM(BPF_JNE, BPF_REG_7, htobe16(protocol), 0),
  /*
   * Call into BPF_FUNC_skb_load_bytes to load the dst/src IP address
   *
   * R1: Pointer to the skb
   * R2: Data offset
   * R3: Destination buffer on the stack (r10 - 4)
   * R4: Number of bytes to read (4)
   */
  BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
  BPF_MOV32_IMM(BPF_REG_2, addr_offset),
  BPF_MOV64_REG(BPF_REG_3, BPF_REG_10),
  BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -addr_size),
  BPF_MOV32_IMM(BPF_REG_4, addr_size),
  BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_skb_load_bytes),
  /*
   * Call into BPF_FUNC_map_lookup_elem to see if the address matches any entry in the
   * LPM trie map. For this to work, the prefixlen field of 'struct bpf_lpm_trie_key'
   * has to be set to the maximum possible value.<
   *
   * On success, the looked up value is stored in R0. For this application, the actual
   * value doesn't matter, however; we just set the bit in @verdict in R8 if we found any
   * matching value.
   */
  BPF_LD_MAP_FD(BPF_REG_1, map_fd),
  BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
  BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -addr_size - sizeof(uint32_t)),
  BPF_ST_MEM(BPF_W, BPF_REG_2, 0, addr_size * 8),
  BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
  BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
  BPF_ALU32_IMM(BPF_OR, BPF_REG_8, verdict),
};

This is a bit raw, and somewhat complex to write, so people worked on C to eBPF compilers, and the feature landed in LLVM: we can use clang to write eBPF programs! It will generate the bytecode, that can then be loaded through the bpf() syscall.

This is still a bit complex, since the eBPF program might need access to some internal data structures of the kernel, and those change depending on kernel versions and configuration options. And we still need to set up the shared data structures with the userland program that will gather data.

That’s why the BCC project provides an easy to use interface to compile and load eBPF programs. They made it so simple that you can write a python script to compile, load and interact with your program:

from bcc import BPF
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\n"); return 0; }').trace_print()

They provide a lot of useful examples and a nice tutorial to get started writing eBPF tracers.

Unfortunately, those tools make a tradeoff that’s slightly annoying for me: they require installing BCC, which requires Python, LLVM and the complete Linux sources, on the target machines. It might be possible to precompile the programs though, but it does not look like it’s a common use case with BCC.

So, maybe there’s a nice way to precompile those programs, store them as bytecode, then load them with a small agent that does not need LLVM and the kernel sources to work? It turns out it is possible, thanks to the gobpf project, who split their ELF loading code from the BCC part a year ago.

And, now, you’ll see where I am going with this. Being one of those annoying Rust developers who want to rewrite everything in their favorite language, I thought “hey, maybe I can Rust that thing too!”

Since it is possible to compile to eBPF bytecode from C, it is possible to compile LLVM IR (the kind of bytecode LLVM generates from the code before compiling it to the target CPU’s assembly) to eBPF. Look for “LLVM IR debugging” in this link for an example. And I know I can compile Rust to that LLVM IR, and everything should work out, as long as Rust’s LLVM version is the same as the system’s version.

So I created a small Rust project (code available here), and wrote the following build script:

#!/bin/sh
cargo rustc --release -- --emit=llvm-ir
cp target/release/deps/hello-*.ll hello.ll
cargo rustc --release -- --emit=llvm-bc
cp target/release/deps/hello-*.bc hello.bc
llc-4.0 hello.bc -march=bpf -filetype=obj -o hello.o

I generated the ll file to take a look at the IR in text form)

And now, the code! I used the example program from this blog post as inspiration, and came up with this:

use std::mem::transmute;
use std::ffi::CStr;

#[no_mangle]
#[link_section = "license"]
pub static _license: [u8; 4] = [71u8, 80, 76, 0]; //b"GPL\0"

#[no_mangle]
#[link_section = "version"]
pub static _version: u32 = 0xFFFFFFFE;

#[no_mangle]
#[link_section = "kprobe/SyS_clone"]
pub extern "C" fn kprobe__sys_clone(ctx: *mut u8) -> i32 {
  let BPF_FUNC_trace_printk = unsafe {
    transmute::<i32>(6)
  };

let msg: [u8; 17] = [104u8, 101, 108, 108, 111, 32, 102, 114, 111, 109, 32, 114, 117, 115, 116, 10, 0]; //b"hello from Rust\0"
  BPF_FUNC_trace_printk((&msg).as_ptr(), 17);
return 0;
}

First, the constants are in their own ELF section, this is expected by gobpf’s elf loader. Apparently, I cannot write pub static _license: &'static [u8] = b"GPL\0", because the _license symbol would then be a relocation of the actual string.

#[no_mangle]
#[link_section = "license"]
pub static _license: [u8; 4] = [71u8, 80, 76, 0];//b"GPL\0";

#[no_mangle]
#[link_section = "version"]
pub static _version: u32 = 0xFFFFFFFE;

Now, the function: gobpf expects a section with the name of the function we will try to hook later.

#[no_mangle]
#[link_section = "kprobe/SyS_clone"]
pub extern "C" fn kprobe__sys_clone(ctx: *mut u8) -> i32 {

Now we might need to import functions. They are defined in a C enum, but interpreted as function pointers. So calling the printk function from there amounts writing the instruction call 6. So we transmute the number 6 into a function. I know, ewww, but it works :D

let BPF_FUNC_trace_printk = unsafe {
  transmute::<i32>(6)
};

And now, the last part, actually printing something. To avoid the previous issue of a constant string appearing as a relocated symbol on which gobpf will throw an error, I defined it as a local constant, then called BPF_FUNC_trace_printk on it. That should be harmless, right?

let msg: [u8; 17] = [104u8, 101, 108, 108, 111, 32, 102, 114, 111, 109, 32, 114, 117, 115, 116, 10, 0]; //b"hello from Rust\0"
BPF_FUNC_trace_printk((&msg).as_ptr(), 17);

So now, let’s take a look at the generated LLVM IR (the ll file):

; ModuleID = 'hello0-b4990b5a434d0f01306c6e79c17f427.rs'
source_filename = "hello0-b4990b5a434d0f01306c6e79c17f427.rs"
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
@_license = local_unnamed_addr constant [4 x i8] c"GPL\00", section "license", align 1
@_version = local_unnamed_addr constant i32 -2, section "version", align 4

; Function Attrs: nounwind uwtable
define i32 @kprobe__sys_clone(i8* nocapture readnone %ctx) unnamed_addr #0 section "kprobe/SyS_clone" {
start:
  %msg = alloca [17 x i8], align 16
  %0 = getelementptr inbounds [17 x i8], [17 x i8]* %msg, i64 0, i64 0
  call void @llvm.lifetime.start(i64 17, i8* nonnull %0)
  %1 = bitcast [17 x i8]* %msg to *
  store  , * %1, align 16
  %2 = getelementptr inbounds [17 x i8], [17 x i8]* %msg, i64 0, i64 16
  store i8 0, i8* %2, align 16
  %3 = call i32 (i8*, i64, ...) inttoptr (i64 6 to i32 (i8*, i64, ...)*)(i8* nonnull %0, i64 17) #2
  call void @llvm.lifetime.end(i64 17, i8* nonnull %0)
  ret i32 0
}

; Function Attrs: argmemonly nounwind
declare void @llvm.lifetime.start(i64, i8* nocapture) #1

; Function Attrs: argmemonly nounwind
declare void @llvm.lifetime.end(i64, i8* nocapture) #1

attributes #0 = { nounwind uwtable "probe-stack"="__rust_probestack" }
attributes #1 = { argmemonly nounwind }
attributes #2 = { nounwind }

So, it generates the correct symbols and sections for _license and _version. It apparently generates a big store instruction for the string we’ll print. And the function call looks like this call i32 (i8*, i64, ...) inttoptr (i64 6 to i32 (i8*, i64, ...)*)(i8* nonnull %0, i64 17) where we cast 6 to a function pointer: inttoptr (i64 6 to i32 (i8*, i64, ...)*).

So that should generate correct BPF bytecode, right? Let’s check that with the command llvm-objdump-4.0 -S hello.o:

hello.o:    file format ELF64-BPF

Disassembly of section kprobe/SyS_clone:
kprobe__sys_clone:
       0:   b7 01 00 00 0a 00 00 00     r1 = 10
       1:   73 1a ef ff 00 00 00 00     *(u8 *)(r10 - 17) = r1
       2:   b7 01 00 00 74 00 00 00     r1 = 116
       3:   73 1a ee ff 00 00 00 00     *(u8 *)(r10 - 18) = r1
       4:   b7 01 00 00 73 00 00 00     r1 = 115
       5:   73 1a ed ff 00 00 00 00     *(u8 *)(r10 - 19) = r1
       6:   b7 01 00 00 75 00 00 00     r1 = 117
       7:   73 1a ec ff 00 00 00 00     *(u8 *)(r10 - 20) = r1
       8:   b7 01 00 00 6d 00 00 00     r1 = 109
       9:   73 1a e9 ff 00 00 00 00     *(u8 *)(r10 - 23) = r1
      10:   b7 01 00 00 72 00 00 00     r1 = 114
      11:   73 1a eb ff 00 00 00 00     *(u8 *)(r10 - 21) = r1
      12:   73 1a e7 ff 00 00 00 00     *(u8 *)(r10 - 25) = r1
      13:   b7 01 00 00 66 00 00 00     r1 = 102
      14:   73 1a e6 ff 00 00 00 00     *(u8 *)(r10 - 26) = r1
      15:   b7 01 00 00 20 00 00 00     r1 = 32
      16:   73 1a ea ff 00 00 00 00     *(u8 *)(r10 - 22) = r1
      17:   73 1a e5 ff 00 00 00 00     *(u8 *)(r10 - 27) = r1
      18:   b7 01 00 00 6f 00 00 00     r1 = 111
      19:   73 1a e8 ff 00 00 00 00     *(u8 *)(r10 - 24) = r1
      20:   73 1a e4 ff 00 00 00 00     *(u8 *)(r10 - 28) = r1
      21:   b7 01 00 00 6c 00 00 00     r1 = 108
      22:   73 1a e3 ff 00 00 00 00     *(u8 *)(r10 - 29) = r1
      23:   73 1a e2 ff 00 00 00 00     *(u8 *)(r10 - 30) = r1
      24:   b7 01 00 00 65 00 00 00     r1 = 101
      25:   73 1a e1 ff 00 00 00 00     *(u8 *)(r10 - 31) = r1
      26:   b7 01 00 00 68 00 00 00     r1 = 104
      27:   73 1a e0 ff 00 00 00 00     *(u8 *)(r10 - 32) = r1
      28:   b7 01 00 00 00 00 00 00     r1 = 0
      29:   73 1a f0 ff 00 00 00 00     *(u8 *)(r10 - 16) = r1
      30:   bf a1 00 00 00 00 00 00     r1 = r10
      31:   07 01 00 00 e0 ff ff ff     r1 += -32
      32:   b7 02 00 00 11 00 00 00     r2 = 17
      33:   85 00 00 00 06 00 00 00     call 6
      34:   b7 00 00 00 00 00 00 00     r0 = 0
      35:   95 00 00 00 00 00 00 00     exit

So, the lines 0 to 29 are there to load our string (instead of pointing to a constant somewhere). The expected call 6 instruction is on line 33!

To load that program, we can now use the example Go program from the blog post (slightly modified to accept a program name as argument): sudo ./bpf-load hello.o. The eBPF program will the hook the sys_clone function and print a hello every time it is called. You can see the trace with the command sudo cat /sys/kernel/debug/tracing/trace_pipe.

And now you can be a happy Rust developer like me because again, you put some Rust where you were not supposed to!

you: *sobbing* please stop...you can't rewrite everything in Rust!
me: *👉video games* Rust
you: nooo😢
me: *👉kernel* Rust
you: stop that😭
me: *👉frontend apps* Rust & wasm
you: 😱
— Geoffroy Couprie (@gcouprie) January 11, 2018

Now, this is a small hack. To make it more useful, here is what we would need:

a small library to import BPF functions instead of transmuting the number every time
that small library should also have a nice way to interact with BPF maps to transmit data to userland
a userland library (in Rust, of course) that can set up maps and load eBPF programs. I should mention here that Julia Evans is currently working on a port of gobpf’s BCC part in Rust for a Ruby profiling tool! The ELF part might not be too far :)

That’s all, I’ll post more once I get more useful code working! PS/ if you want to learn more about BPF, read this great list!

Rust 2018: maybe don’t be too stable

2018-01-10T19:25:14+01:00

I initially did not want to write a post with what I want and foresee for Rust in 2018, because I'm already very happy with it! I have spent more than 4 years tinkering with the language, experimenting, and I love the freedom I get when playing with low level stuff. In those 4 years, I discovered a wonderful, welcoming community and made some awesome friends. So, yes, I'm happy with Rust as it is :)

But some of the recent #Rust2018 posts made me react a bit. I'm interested in learning what other people see in Rust, so I read almost all of them, and there's an easy trend to follow. Rust should be stabilized. Rust should be boring and safe. Crates should be stabilized. We should have definitive crates for some purposes like HTTP clients or async programming.
This is not surprising, since there's already been a lot of focus on stability in 2017, with the impl period, the merge of the Rust epochs RFC, and the fact that more and more companies start relying on Rust.
We want Rust to be appealing to (big(ger)) companies, and to that end we need good compatibility between Rust versions, a high quality ecosystem of crates that work on stable Rust versions. We want newcomers to have a well prepared toolbox for their first projects.

Before that stabilization goal appeared, Rust looked a bit chaotic, with new features coming every 6 weeks, new crates popping up here and there, people hacking something quickly and publishing it the next minute. And this is something I love about this language.
People try stuff, cargo lets them publish it easily, Rust makes sure it's running smoothly. Sure, there's a lot of redundant crates, most of them are far from the big "1.0 stable" target, but it's fine.
This language and its community are full of that unabashed optimism that makes newcomers go "hey, should I really try to write my own kernel? OF COURSE I SHOULD". Should I try to make cool stuff with Web Assembly while it barely landed in nightly? YESSSSSS
I have seen over and over shitposting on twitter that ends up with people hacking on a cool new project. I have seen people publish a crate competing with another well known one, that will then send a PR for their idea to the bigger crate the next day.
I am overly enthusiastic about this, to the point that opening /r/rust often feels like Christmas: what new toys will we get today?

So, to be clear, I am all for getting more stuff stable. We need a stable, asynchronous hyper. We need futures to work. We need impl trait and various other Rust features that will appear in the following months or years. What we do not need is the attitude that wants everything to crystallize.
How many times have I seen people criticising the "yet another" asynchronous IO/command line argument system/web framework/parser, with the usual arguments that this is lost focus, redundant, that why didn't they try to do that in $BIG_PROJECT. This is fine.
Go on, make other parser libraries to compete with nom, keep me on my toes. Try other approaches than tokio. Test different approaches to writing web applications.

The underlying idea for me is that Rust is still incredibly young, extremely enthusiastic, and we still don't fully know how to write Rust. So, yes, we need some parts of Rust to stabilize, but we must balance that with its movement. What is stable and "the way we do things" now might not be the way to go in a year or so.

Let people experiment and lose focus. Keep hacking on cool stuff.

Adventures in logging

2017-08-23T16:26:05+02:00

After working on the Sōzu HTTP reverse proxy for a while, I came up with an interesting approach to logging. Now why would I come up with my own logger, when there are existing solutions in Rust? Mainly, log and slog. That logging library grew up from testing things out with log, and changing requirements along the way.

Beginning with log and env_logger

Like a lot of other Rust developers, I started out with log and env_logger:

#[macro_use]
extern crate log;
extern crate env_logger;

fn main() {
env_logger::init().unwrap();

info!("starting up");
}

It's nice and easy: every library that depends on log will use that same set of logging macros (error, info, warn, debug, trace) that will use whatever global logger was defined. Here we use env_logger to define one.

env_logger is useful because it can apply a filter to the log, from an
environment variable:

[code lang=text]
# will show the logs above info level
RUST_LOG=info ./main

# will show the logs above info level, but also logs above debug level
# for the dependency 'dep1'
RUST_LOG=info,dep1=debug ./main
[/code]

You can also define the filter by module or apply a regular expression.

Custom formatter

env_logger allows you to build your own log formatter. This feature is especially important for me, as I like to add metadata to my logs.

Defining a custom formatter with env_logger is quite straightforward:

[code lang=text]
let format = |record: &LogRecord| {
format!("{} - {}", record.level(), record.args())
};

let mut builder = LogBuilder::new();
builder.format(format).filter(None, LogLevelFilter::Info);

if env::var("RUST_LOG").is_ok() {
builder.parse(&env::var("RUST_LOG").unwrap());
}

builder.init().unwrap();
[/code]

It is easily combined with the filtering and usage of the RUST_LOG environment variable.

Where things get annoying: reducing allocations

If you take a look at env_logger, you'll realize that it will allocate a String for every log line that will be written, using a formatting closure.

Let's get one thing out of the way first: I completely agree with the idea you should not try to optimize stuff too much. But I'm in the case of a networking component that will handle a lot of traffic. I had debugging sessions where I generated tens of gigabytes of logs in a few seconds, and needed almost all of them, to debug async IO issues. In those cases, the time spent allocating and deallocating log lines becomes relevant.

So, how would I get a custom log formatter that does not allocate much? As it turns out, when you tell log to use your logger with log::set_logger, it requires something that implements Log. The logger's log method receives a LogRecord, a structure that's created on the fly from LogLocation, LogMetadata and Arguments.
The first two are internal to log, I can't create them myself. The last one is interesting.

Arguments can be created from the format_args macro. That structure will roughly contain the format string split in the various substrings that appear between arguments. if you do println!("hello {}!", name), you would get a structure that contains "hello ", the content of name and "!". println! and other macros use this.

You can then use that Arguments with io::Write::write_fmt to write it directly to, say, a file or a socket. And it is implemented so that the individual parts are written one after another instead of allocating one big string.

So, how do I use that?

Well, it turns out that, basically, I can't. If I implement Log, I can get a Logrecord which gives me a &Arguments, while write requires a Arguments. So now I have to clone it, which defeats a bit the purpose.

So let's write our own then

There was another reason for the custom logging library: using a custom logging backend. Having the option between stdout and stderr is fine, but I might want to send them to a file or a socket.

So I started writing a specific logging library for sozu. First, copying the log filtering from env_logger. That part is mostly straightforward, but that's still a lot of code to copy around.

The logging macros specify a logging level then they all call the same common macro.

[code lang=text]
#[macro_export]
macro_rules! error {
($format:expr, $($arg:tt)*) => {
log!($crate::logging::LogLevel::Error, $format, "ERROR", $($arg)*);
};
($format:expr) => {
log!($crate::logging::LogLevel::Error, $format, "ERROR");
};
}
[/code]

The main logging macro has two interesting parts. First, we define some static metadata (that's coming from the log crate):

[code lang=text]
static _META: $crate::logging::LogMetadata = $crate::logging::LogMetadata {
level: $lvl,
target: module_path!(),
};
[/code]

That object will not be allocated over and over, all the data in there will be defined at compile time.

Then we call the logger itself (ignore the line with try_lock for now):

[code lang=text]
if let Ok(mut logger) = LOGGER.try_lock() {
logger.log(
&_META,
format_args!(
concat!("{}\t{}\t{}\t{}\t{}\t", $format, '\n'),
::time::now_utc().rfc3339(), ::time::precise_time_ns(), *$crate::logging::PID,
$level_tag, *$crate::logging::TAG));
}
[/code]

So we give this metadata structure to our logger, then we make an Arguments structure with format_args!. The concat! macro is there to concatenate the formatting string with the custom prefix. That way, I could write debug!("hello {}", name) and have the resulting format string be "{}\t{}\t{}\t{}\t{}\thello {}\n", generated at compile time and transformed through the format_args call.

I added the date in ISO format, along with a monotonic timestamp (that becomes handy when multiple workers might write logs concurrently), the process identifier, the log level and a process wide logging tag (to better identify workers).

So this starts looking good, right? Now how do we write this to configurable backends? Some backends already implement io::Write, others will need an intermediary buffer:

[code lang=text]
pub fn log<'a>(&mut self, meta: &LogMetadata, args: Arguments) {
if self.enabled(meta) {
match self.backend {
LoggerBackend::Stdout(ref mut stdout) => {
stdout.write_fmt(args);
},
LoggerBackend::Unix(ref mut socket) => {
socket.send(format(args).as_bytes());
},
LoggerBackend::Udp(ref mut socket, ref address) => {
socket.send_to(format(args).as_bytes(), address);
}
LoggerBackend::Tcp(ref mut socket) => {
socket.write_fmt(args);
},
}
}
}
[/code]

For Unix sockets and UDP, instead of allocating on the fly, it should probably use a buffer (hey, anyone wants to implement that?). Stdout and a TcpStream can be written to directly. Adding buffers might still be a good idea here, depending on what you want, because that write could fail. Would you like a logger that will send a partial log if it can't write on the socket, or one using a buffer that can be filled up?

So, now, what's next? Originally, sozu worked as one process with multiple threads, but evolved as a bunch of single threaded processes. But that raises an interesting question. How do you write logs concurrently?

Highly concurrent logs

It turns out that problem is not really easy. Most solutions end up in this list:

every thread or process writes to stdout or a file at the same time
synchronized access to the logging output
one common logger everybody sends to
every thread or process has its own connection to syslog (or even its own file to write to)

The first solution is easy, but has a few issues. First, writing to stdout is slow, and it can quickly overwhelm your terminal (yes, I know you can redirect to a file). Second, it's not really synchronized, so you might end up with incoherently interleaved log lines.

So we often move to the second solution, where access to the logging stream is protected by a mutex. Now you get multiple threads or processes that might spend their time waiting on each other for serializing and writing logs. Having all threads sharing one lock can quickly affect your performance. It's generally a better idea to have every thread or process running independently from the others (it's one of the principles in sozu's architecture, you can learn more about it in this french talk).

Alright then, moving on to the third solution: let's have one of the threads or processes handle the logging, and send the logs to it via cross thread channels or IPC. That will surely be easier than having everybody share a lock, right? This is also intersting because you can offload serialization and filtering to the logging thread. Unfortunately, that means one thread will handle all the logs, and it can be overwhelming. That also means a lot of data moving between workers (if using processes).

The last solution relies on external services: use the syslog daemon on your system, or a log aggregator somewhere on another machine, that every worker will send the logs to. Let that service interleave logs, and maybe scale up if necessary. Since in a large architecture, you might have such an aggregator, talking to it directly might be a good idea (oh hey BTW I wrote a syslog crate if you need).

With sozu, I ended up with a mixed solution. You can send the logs to various backends. If you choose stdout, then all workers will write to it directly without synchronization which will be mostly fine if you don't have a large traffic. But if you want, each worker can open its own connection to the logging service.

Now that concurrency is taken care of, there's a last issue that has annoyed me for months: how to use the logger from everywhere in the code, when it's obviously one big global mutable object?

the dreaded logging singleton

One thing I like about the macros from the log crate: you can use them anywhere in your code, and it will work. The other approach, used in languages like Haskell, or proposed by slog, is to carry your logger around, in function arguments or structure members. I can understand the idea, but I don't like it much, because I'll often need to add a debug call anywhere in the code, and when it's deep in a serie of five method calls, with those methods coming from traits implemented here and there, updating the types quickly gets annoying.

So, even if the idea of that global mutable logger singleton usually looks like a bad pattern, it can still be useful. In log, the log macro calls the log::Log::log method, getting the logger instance from the logger method. That method gets the logger instance from a global pointer to the logger, with an atomic integer used to indicate if the logger was initialized:

[code lang=text]
static mut LOGGER: *const Log = &NopLogger;
static STATE: AtomicUsize = ATOMIC_USIZE_INIT;

const UNINITIALIZED: usize = 0;
const INITIALIZING: usize = 1;
const INITIALIZED: usize = 2;

[...]

pub fn logger() -> &'static Log {
unsafe {
if STATE.load(Ordering::SeqCst) != INITIALIZED {
static NOP: NopLogger = NopLogger;
&NOP
} else {
&*LOGGER
}
}
}
[/code]

So how can that global mutable pointer be used from any thread? That's because the Log trait requires Send and Sync. As a reminder, Send means it can be sent to another thread, Sync means it can be shared between threads.

That's a cool trick, but usually we employ another pattern:

[code lang=text]
lazy_static! {
pub static ref LOGGER: Mutex<Logger> = Mutex::new(Logger::new());
pub static ref PID: i32 = unsafe { libc::getpid() };
pub static ref TAG: String = LOGGER.lock().unwrap().tag.clone();
}
[/code]

lazy_static allows you to define static variables that will be initialized at runtime, at first access (using the same pattern as log with std::sync::Once. Since our logger's log method needs an &mut, it's wrapped in a Mutex. That's where the call to try_lock comes from.

We might think the mutex is costly compared to log's solution, but remember that the logger instance has to be Sync, so depending on your implementation, there might be some synchronization somewhere. Except that for sozu, it's not the case! Each worker is single threaded, and has its own instance of the logger (possibly with each of them a connection to the logging service). Can't we have a logging system that does not require that mutex used everywhere?

Removing the last Mutex

This is a problem that annoyed me for months. It's not that I really mind the cost of that mutex (since no other thread ever touches it). It's just that I'd feel better not using one when I don't need it :)

And the solution to that problem got quite interesting. To mess around a bit, here's a playground of the logging solution based on lazy_static. You'll see why the code in Foo::modify is important.

There's a feature you can use to have a global variable available anywhere in a thread: thread_local. It uses thread local storage with an UnsafeCell to initialize and provide a variable specific to each thread:

[code lang=text]
thread_local!(static FOO: RefCell<u32> = RefCell::new(1));

FOO.with(|f| {
assert_eq!(*f.borrow(), 1);
*f.borrow_mut() = 2;
});
[/code]

So I tried to use this with my logger, but encountered an interesting bug, as you can see in another playground. I replaced my logging macro with this:

[code lang=text]
#[macro_export]
macro_rules! log {
($format:expr, $($arg:tt)+) => ({
{
LOGGER.with(|l| {
l.borrow_mut().log(
format_args!(
concat!("\t{}\t", $format, '\n'),
"a", $($arg)+));
});
}
});
}
[/code]

And got this error:

[code lang=text]
error[E0502]: cannot borrow `self` as immutable because `self.opt.0` is also borrowed as mutable
--> src/main.rs:36:21
|
36 | LOGGER.with(|l| {
| ^^^ immutable borrow occurs here
...
54 | if let Some(ref mut s) = self.opt {
| --------- mutable borrow occurs here
55 | log!("changing {} to {}", self.bar, new);
| -----------------------------------------
| | |
| | borrow occurs due to use of `self` in closure
| in this macro invocation
56 | }
| - mutable borrow ends here
[/code]

When implementing this inside sozu's source, I got about 330 errors like this one... So what happened? That with method requires a closure. Since we use a macro, if we use self.bar as argument, it will appear inside the closure. That becomes an issue with anything that has been already mutably borrowed somewhere.

I tried a few things, like calling format_args outside the closure, but I get the error error[E0597]: borrowed value does not live long enough. This is apparently a common problem with format_args.

But the real solution came from tomaka, with some macro trickery, as seen in one last playground:

[code lang=text]
#[macro_export]
macro_rules! log {
($format:expr $(, $args:expr)*) => ({
log!(__inner__ $format, [], [a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v]
$(, $args)*)
});

(__inner__ $format:expr, [$($transformed_args:ident),*], [$first_ident:ident $(, $other_idents:ident)*], $first_arg:expr $(, $other_args:expr)*) => ({
let $first_ident = &$first_arg;
log!(__inner__ $format, [$($transformed_args,)* $first_ident], [$($other_idents),*] $(, $other_args)*);
});

(__inner__ $format:expr, [$($final_args:ident),*], [$($idents:ident),*]) => ({
LOGGER.with(move |l| {
//let mut logger = *l.borrow_mut();
l.borrow_mut().log(
format_args!(
concat!("\t{}\t", $format, '\n'),
"a" $(, $final_args)*));
});
});
}
[/code]

The basic idea is that we could avoid the borrowing issue by doing an additional borrow. But since some of the arguments might by expressions (like 1+1 or self.size), we will store the reference to it in a local variable, with let $first_ident = &$first_arg;.

We cannot create variable or function names in macros out of thin air (sadly, because that would be extremely useful), so we instead do recursive macros calls, consuming arguments one after another.
In [$($transformed_args:ident),*], [$first_ident:ident $(, $other_idents:ident)*], $first_arg:expr $(, $other_args:expr)*,
transformed_args accumulates the idents (variable names) in which we stored the data.

[$first_ident:ident $(, $other_idents:ident)*] is matching on the list that started as [a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v], to get the first one in that list (storing it in $first_ident), and using it as variable names. As you might have guessed, that means I won't be able to use a log line with more than 21 arguments. That's a limitation I can live with.

The $first_arg:expr $(, $other_args:expr)* part matches on the log call's arguments, and gets the first in the list as $first_arg. We then use those in the line let $first_ident = &$first_arg; and recursively call log!, adding the variable name $first_ident to the list of transformed arguments, and the rest of the variable names list and the log call's arguments:

[code lang=text]
log!(__inner__ $format, [$($transformed_args,)* $first_ident], [$($other_idents),*] $(, $other_args)*);
[/code]

Once all of the logger's arguments are consumed, we can call format_args on the list of $transformed_args:

[code lang=text]
(__inner__ $format:expr, [$($transformed_args:ident),*], [$($idents:ident),*]) => ({
LOGGER.with(move |l| {
//let mut logger = *l.borrow_mut();
l.borrow_mut().log(
format_args!(
concat!("\t{}\t", $format, '\n'),
"a" $(, $transformed_args)*));
});
});
});
[/code]

and it works!

So, that last part may not be completely relevant to your logger implementation, but I thought
it was quite cool :)

Despite my issues with the log crate, it's quite useful and supported by a lot of libraries. It is currently getting a lot better as part of the libz blitz. I'd also encourage you to check out slog. I haven't felt the need to integrate it in sozu yet, but it can be interesting for new projects, as it comes with an ecosystem of composable libraries to extend it.

How to rewrite your project in Rust

2017-07-12T17:15:01+02:00

In a previous post, I explained why rewriting existing software in Rust could be a good idea. The main point being that you should not rewrite the whole application, but replace the weaker parts without disturbing most of the code, to strengthen the codebase without disruption.

I also provided pointers to projects where other people and I did it succesfully, but without giving too many details. So let's get a real introduction to Rust rewrites now. This article requires a little bit of knowledge about Rust, but you should be able to follow it even as a
beginner.

As a reminder, here are the benefits Rust bring into a rewrite:

it can easily call C code
it can easily be called by C code (it can export C compatible functions and structures)
it does not need a garbage collector
if you want, it does not even need to handle allocations
the Rust compiler can produce static and dynamic libraries, and even object files
the Rust compiler avoids most of the memory vulnerabilities you get in C (yes, I had to mention it)
Rust is easier to maintain than C (this is discutable, but not the point of this article)

As it turns out, this is more or less the plan to replace C code with Rust:

import C structures and functions in Rust
import Rust structures and functions from C
reuse the host application's memory allocations whenever possible
write code (yes, we have to do it at some point)
produce artefacts that can be linked with the host application
integrate with the build system

We'll see how to apply this with examples from the Rust VLC plugin.

Import C structures and functions in Rust

Rust can easily use C code directly, by writing functions and structures definitions. A lot of the techniques you would use for this come from the "unsafe Rust" chapter of "The Rust Programming Language" book. For the following C code:

[code lang=C]
struct vlc_object_t {
const char *object_type;
char *header;
int flags;
bool force;
libvlc_int_t *libvlc;
vlc_object_t *parent;
};
[/code]

You would get the following Rust structure:

[code lang=C]
extern crate libc;
use libc::c_char;

#[repr(C)]
pub struct vlc_object_t {
pub psz_object_type: *const c_char,
pub psz_header: *mut c_char,
pub i_flags: c_int,
pub b_force: bool,
pub p_libvlc: *mut libvlc_int_t,
pub p_parent: *mut vlc_object_t,
}
[/code]

the #[repr(C)] tag indicates to the compiler that the structure should have a memory layout similar to the one generated by a C
compiler. We import types from the libc crate, like c_char. Those types are platform dependent (with their different form already handled in libc). Here, we use a lot of raw pointers (indicated by *), which means by using this structure directly, we're basically writing C, which is no good! A good approach, as we'll see later, is to write safer wrappers above those C bindings.

Importing C functions is quite straightforward too:

[code lang=C]
ssize_t vlc_stream_Peek(stream_t *, const uint8_t **, size_t);
ssize_t vlc_stream_Read(stream_t *, void *buf, size_t len);
uint64_t vlc_stream_Tell(const stream_t *);
[/code]

These C function declarations would get translated to:

[code lang=C]
#[link(name = "vlccore")]
extern {
pub fn vlc_stream_Peek(stream: *mut stream_t, buf: *mut *const uint8_t, size: size_t) -> ssize_t;
pub fn vlc_stream_Read(stream: *mut stream_t, buf: *const c_void, size: size_t) -> ssize_t;
pub fn vlc_stream_Tell(stream: *const stream_t) -> uint64_t;
}
[/code]

The #[link(name = "vlccore")] tag indicates to which library we are linking. It is equivalent to passing a -lvlccore argument to the linker. Libvlccore is a library all VLC plugins must link to. Those functions are declared like regular Rust functions, but like the previous structures, will mainly work on raw pointers.

bindgen

You can always write all your bindings manually like this, but when the amount of code to import is a bit large, it can be a good idea to employ the awesome bindgen tool, that will generate Rust code from C headers.

It can work as a command line tool, but can also work at compile time from a build script. First, add the dependency to your Cargo.toml file:

[code lang=toml]
[build-dependencies.bindgen]
version = "^0.25"
[/code]

You can then write your build script like this:

[code lang=C]
extern crate bindgen;
use std::fs::File;
use std::io::Write;
use std::path::Path;

fn main() {
let include_arg = concat!("-I", env!("INCLUDE_DIR"));
let vlc_common_path = concat!(env!("INCLUDE_DIR"), "/vlc_common.h");

let _ = bindgen::builder()
.clang_arg(include_arg)
.clang_arg("-include")
.clang_arg(vlc_common_path)
.header(concat!(env!("INCLUDE_DIR"), "/vlc_block.h"))
.hide_type("vlc_object_t")
.whitelist_recursively(true)
.whitelisted_type("block_t")
.whitelisted_function("block_Init")
.raw_line("use ffi::common::vlc_object_t;")
.use_core()
.generate().unwrap()
.write_to_file("src/ffi/block.rs");
}
[/code]

So there's a lot to unpack here, because bindgen is very flexible:

we use clang_arg to pass the include folder path and pre include a header everywhere (vlc_common.h is included pretty puch everywhere in VLC)
the header method specifies the header from which we will import definitions
hide_type prevents redefinition of elements we already defined (liek the ones from the common header)
whitelisted_type and whitelisted_function specify types and functions for which bindgen will create definitions
raw_line writes its argument at the top of the file. I apply it to reuse definitions from other files
write_to_file writes the whole definition to the specified path

You can apply that process to any C header you must import. With the build script, it can run every time the library is compiled, but be careful, generating a lot of headers can take some time. It might be a good idea to pregenerate them and commit the generated files, and update them from time to time.

It is usually a good idea to separate the imported definitions in another crate with the -sys suffix, and write the safe code in the main crate.
As an example, see the crates openssl and openssl-sys.

Writing safe wrappers

Previously, we imported the C function ssize_t vlc_stream_Read(stream_t *, void *buf, size_t len) as the Rust version pub fn vlc_stream_Read(stream: *mut stream_t, buf: *const c_void, size: size_t) -> ssize_t but kept an unsafe interface. Since we want to use those functions safely, we can now make a better wrapper:

[code lang=C]
use ffi;

pub fn stream_Read(stream: *mut stream_t, buf: &mut [u8]) -> ssize_t {
unsafe {
ffi::vlc_stream_Read(stream, buf.as_mut_ptr() as *mut c_void, buf.len())
}
}
[/code]

Here we replaced the raw pointer to memory and the length with a mutable slice. We still use a raw pointer to the stream_t instance, maybe we can do better:

[code lang=C]
use ffi;

pub struct Stream(*mut stream_t);

pub fn stream_Read(stream: Stream, buf: &mut [u8]) -> ssize_t {
unsafe {
ffi::vlc_stream_Read(stream.0, buf.as_mut_ptr() as *mut c_void, buf.len())
}
}
[/code]

Be careful if you plan to implement Drop for this type: is the Rust code supposed to free that object? Is there some reference counting involved? Here is an example of Drop implementation from the openssl crate:

[code lang=C]
pub struct SslContextBuilder(*mut ffi::SSL_CTX);

impl Drop for SslContextBuilder {
fn drop(&mut self) {
unsafe { ffi::SSL_CTX_free(self.as_ptr()) }
}
}
[/code]

Remember that it's likely the host application has a lot of infrastructure to keep track of memory, and as a rule, we should reuse the tools it offers for the code at the interface between Rust and C. See the Rust FFI omnibus for more examples of safe wrappers you can write.

Side note: as of now (2017/07/10) custom allocators are still not stable

Exporting Rust code to be called from C

Since the host application is written in C, it might need to call your code. This is quite easy in Rust: you need to write unsafe wrappers.

Here we will use as example the inverted index library for mobile apps I wrote for a conference. In this library, we have an Index type that we want to use from Java. Here is its definition:

[code lang=C]
#[repr(C)]
pub struct Index {
pub index: HashMap<String, HashSet<i32>>,
}
[/code]

This type has a few method we want to provide:

[code lang=C]
impl Index {
pub fn new() -> Index {
Index {
index: HashMap::new(),
}
}

pub fn insert(&mut self, id: i32, data: &str) {
[...]
}

pub fn search_word(&self, word: &str) -> Option<&HashSet<i32>> {
self.index.get(word)
}

pub fn search(&self, text: &str) -> HashSet<i32> {
[...]
}
}
[/code]

First, we need to write the functions to allocate and deallocate our index. Every use from C will be wrapped in a Box.

[code lang=C]
#[no_mangle]
pub extern "C" fn index_create() -> *mut Index {
Box::into_raw(Box::new(Index::new()))
}
[/code]

The Box type indicates and owns a heap allocation. When the box is dropped, the underlying data is dropped as well and the memory is freed. The following function takes ownership of its argument, so it is dropped at the end.

[code lang=C]
#[no_mangle]
pub extern "C" fn index_free(ptr: *mut Index) {
let _ = unsafe { Box::from_raw(ptr) };
}
[/code]

Now that allocation is handled, we can work on a real method. The following method takes an index, and id for a text, and the text itself, as a C string (ie, terminated by a null character).

Since we're kinda writing C in Rust here, we have to first check if the pointers are null. Then we can transform the C string in a slice. Then we check if it is correctly encoded as UTF-8 before inserting it into our index.

[code lang=C]
#[no_mangle]
pub extern "C" fn index_insert(index: *mut Index, id: i32, raw_text: *const c_char) {
unsafe { if index.is_null() || raw_text.is_null() { return } };
let slice = unsafe { CStr::from_ptr(raw_text).to_bytes() };
if let Ok(text) = str::from_utf8(slice) {
(*index).insert(id, text);
}
}
[/code]

Most of the code for those kinds of wrappers is just there to transform between C and Rust types and checking that the arguments coming from C code are correct. Even if we have to trust the host application, we should program defensively at the boundary.

There are other methods we could implement for the index, we'll leave those as exercise for the reader :)

Now, we need to write the C definitions to import those functions and types:

[code lang=C]
typedef struct Index Index;

Index* index_create();
void index_free(Index* index);
void index_insert(Index* index, int32_t id, char const* raw_text);
[/code]

We defined Index as an opaque type here. Since Rust structures can be compatible with C structures, we could export the real type, but since it only contains a Rust specific type, HashMap, it is better to hide it completely and write accessors and wrappers.

Generating bindings with rusty-cheddar

Writing function imports from C to Rust is tedious, so we have bindgen for this. We also have a great tool to go the other way: rusty-cheddar.

In the same way, it can be used from a build script:

[code lang=C]
extern crate cheddar;

fn main() {
cheddar::Cheddar::new().expect("could not read definitions")
.run_build("include/main.h");
cheddar::Cheddar::new().expect("could not read definitions")
.module("index").expect("malformed module path")
.insert_code("#include \"main.h\"")
.run_build("include/index.h");
}
[/code]

Here we run rusty-cheddar a first time without specifying the module: it will default to generate a header for the definitions in src/lib.rs.
The second run specifies a different module, and can insert a file inclusion at the top.

It can be a good idea to commit the generated headers, since you will see immediately if you changed the interface in a breaking way.

Integrating with the build system

As you might know, we can make dynamic libraries and executables with rustc and cargo. But often, the host application will have its own build system, and it might disagree with the way cargo builds its projects. So we have multiple strategies:

build Rust code separately, store libraries and headers in Maven or something (don't laugh, I've worked with such a system once, and it was actually great)
try to let rustc build dynamic libraries from inside the build system. We tried that for VLC and it was not great at all
build a static library from inside or outside the build system, include it in the libraries at link. This was done in Rusticata
build an object file and let the build system link it. This is what we ended up doing with VLC

Building a static library is as easy as specifying crate-type = ["staticlib"] in your Cargo.toml file. To build an object file, use the command cargo rustc --release -- --emit obj. You can see how we added it to the autotools usage in VLC.

Unfortunately, for this part we still do not have automated ways to fix the issues. Maybe with some time, people will write scripts for autotools,
CMake and others to handle Rust and Cargo.

Side note on reproducible builds: if you want to fix the set of Rust dependencies used in your project and make them always available, you can use cargo-vendor to store them in a specific folder

As you might have guessed, this is the most complex part, for which I have no good generic answer. I'd recommend that you spend the most time on this during the project's prototyping phase: import very little C code, export very little Rust code, try to make it build entirely from within the host application's build system. Once this is done, extending the project will get much easier. You really don't want to discover this task at the end of your project and try to retrofit your code in there.

Going further

While this article just explores the surface of Rust rewrites, I hope it provides a good starting point on the tools and techniques you can apply.
Any rewrite will be a large and complex project, but the result is worth the effort. The code you will write will be stronger, and Rust's type system will force you to review the assumptions made in the C version. You might even find better ways to write it once you start refactoring your code in a more Rusty way, safely hidden behind your wrappers.

Why you should, actually, rewrite it in Rust

2017-07-10T16:04:16+02:00

You might have seen those obnoxious "you should rewrite it in Rust comments" here and there:

in a blogpost
in a Github issue
you might have heard about that Rust Evangelism Strike Force
it even has a joke twitter account

It's like at every new memory vulnerability in well known software, there’s that one person saying Rust would have avoided the issue. We get it, it’s annoying, and it does not help us grow Rust. This attitude is generally frowned upon in the Rust community. You can't just show up into someone’s project telling them to rewrite everything.

so, why am I writing this? Why would I try to convince you, now, that you should actually rewrite your software in Rust?

That's because I have been working on this subject for a long time now:

I did multiple talks on it
I even co-wrote a paper
I did it both as client and personal work

So, I'm commited to this, and yes, I believe you should rewrite some code in Rust. But there's a right way to do it.

Why rewrite stuff?

Our software systems are built on sand. We got pretty good at maintaining and fixing them over the years, but the cracks are showing. We still have not fixed definitely most of the low level vulnerabilities: stack buffer overflow (yes, those still exist), heap overflow, use after free, double free, off by one; the list goes on. We have some tools, like DEP, ASLR, stack canaries, control flow integrity, fuzzing. Large projects with funding, like Chrome, can resort to sandboxing parts of their application. The rest of us can still run those applications inside a virtual machine. This situation will not improve. There's a huge amount of old (think 90s), bad quality, barely maintained code that we reuse everywhere endlessly. The good thing with hardware is that at some point, it gets replaced. Software just gets copied again. Worse, with the development of IoT, a lot of the code that ships will never be updated. It's likely that some of those old libraries will still be there 15, 20 years from now.

Let's not shy away from the issue here. Most of those are written in C or C++ (and usually an old version). It is well known that it is hard to write correct, reliable software in those languages. Think of all the security related things you have to keep track of in a C codebase:

pointer arithmetic
allocations and deallocations
data is mutable by default
functions return integers to mean pointers and error codes. Errors can be implicitely ignored
type casts, overflows and underflows are hard to track
buffer bounds in indexing and copying
all the undefined behaviours

Of course, some developers can do this work. Of course, there are sanitizers. But it's an enormous effort to perform everyday for every project.

Those languages are well suited for low level programming, but require extreme care and expertise to avoid most of those issues. And even then, we assume the developers will always be well rested, focused and careful. We're only humans, after all. Note that in 2017, there are still people claiming that a C developer with sufficient expertise would avoid all those issues. It's time we put this idea to rest. Yes, some projects can avoid a lot of vulnerabilities, with a team of good developers, frequent code reviews, a restricted set of features, funding, tools, etc. Most projects cannot. And as I said earlier, a lot of the code is not even maintained.

So we have to do something. We must make our software foundations stronger. That means fixing operating systems, drivers, libraries, command line tools, servers, everything. We might not be able to fix most of it today, or the next year, but maybe 10 years from now the situation will have improved.

Unfortunately, we cannot rewrite everything. If you ever attempted to rewrite a project from scratch, you'd know that while you can avoid some of the mistakes you made before, you will probably introduce a lot of regressions and new bugs. It's also wrong on the human side: if there are maintainers for the projects, they would need to work on the new and old one at the same time. Worse, you would have to teach them the new approach, the new language (which they might not like), and plan for an upgrade to the new project for all users.

This is not doable, and this is the part most people asking for project rewrites in Rust do not understand. What I'm advocating for is much simpler: surgically replace weaker parts but keep most of the project intact.

How

Most of the issues will happen around IO and input data handling, so it makes sense to focus on it. It happens there because that's where the code manipulates buffers, parsers, and uses a lot of pointer calculations. It is also the least interesting part for software maintainers, since it is usually not where you add useful features, business logic, etc. And this logic is usually working well, so you do not want to replace it. If we could rewrite a small part of an application or library without disrupting the rest of the code, we would get most of the benefits without the issues of a full rewrite. It is the exact same project, with the same interface, same distribution packaging as before, same developer team. We would just make an annoying part of the software stronger and more maintainable.

This is where Rust comes in. It is focused on providing memory safety, thread safety while keeping the code performant and the developer productive. As such, it is generally easier to get safe, reliable code in production while writing basic Rust, than a competent, well rested C developer using all the tools available could do.

Most of the other safe languages have strong requirements, like a runtime and a garbage collector. And usually, they expect to be the host application (how many languages assume they will handle the process's entry point?). Here, we are guests in someone else's house. We must integrate nicely and quietly.

Rust is a strong candidate for this because:

it can easily call C code
it can easily be called by C code (it can export C compatible functions and structures)
it does not need a garbage collector
if you want, it does not even need to handle allocations
the Rust compiler can produce static and dynamic libraries, and even object files
the Rust compiler avoids most of the memory vulnerabilities you get in C (yes, I had to mention it)

So you can actually take a piece of C code inside an existing project, import the C structures and functions to access them from Rust, rewrite the code in Rust, export the functions and structures from Rust, compile it and link it with the rest of the project.

If you don't believe it's possible, take a look at these two examples:

Rusticata integrates Rust parsers written with nom in Suricata, an intrusion detection system
a VLC media player plugin to parse FLV files, written entirely in Rust

You get a lot of benefits from this approach. First, Rust has great package management with Cargo and crates.io. That means you can separate some of the work in different libraries. See as an example the list of parsers from the Rusticata project. You can test them independently, and even reuse them in other projects. The FLV parser I wrote for VLC can also work in a Rust GStreamer plugin You can also make a separate library for the glue with the host application. I'm working on vlc_module exactly for that purpose: making Rust VLC plugins easier to write.

This approach works well for applications with a plugin oriented architecture, but you can also rewrite core parts of an application or library. The biggest issue is high coupling of C code, but it is usually easy to rewrite bit by bit by keeping a common interface. Whenever you have rewritten some coupled parts of of a project, you can take time to refactor it in a more Rusty way, and leverage the type system to help you. A good example of this is the rewrite of the Zopfli library from C to Rust.

This brings us to another important part of that infrastructure rewrite work: while we can rewrite part of an existing project without being too intrusive, we can also rewrite a library entirely, keeping exactly the same C API. You can have a Rust library, dynamic or static, with the exact same C header, that you could import in a project to replace the C one. This is a huge result. It's like replacing a load-bearing wall in an existing building. This is not an easy thing to realize, but once it's done, you can improve a lot of projects at once, provided your distribution's package manager supports that replacement, or other projects take the time to upgrade.

This is a lot of work, but every time we advance a little, everybody can benefit from it, and it will add up over the years. So we might as well start now.

Currently, I'm focused on VLC. This is a good target because it's a popular application that's often part of the basic stack of any computer (browser, office suite, media player). So it's a big target. But take a look at the list of dependencies in most web applications, or the dependency graph of common distributions. There is a lot of low hanging fruit there.

Now, how would you actually perform those rewrites? You can check out the next post and the paper explaining how we did it in Rusticata and VLC.

This year in nom: 2.0 is here!

2016-11-25T10:02:05+01:00

Nearly one year ago, on November 15th 2015, I released the 1.0 version of nom, the fast parser combinators library I wrote in Rust. A lot happened around that project, and I have been really happy to interact with nom users around the world.

TL;DR: it's new nom day! The 2.0 release is here! Read the changelog. Follow the upgrade documentation if it breaks stuff.

Interesting usage

I wouldn't be able to list all the projects using nom on this page, even the subset present on crates.io, but here are a few examples of what people built with nom:

semver briefly shipped with nom in February thanks to Steve Klabnik, until he replaced it with a regexp based solution (no hard feelings, I'd have done the same)
tomllib, a complete TOML implementation written by Joel Self
a JVM, because why not! Great work coming from a team of students at the University of Pennsylvania
Tagua VM, a great PHP implementation in Rust by Ivan Enderlin
syn, the Rust item parser written by David Tolnay everybody uses with the macros 1.1 feature to generate code from structures or enums, actually ships with its own fork of nom! It was forked to remove the incomplete data handling, and reduce compilation times
shen-rust, a complete implementation of the Shen language in Rust that was presented at Strangeloop 2016 by Aditya Siram
a series of parsers (DER, NTP, SNMP, IPSec, TLS) were developed for its integration in the Suricata network analysis tool. This work was presented at Suricon 2016 by Pierre Chifflier

And a lot of other projects. As a side note, people apparently like to build parsers for flac, bittorrent and bitcoin stuff, LISP and Scheme tokenizers and, oddly, ASN.1 libraries :D

I have been really humbled by what people achieved with this little library, and I hope it will enable even more awesome projects!

Growth and stabilization

The goal before 1.0 was to get a usable parsing library, and after 1.0, to add features people were missing and explore new ideas. A lot of code was contributed for bitstream and string parsing, and adding a lot of useful combinators like "peek!", "separated_list!" or "tuple!".

Unfortunately, a few parts of nom got increasingly painful to maintain and support, so the 2.0 was a good opportunity to clean them up, and add more features while we're at it.

The "chain!" combinator, which everybody uses to parse a sequence of things and accumulate the results in structs or tuple, is now deprecated, and will be replaced by "do_parse!", a simpler alternative. There are also a lot of specific helpers to make your code nicer, like "pair!", "preceded!", "delimited!", "separated_pair!", "separated_list!" and "delimited!". Yes, I went to great lengths to make sure you stop using chain :)

The "length_value!" and other associated combinators were refactored, to have more sensible names and behaviours. "eof", eol" and the basic token parsers like "digit" or "alphanumeric" got the same treatment. Those can be a source of issues in the upgrade to 2.0, but if the new behaviour does not work in your project, replacing them is still easy with the "is_a!" combinator and others.

At last, I changed the name of the "error!" macro that was conflicting with the one from the log crate. I hoped that by waiting long enough, the log people would change their macro, but it looks like I lost :p

New combinators

A few new simple combinators are here:

the previously mentioned "do_parse!" makes nicer code than "chain!":

The "chain!" version uses this weird closure-like syntax (while not actually using a closure) with a comma ending the parser list:

named!(filetype_parser<&[u8],FileType>, 
 chain!( 
 m: brand_name ~ 
 v: take!(4) ~ 
 c: many0!(brand_name) , 
 ||{ FileType{
   major_brand: m,
   major_brand_version:v,
   compatible_brands: c
 } } 
));

The "do_parse!" version only uses ">>" as separating token, and returns a value as a tuple. If the tuple contains only value, (A) is conveniently equivalent to A.

named!(filetype_parser<&[u8],FileType>, 
 do_parse!( 
   m: brand_name >> 
   v: take!(4) >> 
   c: many0!(brand_name) >> 
   (FileType{
     major_brand: m,
     major_brand_version:v,
     compatible_brands: c
   }) 
));

"chain!" had too many features, like a "?" indicating a parser was optional (which you can now do with "opt!"), and you could declare one of the values as mutable. All of those and the awkward syntax made it hard to maintain. Still, it was one of the first useful combinators in nom, and it can now happily retire

"permutation!" applies its child parser in any order, as long as all of them succeed once

  fn permutation() {
    named!(perm<(&[u8], &[u8], &[u8])>,
      permutation!(tag!("abcd"), tag!("efg"), tag!("hi"))
    );

    let expected = (&b"abcd"[..], &b"efg"[..], &b"hi"[..]);

    let a = &b"abcdefghijk"[..];
    assert_eq!(perm(a), Done(&b"jk"[..], expected));
    let b = &b"efgabcdhijk"[..];
    assert_eq!(perm(b), Done(&b"jk"[..], expected));
    let c = &b"hiefgabcdjk"[..];
    assert_eq!(perm(c), Done(&b"jk"[..], expected)
}

This one was very interesting to write :)

"tag_no_case!" works like "tag!", but compares independently from the case. This works great for ASCII strings, since the comparison requires no allocation, but the UTF-8 case is trickier, and I'm still looking for a correct way to handle it
"named_attr!" creates functions like "named!" but can add attributes like documentation. This was a big pain point, now nom parsers can have documentation generated by rustdoc
"many_till!" applies repeatedly its first child parser until the second succeeds

Whitespace separated formats

This is one of the biggest new additions, and a feature that people wanted for a long time. A lot of the other Rust parser libraries are designed with programming languages parsing in mind, while I started nom mainly to parse binary formats, like video containers. Those libraries usually handle whitespace parsing for you, and you only need to specify the different elements of your grammars. You essentially work on a list of already separated elements.

Previously, with nom, you had to explicitely parse the spaces, tabs and end of lines, which made the parsers harder to maintain. What we want in the following example is to recognize a "(", an expression, then a ")", and return the expression, but we have to introduce a lot more code:

named!(parens<i64>, delimited!(
    delimited!(opt!(multispace), tag!("("), opt!(multispace)),
    expr,
    delimited!(opt!(multispace), tag!(")"), opt!(multispace))
  )
);

This new release introduces "ws!", a combinator that will automatically insert the separators everywhere:

named!(parens<i64>, ws!(delimited!( tag!("("), expr, tag!(")") )) );

By default, it removes spaces, tabs, carriage returns and line feed, but you can easily specify your own separator parser and make your own version of "ws!".

This makes whitespace separated formats very easy to write. See for example the quickly put together, probably not spec compliant JSON parser I added as test.

If you're working on a language parsers, this should help you greatly.

Architecture changes

Error management

The error management system that accumulated errors and input positions as it backtracks through the parser tree is great for some projects like language parsers, but others were not using it and got a penalty because of vectors allocation and deallocation.

In the 2.0 release, this error management system is now activated by the "verbose-errors" feature. Projects that don't use it should build correctly right away, and their parsers could get 30% to 50% faster!

Input types

One of nom's original assumptions was that it should work on byte slices and strings instead of byte or char iterators, because the CPU likes contiguous data. As always, the reality is a bit more complex than that, but it worked well and made the code very simple: I only passed subslices from one parser to the next.

But I wrongly assumed that because of that design, nom could only work on contiguous data. Carl Lerche made the interesting point that there are few points where nom actually needs to read a serie of bytes or chars and those could accomodate other data structures like ropes or a list of buffers.

So I got to work on an abstraction for input types that would work for &[u8] and &str, but also for other types. In the process, I was able to factor most of the &str specific combinators with the &[u8] ones. This will make them easier to maintain in the future.

The result of that work is a list of traits that any input type should implement to be usable with nom. I experimented a bit with the BlockBuf type, and this approach looks promising. I expect that people will find cool applications for this, like parsers returning references to not yet loaded data, or blocking a coroutine on a tag comparison until the data is available.

A smooth upgrade process

For the 1.0 release, I choose a few projects using nom, and tried to build them to test the new version and document the upgrade. This was so useful that I did it again for 2.0, so if you're lucky, you maintain one of the 30 crates I tested, and you received a pull request doing that upgrade for you. Otherwise, I wrote an upgrade documentation that you can follow to fix the migration issues. You're still lucky, though, because most crates will build (or only require a one line fix in Cargo.toml).

I'll write soon about that process and the benefits you can get by applying it to your projects.

The future

I have a lot of ideas for the next version, also a lot of pull requests to merge and issues to fix. Not everything could make it into the 2.0, otherwise I would never have released it.

In short, the plan:

rewrite completely the producers and consumers system. It is not very usable right now. It could be replaced by an implementation based on futures
improve the performance. I got a good enough library by choosing the most naive solutions, but there are lots of points I could improve (especially in helping LLVM generate faster code)
implement a new serialization library. I believe there is some room for a serialization system that does not rely on automatic code generation, and it would go well with nom
continue my work on writing nom demuxers for VLC media player. I have a good proof of concept, now I need to make it production ready
add new, interesting examples: indentation based programming languages, tokio integration, integration in high performance networking systems
I'll release very soon a large networking tool that relies heavily on nom. Expect some big news :)

That's it, now go and upgrade your code, you'll enjoy this new version!

Unhandled Expression

nom 5 is here

Why?

Using functions

Error management

Streaming VS complete parsers

Various other changes

Performance

Thanks

FOSS is free as in toilet

No, pest is not faster than nom

The real benchmark

nom 4.0: faster, safer, simpler parsers

Moving from IResult to Result

Dealing with Incomplete usage

Better documentation

Various fixes

Performance

the future for nom

PoC: compiling to eBPF from Rust

Rust 2018: maybe don’t be too stable

Adventures in logging

Beginning with log and env_logger

Custom formatter

Where things get annoying: reducing allocations

So let's write our own then

Highly concurrent logs

the dreaded logging singleton

Removing the last Mutex

How to rewrite your project in Rust

Import C structures and functions in Rust

bindgen

Writing safe wrappers

Exporting Rust code to be called from C

Generating bindings with rusty-cheddar

Integrating with the build system

Going further

Why you should, actually, rewrite it in Rust

Why rewrite stuff?

How

This year in nom: 2.0 is here!

Interesting usage

Growth and stabilization

New combinators

Whitespace separated formats

Architecture changes

Error management

Input types

A smooth upgrade process

The future

Moving from `IResult` to `Result`