PoC: compiling to eBPF from Rust

I have been playing with eBPF (extended Berkeley Packet Filters), a neat feature present in recent Linux versions (it evolved from the much older BPF filters). It is a virtual machine running in the kernel, to which you can send code from userland, and that code can be used to filter packets or trace parts of the kernel code.

What makes eBPF really nice is how the kernel handles it. You send a program in bytecode format to the kernel, it then checks it, verifying, for example, that there are no loops, thus guaranteeing that the program will terminate, and it will then apply JIT compilation, making the resulting code quite fast. Even better, that code can be loaded and unloaded at any time through a syscall, and you can set up shared data structures between the eBPF program and your own, to efficiently gather data.

As an example, you can use eBPF (and the XDP – eXpress Data Path – feature) to write very efficient firewalls, or employ BCC (BPF Compiler Collection) to trace a process’s IO events.

I’m looking at how we could use that to trace applications on our infrastructure at Clever Cloud. There are a few things we should know about the tooling first.

At the beginning, people wrote their program using the bytecode directly:

/* Compare IPv4 with one word instruction (32bit) /
struct bpf_insn insn[] = {
If skb->protocol != ETH_P_IP, skip this whole block. The offset will be set later. */
BPF_JMP_IMM(BPF_JNE, BPF_REG_7, htobe16(protocol), 0),

* Call into BPF_FUNC_skb_load_bytes to load the dst/src IP address
* R1: Pointer to the skb
* R2: Data offset
* R3: Destination buffer on the stack (r10 - 4)
* R4: Number of bytes to read (4)

BPF_MOV32_IMM(BPF_REG_2, addr_offset),

BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -addr_size),

BPF_MOV32_IMM(BPF_REG_4, addr_size),
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_skb_load_bytes),

* Call into BPF_FUNC_map_lookup_elem to see if the address matches any entry in the
* LPM trie map. For this to work, the prefixlen field of 'struct bpf_lpm_trie_key'
* has to be set to the maximum possible value.
* On success, the looked up value is stored in R0. For this application, the actual
* value doesn't matter, however; we just set the bit in @verdict in R8 if we found any
* matching value.

BPF_LD_MAP_FD(BPF_REG_1, map_fd),
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -addr_size - sizeof(uint32_t)),
BPF_ST_MEM(BPF_W, BPF_REG_2, 0, addr_size * 8),

BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
BPF_ALU32_IMM(BPF_OR, BPF_REG_8, verdict),

This is a bit raw, and somewhat complex to write, so people worked on C to eBPF compilers, and the feature landed in LLVM: we can use clang to write eBPF programs! It will generate the bytecode, that can then be loaded through the bpf() syscall.

This is still a bit complex, since the eBPF program might need access to some internal data structures of the kernel, and those change depending on kernel versions and configuration options. And we still need to set up the shared data structures with the userland program that will gather data.

That’s why the BCC project provides an easy to use interface to compile and load eBPF programs. They made it so simple that you can write a python script to compile, load and interact with your program:

from bcc import BPF

BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\n"); return 0; }').trace_print()

They provide a lot of useful examples and a nice tutorial to get started writing eBPF tracers.

Unfortunately, those tools make a tradeoff that’s slightly annoying for me: they require installing BCC, which requires Python, LLVM and the complete Linux sources, on the target machines. It might be possible to precompile the programs though, but it does not look like it’s a common use case with BCC.

So, maybe there’s a nice way to precompile those programs, store them as bytecode, then load them with a small agent that does not need LLVM and the kernel sources to work? It turns out it is possible, thanks to the gobpf project, who split their ELF loading code from the BCC part a year ago.

And, now, you’ll see where I am going with this. Being one of those annoying Rust developers who want to rewrite everything in their favorite language, I thought “hey, maybe I can Rust that thing too!”

Since it is possible to compile to eBPF bytecode from C, it is possible to compile LLVM IR (the kind of bytecode LLVM generates from the code before compiling it to the target CPU’s assembly) to eBPF. Look for “LLVM IR debugging” in this link for an example. And I know I can compile Rust to that LLVM IR, and everything should work out, as long as Rust’s LLVM version is the same as the system’s version.

So I created a small Rust project, and wrote the following build script:

cargo rustc --release -- --emit=llvm-ir
cp target/release/deps/hello-*.ll hello.ll
cargo rustc --release -- --emit=llvm-bc
cp target/release/deps/hello-*.bc hello.bc

llc-4.0 hello.bc -march=bpf -filetype=obj -o hello.o

(I generated the ll file to take a look at the IR in text form)

And now, the code!

I used the example program from this blog post as inspiration, and came up with this:

use std::mem::transmute;
use std::ffi::CStr;

#[link_section = "license"]
pub static _license: [u8; 4] = [71u8, 80, 76, 0]; //b"GPL\0"
#[link_section = "version"]
pub static _version: u32 = 0xFFFFFFFE;

#[link_section = "kprobe/SyS_clone"]
pub extern "C" fn kprobe__sys_clone(ctx: *mut u8) -> i32 {
let BPF_FUNC_trace_printk = unsafe {
transmute:: i32>(6)

let msg: [u8; 17] = [104u8, 101, 108, 108, 111, 32, 102, 114, 111, 109, 32, 114, 117, 115, 116, 10, 0]; //b"hello from Rust\0"
BPF_FUNC_trace_printk((&msg).as_ptr(), 17);

return 0;

Firs, the constants are in their own ELF section, this is expected by gobpf’s elf loader. Apparently, I cannot write pub static _license: &'static [u8] = b"GPL\0", because the _license symbol would then be a relocation of the actual string.

#[link_section = "license"]
pub static _license: [u8; 4] = [71u8, 80, 76, 0];//b"GPL\0";
#[link_section = "version"]
pub static _version: u32 = 0xFFFFFFFE;

Now, the function: gobpf expects a section with the name of the function we will try to hook later.

#[link_section = "kprobe/SyS_clone"]
pub extern "C" fn kprobe__sys_clone(ctx: *mut u8) -> i32 {

Now we might need to import functions. They are defined in a C enum, but interpreted as function pointers. So calling the printk function from there amounts writing the instruction call 6.
So we transmute the number 6 into a function. I know, ewww, but it works 😀

let BPF_FUNC_trace_printk = unsafe {
transmute:: i32>(6)

And now, the last part, actually printing something. To avoid the previous issue of a constant string appearing as a relocated symbol on which gobpf will throw an error, I defined it as a local constant, then called BPF_FUNC_trace_printk on it. That should be harmless, right?

let msg: [u8; 17] = [104u8, 101, 108, 108, 111, 32, 102, 114, 111, 109, 32, 114, 117, 115, 116, 10, 0]; //b"hello from Rust\0"
BPF_FUNC_trace_printk((&msg).as_ptr(), 17);

So now, let’s take a look at the generated LLVM IR (the ll file):

; ModuleID = 'hello0-b4990b5a434d0f01306c6e79c17f427.rs'
source_filename = "hello0-b4990b5a434d0f01306c6e79c17f427.rs"
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

@_license = local_unnamed_addr constant [4 x i8] c"GPL\00", section "license", align 1
@_version = local_unnamed_addr constant i32 -2, section "version", align 4

; Function Attrs: nounwind uwtable
define i32 @kprobe__sys_clone(i8* nocapture readnone %ctx) unnamed_addr #0 section "kprobe/SyS_clone" {
%msg = alloca [17 x i8], align 16
%0 = getelementptr inbounds [17 x i8], [17 x i8]* %msg, i64 0, i64 0
call void @llvm.lifetime.start(i64 17, i8* nonnull %0)
%1 = bitcast [17 x i8]* %msg to *
store , * %1, align 16
%2 = getelementptr inbounds [17 x i8], [17 x i8]* %msg, i64 0, i64 16
store i8 0, i8* %2, align 16
%3 = call i32 (i8*, i64, ...) inttoptr (i64 6 to i32 (i8*, i64, ...))(i8 nonnull %0, i64 17) #2
call void @llvm.lifetime.end(i64 17, i8* nonnull %0)
ret i32 0

; Function Attrs: argmemonly nounwind
declare void @llvm.lifetime.start(i64, i8* nocapture) #1

; Function Attrs: argmemonly nounwind
declare void @llvm.lifetime.end(i64, i8* nocapture) #1

attributes #0 = { nounwind uwtable "probe-stack"="__rust_probestack" }
attributes #1 = { argmemonly nounwind }
attributes #2 = { nounwind }

So, it generates the correct symbols and sections for _license and _version. It apparently generates a big store instruction for the string we’ll print. And the function call looks like this call i32 (i8*, i64, ...) inttoptr (i64 6 to i32 (i8*, i64, ...)*)(i8* nonnull %0, i64 17) where we cast 6 to a function pointer: inttoptr (i64 6 to i32 (i8*, i64, ...)*).

So that should generate correct BPF bytecode, right? Let’s check that with the command llvm-objdump-4.0 -S hello.o:

hello.o: file format ELF64-BPF

Disassembly of section kprobe/SyS_clone:
0: b7 01 00 00 0a 00 00 00 r1 = 10
1: 73 1a ef ff 00 00 00 00 *(u8 *)(r10 - 17) = r1
2: b7 01 00 00 74 00 00 00 r1 = 116
3: 73 1a ee ff 00 00 00 00 *(u8 *)(r10 - 18) = r1
4: b7 01 00 00 73 00 00 00 r1 = 115
5: 73 1a ed ff 00 00 00 00 *(u8 *)(r10 - 19) = r1
6: b7 01 00 00 75 00 00 00 r1 = 117
7: 73 1a ec ff 00 00 00 00 *(u8 *)(r10 - 20) = r1
8: b7 01 00 00 6d 00 00 00 r1 = 109
9: 73 1a e9 ff 00 00 00 00 *(u8 *)(r10 - 23) = r1
10: b7 01 00 00 72 00 00 00 r1 = 114
11: 73 1a eb ff 00 00 00 00 *(u8 *)(r10 - 21) = r1
12: 73 1a e7 ff 00 00 00 00 *(u8 *)(r10 - 25) = r1
13: b7 01 00 00 66 00 00 00 r1 = 102
14: 73 1a e6 ff 00 00 00 00 *(u8 *)(r10 - 26) = r1
15: b7 01 00 00 20 00 00 00 r1 = 32
16: 73 1a ea ff 00 00 00 00 *(u8 *)(r10 - 22) = r1
17: 73 1a e5 ff 00 00 00 00 *(u8 *)(r10 - 27) = r1
18: b7 01 00 00 6f 00 00 00 r1 = 111
19: 73 1a e8 ff 00 00 00 00 *(u8 *)(r10 - 24) = r1
20: 73 1a e4 ff 00 00 00 00 *(u8 *)(r10 - 28) = r1
21: b7 01 00 00 6c 00 00 00 r1 = 108
22: 73 1a e3 ff 00 00 00 00 *(u8 *)(r10 - 29) = r1
23: 73 1a e2 ff 00 00 00 00 *(u8 *)(r10 - 30) = r1
24: b7 01 00 00 65 00 00 00 r1 = 101
25: 73 1a e1 ff 00 00 00 00 *(u8 *)(r10 - 31) = r1
26: b7 01 00 00 68 00 00 00 r1 = 104
27: 73 1a e0 ff 00 00 00 00 *(u8 *)(r10 - 32) = r1
28: b7 01 00 00 00 00 00 00 r1 = 0
29: 73 1a f0 ff 00 00 00 00 *(u8 *)(r10 - 16) = r1
30: bf a1 00 00 00 00 00 00 r1 = r10
31: 07 01 00 00 e0 ff ff ff r1 += -32
32: b7 02 00 00 11 00 00 00 r2 = 17
33: 85 00 00 00 06 00 00 00 call 6
34: b7 00 00 00 00 00 00 00 r0 = 0
35: 95 00 00 00 00 00 00 00 exit

So, the lines 0 to 29 are there to load our string (instead of pointing to a constant somewhere). The expected call 6 instruction is on line 33!

To load that program, we can now use the example Go program from the blog post (slightly modified to accept a program name as argument): sudo ./bpf-load hello.o. The eBPF program will the hook the sys_clone function and print a hello every time it is called. You can see the trace with the command sudo cat /sys/kernel/debug/tracing/trace_pipe.

And now you can be a happy Rust developer like me because again, you put some Rust where you were not supposed to!

Now, this is a small hack. To make it more useful, here is what we would need:

  • a small library to import BPF functions instead of transmuting the number every time
  • that small library should also have a nice way to interact with BPF maps to transmit data to userland
  • a userland library (in Rust, of course) that can set up maps and load eBPF programs. I should mention here that Julia Evans is currently working on a port of gobpf’s BCC part in Rust for a Ruby profiling tool! The ELF part might not be too far 🙂

That’s all, I’ll post more once I get more useful code working!

PS/ if you want to learn more about BPF, read this great list!


Adventures in logging

After working on the Sōzu HTTP reverse proxy for a while, I came up with an interesting approach to logging. Now why would I come up with my own logger, when there are existing solutions in Rust? Mainly, log and slog. That logging library grew up from testing things out with log, and changing requirements along the way.

Beginning with log and env_logger

Like a lot of other Rust developers, I started out with log and env_logger:

extern crate log;
extern crate env_logger;

fn main() {

  info!("starting up");

It’s nice and easy: every library that depends on log will use that same set of logging macros (error, info, warn, debug, trace) that will use whatever global logger was defined. Here we use env_logger to define one.

env_logger is useful because it can apply a filter to the log, from an
environment variable:

# will show the logs above info level
RUST_LOG=info ./main

# will show the logs above info level, but also logs above debug level
# for the dependency 'dep1'
RUST_LOG=info,dep1=debug ./main

You can also define the filter by module or apply a regular expression.

Custom formatter

env_logger allows you to build your own log formatter. This feature is especially important for me, as I like to add metadata to my logs.

Defining a custom formatter with env_logger is quite straightforward:

let format = |record: &LogRecord| {
  format!("{} - {}", record.level(), record.args())

let mut builder = LogBuilder::new();
builder.format(format).filter(None, LogLevelFilter::Info);

if env::var("RUST_LOG").is_ok() {


It is easily combined with the filtering and usage of the RUST_LOG environment variable.

Where things get annoying: reducing allocations

If you take a look at env_logger, you’ll realize that it will allocate a String for every log line that will be written, using a formatting closure.

Let’s get one thing out of the way first: I completely agree with the idea you should not try to optimize stuff too much. But I’m in the case of a networking component that will handle a lot of traffic. I had debugging sessions where I generated tens of gigabytes of logs in a few seconds, and needed almost all of them, to debug async IO issues. In those cases, the time spent allocating and deallocating log lines becomes relevant.

So, how would I get a custom log formatter that does not allocate much? As it turns out, when you tell log to use your logger with log::set_logger, it requires something that implements Log. The logger’s log method receives a LogRecord, a structure that’s created on the fly from LogLocation, LogMetadata and Arguments.
The first two are internal to log, I can’t create them myself. The last one is interesting.

Arguments can be created from the format_args macro. That structure will roughly contain the format string split in the various substrings that appear between arguments. if you do println!("hello {}!", name), you would get a structure that contains "hello ", the content of name and "!". println! and other macros use this.

You can then use that Arguments with io::Write::write_fmt to write it directly to, say, a file or a socket. And it is implemented so that the individual parts are written one after another instead of allocating one big string.

So, how do I use that?

Well, it turns out that, basically, I can’t. If I implement Log, I can get a Logrecord which gives me a &Arguments, while write requires a Arguments. So now I have to clone it, which defeats a bit the purpose.

So let’s write our own then

There was another reason for the custom logging library: using a custom logging backend. Having the option between stdout and stderr is fine, but I might want to send them to a file or a socket.

So I started writing a specific logging library for sozu. First, copying the log filtering from env_logger. That part is mostly straightforward, but that’s still a lot of code to copy around.

The logging macros specify a logging level then they all call the same common macro.

macro_rules! error {
    ($format:expr, $($arg:tt)*) => {
        log!($crate::logging::LogLevel::Error, $format, "ERROR", $($arg)*);
    ($format:expr) => {
        log!($crate::logging::LogLevel::Error, $format, "ERROR");

The main logging macro has two interesting parts. First, we define some static metadata (that’s coming from the log crate):

static _META: $crate::logging::LogMetadata = $crate::logging::LogMetadata {
  level:  $lvl,
  target: module_path!(),

That object will not be allocated over and over, all the data in there will be defined at compile time.

Then we call the logger itself (ignore the line with try_lock for now):

if let Ok(mut logger) = LOGGER.try_lock() {
      concat!("{}\t{}\t{}\t{}\t{}\t", $format, '\n'),
        ::time::now_utc().rfc3339(), ::time::precise_time_ns(), *$crate::logging::PID,
        $level_tag, *$crate::logging::TAG));

So we give this metadata structure to our logger, then we make an Arguments structure with format_args!. The concat! macro is there to concatenate the formatting string with the custom prefix. That way, I could write debug!("hello {}", name) and have the resulting format string be "{}\t{}\t{}\t{}\t{}\thello {}\n", generated at compile time and transformed through the format_args call.

I added the date in ISO format, along with a monotonic timestamp (that becomes handy when multiple workers might write logs concurrently), the process identifier, the log level and a process wide logging tag (to better identify workers).

So this starts looking good, right? Now how do we write this to configurable backends? Some backends already implement io::Write, others will need an intermediary buffer:

pub fn log<'a>(&mut self, meta: &LogMetadata, args: Arguments) {
    if self.enabled(meta) {
        match self.backend {
            LoggerBackend::Stdout(ref mut stdout) => {
            LoggerBackend::Unix(ref mut socket) => {
            LoggerBackend::Udp(ref mut socket, ref address) => {
                socket.send_to(format(args).as_bytes(), address);
            LoggerBackend::Tcp(ref mut socket) => {

For Unix sockets and UDP, instead of allocating on the fly, it should probably use a buffer (hey, anyone wants to implement that?). Stdout and a TcpStream can be written to directly. Adding buffers might still be a good idea here, depending on what you want, because that write could fail. Would you like a logger that will send a partial log if it can’t write on the socket, or one using a buffer that can be filled up?

So, now, what’s next? Originally, sozu worked as one process with multiple threads, but evolved as a bunch of single threaded processes. But that raises an interesting question. How do you write logs concurrently?

Highly concurrent logs

It turns out that problem is not really easy. Most solutions end up in this list:

  • every thread or process writes to stdout or a file at the same time
  • synchronized access to the logging output
  • one common logger everybody sends to
  • every thread or process has its own connection to syslog (or even its own file to write to)

The first solution is easy, but has a few issues. First, writing to stdout is slow, and it can quickly overwhelm your terminal (yes, I know you can redirect to a file). Second, it’s not really synchronized, so you might end up with incoherently interleaved log lines.

So we often move to the second solution, where access to the logging stream is protected by a mutex. Now you get multiple threads or processes that might spend their time waiting on each other for serializing and writing logs. Having all threads sharing one lock can quickly affect your performance. It’s generally a better idea to have every thread or process running independently from the others (it’s one of the principles in sozu’s architecture, you can learn more about it in this french talk).

Alright then, moving on to the third solution: let’s have one of the threads or processes handle the logging, and send the logs to it via cross thread channels or IPC. That will surely be easier than having everybody share a lock, right? This is also intersting because you can offload serialization and filtering to the logging thread. Unfortunately, that means one thread will handle all the logs, and it can be overwhelming. That also means a lot of data moving between workers (if using processes).

The last solution relies on external services: use the syslog daemon on your system, or a log aggregator somewhere on another machine, that every worker will send the logs to. Let that service interleave logs, and maybe scale up if necessary. Since in a large architecture, you might have such an aggregator, talking to it directly might be a good idea (oh hey BTW I wrote a syslog crate if you need).

With sozu, I ended up with a mixed solution. You can send the logs to various backends. If you choose stdout, then all workers will write to it directly without synchronization which will be mostly fine if you don’t have a large traffic. But if you want, each worker can open its own connection to the logging service.

Now that concurrency is taken care of, there’s a last issue that has annoyed me for months: how to use the logger from everywhere in the code, when it’s obviously one big global mutable object?

the dreaded logging singleton

One thing I like about the macros from the log crate: you can use them anywhere in your code, and it will work. The other approach, used in languages like Haskell, or proposed by slog, is to carry your logger around, in function arguments or structure members. I can understand the idea, but I don’t like it much, because I’ll often need to add a debug call anywhere in the code, and when it’s deep in a serie of five method calls, with those methods coming from traits implemented here and there, updating the types quickly gets annoying.

So, even if the idea of that global mutable logger singleton usually looks like a bad pattern, it can still be useful. In log, the log macro calls the log::Log::log method, getting the logger instance from the logger method. That method gets the logger instance from a global pointer to the logger, with an atomic integer used to indicate if the logger was initialized:

static mut LOGGER: *const Log = &NopLogger;
static STATE: AtomicUsize = ATOMIC_USIZE_INIT;

const UNINITIALIZED: usize = 0;
const INITIALIZING: usize = 1;
const INITIALIZED: usize = 2;


pub fn logger() -> &'static Log {
    unsafe {
        if STATE.load(Ordering::SeqCst) != INITIALIZED {
            static NOP: NopLogger = NopLogger;
        } else {

So how can that global mutable pointer be used from any thread? That’s because the Log trait requires Send and Sync. As a reminder, Send means it can be sent to another thread, Sync means it can be shared between threads.

That’s a cool trick, but usually we employ another pattern:

lazy_static! {
  pub static ref LOGGER: Mutex<Logger> = Mutex::new(Logger::new());
  pub static ref PID:    i32           = unsafe { libc::getpid() };
  pub static ref TAG:    String        = LOGGER.lock().unwrap().tag.clone();

lazy_static allows you to define static variables that will be initialized at runtime, at first access (using the same pattern as log with std::sync::Once. Since our logger’s log method needs an &amp;mut, it’s wrapped in a Mutex. That’s where the call to try_lock comes from.

We might think the mutex is costly compared to log’s solution, but remember that the logger instance has to be Sync, so depending on your implementation, there might be some synchronization somewhere. Except that for sozu, it’s not the case! Each worker is single threaded, and has its own instance of the logger (possibly with each of them a connection to the logging service). Can’t we have a logging system that does not require that mutex used everywhere?

Removing the last Mutex

This is a problem that annoyed me for months. It’s not that I really mind the cost of that mutex (since no other thread ever touches it). It’s just that I’d feel better not using one when I don’t need it 🙂

And the solution to that problem got quite interesting. To mess around a bit, here’s a playground of the logging solution based on lazy_static. You’ll see why the code in Foo::modify is important.

There’s a feature you can use to have a global variable available anywhere in a thread: thread_local. It uses thread local storage with an UnsafeCell to initialize and provide a variable specific to each thread:

thread_local!(static FOO: RefCell<u32> = RefCell::new(1));

FOO.with(|f| {
    assert_eq!(*f.borrow(), 1);
    *f.borrow_mut() = 2;

So I tried to use this with my logger, but encountered an interesting bug, as you can see in another playground. I replaced my logging macro with this:

macro_rules! log {
    ($format:expr, $($arg:tt)+) => ({
            LOGGER.with(|l| {
                        concat!("\t{}\t", $format, '\n'),
                        "a", $($arg)+));

And got this error:

error[E0502]: cannot borrow `self` as immutable because `self.opt.0` is also borrowed as mutable
  --> src/main.rs:36:21
36 |         LOGGER.with(|l| {
   |                     ^^^ immutable borrow occurs here
54 |         if let Some(ref mut s) = self.opt {
   |                     --------- mutable borrow occurs here
55 |             log!("changing {} to {}", self.bar, new);
   |             -----------------------------------------
   |             |                         |
   |             |                         borrow occurs due to use of `self` in closure
   |             in this macro invocation
56 |         }
   |         - mutable borrow ends here

When implementing this inside sozu’s source, I got about 330 errors like this one… So what happened? That with method requires a closure. Since we use a macro, if we use self.bar as argument, it will appear inside the closure. That becomes an issue with anything that has been already mutably borrowed somewhere.

I tried a few things, like calling format_args outside the closure, but I get the error error[E0597]: borrowed value does not live long enough. This is apparently a common problem with format_args.

But the real solution came from tomaka, with some macro trickery, as seen in one last playground:

macro_rules! log {
    ($format:expr $(, $args:expr)*) => ({
        log!(__inner__ $format, [], [a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v]
             $(, $args)*)

    (__inner__ $format:expr, [$($transformed_args:ident),*], [$first_ident:ident $(, $other_idents:ident)*], $first_arg:expr $(, $other_args:expr)*) => ({
        let $first_ident = &$first_arg;
        log!(__inner__ $format, [$($transformed_args,)* $first_ident], [$($other_idents),*] $(, $other_args)*);

    (__inner__ $format:expr, [$($final_args:ident),*], [$($idents:ident),*]) => ({
        LOGGER.with(move |l| {
          //let mut logger = *l.borrow_mut();
              concat!("\t{}\t", $format, '\n'),
              "a" $(, $final_args)*));

The basic idea is that we could avoid the borrowing issue by doing an additional borrow. But since some of the arguments might by expressions (like 1+1 or self.size), we will store the reference to it in a local variable, with let $first_ident = &amp;$first_arg;.

We cannot create variable or function names in macros out of thin air (sadly, because that would be extremely useful), so we instead do recursive macros calls, consuming arguments one after another.
In [$($transformed_args:ident),*], [$first_ident:ident $(, $other_idents:ident)*], $first_arg:expr $(, $other_args:expr)*,
transformed_args accumulates the idents (variable names) in which we stored the data.

[$first_ident:ident $(, $other_idents:ident)*] is matching on the list that started as [a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v], to get the first one in that list (storing it in $first_ident), and using it as variable names. As you might have guessed, that means I won’t be able to use a log line with more than 21 arguments. That’s a limitation I can live with.

The $first_arg:expr $(, $other_args:expr)* part matches on the log call’s arguments, and gets the first in the list as $first_arg. We then use those in the line let $first_ident = &amp;$first_arg; and recursively call log!, adding the variable name $first_ident to the list of transformed arguments, and the rest of the variable names list and the log call’s arguments:

log!(__inner__ $format, [$($transformed_args,)* $first_ident], [$($other_idents),*] $(, $other_args)*);

Once all of the logger’s arguments are consumed, we can call format_args on the list of $transformed_args:

(__inner__ $format:expr, [$($transformed_args:ident),*], [$($idents:ident),*]) => ({
    LOGGER.with(move |l| {
            //let mut logger = *l.borrow_mut();
                        concat!("\t{}\t", $format, '\n'),
                        "a" $(, $transformed_args)*));

and it works!

So, that last part may not be completely relevant to your logger implementation, but I thought
it was quite cool 🙂

Despite my issues with the log crate, it’s quite useful and supported by a lot of libraries. It is currently getting a lot better as part of the libz blitz. I’d also encourage you to check out slog. I haven’t felt the need to integrate it in sozu yet, but it can be interesting for new projects, as it comes with an ecosystem of composable libraries to extend it.

How to rewrite your project in Rust

In a previous post, I explained why rewriting existing software in Rust could be a good idea. The main point being that you should not rewrite the whole application, but replace the weaker parts without disturbing most of the code, to strengthen the codebase without disruption.

I also provided pointers to projects where other people and I did it succesfully, but without giving too many details. So let’s get a real introduction to Rust rewrites now. This article requires a little bit of knowledge about Rust, but you should be able to follow it even as a

As a reminder, here are the benefits Rust bring into a rewrite:

  • it can easily call C code
  • it can easily be called by C code (it can export C compatible functions and structures)
  • it does not need a garbage collector
  • if you want, it does not even need to handle allocations
  • the Rust compiler can produce static and dynamic libraries, and even object files
  • the Rust compiler avoids most of the memory vulnerabilities you get in C (yes, I had to mention it)
  • Rust is easier to maintain than C (this is discutable, but not the point of this article)

As it turns out, this is more or less the plan to replace C code with Rust:

  • import C structures and functions in Rust
  • import Rust structures and functions from C
  • reuse the host application’s memory allocations whenever possible
  • write code (yes, we have to do it at some point)
  • produce artefacts that can be linked with the host application
  • integrate with the build system

We’ll see how to apply this with examples from the Rust VLC plugin.

Import C structures and functions in Rust

Rust can easily use C code directly, by writing functions and structures definitions. A lot of the techniques you would use for this come from the “unsafe Rust” chapter of “The Rust Programming Language” book. For the following C code:

struct vlc_object_t {
    const char   *object_type;
    char         *header;
    int           flags;
    bool          force;
    libvlc_int_t *libvlc;
    vlc_object_t *parent;

You would get the following Rust structure:

extern crate libc;
use libc::c_char;

pub struct vlc_object_t {
  pub psz_object_type: *const c_char,
  pub psz_header:      *mut c_char,
  pub i_flags:         c_int,
  pub b_force:         bool,
  pub p_libvlc:        *mut libvlc_int_t,
  pub p_parent:        *mut vlc_object_t,

the #[repr(C)] tag indicates to the compiler that the structure should have a memory layout similar to the one generated by a C
compiler. We import types from the libc crate, like c_char. Those types are platform dependent (with their different form already handled in libc). Here, we use a lot of raw pointers (indicated by *), which means by using this structure directly, we’re basically writing C, which is no good! A good approach, as we’ll see later, is to write safer wrappers above those C bindings.

Importing C functions is quite straightforward too:

ssize_t  vlc_stream_Peek(stream_t *, const uint8_t **, size_t);
ssize_t  vlc_stream_Read(stream_t *, void *buf, size_t len);
uint64_t vlc_stream_Tell(const stream_t *);

These C function declarations would get translated to:

#[link(name = "vlccore")]
extern {
  pub fn vlc_stream_Peek(stream: *mut stream_t, buf: *mut *const uint8_t, size: size_t) -> ssize_t;
  pub fn vlc_stream_Read(stream: *mut stream_t, buf: *const c_void, size: size_t) -> ssize_t;
  pub fn vlc_stream_Tell(stream: *const stream_t) -> uint64_t;

The #[link(name = "vlccore")] tag indicates to which library we are linking. It is equivalent to passing a -lvlccore argument to the linker. Libvlccore is a library all VLC plugins must link to. Those functions are declared like regular Rust functions, but like the previous structures, will mainly work on raw pointers.


You can always write all your bindings manually like this, but when the amount of code to import is a bit large, it can be a good idea to employ the awesome bindgen tool, that will generate Rust code from C headers.

It can work as a command line tool, but can also work at compile time from a build script. First, add the dependency to your Cargo.toml file:

version = "^0.25"

You can then write your build script like this:

extern crate bindgen;
use std::fs::File;
use std::io::Write;
use std::path::Path;

fn main() {
  let include_arg = concat!("-I", env!("INCLUDE_DIR"));
  let vlc_common_path = concat!(env!("INCLUDE_DIR"), "/vlc_common.h");

  let _ = bindgen::builder()
    .header(concat!(env!("INCLUDE_DIR"), "/vlc_block.h"))
    .raw_line("use ffi::common::vlc_object_t;")

So there’s a lot to unpack here, because bindgen is very flexible:

  • we use clang_arg to pass the include folder path and pre include a header everywhere (vlc_common.h is included pretty puch everywhere in VLC)
  • the header method specifies the header from which we will import definitions
  • hide_type prevents redefinition of elements we already defined (liek the ones from the common header)
  • whitelisted_type and whitelisted_function specify types and functions for which bindgen will create definitions
  • raw_line writes its argument at the top of the file. I apply it to reuse definitions from other files
  • write_to_file writes the whole definition to the specified path

You can apply that process to any C header you must import. With the build script, it can run every time the library is compiled, but be careful, generating a lot of headers can take some time. It might be a good idea to pregenerate them and commit the generated files, and update them from time to time.

It is usually a good idea to separate the imported definitions in another crate with the -sys suffix, and write the safe code in the main crate.
As an example, see the crates openssl and openssl-sys.

Writing safe wrappers

Previously, we imported the C function ssize_t vlc_stream_Read(stream_t *, void *buf, size_t len) as the Rust version pub fn vlc_stream_Read(stream: *mut stream_t, buf: *const c_void, size: size_t) -&gt; ssize_t but kept an unsafe interface. Since we want to use those functions safely, we can now make a better wrapper:

use ffi;

pub fn stream_Read(stream: *mut stream_t, buf: &mut [u8]) -> ssize_t {
  unsafe {
    ffi::vlc_stream_Read(stream, buf.as_mut_ptr() as *mut c_void, buf.len())

Here we replaced the raw pointer to memory and the length with a mutable slice. We still use a raw pointer to the stream_t instance, maybe we can do better:

use ffi;

pub struct Stream(*mut stream_t);

pub fn stream_Read(stream: Stream, buf: &mut [u8]) -> ssize_t {
  unsafe {
    ffi::vlc_stream_Read(stream.0, buf.as_mut_ptr() as *mut c_void, buf.len())

Be careful if you plan to implement Drop for this type: is the Rust code supposed to free that object? Is there some reference counting involved? Here is an example of Drop implementation from the openssl crate:

pub struct SslContextBuilder(*mut ffi::SSL_CTX);

impl Drop for SslContextBuilder {
    fn drop(&mut self) {
        unsafe { ffi::SSL_CTX_free(self.as_ptr()) }

Remember that it’s likely the host application has a lot of infrastructure to keep track of memory, and as a rule, we should reuse the tools it offers for the code at the interface between Rust and C. See the Rust FFI omnibus for more examples of safe wrappers you can write.

Side note: as of now (2017/07/10) custom allocators are still not stable

Exporting Rust code to be called from C

Since the host application is written in C, it might need to call your code. This is quite easy in Rust: you need to write unsafe wrappers.

Here we will use as example the inverted index library for mobile apps I wrote for a conference. In this library, we have an Index type that we want to use from Java. Here is its definition:

pub struct Index {
  pub index: HashMap<String, HashSet<i32>>,

This type has a few method we want to provide:

impl Index {
  pub fn new() -> Index {
    Index {
      index: HashMap::new(),

  pub fn insert(&mut self, id: i32, data: &str) {

  pub fn search_word(&self, word: &str) -> Option<&HashSet<i32>> {

  pub fn search(&self, text: &str) -> HashSet<i32> {

First, we need to write the functions to allocate and deallocate our index. Every use from C will be wrapped in a Box.

pub extern "C" fn index_create() -> *mut Index {

The Box type indicates and owns a heap allocation. When the box is dropped, the underlying data is dropped as well and the memory is freed. The following function takes ownership of its argument, so it is dropped at the end.

pub extern "C" fn index_free(ptr: *mut Index) {
    let _ = unsafe { Box::from_raw(ptr) };

Now that allocation is handled, we can work on a real method. The following method takes an index, and id for a text, and the text itself, as a C string (ie, terminated by a null character).

Since we’re kinda writing C in Rust here, we have to first check if the pointers are null. Then we can transform the C string in a slice. Then we check if it is correctly encoded as UTF-8 before inserting it into our index.

pub extern "C" fn index_insert(index: *mut Index, id: i32, raw_text: *const c_char) {
  unsafe { if index.is_null() || raw_text.is_null() { return } };
  let slice = unsafe { CStr::from_ptr(raw_text).to_bytes() };
  if let Ok(text) = str::from_utf8(slice) {
    (*index).insert(id, text);

Most of the code for those kinds of wrappers is just there to transform between C and Rust types and checking that the arguments coming from C code are correct. Even if we have to trust the host application, we should program defensively at the boundary.

There are other methods we could implement for the index, we’ll leave those as exercise for the reader 🙂

Now, we need to write the C definitions to import those functions and types:

typedef struct Index Index;

Index* index_create();
void   index_free(Index* index);
void   index_insert(Index* index, int32_t id, char const* raw_text);

We defined Index as an opaque type here. Since Rust structures can be compatible with C structures, we could export the real type, but since it only contains a Rust specific type, HashMap, it is better to hide it completely and write accessors and wrappers.

Generating bindings with rusty-cheddar

Writing function imports from C to Rust is tedious, so we have bindgen for this. We also have a great tool to go the other way: rusty-cheddar.

In the same way, it can be used from a build script:

extern crate cheddar;

fn main() {
  cheddar::Cheddar::new().expect("could not read definitions")
  cheddar::Cheddar::new().expect("could not read definitions")
    .module("index").expect("malformed module path")
    .insert_code("#include \"main.h\"")

Here we run rusty-cheddar a first time without specifying the module: it will default to generate a header for the definitions in src/lib.rs.
The second run specifies a different module, and can insert a file inclusion at the top.

It can be a good idea to commit the generated headers, since you will see immediately if you changed the interface in a breaking way.

Integrating with the build system

As you might know, we can make dynamic libraries and executables with rustc and cargo. But often, the host application will have its own build system, and it might disagree with the way cargo builds its projects. So we have multiple strategies:

  • build Rust code separately, store libraries and headers in Maven or something (don’t laugh, I’ve worked with such a system once, and it was actually great)
  • try to let rustc build dynamic libraries from inside the build system. We tried that for VLC and it was not great at all
  • build a static library from inside or outside the build system, include it in the libraries at link. This was done in Rusticata
  • build an object file and let the build system link it. This is what we ended up doing with VLC

Building a static library is as easy as specifying crate-type = ["staticlib"] in your Cargo.toml file. To build an object file, use the command cargo rustc --release -- --emit obj. You can see how we added it to the autotools usage in VLC.

Unfortunately, for this part we still do not have automated ways to fix the issues. Maybe with some time, people will write scripts for autotools,
CMake and others to handle Rust and Cargo.

Side note on reproducible builds: if you want to fix the set of Rust dependencies used in your project and make them always available, you can use cargo-vendor to store them in a specific folder

As you might have guessed, this is the most complex part, for which I have no good generic answer. I’d recommend that you spend the most time on this during the project’s prototyping phase: import very little C code, export very little Rust code, try to make it build entirely from within the host application’s build system. Once this is done, extending the project will get much easier. You really don’t want to discover this task at the end of your project and try to retrofit your code in there.

Going further

While this article just explores the surface of Rust rewrites, I hope it provides a good starting point on the tools and techniques you can apply.
Any rewrite will be a large and complex project, but the result is worth the effort. The code you will write will be stronger, and Rust’s type system will force you to review the assumptions made in the C version. You might even find better ways to write it once you start refactoring your code in a more Rusty way, safely hidden behind your wrappers.

PoC: using LLVM’s profile guided optimization in Rust

call graph

What does profile-guided optimization mean?

Some languages have a JIT (Just In Time) compiler available at runtime, that can optimize the executed code depending on current execution patterns. This is, in large part, the cause of the performance of Lua and the JVM. They can start a bit slow, but by accumulating information on actual running code, they make it faster and faster for the current load. PfLua is a great example: the firewall rules are optimized again and again, until the current network traffic is handled as quickly as possible.

When you use other languages, such as C, you usually cannot optimize the application once it is compiled. Except when you use an optimization technique known as Profile-Guided Optimization. From Wikipedia :

Profile-guided optimization (PGO, sometimes pronounced as pogo), also known as profile-directed feedback (PDF), is a compiler optimization technique in computer programming that uses profiling to improve program runtime performance.

It relies on profiling the compiled application, while it runs with the expected, real world load (web traffic, calculations, etc), and feed this profiling information to the compiler. On the next build, the compiler will have more information on which parts of the program are less used, which branches are taken more often, the expected values in a range, etc. Instead of guessing how the program would behave to choose optimizations, the compiler has true information, and can optimize more precisely. There’s one issue with the process: you need two compilations and a profiling run to generate the final executable. But it gets easier when you automate it, as we can see in the Firefox build process.


While it has been available in other systems for a long time (Visual Studio 2005, the Intel compiler ICC for Itanium), it appeared recently in LLVM.  It has since then been applied successfully to XCode (Objective C, Swift) and LDC, the D compiler.

LLVM has a great feature: it uses an Intermediate Representation code (IR), which is a kind of high level assembly language. It applies its optimizations and machine code generation to that representation. If you make a compiler for a new language, targeting the LLVM IR will give you these features (nearly) for free.

In practice, compiler frontends choose which features they use, so you may not access everything LLVM has to offer. In particular, the Rust compiler, as of now (April 2016), provides a llvm-args option, but that option filters what you can send to LLVM, so we cannot use PGO here.

PGO in Rust

Still, with rustc, you can generate directly the IR, or its binary encoding, named bitcode:

rustc –emit llvm-bc main.rs
# or, with cargo:
cargo rustc — –emit llvm-bc

The approach I tried here is to take that bitcode, and manually apply LLVM’s transformations until I get a compiled executable. This is not really usable for now, especially because I chose an example with very few dependencies. With more dependencies, the compilation and linking will get more complex and unmanageable manually.

LLVM comes with a few commands that you can use to build code manually. The first one is opt, and it applies optimizations and instrumentation on the bitcode file (here, the file target/release/pgo.bc):

opt-3.8 -O2 -pgo-instr-gen -instrprof target/release/pgo.bc -o pgo.bc

The new bitcode file contains code to profile the end application (mainly by counting how often we use each code path). We can now convert that bitcode file to an object file, and link it using clang:

llc-3.8 -O2 -filetype=obj pgo.bc
clang-3.8 -O2 -flto -fprofile-instr-generate pgo.o -L/usr/local/lib/rustlib/x86_64-apple-darwin/lib -lstd-ca1c970e -o pgo

Note: I built my own rustc from source, so your libstd file may not have the same hash. Since Rust (as of April 2016) uses LLVM 3.7, we can use LLVM 3.8’s PGO features, since the bitcode format is apparently backward compatible. I use OS X, and Homebrew’s LLVM 3.8 has compilation issues, so I needed to build the compiler runtime from source. It’s a proof of concept, not production code 😉

We will now run the program we just built, preferably with production data and traffic. It will automatically generate a default.profraw file, containing the profiling information. This file must be transformed to a format that opt will understand with llvm-profdata:

llvm-profdata-3.8 merge -output=pgo.profdata default.profraw

This .profdata file will now be used in the compilation steps:

opt-3.8 -O2 -pgo-instr-use -pgo-test-profile-file=pgo.profdata target/release/pgo.bc -o pgo-opt.bc
llc-3.8 -O2 -filetype=obj pgo-opt.bc
clang-3.8 -O2 -flto -fprofile-instr-use=pgo.profdata pgo-opt.o -L/usr/local/lib/rustlib/x86_64-apple-darwin/lib -lstd-ca1c970e -o pgo-opt

We now have an executable compiled using profiling information. Is it fast?

The benchmarks

The program I tested is a n-body simulation. It was a great test target since libstd is the only dependency, and the load factor depends on a number given as command line argument. Here is a test with time (I know it’s not the most precise benchmarking tool, but for a tenth of second precision, it works alright):

$ time ./target/release/pgo 1000000000

real    1m22.528s
user    1m22.214s
sys     0m0.173s

$ time ./pgo-opt 1000000000

real    1m9.810s
user    1m9.687s
sys     0m0.070s

As it turns out, we gain nearly 15% in running time on this program. Other examples could have less impact, but this is encouraging! So, what happened inside our program?

The generated code

I provide assembly dumps of the normal program, generated with cargo –release, and the one optimized with PGO. Mostly, the code has been reordered, probably to fit better in cache lines. You can also consult PDF files with call graphs: normal, PGO optimized.

The whole code for this article is available here if you want to reproduce the results or tinker with optimizations yourself.

This is a proof of concept, demonstrating that profile guided optimization could work in Rust. It is probably worthy of integration into rustc, but there’s a lot of work before it could be usable. Still, there’s a github issue where you can weigh in, if you would like this optimization in your applications.


Frustrating communication

I’m getting less and less satisfied with Twitter to exchange thoughts. The 140 characters is not the obvious problem, since you can chain messages easily. The issue is that those thoughts are ephemeral. This medium does not optimize for smart discussion with relevant people, but for quick wit from currently available people, before being dumped under a stack of comments on the latest news. The retweeting does not help much, since the primary reason for retweeting are 1. it’s funny 2. it is shocking 3. it’s inspiring, and long last “maybe it’s interesting”. They don’t create much discussion.

Until now, I have primarily used this blog for long posts (thus explaining why I don’t write much here). As my friends say “if it’s more than 3 tweets, write a blog post”.

So in the following months, I’ll try to post short, not well researched but spontaneous articles, instead of ranting in 140 characters.

A world without certificate authorities

love locks
When networks began to expand and people saw the need for secure communication, they designed complex systems based on public key cryptography, that worked more or less. Problem: how do you trust that the key a server sent you is the right one? How can you make sure that it is not somebody else trying to impersonate that website?

Multiple solutions were proposed, and the most promising was a public directory of domain names and associated public keys, maintained by a peer to peer network named KeyCoin. It looked better than so called Web Of Trust solutions, because everybody could agree on what was the correct key for a given domain. As long as nobody hold 51% of the network, no change could happen without being validated by a lot of different peers. The network was maintained by 10000 enthusiast system administrators who took their task very seriously (after all, the security of the whole system depended on their honesty), and nobody had enough computing power to take over the network.

After a while, people began using the system, since it was directly integrated in their browsers, but they did not want to run a node on the network themselves. It was too bothersome, and they could trust the administrators. Also, they had to ask one of them to make a change everytime. The whole process was a bit artisanal.

In the meantime, some people demonstrated the 51% attack on networks of reduced size, and that worried people. They wanted a safe system, one that was not only relying on those sysadmins that could do anything. Who were they anyway? Running that system was still too complex for non technical too run it themselves anyway, so they did not worry enough. But some governments found that rewriting the truth of name/key matching was interesting. Maybe to catch pedophiles, terrorists, criminals. Or maybe to censor websites, I do not know, they told me it was for my own good.

Some smart person found a good solution: if controlling the whole system necessitated owning 51% of the system, the easiest way was to have a lot of machines, enough to counteract the sysadmins. That did not seem risky when people designed the system. Nobody could have enough computing power to take over the whole network, and there would be even more nodes every day.

Yet, that person got enough funding to install tens of thousands of machines and make them join the network. They even provided a nice enough interface for people and businesses to input their domain name and public key, as long as they paid some fees. The sysadmins welcomed him at first, since money coming in the system validated their ideas. Atfer a while, they started worrying, since none of them could keep up with the computing power, but that company asssured them it would never attain 51% of the network.

Other companies jumped on the bandwagon and started to profit from that new business opportunity. Governments started their own server farms to participate too. Problem: now that everybody (except the sysadmins) had a lot of computing power, nobody had enough to control the network entirely.

So they started making alliances. If a few major players work as a team, they can do whatever they want on the network. If one of them decided to try and replace a key on the ledger, others could help it. Of course, once they begun doing that, others wanted to participate. So they created a few rules to join their club. First, you needed to have enough machines. That was a good rule, because that made a big barrier to entry. You could not start as a small player. The other rules? You had to submit to an audit, performed by the other players. Yet another barrier to entry. And once they deemed you acceptable, you had to follow the requests of governments, which were arbitrarily refusing candidates.

Even with the big barriers to entry, a few hundred players came up, often backed by governments. Of course, all ended up in the same team, doing whatever they wanted, as long as nobody was complaining, because anytime one of them had something shady to do, all of them followed automatically.

Since building those big companies required money, they made their clients pay more and more, and to make it easier to accept, provided “premium” options where they show they trust you more, since they took the time to phone your company and ask a few questions.

Some found that big system too centralized, too obedient to states, and decided to fork it. There are separate public ledgers, but they do not come directly embedded in browsers, you need to integrate them yourself, and that’s bothersome. Also, most of those networks have a few hundred nodes at best.

From a nice, decentralized, home made system, we ended up with a centralized system controlled by corporations and governments.

Now let me tell you about that system I designed. It is based on a concept named certificate, a cryptographically signed file that links the public key to a domain name. Now here’s the catch: a certificate represents a key, and is signed by another key, which is represented by another certificate, and so on and so forth until a certificate that signs itself. That system is good, because you just have to embed the root certificate that your friends gives you, and you’ll be able to verify the key of his websites, even if those keys change. And this, without even asking the public ledger, so that is a truly decentralized and more anonymous system! Nothing could go wrong with that, right?