Lately I’ve become quite a fan of Rust, a modern programming language with a focus on memory safety. I’m using it in different projects, mostly for it’s great speed and low resource usage compared to e.g. Python or Node.

In contrast to scripting languages, all Rust applications have to be compiled before they can be executed. This is a process that takes some time, depending on the size of the app and the number of dependencies.

During development, this isn’t an issue thanks to local caching and incremental compilation, which allows the compiler to only re-compile what has changed since the last run.

For my web applications, I use Kubernetes and Docker images for deployment, which means I have to build my applications using Docker as well. In this case, these tricks won’t help us.

Naive solution

The following example Dockerfile shows how to build an Rust application using Docker. We’re using a multi-stage build here, where the first stage uses an base image that contains the Rust compiler, while the second stage uses a slim Debian image which contains the necessary runtime libraries (this service uses PostgreSQL). This approach greatly reduces the final size of the image.

This example is based on the instructions of the Docker Rust base image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# ------------------------------------
# Stage 1: Build the application

FROM rust:latest AS builder

RUN apt-get update && apt-get install --yes libpq-dev

WORKDIR /app

COPY . .

RUN cargo build --release

# ------------------------------------
# Stage 2: Build the final container
#

FROM debian:buster-slim
RUN apt-get update && apt-get install -y libpq-dev && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/my-app /usr/local/bin/my-app
CMD ["my-app"]

This approach is easy, but has a great disadvantage: It’s really slow.

While Docker has some built-in layer caching, we can’t use it here. Even tricks that are used in other language platforms - like copying the dependency lockfile first and the rest of the app laster - cannot help us here.

Fortunately, developers at Mozilla found a lovely solution to share the build cache between systems, even if not using Docker: sccache.

sccache is a ccache-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on local disk or in one of several cloud storage backends.

Using sccache

sccache allows different storage options, including the local disk, S3-compatible object storage and caching systems like Redis or Memcached. In my case, I chose an S3-based storage (MinIO), as it’s working well in externally running systems like CI runners.

As first step, we need to get sccache into the build container. Fortunately the maintainers provide pre-built binaries, that can be easily downloaded and used:

💡 Please check the latest version on sccache releases page instead of just copy-pasting this code, which might be already outdated when you’re reading it.

1
2
3
4
5
6
7
8
# Note that we add wget here
RUN apt-get update && apt-get install --yes libpq-dev wget

# Install sccache to greatly speedup builds in the CI
RUN wget https://github.com/mozilla/sccache/releases/download/v0.2.15/sccache-v0.2.15-x86_64-unknown-linux-musl.tar.gz \
    && tar xzf sccache-v0.2.15-x86_64-unknown-linux-musl.tar.gz \
    && mv sccache-v0.2.15-x86_64-unknown-linux-musl/sccache /usr/local/bin/sccache \
    && chmod +x /usr/local/bin/sccache

Additionally, we need to set some environment variables to configure sccache. Put these directly after the previous code block, the variables must be set before invoking cargo build.

💡 These are set inside the build container only, and won’t leak into the final image.

1
2
3
4
5
6
7

ENV RUSTC_WRAPPER=/usr/local/bin/sccache
ENV AWS_ACCESS_KEY_ID=changeme
ENV AWS_SECRET_ACCESS_KEY=changeme
ENV SCCACHE_BUCKET=sccache
ENV SCCACHE_ENDPOINT=cdn.example.org
ENV SCCACHE_S3_USE_SSL=true

For further details on configuring sccache, please check out their README.txt.

Securing up the storage

For security reasons, I recommend creating an own IAM user and limit it’s permissions to the bucket only. On my MinIO instance, I’ve used the following policy to only grant limited access:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::sccache",
                "arn:aws:s3:::sccache/*"
            ]
        }
    ]
}

Additionally, you might want to define a retention policy to auto-delete old files, else you’ll slowly build up a pile of unused data.

Final notes

While the first build might take slightly longer (because of the additional I/O to upload the cache files), all further compilations should be considerable faster.