Nix: Superglue for immutable infrastructure

Or, how I rationalize my love of Nix.

Nix markets itself as a “package manager.” I’d say “A principled build tool for everything that actually works.”

Nix assembles your immutable infrastructure from thousands of little lego pieces and sticks them together with superglue. That super glue holds infinitely strong but, at your discretion, it becomes malleable like playdough.

The adhesive might smell strongly academic and nausiate you. Then, you realize that a handful of powerful concepts bind everything together and you become intoxicated.

Nix has been used to create a highly customizable collection of packages, Nixpkgs, that is the core of a complete, modern Linux distribution. Imagine! Driving the build of all these different programs that make up a Linux distribution...with one and only one build tool!

You can build your immutable infrastructure on top of that. With distributed, reproducible builds that can be shared between all of your team.

How does Nix work?

As all build systems, Nix splits the build into smaller build steps. One build step could compile a particular program to a binary. Another build step could create a config file. The final build step might combine those into a docker image.

Simple? Nix calls these build steps “derivations.” Here you go, academic smell, but it really just means “build step” with inputs that executes some code that spits out the output that you so desire.

Hermetic and reproducible lego pieces

Nix is a “hermetic build tool.” That means that it is very explicit and picky about the inputs to your derivations:

1. Nix considers the compiler binary and other used tools as part of the inputs. A lot of build tools ignore this and consider your output “up-to-date” even though you just upgraded your compiler.
2. Nix needs to know the exact hashes¹ of all inputs and uses them to detect changes. If you use files from your local file system in the build, Nix calculates their hash automatically for you. If you download it from elsewhere, you need to specify its hash yourself. All derivations are built in a sandbox to ensure that you don’t use undeclared inputs.

This picky behavior makes reproducible builds possible² and allows Nix to pull off some awesome tricks like:

• Caching the outputs of each build step: As long as your compiler produces the same output with the same inputs, why repeat the build if the inputs didn’t change?
• Distributing the build across multiple machines: If I copy all the inputs to another machine, I can also build it there! I only need to copy the inputs that the other machine doesn’t already have so that I can usually skip copying the compiler.

While your normal programming specific build tool (like npm, yarn, cargo, maven, sbt, leiningen) is usually not hermetic, some progamming language agnostic tools are (e.g. bazel, pants). Here we go, rock solid lego pieces. So, what makes Nix unique?

Nix, the language, glues it all together

Nix build files are written in the Nix programming language. At first, it looks like an innocent little language with data types that are roughly equivalent to JSON but have different names: lists=arrays, attribute sets=object/map, strings, numbers, booleans.

But on top of that, Nix is a pure lazy functional language. What does that mean? It is an exciting and unusual choice for a build system! If you already had some experience with pure lazy functional programming and you know these concepts, good for you. For me, these concepts are mind-blowing.

The “pure” means that:

• evaluating a Nix build file purely returns a description of all build steps (“derivations”) and does not yet execute them (that would be impure). Only after you finalized what you want to build, Nix looks at the graph of build steps and executes them -- in parallel, where it can. In practice, that means that you can easily take an existing build graph and return a slightly modified one without having to fear that you already invoked an expensive build step.³
• debugging a Nix build file is easier because values don’t get changed behind the scenes. Once set, you can rely on them staying the same.

“Lazy” in this context means that like a good lazy programmer, it will avoid doing unnecessary work. It will not evaluate all of your Nix code but only the parts that you explicitly use. E.g. you can use an attribute set (think JSON object/map) with all packages in the NixOs distribution, pass it around as a whole and not fear that your evaluation time explodes. Nix will only fully evaluate the packages that you then actually use in your build file.“Lazyness” allows a number of additional simple but powerful patterns that are beyond this article.

What does that mean in practice?

Nix empowers you to bend the build to your will.

• Do you want to ensure that all of your docker images use the exact same version of openssl? Easy.
• Do you want to use mostly the versions of the libraries in NixOS but you want to patch some with your magic source? You can. Everything that depends on the magic source will be automatically rebuilt.
• Do you want to upgrade a risky new library explicitly and independently for every container? Also possible.
• What about building all your docker images in two versions? One version with the currently deployed RPC library and one with the upgraded one for A/B testing. Not a problem! No need to write a second configuration for every container, since you can easily abstract over all of your images. In fact, you probably don’t even need to touch the individual container configurations at all if you do this.

So, what is the catch?

I am in love with Nix and while I try to stick to facts, I cannot be fully trusted. However, not everything is perfect in the Nix world. Here are three gotchas you will encounter digging into Nix:

1. Some of Nix’s concepts are highly unusual and most people need some time to let them sink in. I did!
2. Nix and its community are not very prescriptive. Sounds good? In a way it is, but documentation often tells you what is possible and not so much if it is a good idea. Discovering how to write “good Nix” was quite hard for me.
3. Nix is most fun if all your dependencies are Nix-ified. NixOS/Nixpkgs is a huge package collection and has often got you covered but if not, you’ll have to work some and maybe some more to bootstrap. Since other build systems are a lot more permissive, some tools in your software stack may use some clever hacks - such as downloading a binary in a build script if you are missing a dependency - that are not allowed in Nix. That said, if it is too hard, you can just build this part outside of Nix and import it as a binary.⁴
4. Nix currently works well if either your particular language is well supported or you rather build your things inside nix with the original build tool -- in one step. The second approach is easier but requires Nix to rebuild everything in that step when things change.

You might have guessed that I think these obstacles are well worth it!

How to get started

You can install Nix under any Linux distribution and Mac OS and then start “Nix-ifying” one little corner of your universe. Download it from the Nix homepage.

How to best proceed from there depends on your learning style! There are a lot of resources on the Nix "learn" page.