Getting proprietary binaries, scripts to your co-workers is so hard! And that is a shame because CLIs are awesome:

  • Composable. It is very easy to write a little wrapper around another CLI that adds some functionality.

  • Easy to write. Much easier than a web app.

  • ...

I should write another article on that!

The problem

You say: "How hard is it really? Can't you do a release on GitLab/GitHub and provide some download links? They are already protected by private authentication."

Yeah, sure. But there are some problems:

  • Version consistency. Everyone might download the binaries they need on demand resulting in a unique versioning footprint on every dev machine. Or are you naive enough that your developers update their kubectl version with plugins and third-party tools relying on it when you announce a new Kubernetes upgrade? There are more fun ways to spend your time than asking again and again: Have you upgraded CLI X yet?

  • Packaging different language runtimes. Yeah, your Go or Rust binary might be somewhat easy to distribute. What about a Python script? A TypeScript CLI? So you need to package the runtime environment for your script as well. But not interfere with the local development setup.

The solution?

nix & the nixpkgs* collection.

Nix is many things but it is also a package manager that works on Linux/Mac OS. It does NOT support Windows directly but my coworkers are fine with working in a WSL2 shell.

  • Awesome direnv integration: Imagine the right set of tools automatically available to you if you enter a project directory! And that also sets up commit hooks etc for you. That is direnv + nix. Have a look at devenv if you want to play around with it quickly!

  • nixpkgs collection: A vast collection of prepackaged software, your favorite programming language tooling is most likely among them.

  • Flake ecosystem: Flakes are a simple way to package nix libraries and binaries outside of the main nixpkgs collection.

  • Very flexible packaging options:

    - Already have some binaries? Great! You can integrate them into your
  package bundle by supplying a download URL and a hash for all of them.

- Bundle up your Python script with a runtime environment? Quite easy.

- ...

  • Works alongside other package managers! Your users can keep using home brew or their native package managers for their personal preferences. Nix will not overwrite the global Python installation just because your script needs it. Even if you make tools available via direnv for a project, they are only in your path when you are in the project folder!

Ecosystem

The Nix ecosystem is thriving. E.g. look at the commits in the nixpkgs repository to have a glimpse at the vibrant open source community! There are also multiple popular communication channels where you can get help and discuss your ideas.

But the ecosystem definitely also profits from contributions of various companies including Big Tech and companies fairly specialized on Nix or builds.

Need commercial support? You are covered.

There are several companies that bet on using Nix as a basis for dev tooling so I am sure that package support will even get better.

The rough edges of nix itself

Flake's status*: You remember Flakes that I mentioned above? Well, technically, this is an experimental feature. They have been around for years, whole ecosystems have been built around it, but officially, they might still change in incompatible ways!

I am obviously not the only one that has noticed this as a big problem and the consensus is that while not officially stable, they have such wide-spread use that keeping them working is a big priority nevertheless.

Simple but exotic language: The Nix ecosystem is built around the nix programming language. That is a simple and mostly well-designed language but it is not mainstream. The combination of being dynamically typed and lazy-evaluated by default makes debugging sometimes quite cumbersome.

Nix allows some amazing freedom but it can be hard to orient at times: E.g. Nix doesn't impose much structure on your code and you need to rely on best practices on which there are diverse opinions.

Using nix in the enterprise

In my company, I have just started the journey and only completed the first steps. Here is an outline of what I did and am about to do. Feel free to chime in and suggest better paths for me!

Streamline the nix installation experience

What do I want?

  • Simple installation instructions with as few steps as possible.

  • Should work for developers who already set up some things and for people that have just started in the company.

  • If you run it again, it should simply update everything to the latest software version and config version.

Therefore, I created nix-bootstrap that takes care of:

  • Installing nix with nix-installer. I love Determinate System's mission to get rid of annoying papercuts like e.g. broken installs after Mac OS upgrades.

  • Deterministic versions of nix-installer, nix, and nom. Rather than just always installing the latest ones.

  • Working shell completions for nix commands and other software installed in the user profile. This will hopefully be fixed in the installer script of nix that nix-installer also uses. Meanwhile I included a fix in nix-bootstrap.

  • Installing direnv. direnv is unfortunately out of scope for nix-installer, so I also install it as part of the bootstrap nix profile that I install system-wide and for the installing user.

  • Setting up credentials for private GitLab repositories.

All triggered with a simple shell command:

bash -c "$(curl https://nexxiot-labs.gitlab.io/nix-bootstrap/install.sh)"

I am happy to collaborate with others on improving this or upstreaming fixes to nix/nix-installer.

CI support for nix in GitLab

Generally, the main nix projects are all on GitHub and, therefore, GitHub is better supported by the ecosystem. That said, using GitLab CI with nix is not super difficult.

  • Creating a CI container. Nicely minimal but still everything that is needed. E.g. GitLab CI expects a grep before I even get a chance to execute a nix command to install it. nixpkgs/nix-flakes is nicely minimal and does not include it. I used the new cool kid on the block nix2container but it has a problem that gets only triggered if you build the container itself within that container.

  • Allowing CI access to a nix substituter (= cache). Allowing read/write access to an S3 bucket was easy in our setup but it is much harder to share that access with users -- see below.

  • Allowing nix in GitLab CI to fetch binary artifacts from other private repos. Absolutely not straight-forward to inject the right credentials into the fetch build step, especially since GitLab does not support Basic Auth in its API. Please say I am wrong and there is an easy way!

  • Mac OS binaries. At least for our internal CLI tools, we'd want Mac OS support but we don't have CI runners for Mac OS and cross-compiling is tricky and maybe not even reliably possible. Maybe I'll just test & push them manually into the cache on releases.

Local install support

If users install an unchanged flake, read-access to a binary cache (=substituter in nix lingo) should be sufficient. If that is prefilled by CI and/or manual steps as needed -- all cache requests should be hits.

Unfortunately, we only allow temporary AWS session credentials for our users. Passing those to the nix-daemon is not really possible.

Using Cachix instead would be a serious contender with fair prices. Unfortunately, that would require additional onboarding / offboarding steps for both our IT and the developers itself. Maybe that can be solved with SSO support in the future!

What is more, all systems we interact with, need to be approved by our Internal IT reviews. For AWS S3, since we are already using it, this is not necessary.

So here I am still wondering what the best path forward is!

Local build support

Obviously, local builds should also work when working on internal tool release. So we need to solve the authentication problem to the GitLab API (for fetching binary artifacts) not only in CI but also for every developer.

Maybe I can build sort of a "doctor" script that takes care of ensuring that all necessary secrets are set up and that the nix-daemon can access them?

DevContainer/GitPod support

What about allowing our developers to dabble with nix tools without even installing nix on their computer?

Devcontainer support in Visual Code is quite awesome. A lot of our developers use Intellij -- WSL2 remote support seems to be better than devcontainer support in my testing, so probably not helpful for those users.

Complementarily, GitPod would be a super nice test bed that could be easily used from our GitLab repositories.

Authentication within a devcontainer hopefully just works similarly to doing it locally outside of the container -- with the right volume mounts. But it might be also a big pain!

For GitPod, it is likely a completely different flow.

Cache Pruning

Just expiring files after 30 days or so doesn't work with nix since it breaks some invariants: All dependencies of a derivation are also assumed to be available via the cache (I think).

Therefore, implementing a nice expiry policy is more involved. E.g. maintaining a list of store paths that are still relevant for recent builds and pruning everything else.

A journey...

Puuuh... That is quite a journey ahead of me! I believe it can unlock a lot of productivity by easy tool distribution in my company but it doesn't look that easy!

Thanks for reading until here.

I am thankful for any pointers to make this smoother. Feel free to get in touch with me with the information shared on my GitHub profile: kolloch