Private stores? Temporary credentials? At the time of writing this, definitely not easy! (Please, please prove me wrong!)

At the end of this article I'll make a pragmatic proposal that should be quite easy to implement and improve the user experience significantly.

Stores: What's possible today?

"Stores" store build results and allow you to share them between machines.

You can copy packages (store paths) with their transitive dependencies (closure) between machines explicitly, e.g. with nix copy or nix-copy-closure.

You can also use them as a cache (substituter) during builds: For suitable build steps (derivations), Nix will check if the result has already been build and is available for download.

How can we authenticate towards private stores? That depends on the store type:

How do you provide these credentials? It depends on the context in which the store is used.

  • nix copy: The credentials e.g. for AWS are taken from the invoking user or the environment in which nix runs.
  • Substitution with single-user setup without daemon: Same.
  • Substitution but with nix-daemon: Credentials are resolved in the environment of the nix-daemon or from the config files of the nix-daemon user.

That is already super confusing. You have entered an AWS session with temporary credentials available in environment variables:

  • nix copy just works.
  • nix build cannot use the substituter when it uses the nix-daemon.

Cachix: State of the art for HTTPS stores

Cachix uses a CLI to make this as convenient as possible:

  1. It assumes you are a trusted user that can change nix-daemon settings on the fly.
  2. It updates your nix.conf with a potential private cache and a net-rc file setting.
  3. nix copy takes this config directly and works.
  4. nix build passes this config to the nix-daemon that can hopefully also read your net-rc file.

Passing the config to the nix-daemon is implemented via daemon op SetOptions which changes the Settings globally. That is fine, because the nix-daemon is forked for every connection.

What are the problems?

  • Need to patch your nix-config. This is potentially not that bad!
  • Remote builders need additional server-side config. The config options might actually even be transfered with the SetOptions command, not sure, but your net-rc file is likely unavailable on the remote builder. Might be OK if you don't use them or someone else sets them up for you!
  • Does not work well with temporary credentials. Might be a showstopper in some environments.
  • Only works for HTTP(S) with Basic Authenticationstores. Yeah, well, if you use something else... Doesn't work with AWS S3...

AWS S3: State of the art solution

If you have a single user nix setup, it is rather easy. Is your user authenticated and authorized to access the S3 store? So is nix running as that user.

It becomes more annoying if you use a nix-daemon: You also need to provide the credentials to the nix-daemon. If you use

Pragmatic User-Friendly Solution

I assume that your trust your nix-daemon or remote builder. I think that's fair and the assumption behind the approach that cachix uses.

While netrc works pretty decently for Cachix, it doesn't match AWS well since it doesn't use Basic Authentication. Remember, the complicated logic is already all inside of the nix-daemon via the AWS SDK, we just need to pass a couple of values to the nix-daemon.

I see two major approaches:

  1. Custom logic for AWS: Either extend SetOptions or create a new operation to transfer AWS credentials and store them in special-purpose fields. Modify the credential provider chain to also look there for credentials.

  2. A general way to transfer environment variables from the nix client to the daemon for the current session.

Details for passing environment variables to the daemon

I like the second one better because it would also solve the problem if we integrated solutions for other cloud providers. And it might even be an acceptable workaround for fetches from private sources in builds (more about that later).

I'd restrict this features with some explicit config options.

  1. In the nix-daemon: allowed-env-variables-from-trusted-client as a list of environment variables that may be set from a trusted client. E.g.:

    allowed-env-variables-from-trusted-client = AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION

  2. For the nix client: pass-env-variables as a list of environment variables that are passed whenever a connection to the nix-daemon is opened. E.g.:

    pass-env-variables = AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION

If a client has valid AWS environment variables, this will make it just work, even with temporary credentials.

  • nix copy / nix build will just both work after adding a few standard config options.

  • The config can remain static while the credentials can change.

Yey!

It might even work with remote builds: at least if the SetOptions op is also used there.

Restrictions:

  1. This requires the client to have a valid AWS session whenever you might access the AWS S3 store. It cannot prompt for e.g. your MFA code on demand.

  2. If the nix-daemon implementation ever changes and does not fork on every new connection, then changing the global environment variables of the process is also a nono. Note though, that a lot of the implementation would need to change in that case.

Conclusion

I can definitely think of other solutions here but this seems to have the right mix of "good enough", "not too specialized" and "easy to implement."

I might be wrong, though, and would love to hear your thoughts!

If you want to know more about my current journey and context, check out Nix: Distributing private/public binaries.

Feel free to get in touch with me with the information shared on my GitHub profile: kolloch