Private stores? Temporary credentials? At the time of writing this, definitely not easy! (Please, please prove me wrong!)
At the end of this article I'll make a pragmatic proposal that should be quite easy to implement and improve the user experience significantly.
Stores: What's possible today?
"Stores" store build results and allow you to share them between machines.
You can copy packages (store paths) with their transitive dependencies (closure)
between machines explicitly, e.g. with
nix copy or
You can also use them as a cache (substituter) during builds: For suitable build steps (derivations), Nix will check if the result has already been build and is available for download.
How can we authenticate towards private stores? That depends on the store type:
SSH. Well with the usual SSH mechanisms.
HTTPS. You can use
.netrcto provide basic auth credentials.
AWS S3. Uses the AWS SDK's
Aws::Auth::AWSCredentialsProviderwhich e.g. search for credentials in environment variables and configuration files.
How do you provide these credentials? It depends on the context in which the store is used.
nix copy: The credentials e.g. for AWS are taken from the invoking user or the environment in which nix runs.
- Substitution with single-user setup without daemon: Same.
- Substitution but with nix-daemon: Credentials are resolved in the environment of the nix-daemon or from the config files of the nix-daemon user.
That is already super confusing. You have entered an AWS session with temporary credentials available in environment variables:
nix copyjust works.
nix buildcannot use the substituter when it uses the nix-daemon.
Cachix: State of the art for HTTPS stores
Cachix uses a CLI to make this as convenient as possible:
- It assumes you are a trusted user that can change nix-daemon settings on the fly.
- It updates your
nix.confwith a potential private cache and a
nix copytakes this config directly and works.
nix buildpasses this config to the nix-daemon that can hopefully also read your net-rc file.
Passing the config to the nix-daemon is implemented via daemon op
which changes the
Settings globally. That is fine, because the nix-daemon is
forked for every connection.
What are the problems?
- Need to patch your
nix-config. This is potentially not that bad!
- Remote builders need additional server-side config. The config options might
actually even be transfered with the
SetOptionscommand, not sure, but your net-rc file is likely unavailable on the remote builder. Might be OK if you don't use them or someone else sets them up for you!
- Does not work well with temporary credentials. Might be a showstopper in some environments.
- Only works for HTTP(S) with Basic Authenticationstores. Yeah, well, if you use something else... Doesn't work with AWS S3...
AWS S3: State of the art solution
If you have a single user nix setup, it is rather easy. Is your user authenticated and authorized to access the S3 store? So is nix running as that user.
It becomes more annoying if you use a nix-daemon: You also need to provide the credentials to the nix-daemon. If you use
Pragmatic User-Friendly Solution
I assume that your trust your nix-daemon or remote builder. I think that's fair and the assumption behind the approach that cachix uses.
netrc works pretty decently for Cachix, it doesn't match AWS well since
it doesn't use Basic Authentication. Remember, the complicated logic is already
all inside of the nix-daemon via the AWS SDK, we just need to pass a couple of
values to the nix-daemon.
I see two major approaches:
Custom logic for AWS: Either extend
SetOptionsor create a new operation to transfer AWS credentials and store them in special-purpose fields. Modify the credential provider chain to also look there for credentials.
A general way to transfer environment variables from the nix client to the daemon for the current session.
Details for passing environment variables to the daemon
I like the second one better because it would also solve the problem if we integrated solutions for other cloud providers. And it might even be an acceptable workaround for fetches from private sources in builds (more about that later).
I'd restrict this features with some explicit config options.
In the nix-daemon:
allowed-env-variables-from-trusted-clientas a list of environment variables that may be set from a trusted client. E.g.:
allowed-env-variables-from-trusted-client = AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION
For the nix client:
pass-env-variablesas a list of environment variables that are passed whenever a connection to the nix-daemon is opened. E.g.:
pass-env-variables = AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION
If a client has valid AWS environment variables, this will make it just work, even with temporary credentials.
nix buildwill just both work after adding a few standard config options.
The config can remain static while the credentials can change.
It might even work with remote builds: at least if the
SetOptions op is also
This requires the client to have a valid AWS session whenever you might access the AWS S3 store. It cannot prompt for e.g. your MFA code on demand.
If the nix-daemon implementation ever changes and does not fork on every new connection, then changing the global environment variables of the process is also a nono. Note though, that a lot of the implementation would need to change in that case.
I can definitely think of other solutions here but this seems to have the right mix of "good enough", "not too specialized" and "easy to implement."
I might be wrong, though, and would love to hear your thoughts!
If you want to know more about my current journey and context, check out Nix: Distributing private/public binaries.
Feel free to get in touch with me with the information shared on my GitHub profile: kolloch