Shell Scripts with Nix

1. Purpose

The goal of this page is to explore and document “best practises” for writing shell scripts and packaging them for the Nix package manager.

  • This article describes software which is both: unstable, and something I’m still learning about. As a consequence, this advice may change over time.

2. Introduction

One of my favourite things about Nix is how easy it is to transform my one-off shell scripts into halfway-usable “system applications.”

I used to maintain a ~/bin/ directory, full of poorly maintained scripts. Each one a reminder of some repetitive task I was trying to conquer and whatever scripting language had my attention at the time. A smattering of shell, ruby, and python scripts; and for some reason, a few tiny main.c apps?

I’ll flatter myself by saying that most of these scripts have no purpose left aside from being museum exhibits; however, some are still useful from time to time. For example, there’s that one bash script that renames all the .raw files from the camera to the naming convention I use. And there’s that one ruby script that knows to generate the complex “filter” DSL1 that ffmpeg uses.

The problem I run into with these old and rarely used scripts is that, when I do need to use them, I can never remember how to actually use them. Shell scripts almost always execute other applications. Are those even installed on my PC anymore? Am I even using the same computer or Linux distro I used when I originally wrote this script? Probably not. Typically I wind up burning a half-hour re-reading the script and searching online for what packages I need from my distro’s package manager.

A secondary difficulty I run into with these one-off shell scripts is actually being able to access them and execute them. This is easy enough when you’re manually executing a script from the terminal, but it’s not always straight forward when I want some other kind of app to be able to use my script. Consider:

  • I often want Firefox, or other GUI apps, to recommend my scripts when opening a file with a particular MIME type.
  • Many Linux daemons run under their own users. It can be a challenge to find a way to get my script onto their $PATH.

For all intents, these one-off shell scripts are the same as the apps I installed from my package manager. When I install proper apps from the net, my package manager ensures that all their dependencies are also installed and that all their files are installed where they need to be; moreover, it’s a cinch to remove packages when I no longer need them. I want all this for the scripts that I write myself! I am the user, after all. Shouldn’t I get the best treatment? 😇

This is something I’ve been battling with for many years. I’ve experimented with RedHat’s RPMs and ArchLinux’s PKGBUILD system. I don’t want to say either system is bad, but they’re weren’t nearly convenient enough for someone as lazy as I am. The title of this article is a bit of a spoiler, but I’ve found the best results with Nix.

2.1. A warning about Nix

“Nix” is a package manager2. It’s not be confused with “NixOS:” a whole Linux distro designed around the Nix package manager. NixOS aside, Nix is meant to be three cases, it’s worked great. For the last year, I’ve had about a dozen PCs, distro-agnostic. So far, I’ve used it with Fedora, MacOS, and NixOS. In all laptops, and servers that rely on it and it’s been a relatively enjoyable experience.

That being said, the majority of the Nix features that I rely on are available only on the “unstable3” branch. Nix is already very young software, and I would only recommend switching to its unstable branch if you really, really enjoy tinkering with software. Nix’s unstable branch is very unstable. You’ve been warned!

To be more specific: this article relies heavily on Nix Flakes. As of August 2021, flakes are only available through the unstable branch.

3. A simple example

There is one Nix function that I reach for time I want to write a shell script is writeShellScriptBin4, provided by nixpkgs. This function takes two arguments: the filename for your script and its contents.

3.1. Example script

For this example, we’re going to create a very basic shell script. This one:

1: DATE=$(ddate +'the %e of %B%, %Y')
2: cowsay Hello, world! Today is $DATE.

This very-useful shell script has two dependencies:

ddate
“prints the date in Discordian date format.”
cowsay
“a program that generates ASCII art pictures of a cow with a message.”

With my original ~/bin/ version of this script, I just have to remember that I need these apps installed for my script to do anything.

The following snippet shows a simple Nix Flake that defines the “package” (known as “derivations” in Nix) for our shell script.

 1: {
 2:   description = "A simple script";
 3: 
 4:   outputs = { self, nixpkgs }: {
 5:     defaultPackage.x86_64-linux = self.packages.x86_64-linux.my-script;
 6: 
 7:     packages.x86_64-linux.my-script =
 8:       let
 9:         pkgs = import nixpkgs { system = "x86_64-linux"; };
10:       in
11:       pkgs.writeShellScriptBin "my-script" ''
12:         DATE="$(${pkgs.ddate}/bin/ddate +'the %e of %B%, %Y')"
13:         ${pkgs.cowsay}/bin/cowsay Hello, world! Today is $DATE.
14:       '';
15:   };
16: }

3.1.1. Test out your simple script

We’ll go over this example in a moment, but before that, you’ll probably want to test out your script, to make sure it works before you build it into your system. There are various ways to do this:

# Let's get our bearings first:
$ pwd
~/code
$ ls ./simple-script/
flake.nix

# Execute the "default package" of ./simple-script/flake.nix
$ nix run ./simple-script#
# or...
$ cd ./simple-script && nix run .#

# Execute a named package of ./simple-script/flake.nix
$ nix run ./simple-script#my-script
# or...
$ cd ./simple-script && nix run .#my-script

Regardless of how you ran the script, you should eventually see output like:

 ____________________________________ 
/ Hello, world! Today is the 58th of \
\ Bureaucracy 3187.                  /
 ------------------------------------ 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

3.1.2. Breakdown

5: defaultPackage.x86_64-linux = self.packages.x86_64-linux.my-script;

On line 5, we specify that this derivation has a default app, named my-script. We haven’t yet said what my-script is, but when we do, you better believe that it’ll be the default!

The default package is optional, but when present it identifies the derivation’s “main app.” Packages can come with only a single executable or a suite of them, but even with many, typically there is a single flagship app that defines the package. Consider ffmpeg. The titular ffmpeg is clearly the primary executable, but most package managers also include the utility apps, ffplay and ffprobe.

Not every derivation will have a default package, but it’s more convenient to use when available.

7: packages.x86_64-linux.my-script = { ... };

At line 7, we begin to define the my-script derivation. This was previous referenced on line 5, where we declared that it’ll be our default package. Our package will only include 1 script, but you can define a bunch here.

The name used here, my-script is arbitrary; it won’t be used as the name of the executable script that will eventually be installed on our system. It’s more of a variable name, or the key in a dictionary5. We only use it to refer to this derivation within our Nix code.

9: pkgs = import nixpkgs { system = "x86_64-linux"; };

Here we’re importing the entire upstream “nixpkgs” repository and storing the entire set in the pkgs variable. We can later reference upstream derivations with pkgs.foo-bar. There are all the derivations managed by the NixOS maintainers and community6.

  • You’ve probably noticed x86_64-linux both on this line and also lines 5 and 7. This tells Nix that our script is suitable only for 64-bit Linux machines. This is only to keep our examples simple. Shell scripts are typically pretty portable, so we’ll soon go over how to provide this script on more systems shortly.
11: pkgs.writeShellScriptBin "my-script" ''
12:   DATE="$(${pkgs.ddate}/bin/ddate +'the %e of %B%, %Y')"
13:   ${pkgs.cowsay}/bin/cowsay Hello, world! Today is $DATE.
14: '';

On line 11, we actually define packages.x86_64-linux.my-script. ie: what actually gets installed on the system. This is easy for us, as we farm out this work to writeShellScriptBin4, provided by nixpkgs. This function takes two arguments: the filename for your script and its contents. It then takes care of the packaging for us! It will also run a linter across our script, to help us avoid installing busted trash on our system.

You’ll have noticed that our two-line script has changed slightly. For instance, the reference to cowsay, on line 13, now looks like "${pkgs.cowsay}/bin/cowsay". When Nix builds our script, ${pkgs.cowsay} will be replaced with the absolute path of the cowsay package’s root directory, so we’re referencing the cowsay executable by it’s absolute path: no need to worry about PATH at all here!

4. A more “pure” version

So far, our script is working great and we even have our dependencies being installed automatically for us. That’s pretty rad, but there are a few problems with this approach, with one being downright unacceptable: we have to rewrite our, just for NixOS! That’s really bad, and will make future maintenance really difficult. If we want to run this script on multiple OSes, or even multiple Linux distros, we’re going to need to maintain parallel branches. No way!

We definitely need a way to package our script that let’s us use the original script as-is. So let’s do that. Fortunately for us, NixOS provides a couple handy utilities to make this possible:

4.1. The updated script

Here is the entire updated package for our script. Skip over it and we’ll discuss the bits that have changed:

 1: {
 2:   description = "A better version";
 3: 
 4:   outputs = { self, nixpkgs }: {
 5:     defaultPackage.x86_64-linux = self.packages.x86_64-linux.my-script;
 6: 
 7:     packages.x86_64-linux.my-script =
 8:       let
 9:         pkgs = import nixpkgs { system = "x86_64-linux"; };
10: 
11:         my-name = "my-script";
12:         my-script = pkgs.writeShellScriptBin my-name ''
13:           DATE=$(ddate +'the %e of %B%, %Y')
14:           cowsay Hello, world! Today is $DATE.
15:         '';
16:         my-buildInputs = with pkgs; [ cowsay ddate ];
17:       in pkgs.symlinkJoin {
18:         name = my-name;
19:         paths = [ my-script ] ++ my-buildInputs;
20:         buildInputs = [ pkgs.makeWrapper ];
21:         postBuild = "wrapProgram $out/bin/${my-name} --prefix PATH : $out/bin";
22:       };
23:   };
24: }

4.1.1. Breakdown

11: my-name = "my-script";
12: my-script = pkgs.writeShellScriptBin my-name ''
13:   DATE=$(ddate +'the %e of %B%, %Y')
14:   cowsay Hello, world! Today is $DATE.
15: '';

On lines 11-15, we create a derivation for our script7. This version is exactly the same as the very first one we created back in 3.1. This isn’t the end of the story though, as we once again have lost track of what our dependencies (cowsay and ddate) are! This version of the script assumes these apps will be available on the PATH.

16: my-buildInputs = with pkgs; [ cowsay ddate ];

On line 16, we create a variable to store the list of dependencies our script has. Now we just need to deploy our script in such a way that these are available in its PATH at runtime.

17: pkgs.symlinkJoin {
18:   name = my-name;
19:   paths = [ my-script ] ++ my-buildInputs;
20:   buildInputs = [ pkgs.makeWrapper ];
21:   postBuild = "wrapProgram $out/bin/${my-name} --prefix PATH : $out/bin";
22: };

symlinkJoin is where the magic happens. This Nix function creates a derivation that combines the files of several different packages into a single package. You can see on line 19, we provide a list of all the packages to join together (our script, plus our two dependencies). This will ensure that all of our dependencies are installed when our script is installed, but we still need to arrange things so these apps are on the PATH at runtime.

The NixOS package makeWrapper provides a very handy shell script, wrapProgram, that will let us specify our script’s PATH. On line 20 we let Nix know that makeWrapper is needed to build this package, and line 21 instructs wrapProgram to append $out/bin to our script’s PATH. $out/bin is the directory where our shell script is installed to and, thanks to symlinkJoin, also where cowsay and ddate live!

4.1.2. Build Results

Build this puppy and let’s have a look at what we’ve actually built here.

$ nix build
$ ls -ogA result/bin

total 16
lrwxrwxrwx 1  66 Dec 31  1969 cowsay -> /nix/store/qznd0pqdw925g7vz0iaynbszlvypdcvq-cowsay-3.04/bin/cowsay
lrwxrwxrwx 1  65 Dec 31  1969 ddate -> /nix/store/g9prmy4a60khk5021w2awrzzfr1fkpgq-ddate-0.2.2/bin/ddate
-r-xr-xr-x 1 259 Dec 31  1969 my-script
lrwxrwxrwx 1  67 Dec 31  1969 .my-script-wrapped -> /nix/store/x6j48dklxm9d7cjwga5arzpcq6n2w461-my-script/bin/my-script

cowsay and ddate are here as promised, but note that we also have two versions of my-script. These two files are the result of wrapProgram:

.my-script-wrapped
This hidden file is a symlink to the script we created on lines 12-15 above (using writeShellScriptBin). If you cat it, you’ll see it’s our shell script, verbatim.
my-script
This is the script users will actually call. If you cat this file, you’ll see a very small shell script that updates the PATH, as we’ve instructed, then replaces itself (exec) with .my-script-wrapped. Magic!

5. The best version (so far)

In section 4, we covered all the important topics in this article, but there are a couple of extra, optional changes we can make to our package to really make it sing. They’re a bit more complex though, so you may want to just skip `em.

  • Our script is currently written inline, in flake.nix itself. That’s not ideal.
  • Our package is currently only available to x86_64-linux OSes. Let’s make it available all over.

5.1. Move script to its own file

Our example script is pretty small, so in section 4, we wrote the whole script inline, on lines 12-15. Most scripts are more than 2 lines long though, so it would be obnoxious to have to write the entire thing this way. Moreover, your editor can do decent syntax highlighting on scripts embedded in a .nix file. More-moreover-still, you might want to provide other ways to build your script and provide a flake.nix for Nix-systems and maybe a PKGBUILD file for Arch-systems. There are many reasons not couple your script and its build-file in the same file, so take your pick.

5.1.1. Extract the source script

First thing’s first: Let’s extract our script to its own file:

#!/usr/bin/env bash
DATE=$(ddate +'the %e of %B%, %Y')
cowsay Hello, world! Today is $DATE.

This is the same script from the beginning of this article… except we’ve added in a shebang on the first line. Now that we’ve specified the script’s interpreter, it’s a proper shell script in its own right!

The reason why we’re adding the shebang at this point is to fully decouple the script from our Nix package. You can run it directly (assuming cowsay and ddate are available on your system) or package it for another OS.

5.1.2. Update flake.nix

The only part of our package we need to change is our my-script variable. The original (inline) definition of this variable is:

12: my-script = pkgs.writeShellScriptBin my-name ''
13:   DATE=$(ddate +'the %e of %B%, %Y')
14:   cowsay Hello, world! Today is $DATE.
15: '';

Our new definition of this variable is:

12: my-src = builtins.readFile ./simple-script.sh;
13: my-script = (pkgs.writeScriptBin my-name my-src).overrideAttrs(old: {
14:   buildCommand = "${old.buildCommand}\n patchShebangs $out";
15: });

We’ve got three changes here:

  1. Instead of our script being inline, we read it from ./simple-script.sh.
  2. We use writeScriptBin instead of writeShellScriptBin. writeShellScriptBin prepends a shebang line, but our script already has one.
  3. We update our package’s buildCommand by appending patchShebangs .. This command will update our script’s interpreter to point to the correct binary on the system. ie: #!/usr/bin/env bash will be replaced with something like #!/nix/store/1flh34xxg9q3fpc88xyp2qynxpkfg8py-bash-4.4-p23/bin/bash.

5.2. Provide our script to every system

So far, every flake.nix package we’ve written in this article has been explicitly for x86_64-linux systems. ie: not 32 bit machines, MacOS, BSD, or a number of other architectures and OSes. Some apps can only work on particular systems, and some apps require special build instructions for particular machines; however, this probably isn’t the case for a simple shell script. Why not package it to work everywhere possible?

Fortunately for us, the solution to this is very simple: use flake-utils. These utilities solve this exact problem, and our script has no special requirements, so just follow its guide. In particular, check out the eachDefaultSystem example.

5.3. The final version of our flake.nix

Here’s the final version of our flake.nix package, which reads our script from ./simple-script.sh, and provides packages for a variety of machine types:

 1: {
 2:   description = "A best script!";
 3: 
 4:   inputs.flake-utils.url = "github:numtide/flake-utils";
 5: 
 6:   outputs = { self, nixpkgs, flake-utils }:
 7:     flake-utils.lib.eachDefaultSystem (system:
 8:       let
 9:         pkgs = import nixpkgs { inherit system; };
10:         my-name = "my-script";
11:         my-buildInputs = with pkgs; [ cowsay ddate ];
12:         my-script = (pkgs.writeScriptBin my-name (builtins.readFile ./simple-script.sh)).overrideAttrs(old: {
13:           buildCommand = "${old.buildCommand}\n patchShebangs $out";
14:         });
15:       in
16:         rec {
17:           defaultPackage = packages.my-script;
18:           packages.my-script = pkgs.symlinkJoin {
19:             name = my-name;
20:             paths = [ my-script ] ++ my-buildInputs;
21:             buildInputs = [ pkgs.makeWrapper ];
22:             postBuild = "wrapProgram $out/bin/${my-name} --prefix PATH : $out/bin";
23:           };
24:         }
25:     );
26: }

Wasn’t that easy!

Footnotes:

2

“Nix” is also the name for the programming language used to write Nix packages. These homonyms can be confusing at times. You can read more about the language at the wiki (Nix Expression Language) and in the Nix manual (Writing Nix Expressions).

3

See the wiki article on Nix Channels and the unstable branch on GitHub.

5

Known as “attribute sets” in Nix jargon. You can review some convenient library functions here: lib/attrsets.nix.

7

We also stash the name of the script in my-name, since we’ll need to use that in a few places.