Patching nixpkgs

1. Introduction

Every Nix flake implicitly gets nixpkgs as an input, but what if it contains a show-stopping bug? It may have been fixed upstream, but these fixes can sometimes take days or weeks to even reach the nixos-unstable branch.

In this article, we’ll describe way one easy way to create a nix.flake that patches nixpkgs.

When you use unstable open-source code, you get what you paid for. Well, I suppose you paid for nothing, but what you’re getting can be unstable. When you’re on the bleeding-edge, you get first-swing at all the newest fun code, but things are going to break. There’s no real way around it. You can pack-up and head back to the stable branch, but let’s barrel-forward instead!

2. When should you try this?

In this article, I’ll use, as an example, a specific bug I ran into yesterday. That being said, you should be able to follow this strategy to bypass any number of bugs you encounter in nixpkgs. In short, give this a spin if the following holds true:

  • You’re experiencing a bug, and suspect it to be from nixpkgs.
  • Someone’s submitted a GitHub pull-request that (probably) fixes this bug.
  • This pull-request may or may-not have been accepted yet, but either way it hasn’t been merged into the branch your nixpkgs is sourced from. (nixos-unstable, in my case).
  • You don’t want to wait!

3. An example bug

Yesterday I was writing a flake.nix for a new project, and as soon as I ran nix develop for the first time, I was hit with this really peculiar error (don’t bother read it):

error: [json.exception.parse_error.101] parse error at line 57, column 178:
syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last
read: '" \n    local fn=\"$1\";\n    local fd;\n    local magic;\n    exec
{fd}< \"$fn\";\n    read -r -n 4 -u \"$fd\" magic;\n    exec {fd}>&-;\n
if [[ \"$magic\" = ''

My project has nothing to do with JSON, and my flake.nix isn’t even 57 lines long, so I was pretty sure this error wasn’t my doing. Fortunately, the error message was unique enough that searching for "nixos""json.exception.parse_error.101" quickly lead me to a relevant GitHub Issue: “get-env.sh interprets hex literals in function bodies.”

I have no idea what this issue is about, or even what these smart programmers are talking about. To be honest, I just don’t have the time right now to figure it out. However, if we scroll down the page, we can see the original poster, winterqt, has already submitted a pull-request will a potential fix: “stdenv: remove isMachO helper function”. What more, it’s already been approved and merged. Thanks for your hard work, winterqt!

Now, just because this fix has been approved and merged, that doesn’t necessarily mean that it’s available in our channel of nixpkgs, even when we’re using nixos-unstable! We can use a neat tool written by Alyssa Ross to check the status of this PR: Nixpkgs Pull Request Tracker: 138186.

pr-tracker-138186.png
Figure 1: The status of PR #138186 at the time this article was written.

It will get to our channel eventually, but we want to work on our project right now!

4. Our flake.nix, plagued by an upstream bug

In our example scenario, we’re just starting a new project, so our flake.nix is still pretty light:

 1: {
 2:   description = "A totally fine flake.nix with a buggy nixpkgs";
 3: 
 4:   # TODO: uncomment the next line to reproduce the example bug described in this article.
 5:   # inputs.nixpkgs.url = github:nixos/nixpkgs?rev=79c444b5bdeaba142d128afddee14c89ecf2a968;
 6: 
 7:   outputs = { self, nixpkgs }:
 8:     let
 9:       system = "x86_64-linux";
10:       pkgs = import nixpkgs { inherit system; };
11:     in
12:     {
13:       devShell.${system} = pkgs.mkShell {
14:       buildInputs = with pkgs; [ ddate ];
15:       shellHook = ''
16:         echo "Welcome to your project. Today is $(ddate)."
17:       '';
18:       };
19:     };
20: }
  • This is on a NixOS system, so we don’t have to specify inputs.nixpkgs.url. You can use the command nix registry list to see the channel which will be used for this input, when unspecified.
  • You can comment out the inputs.nixpkgs.url line in the above code block to experience this specific error. It may result in a large download though.

The above flake.nix is totally fine and on a normal day, running nix develop would present you with something like:

$ nix develop

Welcome to your project. Today is Today is Setting Orange, the 46th day of Bureaucracy in the YOLD 3187.

[jon@ertt.ca:~/my-project]$

But today we get error: [json.exception.parse_error.101] parse error at line 57, column 178: .... So let’s fix it!

5. Patching nixpkgs

5.1. Download the patch

The first thing we need to due is acquire the patch file that will fix our buggy nixpkgs. As mentioned in section 3, GitHub user winterqt has already submitted a potential fix via PR #138186. We’ll use that one.

GitHub has a very convenient feature for download patch files: just slap .patch onto the PR URL! Easy.

PR URL:    https://github.com/NixOS/nixpkgs/pull/138186
Patch URL: https://github.com/NixOS/nixpkgs/pull/138186.patch

Let’s grab it and add it to our project.

$ curl "https://github.com/NixOS/nixpkgs/pull/138186.patch" \
       -Lo "nixos-nixpkgs-138186.patch"
$ git stage "nixos-nixpkgs-138186.patch"

5.2. Apply the patch

To apply our patch, we only need to change line 10 of our flake.nix, from:

10: pkgs = import nixpkgs { inherit system; };

… to this expanded version:

10: nixpkgs-patched = (import nixpkgs { inherit system; }).applyPatches {
11:   name = "nixpkgs-patched-138186";
12:   src = nixpkgs;
13:   patches = [ ./nixos-nixpkgs-138186.patch ];
14: };
15: 
16: pkgs = import nixpkgs-patched { inherit system; };

What we’re doing here is creating a totally new derivation, named nixpkgs-patched-138186, based on the upstream nixpkgs derivation. Then, on the new line 16, we import that derivation for our pkgs. Everything else stays the same. Nice!

When you run nix develop the first time, it will take some time to build our nixpkgs-patched-138186 then you should get the shell you expected.

  • nix develop make take some time to build the first time. Depending on what packages are affected by your patch, quite a few things may need to be rebuilt.
  • The name you choose can be anything, but it should be unique per the patches you’ve applying. You don’t want two different derivations with the same name; you’ll only confuse yourself.

After you run nix develop for the first time, and you’re feeling curious, you can have a peek at the patched version of nixpkgs you’ve created:

$ ls -l /nix/store/*nixpkgs-patched-138186

5.3. Example patched flake.nix

 1: {
 2:   description = "A totally patched flake.nix";
 3: 
 4:   # TODO: uncomment the next line to reproduce the example bug described in this article.
 5:   # inputs.nixpkgs.url = github:nixos/nixpkgs?rev=79c444b5bdeaba142d128afddee14c89ecf2a968;
 6: 
 7:   outputs = { self, nixpkgs }:
 8:     let
 9:       system = "x86_64-linux";
10:       nixpkgs-patched = (import nixpkgs { inherit system; }).applyPatches {
11:         name = "nixpkgs-patched-138186";
12:         src = nixpkgs;
13:         patches = [ ./nixos-nixpkgs-138186.patch ];
14:       };
15:       pkgs = import nixpkgs-patched { inherit system; };
16:     in
17:     {
18:       devShell.${system} = pkgs.mkShell {
19:       buildInputs = with pkgs; [ ddate ];
20:       shellHook = ''
21:         echo "Welcome to your project. Today is $(ddate)."
22:       '';
23:       };
24:     };
25: }

6. Cleaning up

Days, weeks, or months later, when upstream nixpkgs is working fine again, all you have to do is revert your flake.nix back to its original version and delete the .patch file from your project repo. You may also want to run nix store gc to recover a bit of disk space.