This is an alternative to #39. It runs the dependabot CLI locally in the existing weekly update CI action (along with some jq magic). Yet untested in CI, but It also works locally, which is really nice!

Best reviewed commit-by-commit, there's some Nix changes in the first 2 commits.

This work is sponsored by Antithesis :sparkles:

See the nixpkgs_problem.rs file. There's some different ways of construct error messages in there:

• Some are styled with a single-line, some are multi-line (since https://github.com/NixOS/nixpkgs/pull/290743)
• Some expression paths are relative to the Nixpkgs root (e.g. EmptyArgument), some are relative to the expression file (e.g. WrongCallPackagePath, using create_path_expr). Ideally the latter should be used in all cases for more accurate errors
• The field names are sometimes superfluous, e.g. UndefinedAttr has both relative_package_file and package_name, even though the former can be computed from the latter using structure::relative_file_for_package, which e.g. EmptyArgument does.

These should be more consistent overall.

No idea how I didn't notice it missing before..

Must've come from somewhere else :shrug:

Turns out lazyDerivation (added in https://github.com/NixOS/nixpkgs-check-by-name/pull/40/commits/3f361a461c259b738ad8914749c0ea24ff0ca018) ended up filtering out .nativeBuildInputs and co., which mkShell's inputsFrom relies on to figure out what to use for the environment.. :facepalm:

@infinisil I recall you saying something about feeling dependabot is too Github specific. However, I'm not aware of an easier way to keep Github actions up-to-date. Speaking of not being GH specific, worth considering auto-mirroring this repo to another place just for backup? I thought about doing the same for my personal repos, but never got around to it.

Prompted by https://github.com/NixOS/nixpkgs-check-by-name/pull/46#discussion_r1566745927 (cc @willbush). I didn't use a bash formatter, but https://github.com/lovesegfault/beautysh seems to almost match what I've been doing!

For Nix formatting we obviously use https://github.com/NixOS/nixfmt :)

lazyDerivation limits which attributes are exposed, and version is not currently one of them, so we need to explicitly expose it.

This fixes this CD failure:

https://github.com/NixOS/nixpkgs-check-by-name/actions/runs/8637321506/job/23679230914

This was accidentally introduced in #40 (should've made a separate PR for the first commit after all!)

Run automated updates

    Updating crates.io index
    Updating aho-corasick v1.0.4 -> v1.1.3
    Updating anstream v0.5.0 -> v0.6.13
    Updating anstyle v1.0.2 -> v1.0.6
    Updating anstyle-parse v0.2.1 -> v0.2.3
    Updating anstyle-query v1.0.0 -> v1.0.2
    Updating anstyle-wincon v2.1.0 -> v3.0.2
    Updating anyhow v1.0.75 -> v1.0.81
    Updating bitflags v2.4.0 -> v2.5.0
    Removing cc v1.0.83
    Updating clap v4.4.0 -> v4.5.3
    Updating clap_builder v4.4.0 -> v4.5.2
    Updating clap_derive v4.4.0 -> v4.5.3
    Updating clap_lex v0.5.1 -> v0.7.0
    Updating colored v2.0.4 -> v2.1.0
    Updating either v1.9.0 -> v1.10.0
    Updating errno v0.3.2 -> v0.3.8
    Removing errno-dragonfly v0.1.2
    Updating fastrand v2.0.0 -> v2.0.1
    Updating hashbrown v0.12.3 -> v0.14.3
    Updating heck v0.4.1 -> v0.5.0
    Removing hermit-abi v0.3.2
    Removing is-terminal v0.4.9
    Updating itoa v1.0.9 -> v1.0.10
    Updating libc v0.2.147 -> v0.2.153
    Updating linux-raw-sys v0.4.5 -> v0.4.13
    Updating lock_api v0.4.10 -> v0.4.11
    Updating memchr v2.5.0 -> v2.7.1
    Updating memoffset v0.8.0 -> v0.9.0
    Removing once_cell v1.18.0
    Updating parking_lot_core v0.9.8 -> v0.9.9
    Updating proc-macro2 v1.0.66 -> v1.0.79
    Updating quote v1.0.33 -> v1.0.35
    Updating redox_syscall v0.3.5 -> v0.4.1
    Updating regex v1.9.3 -> v1.10.3
    Updating regex-automata v0.3.6 -> v0.4.6
    Updating regex-syntax v0.7.4 -> v0.8.2
    Updating rowan v0.15.11 -> v0.15.15
    Updating rustix v0.38.8 -> v0.38.32
    Updating ryu v1.0.15 -> v1.0.17
    Updating serde v1.0.186 -> v1.0.197
    Updating serde_derive v1.0.186 -> v1.0.197
    Updating serde_json v1.0.105 -> v1.0.114
    Updating smallvec v1.11.1 -> v1.13.2
    Updating strsim v0.10.0 -> v0.11.0
    Updating syn v2.0.29 -> v2.0.53
    Updating temp-env v0.3.5 -> v0.3.6
    Updating tempfile v3.8.0 -> v3.10.1
    Updating unicode-ident v1.0.11 -> v1.0.12
      Adding windows-sys v0.52.0
      Adding windows-targets v0.52.4
      Adding windows_aarch64_gnullvm v0.52.4
      Adding windows_aarch64_msvc v0.52.4
      Adding windows_i686_gnu v0.52.4
      Adding windows_i686_msvc v0.52.4
      Adding windows_x86_64_gnu v0.52.4
      Adding windows_x86_64_gnullvm v0.52.4
      Adding windows_x86_64_msvc v0.52.4

[INFO ] Updating 'nixpkgs' …
Changes:
-    url: https://releases.nixos.org/nixpkgs/nixpkgs-24.05pre598655.e367f7a1fb93/nixexprs.tar.xz
+    url: https://releases.nixos.org/nixpkgs/nixpkgs-24.05pre599295.7872526e9c53/nixexprs.tar.xz
-    hash: 0jbhrysv2dmzd2jilsdsw8y80rzlymwgvaw3jh5ky5k3bkzbi38y
+    hash: 0ix23iwagyyh815bq65gpz7a3xnqvx5r1p6kpzhlqcb7vs6qnwcq
[INFO ] Update successful.

🚀 Catalyst

https://github.com/NixOS/nixpkgs-check-by-name/issues/2#issuecomment-2018895221

📃 Description

PR has changes needed to output error messages with paths relative to expression file instead of nixpkgs root.

• tests: move all-packages.nix to pkgs/top-level

  In this commit I move all the tests/*/all-packages.nix to tests/*/pkgs/top-level/all-packages.nix without changing any code (breaking a lot of tests).

  The reason for this commit is to make it more obvious in the expected files results where changes where needed to get ./pkgs/by-name to become ./../by-name

• Fix relative paths in all-packages.nix files and expected outputs

  This commit just gets tests passing again before I start modifying other code.

• Update errors paths to be relative to the expression file

  Most changes are here to use relative paths to expression file in all cases.

Addresses the previous problem (https://github.com/NixOS/nixpkgs-check-by-name/pull/20#issuecomment-2013998832) with the second commit.

Also using this opportunity to improve how we define what CI should build and make it build the automated update script (but not running it), see the first commit for that.

This makes sure that when we make changes to the update script (like in this PR), it at least gets at least built (though not ran).

This sets up automated releases, mainly so that we can distribute a pre-built version of the tool, allowing Nixpkgs' CI to fetch it without building.

I used the release script to generate the first release, which also contains some info about how to use the artifact: https://github.com/NixOS/nixpkgs-check-by-name/releases/tag/0.1.0

See this Matrix message for historical context.

After this and #15 is merged, we could also extend the automated PRs to regularly update the version, such that we don't forget to do new releases.

I didn't notice xref was failing for a valid reason when merging https://github.com/NixOS/nixpkgs-check-by-name/pull/54

Decided to point at the release.sh which publishes the release.

The current cargo update command updates the version of dependencies in the Cargo.lock file based on the semver compatibility. It will not change the version range specified in the Cargo.toml file.

Most packages in Nixpkgs aren't defined by pkgs/by-name yet, even if they could. While one could migrate them manually, that would take a long time. Instead we should automate this. Figuring out how to do large-scale automated migrations will also be of great benefit to many other future changes in Nixpkgs.

In this issue I'll try to explain the pre-existing work and my ideas on how to go about this in the current codebase, such that others could implement this.

The task

The pkgs/by-name directory has certain validity checks that need to be fulfilled. For the migration in particular, these are most important:

1. Each package directory (e.g. pkgs/by-name/he/hello) must not refer to files outside itself using symlinks or Nix path expressions.
2. Evaluate Nixpkgs with system set to x86_64-linux and check that: a. For each package directory, the pkgs.${name} attribute must be defined as callPackage pkgs/by-name/${shard}/${name}/package.nix args for some args. b. For each package directory, pkgs.lib.isDerivation pkgs.${name} must be true.

A lot of packages under pkgs.* do fulfill these requirements, these are the ones we can migrate automatically. The ones that don't fulfil these requirements typically need some manual refactoring.

After verifying these requirements, migration can happen in two separate stages:

• Move all files needed for the package into the appropriate pkgs/by-name directory, with the entry-point at package.nix.

  Rewrite the packages definition in all-packages.nix appropriately to point to ../by-name/...

• Optionally, if the definition in all-packages.nix has { } as the second callPackage argument, remove the definition completely.

Examples

• pkgs.kody (defined here) can't be migrated automatically, because its default.nix contains ../../../top-level/kodi-packages.nix, therefore not fulfilling requirement 1.
• pkgs.termdown (defined here) can't be migrated automatically, because it's not defined using pkgs.callPackage
• pkgs.qiv (defined here) fulfills all requirements and therefore can be migrated by:
  • Moving pkgs/applications/graphics/qiv/default.nix to pkgs/by-name/qi/qiv/package.nix and adjusting the definition in all-packages.nix to

      qiv = callPackage ../by-name/qi/qiv/package.nix {
        imlib2 = imlib2Full;
      };
    

  • We cannot remove the definition entirely because the second callPackage argument is not { }
• pkgs.scli (defined here) fulfills all requirements and therefore can be migrated by:
  • Moving pkgs/applications/misc/scli/default.nix to pkgs/by-name/sc/scli/package.nix and adjusting the definition in all-packages.nix to

      scli = callPackage ../by-name/sc/scli/package.nix { };
    

  • And because the second callPackage argument is { } we can remove the entire definition.

Ratchet checks

Currently this tool implements ratchet checks, which prevent the introduction of new instances of deprecated patterns. This happens here: https://github.com/NixOS/nixpkgs-check-by-name/blob/86202962eed024d778f2bece8a68e81361898405/src/eval.rs#L515-L517

This says that the ratchet for a specific package is "loose" (aka legacy), meaning we can migrate away from it (therefore tightening the ratchet).

Currently these ratchet states are only used to compare the base branch and the head of a PR, such that we can detect invalid transitions between ratchet states, see ratchet.rs. This allows detect when a PR introduces a new instance of a legacy pattern.

This notably also means that even if somebody merges a PR that introduces new instances of deprecated patterns (happens if a pattern is deprecated between the PRs CI running and it being merged), it won't break master, because these ratchets aren't invalid on their own: Only the transition from tight to loose is invalid.

Using ratchets for migrations

We can also use this ratchet abstraction to determine migrations, because the loose state is exactly what needs to be migrated!

The tool can have two modes:

• (This one currently exists) Take two Nixpkgs instances, verify them individually, then compare the ratchets between them: https://github.com/NixOS/nixpkgs-check-by-name/blob/86202962eed024d778f2bece8a68e81361898405/src/main.rs#L119
• (This one would be new) Take a single Nixpkgs instance, verify it, then run a migration on all loose ratchets.

Thus in addition to defining how ratchets can be compared, we just need a way to define how ratchets can be migrated.

I've started a very rough WIP branch of this before here. Note how:

• There's a new migrate mode on the CLI, which only takes a single Nixpkgs path
• All of the ratchets got a migrate method in addition to a compare one.

Problem: Which files are part of a package?

It's really tricky to figure out which files are part of a package and should therefore be moved. Because outside of pkgs/by-name, each package's entry-point file (usually default.nix) can refer to either more of its own files which should be moved (e.g. ./Cargo.lock), or refer to arbitrary files from other packages in which case it can't be moved without breaking check 1 (like ../../some/file.sh).

I did manage to get that to work almost perfectly in an older version of this tooling. It works by creating an index of all (static) references between Nix files to figure out whether a file was only references by a single package or many packages, and then revert that relation to figure out which packages only referred to files used by that package itself.

This tool was used to create this PR: https://github.com/NixOS/nixpkgs/pull/211832

Notably, the current nixpkgs-check-by-name tool doesn't have such functionality, which also means that when somebody creates a PR that introduces a new package outside of pkgs/by-name, it tells them to migrate it to pkgs/by-name, even if it can't be directly migrated due to it referring to shared files! This is the reason the workaround for packages with multiple versions is necessary.

Problem: It's non-trivial to efficiently rewrite many sections of files

This is what I've been struggling with while experimenting. I wrote some notes on this. Basically it's really hard to use rowan (the parsing library used underneath rnix) to edit files in multiple places.

In this case I'd like to have a basic edit operation implemented that can modify or remove attributes (for all-packages.nix). I did get that working in the older tooling, but it's a bit too hacky (still, it shows that it can be done!).

Todo

This is my current idea on how to progress with this:

1. Port the reference index code from the old tool to nixpkgs-check-by-name, therefore removing the need for the multi-versioned package workaround. This means that we only have a loose ratchet exactly when we can figure out how to migrate automatically.

2. Implement decent abstractions for representing tree changes, including moving files and changing Nix files. And implement those abstractions efficiently using rnix.

3. Make ratchets declare how to migrate from a loose state to a tight one using the above tree change abstraction. This should be almost trivial at this point, since we already know that we can migrate, and information from determining this should allow exactly specifying how we can migrate using the abstraction for doing changes.

4. Set up CI to run the migration for every push to Nixpkgs master, and force push the resulting changes into a separate branch, which can then be merged at any point.

   Bonus: Also make it open a tracking issue for all the packages that can't be migrated manually and update it as they are fixed.

Every future addition of a new ratchet check (for any pattern we want to deprecate) will have to come with an implementation of how to migrate from it, which will then automatically run and be pushed to the branch, which can then be merged again as necessary.

This adds a fully automated changelog and version pipeline, fixing #38. I looked into mainly Towncrier and Changie, but both of them weren't super integrated, so I wrote my own solution, which ended up being fairly simple.

The solution here works like this:

• Every PR has to have a "This change is user-facing" checkbox to the PR description (checked in the default PR template), and either:

  • Uncheck it to indicate it's not a user-facing change

  • Or add a changelog by creating a file in one of changes/unreleased/{major,medium,minor}.

    The file has to be a markdown file with its first line like # (and no other lines starting with #).

• Every week on Tuesday (presumably after the automated update PR from Monday is merged, but can also be triggered manually at any point), a version PR is opened if the main branch has any unreleased changelogs. This PR includes:

  • Depending on whether there are any major, medium or only minor changelogs, the appropriate version number is bumped in Cargo.toml

  • A new changes/released/$MAJOR.$MEDIUM.$MINOR.md file that compiles all previously unreleased changelogs into a single file. Each changelog entry is a Markdown list entry of the form

    - <title> by @<author> in #<PR>
      <optional rest of lines>
    

    The author and PR number is automatically determined based on which PR added the changelog file.

• Once that PR is merged (optionally with manual updates), the release will be published, with the compiled changelog included.

  Currently there's nothing that would ever update the changelog of releases, but I don't expect that to be necessary.

This PR needs more testing and comments before it can be merged.

• [ ] This change is user-facing

This work is sponsored by Antithesis :sparkles:

Recently there was an automated mass removal of lib.mdDoc which now requires subsequent PRs whenever a PR that introduces more instances gets merged.

The approach used by nixpkgs-check-by-name makes sure that this problem doesn't happen. We should consider integrating such treewide removals to benefit from that. This will also be relevant for the upcoming automated pkgs/by-name migration.

In particular, while this tool currently only has infrastructure for detecting when new instances of deprecated patterns are introduced, I've been planning to add infrastructure to also do automated replacements of such deprecated patterns.

For now I haven't gotten to that, but let's use this issue to track such developments.

We should have a changelog and decide the versioning scheme along with it. These are some thoughts regarding that, prompted via https://github.com/NixOS/nixpkgs-check-by-name/pull/37.

I'm not considering changelog tools that use the Git log, because I don't think the Git log is valuable for users. I'd like to require each PR to submit such a changelog entry file (with an option to explicitly declare the PR as not changelog-worthy) (first seen here), and then using a tool to generate the changelog when it's time for a release:

• changelog-d (used by Nix since recently)
• changelogd (what I first thought Nix used..)
• Towncrier (looks very promising)
• Scriv (not as feature-complete, but does have some built-in support for GitHub releases)
• Changie (looks very promising)
• (others?)

Regarding versions, there's various versioning schemes, but I'm liking EffVer the most, and it matches the existing versions 0.1.0 and since recently 0.1.1.

I would like the automated weekly update PRs to also take care of bumping the release and changelog automatically if there's been any changes. To figure out the next version, all change entries should be tagged with the level of effort required to update (one of macro/meso/micro). Then the next version can be generated by increasing the previous one by that level. Though it's not perfect, because 100 micro changes could amount to a meso/macro change. So being able to adjust the automatically generated version number would be good too (also for an eventual 1.0.0 release).

Note that we should take some precaution to not miss PRs that might get merged during which the release/update PR is open, probably by making sure that a commit gets tagged instead of the merged commit.

We want to use the pkgs/by-name convention for other package sets, e.g. https://github.com/NixOS/nixpkgs/pull/299347.

Here's a rough idea how this could look like:

Package set config

Define some Nix config file to determine which package sets exist:

# pkgs/top-level/by-name-sets-config.nix
{
  packageSets = {
    toplevel = {
      attrPath = [ ];
      path = ./pkgs/by-name;
      shardLength = 2;
    };
    mySet = {
      # Any attrpath that will contain the attributes
      attrPath = [ "mySet" ];
      path = ./pkgs/mySet/by-name;
      # No sharding
      shardLength = null;
    };
  };
}

Read the packages in Nix

Create a Nix file to automatically call all of them like the current by-name-overlay.nix:

# pkgs/top-level/by-name-sets.nix
{
  toplevel = callPackage: {
    hello = <...>;
  };
  mySet = callPackage: {
    hi = <...>;
  };
}

For each package set declared, define it by importing by-name-sets.nix, like

mySet = lib.makeScope pkgs.newScope (self:
  (import ./pkgs/top-level/by-name-sets.nix).mySet self.callPackage
) 

Check the packages with the tool

Adjust this tool to read a by-name-sets-config.nix file for determining what it needs to check and how.