> Anyway, I keep picturing all these little kids playing some game in this big field of rye and all. Thousands of little kids, and nobody’s around―nobody big, I mean―except me. And I’m standing on the edge of some crazy cliff. What I have to do, I have to catch everybody if they start to go over the cliff―I mean if they’re running and they don’t look where they’re going I have to come out from somewhere and catch them. That’s all I’d do all day. I’d just be the catcher in the rye and all. I know it’s crazy, but that’s the only thing I’d really like to be. I know it’s crazy.
The python-updates branch is not a "development" branch in the sense of
ci/README.md's classification. That's because it allows force pushes.
When rewrites are possible, cherry-picking from this branch should not
be allowed, because the commit references will potentially end up out of
sync.
These kind of branches are now termed "Work-in-Progress" branches. Up
until recently these branches didn't work well for Pull Requests
targeting them, because Eval wouldn't run on them with a push event and
thus, Eval in the PR couldn't succeed either. That's now fixed, PRs
towards *any* WIP branch should work correctly.
Instead of uploading the outpaths as artifact, this uploads them via
cachix. Most of all, this makes CI less brittle, because Eval in PRs
will still be able to succeed, even if no workflow run for the push
event could be found on the target branch. It will just take longer.
This also makes moving Eval into the Merge Queue easier to do: When
downloading artifacts from a different run, these would always have to
match on the right event, too. By pulling from cachix, the same workflow
can support target branches with merge queue and without merge queue at
the same time. The latter would still use the push event, while the
former could use the merge_group event.
Last but not least, this should fix Eval on PRs targeting `wip-`
branches and any other branches that the push event doesn't trigger on.
These would never find an Eval result from the target branch and could
never show rebuilds accurately. Now these PRs should work at a slightly
higher runtime cost.
This needs the multiline flags, which enables `^` and `$` to match line
start and line end, not start and end of the whole string.
Not sure how this got past testing when initially merged.
This ensures we don't accidentally use aliases in the nixpkgs shell or
other places that depend on the CI-pinned pkgs instance.
Nixpkgs generally — and CI specifically — do not use aliases, because we
want to ensure they are not load-bearing and can be removed safely.
See: https://github.com/NixOS/nixpkgs/blob/ce9979ec1c/pkgs/top-level/release-outpaths.nix#L28
Currently treefmt-nix is still defaulting `programs.nixfmt.package` to
the `nixfmt-rfc-style` alias. This makes sense, as they do not know for
certain which revision of nixpkgs is in use.
We do know, however, so we can explicitly use the non-alias name.
Logging objects to stdout is not possible with `core.info`, so we
fallback to `console.log` instead. There's no functional difference for
these anyway.
This just moves the code over to ci/github-script to make it easy to
test and iterate on locally.
The name `prepare` is chosen, because the script will be extended with
the other steps from "PR / prepare" next.
Splits the "occasionally" case into two, depending on whether the commit
has a diff or was not cherry-picked at all. Prepares the next commit,
where these are conditionally shown only.
This change allows giving a reason via footer of the commit message for
why this commit is not cherry-picked. This avoids having to "explain"
the automated review comment afterwards - instead, this explanation can
be given immediately when writing that commit.
For example, for an update of `xen` on the stable branch, this could be:
```
xen: 4.19.3-unstable-2025-07-09 -> 4.19.3
[... commit message ...]
Not-cherry-picked-because: unstable is on a different minor version
```
This would then be shown as part of the automated review. The severity
of this will be downgraded from "warning" to "important". We still treat
the review as "changes requested", because it would be very complicated
and noisy to handle two different categories of reviews, some with
requested changes and some with comments only.
An alternative would be to not show this review at all. However, given
that the reviewers expectation on backports should already be "if it's
not a clean backport, the automated review will tell me what to look
at", it seems better to show these and have the committer confirm by
dismissing the review. Otherwise we risk merging actually unreviewed
commits.
The filtered checkout we used before was a nice idea, but only worked
for small numbers of commits in a PR. It would fetch the whole history
initially, but only fetch objects on demand. This turns out to be much
too slow in a PR with 18 commits, regularly hitting the 10 minute
timeout, even when running it locally.
The new approach uses regular checkouts again. In contrast to the old
style, before we switched to the filtered checkout, this only fetches
exactly the commits we need - and all of them at once. This is
significantly faster than both other approaches, and scales much better.
A bigger number of commits doesn't have much of an effect, if any at
all.
This has been marked insecure a while ago, as some CVEs have not been
backported. Even if *some* CVEs are fixed, we'd need **all** of them to
be, to get it back into the cache.
Not having it in the cache means, we can not test it in CI. This means
we can't make sure to actually support this version to evaluate Nixpkgs.
The concept of this alias becomes questionable once we move past 2.18,
where Lix was forked. We should probably move to a feature-detection
based approach for lib/minver.nix eventually, too.
With this change, we start running Eval on all available Lix and Nix
versions. Because this requires a lot of resources, this complete test
is only run when `ci/pinned.json` is updated.
The resulting outpaths are checked for consistency with the target
branch. A difference will cause the `report` job to fail, thus blocking
the merge, ensuring Eval consistency for Nixpkgs across different
versions.
This implements a kind of "ratchet style" check: Since we originally
confirmed that the versions currently in Nixpkgs at the time of this
commit match Eval behavior of Nix 2.3, we can ensure consistency with
Nix 2.3 down the road, even without testing for it explicitly.
There had been one regression in Eval consistency for Nix between 2.18
and 2.24 - two tests in `tests.devShellTools` produce different results
between Lix 2.91+ (which was forked from Nix 2.18) and Nix 2.24+. I
assume it's unlikely that such a change would be "fixed" by now, thus I
added an exception for these.
As a bonus, we also present the total time in seconds it takes for Eval
to complete for every tested version in a summary table. This allows us
to easily see performance improvements for Eval due to version updates.
At this stage, this time only includes the "outpaths" step of Eval, but
not the generation of attrpaths beforehand.
This is slightly faster than downloading and extracting a tarball and
additionally allows a sparse checkout. No need to download docs or nixos
for our purpose.
The data is quite noisy, but suggests improvements from anywhere between
5-15 seconds for each job using the pinned nixpkgs.
Using these defaults makes it possible to just run `nix-build ci -A
eval.singleSystem` without passing additional arguments and get a sane
result back. Especially helpful when testing or debugging.
A `chunkSize` of 5000 is conservative to be able to run on systems with
less memory as well. Run-time is not impacted by that, as recent
benchmarks show.
`procps` pulls in 180 MB of systemd, but busybox also provides `kill`.
`busybox` also ships `time`, so no need for that extra dependency.
Using `nativeBuildInputs` pulls in all the -dev outputs of the listed
packages - which we don't need. We only need to run these tools, thus
map to their bin outputs.
Brings down the closure size from 500+ MB to 193 MB for the Eval job.
This probably saves ~10 seconds for the job.
No need to return bytes in these files. Also busybox has problems to
render `free -b` with sizes > 100 GB properly in the next commit,
leading to extraction errors with awk.
When a package's attrpath is renamed it is currently treated as a
rebuild, even though the outpath already exists and is already cached.
This also happens when adding new names for packagesets that already
exist, for example when starting to eval `perlPackages` in CI, which is
just the same as `perl540Packages` currently. It would also happen when
`perlPackages` is switched from `perl540Packages` to `perl999Packages`.
Assuming that `perl999Packages` had already been built before, this
doesn't really cause any rebuilds.
This was run as a test in `doc/tests/check-nix-code-blocks.nix` before,
but its DX can be improved: By including it in `treefmt` we get better
error reporting and auto-fixing, as well as running it on *all* markdown
files (including READMEs etc.) for free.
This gives us Nix 2.30 as `nixVersions.latest`, which enables it for
Eval in CI automatically.
It also gives us markdown-code-runner 0.2.0, which allows it to run with
treefmt.
From the nixpkgs-unstable channel:
https://hydra.nixos.org/eval/1817362#tabs-inputs
Changes for treefmt-nix:
421b56313c...58bd4da459
This reduces my codeownership to Firefox Rapid, ESR and the update
script. My hope is that will reduce my notification fatigue on Firefox
Beta/Devedition PRs that I needlessly get requested to.
