linux: new workflow for kernel changes

See https://github.com/NixOS/nixpkgs/issues/437208#issuecomment-3288623669
Depends on https://github.com/NixOS/org/pull/172

As documented below, the idea is to essentially group all changes
rebuilding all VM tests with kernel updates and merge them together into
`master` whenever the Linux kernels get updated.

This documents the workflow of updates in the nixpkgs manual. While at
it, I removed the README from the packages because

* it's horribly outdated
* I didn't even know it exists which confirms that its discoverability
  was very poor

and added the relevant portions into the nixpkgs manual as well.

.github/workflows/periodic-merge-6h.yml

@@ -35,6 +35,8 @@ jobs:
             into: staging-next
           - from: staging-next
             into: staging
+          - from: master
+            into: staging-nixos
     uses: ./.github/workflows/periodic-merge.yml
     with:
       from: ${{ matrix.pairs.from }}

CONTRIBUTING.md

@@ -430,20 +430,22 @@ gitGraph
 
 Here's an overview of the different branches:
 
-| branch | `master` | `staging-next` | `staging` |
+| branch | `master` | `staging-next` | `staging` | [`staging-nixos`](#test-driver-rebuild) |
-| --- | --- | --- | --- |
+| --- | --- | --- | --- | --- |
-| Used for development | ✔️ | ❌ | ✔️ |
+| Used for development | ✔️ | ❌ | ✔️ | ✔️ |
-| Built by Hydra | ✔️ | ✔️ | ❌ |
+| Built by Hydra | ✔️ | ✔️ | ❌ | ❌ |
-| [Mass rebuilds][mass-rebuild] | ❌ | ⚠️  Only to fix Hydra builds | ✔️ |
+| [Mass rebuilds][mass-rebuild] | ❌ | ⚠️  Only to fix Hydra builds | ✔️ | ❌[^1]  |
-| Critical security fixes | ✔️ for non-mass-rebuilds | ✔️ for mass-rebuilds | ❌ |
+| Critical security fixes | ✔️ for non-mass-rebuilds | ✔️ for mass-rebuilds | ❌ | ✔️ |
-| Automatically merged into | `staging-next` | `staging` | - |
+| Automatically merged into | `staging-next` & `staging-nixos` | `staging` | - | - |
-| Manually merged into | - | `master` | `staging-next` |
+| Manually merged into | - | `master` | `staging-next` | `master` |
 
 The staging workflow is used for all stable branches with corresponding names:
 - `master`/`release-YY.MM`
 - `staging`/`staging-YY.MM`
 - `staging-next`/`staging-next-YY.MM`
 
+[^1]: Except changes that cause no more rebuilds than kernel updates
+
 # Conventions
 
 ## Branch conventions
@@ -495,6 +497,26 @@ In order to help the decision, CI automatically assigns [`rebuild` labels](https
 As a rule of thumb, if the number of rebuilds is **500 or more**, consider targeting the `staging` branch instead of `master`; if the number is **1000 or more**, the pull request causes a mass rebuild, and should target the `staging` branch.
 See [previously merged pull requests to the staging branches](https://github.com/NixOS/nixpkgs/issues?q=base%3Astaging+-base%3Astaging-next+is%3Amerged) to get a sense for what changes are considered mass rebuilds.
 
+Please note that changes to the Linux kernel are an exception to this rule.
+These PRs go to `staging-nixos`, see [the next section for more context](#changes-rebuilding-all-tests).
+
+### Changes rebuilding all NixOS tests
+[test-driver-rebuild]: #changes-rebuilding-all-tests
+
+Changes causing a rebuild of all NixOS tests get a special [`10.rebuild-nixos-tests`](https://github.com/NixOS/nixpkgs/issues?q=state%3Aopen%20label%3A10.rebuild-nixos-tests) label.
+These changes pose a significant impact on the build infrastructure.
+
+Hence, these PRs should either target a `staging`-branch or `staging-nixos`, provided one of following conditions applies:
+
+* The label `10.rebuild-nixos-tests` is set, or
+* The PR is a change affecting the Linux kernel.
+
+The branch gets merged whenever mainline kernel updates or critical security fixes land on the branch.
+This usually happens on a weekly basis.
+
+Backports are not handled by such a branch.
+The relevant PRs from this branch must be backported manually.
+
 ## Commit conventions
 [commit-conventions]: #commit-conventions
 

ci/supportedBranches.js

@@ -8,6 +8,7 @@ const typeConfig = {
   release: ['development', 'primary'],
   staging: ['development', 'secondary'],
   'staging-next': ['development', 'secondary'],
+  'staging-nixos': ['development', 'secondary'],
   'haskell-updates': ['development', 'secondary'],
   nixos: ['channel'],
   nixpkgs: ['channel'],
@@ -19,6 +20,7 @@ const orderConfig = {
   master: 0,
   release: 1,
   staging: 2,
+  'staging-nixos': 2,
   'haskell-updates': 3,
   'staging-next': 4,
 }

doc/packages/linux.section.md

@@ -105,3 +105,56 @@ $ make -C $dev/lib/modules/*/build M=$(pwd)/drivers/net/ethernet/mellanox module
 ```
 
 :::
+
+## Maintainer information {#sec-linux-kernel-maintainer-information}
+
+### Updating kernels {#sec-linux-updates}
+
+Updating all kernels can be done with the following script:
+
+```ShellSession
+$ pkgs/os-specific/linux/kernel/update.sh
+```
+
+The change gets submitted like this:
+
+* File a PR against `staging-nixos`.
+  * Add a `backport release-XX.XX` label for an automated backport.
+    We don't expect many other changes on that branch to require a backport, hence there's no such branch for stable.
+    By using an additional PR, we get the automatic backport against stable without manual cherry-picks.
+* Merge into `staging-nixos`.
+* File as PR from `staging-nixos` against `master`.
+* When all status checks are green, merge.
+
+### Add a new (major) version of the Linux kernel {#sec-linux-add-new-kernel-version}
+
+* When running `./pkgs/os-specific/linux/kernel/update.sh`, new kernel majors get discovered automatically.
+* Prepare all Nix expressions for the new kernel
+  * Instantiate the new kernel in `pkgs/top-level/linux-kernels.nix` in the `kernels`-section.
+    ```nix
+    {
+      linux_X_Y = callPackage ../os-specific/linux/kernel/mainline.nix {
+        branch = "X.Y";
+        kernelPatches = [
+          # any new patches required (it makes to look which patches are used by its predecessor)
+        ];
+      };
+    }
+    ```
+  * Instantiate the package-set in `vanillaPackages`:
+    ```nix
+    {
+      linux_X_Y = recurseIntoAttrs (packagesFor kernels.linux_X_Y);
+    }
+    ```
+  * Update `linux_latest` to the new attribute.
+* __SQUASH__ the changes into the `linux: init at …` commit.
+* If a new hardened is available:
+  * Instantiate a `linux_X_Y_hardened = hardenedKernelsFor kernels.linux_X_Y { };` in `kernels` and
+    `linux_X_Y_hardened = hardenedKernelFor kernels.linux_X_Y { };` in the `packages`-section.
+  * Make sure to remove the hardened variant of the previous kernel version unless it's LTS.
+    We only support the latest and latest LTS version of hardened.
+* If no new hardened kernel is available:
+  * Keep the previously latest kernel until its mainline counterpart gets removed.
+    After that `linux_hardened` points to the latest LTS supported by hardened.
+* __SQUASH__ the changes into the `linux_X_Y_hardened: init at …` commit.

doc/redirects.json

@@ -4306,6 +4306,15 @@
   "sec-linux-kernel": [
     "index.html#sec-linux-kernel"
   ],
+  "sec-linux-kernel-maintainer-information": [
+    "index.html#sec-linux-kernel-maintainer-information"
+  ],
+  "sec-linux-updates": [
+    "index.html#sec-linux-updates"
+  ],
+  "sec-linux-add-new-kernel-version": [
+    "index.html#sec-linux-add-new-kernel-version"
+  ],
   "ex-overriding-kernel-derivation": [
     "index.html#ex-overriding-kernel-derivation"
   ],

pkgs/os-specific/linux/kernel/README.md

@@ -1,47 +0,0 @@
-# How to add a new (major) version of the Linux kernel to Nixpkgs:
-
-1.  Copy the old Nix expression (e.g., `linux-2.6.21.nix`) to the new one (e.g., `linux-2.6.22.nix`) and update it.
-
-2.  Add the new kernel to the `kernels` attribute set in [`linux-kernels.nix`](../../../top-level/linux-kernels.nix) (e.g., create an attribute `kernel_2_6_22`).
-
-3.  Update the kernel configuration:
-
-    1. While in the Nixpkgs repository, enter the development shell for that kernel:
-
-       ```console
-       $ nix-shell -A linuxKernel.kernels.linux_2_6_22
-       ```
-
-    2. Unpack the kernel:
-
-       ```console
-       [nix-shell]$ pushd $(mktemp -d)
-       [nix-shell]$ unpackPhase
-       ```
-
-    3. For each supported platform (`i686`, `x86_64`, `uml`) do the following:
-
-       1. Make a copy from the old config (e.g., `config-2.6.21-i686-smp`) to the new one (e.g., `config-2.6.22-i686-smp`).
-
-       2. Copy the config file for this platform (e.g., `config-2.6.22-i686-smp`) to `.config` in the unpacked kernel source tree.
-
-       3. Run `make oldconfig ARCH={i386,x86_64,um}` and answer all questions. (For the uml configuration, also add `SHELL=bash`.) Make sure to keep the configuration consistent between platforms (i.e., don’t enable some feature on `i686` and disable it on `x86_64`).
-
-       4. If needed, you can also run `make menuconfig`:
-
-          ```ShellSession
-          $ nix-shell -p ncurses pkg-config
-          $ make menuconfig ARCH=arch
-          ```
-
-       5. Copy `.config` over the new config file (e.g., `config-2.6.22-i686-smp`).
-
-4.  Test building the kernel:
-
-```ShellSession
-nix-build -A linuxKernel.kernels.kernel_2_6_22
-```
-
-If it compiles, ship it! For extra credit, try booting NixOS with it.
-
-5.  It may be that the new kernel requires updating the external kernel modules and kernel-dependent packages listed in the `linuxPackagesFor` function in `linux-kernels.nix` (such as the NVIDIA drivers, AUFS, etc.). If the updated packages aren’t backwards compatible with older kernels, you may need to keep the older versions around.