gador/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #281412 from tweag/by-name-update-contrib-docs

Browse Source 
check-by-name: Fix CI, update contributor docs, remove legacy script
This commit is contained in:
Silvan Mosberger
2024-01-17 21:28:57 +01:00
committed by GitHub
parent 0f6090448a 1516b35afa
commit 1cb065db34
5 changed files with 27 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/check-by-name.yml
+4 -3
View File
@@ -4,8 +4,8 @@
# When you make changes to this workflow, also update pkgs/test/nixpkgs-check-by-name/scripts/run-local.sh adequately
name: Check pkgs/by-name
# The pre-built tool is fetched from a channel,
# making it work predictable on all PRs.
# The tool is pinned to a pre-built version on Hydra,
# see pkgs/test/nixpkgs-check-by-name/scripts/README.md
on:
# Using pull_request_target instead of pull_request avoids having to approve first time contributors
pull_request_target
@@ -92,9 +92,10 @@ jobs:
echo "base=$base" >> "$GITHUB_ENV"
- uses: cachix/install-nix-action@7ac1ec25491415c381d9b62f0657c7a028df52a7 # v24
- name: Fetching the pinned tool
# Update the pinned version using pkgs/test/nixpkgs-check-by-name/scripts/update-pinned-tool.sh
run: |
# Get the direct /nix/store path from the pin to avoid having to evaluate Nixpkgs
toolPath=$(jq -r .ci-path pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json)
toolPath=$(jq -r '."ci-path"' pkgs/test/nixpkgs-check-by-name/scripts/pinned-tool.json)
# This asks the substituter for the path, which should be there because Hydra will have pre-built and pushed it
nix-store --realise "$toolPath" --add-root result
- name: Running nixpkgs-check-by-name
nixos/release-combined.nix
+2 -2
View File
@@ -171,8 +171,8 @@ in rec {
(onSystems ["x86_64-linux"] "nixpkgs.mesa_i686") # i686 sanity check + useful
["nixpkgs.tarball"]
# Ensure that nixpkgs-check-by-name is available in all release channels and nixos-unstable,
# so that a pre-built version can be used in CI for PR's on the corresponding development branches.
# Ensure that nixpkgs-check-by-name is available in nixos-unstable,
# so that a pre-built version can be used in CI for PR's
# See ../pkgs/test/nixpkgs-check-by-name/README.md
(onSystems ["x86_64-linux"] "nixpkgs.tests.nixpkgs-check-by-name")
];
pkgs/test/nixpkgs-check-by-name/README.md
+5 -16
View File
@@ -1,9 +1,13 @@
# Nixpkgs pkgs/by-name checker
This directory implements a program to check the [validity](#validity-checks) of the `pkgs/by-name` Nixpkgs directory.
It is being used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).
This is part of the implementation of [RFC 140](https://github.com/NixOS/rfcs/pull/140).
A [pinned version](./scripts/pinned-tool.json) of this tool is used by [this GitHub Actions workflow](../../../.github/workflows/check-by-name.yml).
See [./scripts](./scripts/README.md#update-pinned-toolsh) for how to update the pinned version.
The source of the tool being right inside Nixpkgs allows any Nixpkgs committer to make updates to it.
## Interface
The interface of the tool is shown with `--help`:
@@ -96,18 +100,3 @@ Tests are declared in [`./tests`](./tests) as subdirectories imitating Nixpkgs w
- `expected` (optional):
A file containing the expected standard output.
The default is expecting an empty standard output.
## Hydra builds
This program will always be available pre-built for `x86_64-linux` on the `nixos-unstable` channel and `nixos-XX.YY` channels.
This is ensured by including it in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).
This allows CI for PRs to development branches `master` and `release-XX.YY` to fetch the pre-built program from the corresponding channel and use that to check the PR. This has the following benefits:
- It allows CI to check all PRs, even if they would break the CI tooling.
- It makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.
- It improves security, since we don't have to build potentially untrusted code from PRs.
The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).
- It allows anybody to make updates to the tooling and for those updates to be automatically used by CI without needing a separate release mechanism.
The tradeoff is that there's a delay between updates to the tool and those updates being used by CI.
This needs to be considered when updating the [API](#api).
pkgs/test/nixpkgs-check-by-name/scripts/README.md
+16 -2
View File
@@ -19,6 +19,20 @@ Arguments:
## `./update-pinned-tool.sh`
Updates the pinned CI tool in [`./pinned-tool.json`](./pinned-tool.json) to the
[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux)
[latest version from the `nixos-unstable` channel](https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tests.nixpkgs-check-by-name.x86_64-linux).
This script needs to be called manually when the CI tooling needs to be updated.
The `pinned-tool.json` file gets populated with both:
- The `/nix/store` path for `x86_64-linux`, such that CI doesn't have to evaluate Nixpkgs and can directly fetch it from the cache instead.
- The Nixpkgs revision, such that the `./run-local.sh` script can be used to run the checks locally on any system.
To ensure that the tool is always pre-built for `x86_64-linux` in the `nixos-unstable` channel,
it's included in the `tested` jobset description in [`nixos/release-combined.nix`](../../../nixos/release-combined.nix).
Why not just build the tooling right from the PRs Nixpkgs version?
- Because it allows CI to check all PRs, even if they would break the CI tooling.
- Because it makes the CI check very fast, since no Nix builds need to be done, even for mass rebuilds.
- Because it improves security, since we don't have to build potentially untrusted code from PRs.
The tool only needs a very minimal Nix evaluation at runtime, which can work with [readonly-mode](https://nixos.org/manual/nix/stable/command-ref/opt-common.html#opt-readonly-mode) and [restrict-eval](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval).
This script is called manually once the CI tooling needs to be updated.
pkgs/test/nixpkgs-check-by-name/scripts/fetch-pinned-tool.sh
-34
View File
@@ -1,34 +0,0 @@
#!/usr/bin/env bash
# Legacy script, can be removed once the PR removing its use is merged.
# It's not used anymore because CI and local runs both use pinned-tool.json as their source of truth now,
# though in different ways since local runs need to support arbitrary platforms
# Try to not use nix-shell here to avoid fetching Nixpkgs,
# especially since this is used in CI
# The only dependency is `jq`, which in CI is implicitly available
# And when run from ./run-local.sh is provided by that parent script
set -o pipefail -o errexit -o nounset
trace() { echo >&2 "$@"; }
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
pin_file=$SCRIPT_DIR/pinned-tool.json
if (( $# < 1 )); then
trace "Usage: $0 fetch OUTPUT_PATH"
trace "OUTPUT_PATH: The output symlink path for the tool"
exit 1
fi
output=$1
trace "Reading $pin_file.. "
rev=$(jq -r .rev "$SCRIPT_DIR"/pinned-tool.json)
trace -e "Git revision is \e[34m$rev\e[0m"
path=$(jq -r '."ci-path"' "$SCRIPT_DIR"/pinned-tool.json)
trace "Tooling path is $path"
trace -n "Fetching the prebuilt version of nixpkgs-check-by-name.. "
nix-store --add-root "$output" -r "$path" >/dev/null
realpath "$output"