From 48d5f38c6a2e5d21c22abb2485aaeeb66ca8a50a Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Wed, 22 Jun 2022 10:13:16 -0600 Subject: [PATCH 01/11] doc/builders/fetchers: Add examples, reduce sentence complexity --- doc/builders/fetchers.chapter.md | 33 +++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index 70380248f8c6..29064f16b4f3 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -1,12 +1,39 @@ # Fetchers {#chap-pkgs-fetchers} -When using Nix, you will frequently need to download source code and other files from the internet. For this purpose, Nix provides the [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) feature and Nixpkgs provides various functions that implement the actual fetching from various protocols and services. +Nixpkgs provides fetchers for different protocols and services. Fetcher's are functions that simplify downloading files. ## Caveats -Because fixed output derivations are _identified_ by their hash, a common mistake is to update a fetcher's URL or a version parameter, without updating the hash. **This will cause the old contents to be used.** So remember to always invalidate the hash argument. +Fetcher's store their output in the nix store, or cache, using Nix's [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs). Later, Nix can reuse the downloaded contents via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: -For those who develop and maintain fetchers, a similar problem arises with changes to the implementation of a fetcher. These may cause a fixed output derivation to fail, but won't normally be caught by tests because the supposed output is already in the store or cache. For the purpose of testing, you can use a trick that is embodied by the [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function. It uses the derivation `name` to create a unique output path per fetcher implementation, defeating the caching precisely where it would be harmful. +```nix +fetchurl { + url = "http://www.example.org/hello-1.0.tar.gz"; + sha256 = "1111111111111111111111111111111111111111111111111111"; +}; +``` + +A common mistake is to update a fetcher’s URL, or a version parameter, without updating the hash. + +```nix +fetchurl { + url = "http://www.example.org/hello-1.1.tar.gz"; + sha256 = "1111111111111111111111111111111111111111111111111111"; +}; +``` + +**This will reuse the old contents**. Remember to invalidate the hash argument by setting the sha256 to an empty string. + +```nix +fetchurl { + url = "http://www.example.org/hello-1.1.tar.gz"; + sha256 = ""; +}; +``` + +Hash mismatches generate an error message containing the correct sha256. + +A similar problem arises when changing a fetcher's implementation. Changes may cause a fixed output derivation to fail. Failures can go undetected if the output of the derivation already exists in the nix store or cache. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent this. It uses the derivation hash in the output's name to invalidate the cache. ## `fetchurl` and `fetchzip` {#fetchurl} From 24e95a44613d9731e459ba98a9cfc011c4edd922 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Wed, 29 Jun 2022 12:50:16 -0600 Subject: [PATCH 02/11] Remove possessive apostrophes --- doc/builders/fetchers.chapter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index 29064f16b4f3..3ed3c103d750 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -1,10 +1,10 @@ # Fetchers {#chap-pkgs-fetchers} -Nixpkgs provides fetchers for different protocols and services. Fetcher's are functions that simplify downloading files. +Nixpkgs provides fetchers for different protocols and services. Fetchers are functions that simplify downloading files. ## Caveats -Fetcher's store their output in the nix store, or cache, using Nix's [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs). Later, Nix can reuse the downloaded contents via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: +Fetchers store their output in the nix store, or cache, using Nix's [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs). Later, Nix can reuse the downloaded contents via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: ```nix fetchurl { From 6a9987a0951e2657fe57787bc2d5aafcba4b1ac9 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Mon, 4 Jul 2022 14:39:11 -0600 Subject: [PATCH 03/11] Updated wording from code review --- doc/builders/fetchers.chapter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index 3ed3c103d750..a0988f3c4f98 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -4,7 +4,7 @@ Nixpkgs provides fetchers for different protocols and services. Fetchers are fun ## Caveats -Fetchers store their output in the nix store, or cache, using Nix's [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs). Later, Nix can reuse the downloaded contents via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: +Fetchers create [_fixed output derivations_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) from downloaded files. Later, Nix can reuse the downloaded files via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: ```nix fetchurl { @@ -33,7 +33,7 @@ fetchurl { Hash mismatches generate an error message containing the correct sha256. -A similar problem arises when changing a fetcher's implementation. Changes may cause a fixed output derivation to fail. Failures can go undetected if the output of the derivation already exists in the nix store or cache. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent this. It uses the derivation hash in the output's name to invalidate the cache. +A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the nix store or cache, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. ## `fetchurl` and `fetchzip` {#fetchurl} From f25dd50085bb7d76550e092546d92c3bafa5b0f8 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Mon, 18 Jul 2022 11:26:51 -0600 Subject: [PATCH 04/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Robert Hensing --- doc/builders/fetchers.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index a0988f3c4f98..ce182b1c264d 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -4,7 +4,7 @@ Nixpkgs provides fetchers for different protocols and services. Fetchers are fun ## Caveats -Fetchers create [_fixed output derivations_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) from downloaded files. Later, Nix can reuse the downloaded files via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: +Fetchers create [_fixed output derivations_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) that downloaded files. Later, Nix can reuse the downloaded files via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: ```nix fetchurl { From 53ffebbe1ff7985875796b32fc2f0432a5ec0464 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:50:05 -0600 Subject: [PATCH 05/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Valentin Gagarin --- doc/builders/fetchers.chapter.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index ce182b1c264d..e97d6342461f 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -1,6 +1,7 @@ # Fetchers {#chap-pkgs-fetchers} -Nixpkgs provides fetchers for different protocols and services. Fetchers are functions that simplify downloading files. +Building software with Nix often requires downloading source code and other files from the internet. +`nixpkgs` provides *fetchers* for different protocols and services. Fetchers are functions that simplify downloading files. ## Caveats From 0444078dabdef0a4fdb64e9a607cf48a431c598c Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:50:22 -0600 Subject: [PATCH 06/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Valentin Gagarin --- doc/builders/fetchers.chapter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index e97d6342461f..e52d01cbb8eb 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -34,7 +34,7 @@ fetchurl { Hash mismatches generate an error message containing the correct sha256. -A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the nix store or cache, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. +A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. ## `fetchurl` and `fetchzip` {#fetchurl} From 6d0bf6ae0549c37caae3f337219e622701b30402 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:51:09 -0600 Subject: [PATCH 07/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Valentin Gagarin --- doc/builders/fetchers.chapter.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index e52d01cbb8eb..ce6c214259a3 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -32,7 +32,9 @@ fetchurl { }; ``` -Hash mismatches generate an error message containing the correct sha256. +Use the resulting error message to determine the correct hash. + + A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations. From 0685e4a92984b7831a4256cc7e5463668f20e45c Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:51:38 -0600 Subject: [PATCH 08/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Valentin Gagarin --- doc/builders/fetchers.chapter.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index ce6c214259a3..26b70b1fde2b 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -23,7 +23,8 @@ fetchurl { }; ``` -**This will reuse the old contents**. Remember to invalidate the hash argument by setting the sha256 to an empty string. +**This will reuse the old contents**. +Remember to invalidate the hash argument, in this case by setting the `sha256` attribute to an empty string. ```nix fetchurl { From 0e91d53cc541bc4676e993f4d841cab1457402f9 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:52:08 -0600 Subject: [PATCH 09/11] Update doc/builders/fetchers.chapter.md Co-authored-by: Valentin Gagarin --- doc/builders/fetchers.chapter.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index 26b70b1fde2b..3d89410f9723 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -5,7 +5,11 @@ Building software with Nix often requires downloading source code and other file ## Caveats -Fetchers create [_fixed output derivations_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) that downloaded files. Later, Nix can reuse the downloaded files via their hash. While the caching improves performance, it can lead to some confusion. For example, consider the following fetcher: +Fetchers create [fixed output derivations](https://nixos.org/manual/nix/stable/#fixed-output-drvs) from downloaded files. +Nix can reuse the downloaded files via the hash of the resulting derivation. + +The fact that the hash belongs to the Nix derivation output and not the file itself can lead to confusion. +For example, consider the following fetcher: ```nix fetchurl { From 9c292dd02ee67c47b15789a5709d9a544ed392b7 Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 10:54:02 -0600 Subject: [PATCH 10/11] Updating sha256 to more realistic example --- doc/builders/fetchers.chapter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index 3d89410f9723..f4c81d4d269d 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -14,7 +14,7 @@ For example, consider the following fetcher: ```nix fetchurl { url = "http://www.example.org/hello-1.0.tar.gz"; - sha256 = "1111111111111111111111111111111111111111111111111111"; + sha256 = "0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm"; }; ``` @@ -23,7 +23,7 @@ A common mistake is to update a fetcher’s URL, or a version parameter, without ```nix fetchurl { url = "http://www.example.org/hello-1.1.tar.gz"; - sha256 = "1111111111111111111111111111111111111111111111111111"; + sha256 = "0v6r3wwnsk5pdjr188nip3pjgn1jrn5pc5ajpcfy6had6b3v4dwm"; }; ``` From 3c2cd351d49041373a74897472e905b38697489f Mon Sep 17 00:00:00 2001 From: Cody Hiar Date: Tue, 2 Aug 2022 11:07:36 -0600 Subject: [PATCH 11/11] Add example error message. --- doc/builders/fetchers.chapter.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/builders/fetchers.chapter.md b/doc/builders/fetchers.chapter.md index f4c81d4d269d..947afe8e9fdb 100644 --- a/doc/builders/fetchers.chapter.md +++ b/doc/builders/fetchers.chapter.md @@ -39,7 +39,11 @@ fetchurl { Use the resulting error message to determine the correct hash. - +``` +error: hash mismatch in fixed-output derivation '/path/to/my.drv': + specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= + got: sha256-RApQUm78dswhBLC/rfU9y0u6pSAzHceIJqgmetRD24E= +``` A similar problem arises while testing changes to a fetcher's implementation. If the output of the derivation already exists in the Nix store, test failures can go undetected. The [`invalidateFetcherByDrvHash`](#tester-invalidateFetcherByDrvHash) function helps prevent reusing cached derivations.