278 Commits

Author SHA1 Message Date
Philip Taron 5b5c421f52 nix-prefetch-git: also disable legacy auto-gc
Commit 88cfc54552 ("nix-prefetch-git: disable maintenance mode via
environment variables") disabled maintenance.auto (git's new
auto-maintenance) via GIT_CONFIG_*, which propagates to submodules. But
the legacy auto-gc path (gc.auto plus detached gc.autoDetach) is still
enabled, and it keeps repacking .git/modules/<sub>/objects after a nested
submodule reports "checked out", racing the later `rm -rf .git`:

- rm empties objects/pack, the detached gc drops a fresh pack in, and
  rmdir fails with "Directory not empty" (exit 123); or
- removal partially completes, leaving varying .git remnants, yielding a
  non-deterministic NAR hash.

Disable gc.auto/gc.autoDetach alongside maintenance.auto, using the same
GIT_CONFIG_* mechanism so the settings reach every git invocation and its
children (including nested submodule fetches). Fold the repeated
boilerplate into a small git_config_env helper.

The successful output is unchanged (.git is removed either way), so
pinned fetchgit hashes stay valid; this only makes removal deterministic.

Fixes #524215.

Assisted-by: Claude Code (claude-opus-4-8)
2026-06-11 08:25:04 -07:00
Heitor Augusto 6c103ed7ba nix-prefetch-git: disable maintenance mode via environment variables (#525255) 2026-06-04 06:48:05 +00:00
Philip Taron 0c469218f3 build-support: remove assertMsg, throwIf, and throwIfNot usage (#524535) 2026-05-28 18:09:32 +00:00
Pratham Patel 88cfc54552 nix-prefetch-git: disable maintenance mode via environment variables
The commit 011471c7f2 ("fetchgit: disable git maintenance") disabled
git's maintenance mode, but that does not affect git repositories with
submodules.

Switch to using the GIT_CONFIG_* environment variables. This is done in
a way to make the approach more generic, but also kinda not the best
("insane") so that it works with any future change, even with
`impureEnvVars`.

Maybe `git -c maintenance.auto=false` can work but I cannot figure out
how to use that here. I would prefer disabling maintenance mode in Git's
derivation itself but did not find any build-time knob to toggle that.
2026-05-28 22:43:24 +05:30
Philip Taron 7b0f8708ea fetchgit: disable git maintenance (#524430) 2026-05-28 00:07:30 +00:00
Hans Joachim Kliemeck 8478c2384b stdenv: Package URL (PURL) metadata fields 2026-05-26 21:47:54 +02:00
Eman Resu 661fe16b08 build-support/fetchgit: remove assertMsg usage 2026-05-26 14:14:41 -04:00
Adam Dinwoodie 011471c7f2 fetchgit: disable git maintenance
By default, Git commands will trigger background processes to perform
repository maintenance.  As of Git v2.54.0, and in particular as of
452b12c2e0 (builtin/maintenance: use "geometric" strategy by default,
2026-02-24), Git has changed the strategy it uses to decide whether
maintenance needs performing, and as a result will kick off background
maintenance processes much more often.

These background processes will potentially change the contents of the
`.git` directory while `nix-prefetch-git` is deleting it.  This results in
effectively random failures, when files are added to the `.git`
directory during the `rm -rf` operation (causing it to fail) or after
the operation has completed (causing the `.git` directory to be
recreated and resulting in a FOD hash mismatch).

This also causes problems for the determinism of Git's own tests.
They've resolved this in 09505b1115 (t: fix races caused by background
maintenance, 2026-02-24) by having the maintenance processes run in the
foreground.  The discussion in that commit explains they considered just
disabling maintenance entirely, but opted for the foreground option
because disabling maintenance would mean the tests looked even less like
how most Git users work with Git.

`nix-prefetch-git` is nothing like most Git users.  In almost every
case, the `.git` directory will be deleted during or immediately after
the script is called.  The repositories certainly don't benefit from
ongoing maintenance.  To avoid the non-determinism created by the
background maintenance processes, just disable maintenance entirely.

This is done with `git config` after initialising the repository, rather
than `git config --global` earlier in the script, as `git config
--global` will fail if `fetchgit` was called with a `gitConfig`
argument.

It would be possible to set this option with
`GIT_CONFIG_[COUNT|KEY_<n>|VALUE_<n>]` environment variables; I've opted
against that because (a) they're less familiar to most people who might
want to come to edit this script in future and (b) because if someone is
using those values themselves via `impureEnvVars` or similar, making our
configuration play nicely with someone else's would require some fairly
complex logic.

I haven't made this a configurable option: any use of `nix-prefetch-git`
will run with maintenance disabled, as I cannot think of a use case
where it would be useful.  If there is such a use case, it would be
straightforward to extend `nix-prefetch-git` and `fetchgit` to allow
users to change the behaviour.

Fixes #524215.
2026-05-26 13:43:37 +01:00
Colin 348bfd5f6d tests.fetchgit.withGitConfig: fix build 2026-03-02 17:56:54 +00:00
Emily ad90665682 tests.fetchgit: pass system and the existing config 2026-02-19 16:07:50 +00:00
HigherOrderLogic afa98a746b fetchgit: remove unused argument 2026-02-07 02:53:27 +00:00
Yueh-Shun Li 09030960d5 nix-prefetch-git: restore the tag-related clone flags when leaving .git
When leaveDotGit == true,
restore the cloning behaviour before
commit 7e085677f9 ("nix-prefetch-git: dont't fetch tags when deep clone unless leaving .git")
to preserve the fragile hashes of .git sources.

Clean up fetchTagsCompat,
as fetchTagsCompat is no longer needed
after restoring the old tag-fetching flags.
2025-12-24 21:59:46 +08:00
Yueh-Shun Li 62b2dbef68 nix-prefetch-git: Name FETCHED_HEAD "$ref" only for refs/tags/* 2025-12-23 04:50:05 +08:00
Philip Taron cca4ec8008 fetchgit: take postCheckout to allow gathering revision and commit metadata without leaving .git (#465497) 2025-12-17 19:41:13 +00:00
Yueh-Shun Li 89746696de tests.fetchgit: add submodule-revision-count 2025-12-17 01:19:46 +08:00
Yueh-Shun Li 7e085677f9 nix-prefetch-git: dont't fetch tags when deep clone unless leaving .git
Co-authored-by: Adam Dinwoodie <adam@dinwoodie.org>
2025-12-17 01:19:43 +08:00
Yueh-Shun Li d4dd261ab5 nix-prefetch-git: fetch the fetching tag for NIX_PREFETCH_GIT_CHECKOUT_HOOK
Co-authored-by: Adam Dinwoodie <adam@dinwoodie.org>
2025-12-17 01:14:55 +08:00
Yueh-Shun Li 689380e029 fetchurl, fetchgit: use __structuredAttrs = true and pass curlOptsList and sparseCheckout as lists (#464475) 2025-12-11 18:25:29 +00:00
Yueh-Shun Li 4e37d03ba2 fetchgit: reference hash from finalAttrs.hash 2025-12-10 01:07:13 +08:00
Yueh-Shun Li 982cafd8c6 fetchgit: use __structuredAttrs = true 2025-12-09 12:35:42 +08:00
Yueh-Shun Li cbd24f4a8a tests.fetchgit: add simple-tag 2025-12-09 12:27:32 +08:00
Yueh-Shun Li fc295b1e28 fetchgit: take postCheckout to allow collecting revision without leaving .git 2025-12-09 12:27:27 +08:00
Yueh-Shun Li ea1e4d73d8 fetchgit: adjust format for sparseCheckout default
Co-authored-by: Matt Sturgeon <matt@sturgeon.me.uk>
2025-12-08 21:52:45 +08:00
Yueh-Shun Li 5c51547b22 fetchgit: sparseCheckout: reference depending attributes from finalAttrs 2025-12-04 15:28:38 +08:00
Yueh-Shun Li 8c85578602 fetchgit: format expression after eliminating global assertions 2025-12-04 15:26:16 +08:00
Yueh-Shun Li d1275c4ff1 fetchgit: move sparseCheckout type check down to sparseCheckoutText 2025-12-04 15:25:40 +08:00
Yueh-Shun Li dd264228ff fetchgit: default sparseCheckout to null and handle the default internally 2025-12-04 15:25:39 +08:00
Yueh-Shun Li 83b42e34ec fetchgit: keep sparseCheckout a list and stringify as sparseCheckoutText 2025-12-04 15:25:39 +08:00
Yueh-Shun Li ea5b3c6d5d fetchgit: nonConeMode: reference depending attributes from finalAttrs 2025-12-04 15:25:35 +08:00
Yueh-Shun Li 6c5dc5d57f fetchgit: default nonConeMode to null and manage its default internally 2025-12-04 15:22:33 +08:00
Yueh-Shun Li 9fe4e65f3c fetchgit: default argument leaveDotGit to null 2025-11-17 02:09:55 +08:00
Yueh-Shun Li 1d6c7d6650 fetchgit: move assertions down to argument values 2025-11-16 04:04:51 +08:00
Yueh-Shun Li 2946555411 fetchgit: expose getRevWithTag as fetchgit.getRevWithTag 2025-11-03 21:31:04 +08:00
Yueh-Shun Li 21677a6bd8 fetchgit: pass rev as revCustom and make overridable 2025-11-03 21:31:04 +08:00
Yueh-Shun Li dbf76b7c1b fetchgit: pass tag directly and make overridable 2025-11-03 21:31:04 +08:00
Yueh-Shun Li 076076e946 fetchgit: abstract and simplify revWithTag 2025-11-03 21:31:04 +08:00
Yueh-Shun Li 38f61c3f10 fetchgit: take derivationArgs 2025-11-03 21:31:03 +08:00
Yueh-Shun Li aeddd850c6 fetchurl, fetchzip, fetchgit: format Nix expression after lib.extendMkDerivation 2025-10-28 06:41:42 +08:00
Yueh-Shun Li ae5b3bdb05 fetchgit: restructure with lib.extendMkDerivation (not formatted) 2025-10-28 06:38:46 +08:00
Yueh-Shun Li a7023948c0 fetchgit: take passthru 2025-10-28 01:47:34 +08:00
d068328 e47455f51d Revert "stdenv: pURL implementation (#421125)"
This reverts commit 5427115670, reversing
changes made to 88ee2ac331.
2025-10-18 20:41:01 +00:00
Arian van Putten 5427115670 stdenv: pURL implementation (#421125) 2025-10-17 09:30:32 +00:00
Hans Joachim Kliemeck 028af7c17d stdenv: pURL review suggestions - replace merge 2025-10-14 17:02:51 +00:00
Hans Joachim Kliemeck bacccc39a9 stdenv: pURL - fix chaining case (github&submodules using fetchgit) 2025-10-14 14:53:13 +00:00
h0nIg 4e2614fc07 stdenv: pURL implementation 2025-10-03 21:19:09 +00:00
Alexander Bantyev 80d9975bd4 config: add a gitConfig/gitConfigFile option
Adds a `gitConfig` option (and `gitConfigFile`), to set a default
`gitConfigFile` argument for `fetchgit`.
2025-09-29 22:34:54 +02:00
Alexander Bantyev 296399ec68 fetchgit: add a gitConfigFile argument
Passing a `gitConfigFile` argument as a string allows setting a custom
git config (as `$GIT_CONFIG_GLOBAL`). The most obvious use is to
override URLs with custom mirrors, but others are possible too.
2025-09-29 22:34:11 +02:00
Silvan Mosberger 7d75d07d44 nix-prefetch-git: Add regression test for avoiding fetching cached paths 2025-09-24 22:18:32 +04:00
Alexander Bantyev f12f3dfb21 nix-prefetch-git: add --no-add-path argument
Adds a new argument to `nix-prefetch-git`: `--no-add-path`. It disables
adding the path to the store. It is useful when working with a read-only
store.
2025-09-24 22:18:32 +04:00
Philip Taron 754ee2fc40 fetchgit: Add rootDir argument (#427165) 2025-08-10 17:38:43 -07:00