npm's locking process resolves Git dependencies to a given commit, separated
by a `#`. Before this change, any encounter of these URLs would cause `fetchGit`
to fail, as they're not valid repo URLs.
Up to now, existing symlinks were replaced by removing the existing link
and then creating a new one. This left a window where the link did not
exist, which would break cases where the hook was running concurrently
with other things using node_modules.
In my case, I would run multiple webpack builds at once like:
nix develop -c webpack build --mode production
nix develop -c webpack build --mode development
If one of the shells was ready faster than the other, webpack would
start running, then the second would run the linkNodeModules hook, and
the earlier webpack build would run into missing files as the second
linkNodeModules hook removed symlinks to replace them.
This change mitigates this race condition in two ways:
- Not replacing the symlink if it already points to the right target
(this also avoids superfluous filesystem writes)
- Replacing the symlink atomically, using a rename,
otherwise. The symlink was replaced atomically prior to
db7050bb88 as well, but the name of the
temporary symlink was always the same, which led to a similar race
condition. This change reintroduces the atomic replacement, but adds
the PID to the temporary filename in order to avoid conflicting with
other instances of the hook running concurrently.
Upstream projects' `package-lock.json` file might include many deps with
no resolved line in them, as described here:
https://github.com/npm/cli/issues/6301
This can cause dependencies to be missing after generating a hash for
`.npmDeps`. If such dependencies are encountered, it's nice to print
them and warn the user that an upstream patch might be needed.
Co-Authored-By: Jörg Thalheim <joerg@thalheim.io>
The dist object in npm packuments contains fields that can change
after a package is published, causing hash mismatches in
fixed-output derivations:
- signatures: changes when npm rotates registry signing keys
(old key SHA256:jl3bwswu80Pjj... expired 2025-01-29)
- npm-signature: legacy format being progressively removed
- attestations: provenance metadata added post-publication
Only keep the three fields npm actually needs during install:
tarball, integrity, and shasum. Also strip the informational-only
fileCount and unpackedSize fields.
This is the same approach already used for top-level and
version-level field whitelisting.
Ref: https://github.com/numtide/llm-agents.nix/issues/2459
The 'deprecated' field can be modified by package maintainers at any time
via 'npm deprecate', even for already-published versions. This causes
hash mismatches when the npm registry adds deprecation notices to packages
that were previously not deprecated.
For lockfile-based installs, the version is already resolved, so the
deprecation hint used by npm-pick-manifest for version selection is not
needed. The only behavioral change is that users won't see deprecation
warnings during 'npm install', which is an acceptable tradeoff for
reproducible builds.
This was discovered when tar@7.5.2 was deprecated upstream, breaking
builds in downstream projects using fetcherVersion=2.
Switch from stripping known volatile fields to using an explicit whitelist
of allowed fields. This is more robust against upstream changes that add
new fields which could affect hash stability.
Top-level: only name and versions (dist-tags and time not needed for
lockfile installs where versions are already resolved)
Version-level: identity, all dependency types, dist, bin, platform
constraints (engines/os/cpu), scripts, and deprecated flag.
Based on analysis of pacote, npm-pick-manifest, npm-install-checks, and
arborist - only fields actually read during npm install are included.
Fixed-output derivations for npm dependencies fail with hash mismatches
when rebuilt on different days. The root cause is that npm registry
metadata (packuments) contain volatile fields that change whenever
upstream publishes a new version.
For example, the TypeScript packument changes daily due to nightly
releases, even though the lockfile pins a specific version. The cached
packument includes _rev, time, modified, and the full versions list -
all of which drift over time.
Strip these volatile fields and filter the versions map to only include
versions actually referenced in the lockfile.
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
npm lockfiles can contain package aliases where the lockfile key differs
from the actual package name (e.g., "string-width-cjs" aliasing
"string-width"). Previously we always used the lockfile key, causing us
to fetch packuments for the wrong package.
Use the package's own "name" field when present, falling back to the
lockfile key. This ensures we fetch the correct packument for aliased
packages, fixing non-deterministic builds where the wrong packument
fetch could succeed or fail depending on network timing.
The current approach parses tarball URLs to extract package names for
packument fetching. This is fragile as it only handles npmjs.org URLs
and requires special-casing other registries.
Use lockfile keys directly instead. The lockfile already contains the
canonical package names in the form "node_modules/@scope/name", so we
can simply strip the prefix rather than parsing URLs.
This handles all registries uniformly and eliminates the URL parsing
code along with its tests.
