If we introduce version 3, current packages that use the non-existent
version 3 will get their hashes broken. Prevent stupid LLMs from
introducing more trouble when hallucinating more non-existent versions.
Upstream projects' `package-lock.json` file might include many deps with
no resolved line in them, as described here:
https://github.com/npm/cli/issues/6301
This can cause dependencies to be missing after generating a hash for
`.npmDeps`. If such dependencies are encountered, it's nice to print
them and warn the user that an upstream patch might be needed.
Co-Authored-By: Jörg Thalheim <joerg@thalheim.io>
The dist object in npm packuments contains fields that can change
after a package is published, causing hash mismatches in
fixed-output derivations:
- signatures: changes when npm rotates registry signing keys
(old key SHA256:jl3bwswu80Pjj... expired 2025-01-29)
- npm-signature: legacy format being progressively removed
- attestations: provenance metadata added post-publication
Only keep the three fields npm actually needs during install:
tarball, integrity, and shasum. Also strip the informational-only
fileCount and unpackedSize fields.
This is the same approach already used for top-level and
version-level field whitelisting.
Ref: https://github.com/numtide/llm-agents.nix/issues/2459
The 'deprecated' field can be modified by package maintainers at any time
via 'npm deprecate', even for already-published versions. This causes
hash mismatches when the npm registry adds deprecation notices to packages
that were previously not deprecated.
For lockfile-based installs, the version is already resolved, so the
deprecation hint used by npm-pick-manifest for version selection is not
needed. The only behavioral change is that users won't see deprecation
warnings during 'npm install', which is an acceptable tradeoff for
reproducible builds.
This was discovered when tar@7.5.2 was deprecated upstream, breaking
builds in downstream projects using fetcherVersion=2.
Switch from stripping known volatile fields to using an explicit whitelist
of allowed fields. This is more robust against upstream changes that add
new fields which could affect hash stability.
Top-level: only name and versions (dist-tags and time not needed for
lockfile installs where versions are already resolved)
Version-level: identity, all dependency types, dist, bin, platform
constraints (engines/os/cpu), scripts, and deprecated flag.
Based on analysis of pacote, npm-pick-manifest, npm-install-checks, and
arborist - only fields actually read during npm install are included.
Fixed-output derivations for npm dependencies fail with hash mismatches
when rebuilt on different days. The root cause is that npm registry
metadata (packuments) contain volatile fields that change whenever
upstream publishes a new version.
For example, the TypeScript packument changes daily due to nightly
releases, even though the lockfile pins a specific version. The cached
packument includes _rev, time, modified, and the full versions list -
all of which drift over time.
Strip these volatile fields and filter the versions map to only include
versions actually referenced in the lockfile.
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
npm lockfiles can contain package aliases where the lockfile key differs
from the actual package name (e.g., "string-width-cjs" aliasing
"string-width"). Previously we always used the lockfile key, causing us
to fetch packuments for the wrong package.
Use the package's own "name" field when present, falling back to the
lockfile key. This ensures we fetch the correct packument for aliased
packages, fixing non-deterministic builds where the wrong packument
fetch could succeed or fail depending on network timing.
The current approach parses tarball URLs to extract package names for
packument fetching. This is fragile as it only handles npmjs.org URLs
and requires special-casing other registries.
Use lockfile keys directly instead. The lockfile already contains the
canonical package names in the form "node_modules/@scope/name", so we
can simply strip the prefix rather than parsing URLs.
This handles all registries uniformly and eliminates the URL parsing
code along with its tests.
Add a cacheVersion parameter to fetchNpmDeps and npmDepsCacheVersion to
buildNpmPackage. When set to 2, prefetch-npm-deps will also fetch and
cache packuments (package metadata) in addition to tarballs.
npm can request packuments with two different Accept headers:
- corgiDoc: abbreviated metadata (default)
- fullDoc: full metadata (used for workspaces)
npm's cache policy requires headers to match, so we cache both versions.
This is opt-in via cacheVersion to avoid breaking existing hashes.
Set npmDepsCacheVersion = 2 for projects using npm workspaces.
Also fix cacache index format to properly separate multiple entries
with newlines, and update map_cache() to parse multi-line index files.
Marie Ramlow