This hook is called in the fixupPhase via
```
local output
for output in $(getAllOutputNames); do
prefix="${!output}" runHook fixupOutput
done
```
Without `__structuredAttrs`, `getAllOutputNames` returns the `output` array, in order.
However, with `__structuredAttrs`, it returns the keys of the `output` associative array,
which are no longer necessarily ordered in the same way.
In the case of some packages (e.g. `mate-panel-with-applets`) this means that instead of
`[ "out" "man" ]`, we process `[ "man" "out" ]`. Running the hook for `"man"` then sets
`wrapGAppsHookHasRun` and no wrapping is done for `"out"`, which is what was really needed.
Instead, keep track of whether the hook has run on a per-output basis. That way, the order
does not matter and any executables that are spread around multiple outputs are wrapped.
This commit was created by a combination of scripts and tools:
- an ast-grep script to prefix things in meta with `lib.`,
- a modified nixf-diagnose / nixf combination to remove unused `with
lib;`, and
- regular nixfmt.
Co-authored-by: Wolfgang Walther <walther@technowledgy.de>
This optimizes the performance of the hook on machines with multiple cores.
The previous implementation was slow, as it was launching two extra processes per symlink (readlink and ln) while not making use of multiprocessing.
The following improvements were made:
- don't even execute the hook if dontRewriteSymlinks is set
- replace multiple find calls with single find call over all outputs
- use xargs -P to process symlinks in parallel
- add test to check that the hook operates as expected
without `-n`, gzip leaks the file timestamp into the compressed file,
which is likely to leak the build timestamp into the output.
This fixes#434930, a regression introduced in c5252e1 / #406922
otherwise systemd fails to build because autoPatchelfHook patches libs
with debug objects and this creates circular dependencies from out to
debug (of course, debug depends on out)
example of faulty behavior:
searching for dependencies of /nix/store/c8cd7b82py02f0rkags351nhg82wwjm6-systemd-minimal-257.5/bin/systemd-delta
libsystemd-shared-257.so -> found: /nix/store/icpqrawjhsw4fbi4i2hp7cxvf3kbzq7m-systemd-minimal-257.5-debug/lib/debug
libc.so.6 -> found: /nix/store/7lyblga0bzjk0sl93vp4aiwbvb57vp2x-glibc-2.40-66/lib
A debuginfod support must be able to map a build-id to
- debug symbols
- the original elf file for which the debug symbols where separated
- the corresponding source files
Currently, hydra provides an index from build-id to the nar of the debug
output containing the debug symbols.
Add symlinks in these outputs so that we can recover the store path of
the source and original elf file. We can then fetch them by the normal
binary cache protocol.
About source files: to minimize storage demands, in the ideal case,
software would be built from the source store path $src and the
debuginfod server would just have to serve source files from this store
path. In practice, source files are sometimes patched as part of the
build. This commit stores the modified files in the debug output is a so
called source overlay so that the debuginfod serve can serve the patched
content of the file.
The checksum was chosen as follows (where big is 4GB of zeros):
$ hyperfine -L s sysv,bsd,crc,sha1,sha224,sha256,sha384,sha512,blake2b,sm3 'cksum -a {s} big'
Benchmark 1: cksum -a sysv big
Time (mean ± σ): 854.5 ms ± 270.5 ms [User: 245.3 ms, System: 601.8 ms]
Range (min … max): 760.5 ms … 1623.8 ms 10 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (1.624 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
Benchmark 2: cksum -a bsd big
Time (mean ± σ): 5.838 s ± 0.045 s [User: 5.118 s, System: 0.693 s]
Range (min … max): 5.767 s … 5.897 s 10 runs
Benchmark 3: cksum -a crc big
Time (mean ± σ): 829.9 ms ± 28.6 ms [User: 274.5 ms, System: 551.0 ms]
Range (min … max): 803.2 ms … 904.8 ms 10 runs
Benchmark 4: cksum -a sha1 big
Time (mean ± σ): 2.553 s ± 0.010 s [User: 1.912 s, System: 0.631 s]
Range (min … max): 2.543 s … 2.575 s 10 runs
Benchmark 5: cksum -a sha224 big
Time (mean ± σ): 2.716 s ± 0.018 s [User: 2.054 s, System: 0.645 s]
Range (min … max): 2.695 s … 2.743 s 10 runs
Benchmark 6: cksum -a sha256 big
Time (mean ± σ): 2.751 s ± 0.029 s [User: 2.057 s, System: 0.674 s]
Range (min … max): 2.712 s … 2.812 s 10 runs
Benchmark 7: cksum -a sha384 big
Time (mean ± σ): 5.600 s ± 0.049 s [User: 4.820 s, System: 0.753 s]
Range (min … max): 5.515 s … 5.683 s 10 runs
Benchmark 8: cksum -a sha512 big
Time (mean ± σ): 5.543 s ± 0.021 s [User: 4.751 s, System: 0.768 s]
Range (min … max): 5.523 s … 5.579 s 10 runs
Benchmark 9: cksum -a blake2b big
Time (mean ± σ): 5.091 s ± 0.025 s [User: 4.306 s, System: 0.764 s]
Range (min … max): 5.048 s … 5.125 s 10 runs
Benchmark 10: cksum -a sm3 big
Time (mean ± σ): 14.220 s ± 0.120 s [User: 13.376 s, System: 0.783 s]
Range (min … max): 14.077 s … 14.497 s 10 runs
Summary
cksum -a crc big ran
1.03 ± 0.33 times faster than cksum -a sysv big
3.08 ± 0.11 times faster than cksum -a sha1 big
3.27 ± 0.11 times faster than cksum -a sha224 big
3.31 ± 0.12 times faster than cksum -a sha256 big
6.13 ± 0.21 times faster than cksum -a blake2b big
6.68 ± 0.23 times faster than cksum -a sha512 big
6.75 ± 0.24 times faster than cksum -a sha384 big
7.03 ± 0.25 times faster than cksum -a bsd big
17.13 ± 0.61 times faster than cksum -a sm3 big
unfortunately, crc (and sysv) are not supported by --check, so they are
disqualified. sha1 sha224 and sha256 are sensibly as fast as one
another, so let's use a non broken one, even though cryptographic
qualities are not needed here.
The no-broken-symlinks hook does not fail if bad links exist out of
store, but /build is also a special directory for nix derivations: the
build directory in the builder mount namespace.
There should be no link to /build in the output derivation, so also
error on these directories (through $TMPDIR which default to that)
Closes#410508
Arne Keller