The streamLayeredImage wrapper invokes the image streamer at build time on
the build platform. It was built with makeWrapper, which bakes
targetPackages.runtimeShell into the shebang. Under cross compilation that
is the host shell, so the wrapper is not executable on the builder and
crashes (e.g. a Linux image built on a Darwin builder segfaults the
streamer).
This was fixed by switching to pkgsBuildBuild.makeWrapper (build == host == target)
so the shebang points at the build-platform shell.
The nobody/nogroup pair of should only ever be used by NFS to map remote
uid/guids that are missing locally. Every other use is an abuse and
generally a security concern.
Specifically, they're NOT some special user/group that can be used to
run programs without giving them any privileges. In fact, the opposite
is true: the more programs you run as nobody/nogroup the more valuable
files and resources become accessible to "nobody" and consequently
exposed to a potentially compromised program.
This removes mentions of nobody/nogroup from documentation and examples.
dockerTools streamNixShellImage implements a facsimile of the
environment created by the Nix build process. In order to keep track of
the Nix features implemented, the streamNixShellImage code has comments
containing hyperlinks indicating the corresponding Nix C++ code in the
GitHub repository.
As Nix 2.32.0 has now been released [1], this patch updates the GitHub
links to correspond to the code in the new version.
[1] https://discourse.nixos.org/t/nix-2-32-0-released/70528
This broke with #417132 where the previous behaviour of suppressing
errors of the export into $NIX_BUILD_TOP/env-vars when dumping env vars.
This error seems to stem from the non-existance of $NIX_BUILD_TOP. This
fix ensures the NIX_BUILD_TOP propagates into the proot environment.
Fixes build failure for nixos.tests.docker-tools.
The tar command below the changed line relied on the $out directory
being created but it is not being created after the commit below.
97ed6b4565
When building with structuredAttrs, two things would happen:
- sourcing stdenv would fail, because $stdenv is not set.
- the error is swallowed because bash is called without -e.
Thus, the build appears to be successful at first, but could fail later
when anything depends on a sourced stdenv.
Allow customisation of the algorithm used to convert nix references
graph (created from docker image contents) to docker layers.
A collection of building blocks (python functions) is provided, which
use can assembled into a processing pipeline by specifying a list of
operations (and their initial arguments) via a nix list.
Nix references graph if first converted into a python igraph.Graph
object (with each vertex representing a nix path), which is then fed
into the user defined pipeline. Each stage in the pipeline represents a
function call, with initial arguments specified by the user in nix, and
the last argument being the result of the previous stage in the pipeline
(or the initial Graph object). Each step of the pipeline is expected to
produce a data structure consisting of arbitrarily nested lists/dicts
with Graph objects (representing docker layers) at it's leafs. The
result of the last stage in the pipeline is recursively flattened (with
each dict converted into list of values), until a flat list of Graphs
remains. This is then output as a json array of arrays (each Graph
converted into an array of paths).
This functionality is made available via new `layeringPipeline` argument
to the `streamLayeredImage`/`buildLayeredImage` functions. The default
value of the argument has been chosen to to preserve current layering
behaviour.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Setting the image creation timestamp in the image metadata to a
constant date can cause problems with self-hosted container
registries, that need to e.g. prune old images. This timestamp is
also useful for debugging.
However, it is almost never useful to set the filesystem timestamp to
a constant value. Doing so not only causes the image to possibly no
longer be reproducible, but also removes any possibility of
deduplicating layers with other images, causing unnecessary storage
space usage.
Therefore, this commit introduces "mtime", a new parameter to
streamLayeredImage, which allows specifying the filesystem timestamps
separately from "created". For backwards compatibility, "mtime"
defaults to the value of "created".
In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
* streamLayeredImage: self-document the script
'podman load' doesn't let me override the name/tag from the image.
This name and tag is dynamic and in a CI environment so I would like to
be able to adjust the tag dynamically.
Since the image is streamed by stream_layered_image.py, there is no need
to stick with the nix-hardcoded image name/tag: the python script can
accept another name
I've added argparse to expose the feature. It has the added benefit of
adding `--help` support to the script, which makes its usage
self-explanatory.
* blackified file
This removes redundant inline docs, because
- users should consult the better docs in the manual,
- contributors should add to the manual, not the inline comments
This is convenient for debugging the underlying streamed image used by
`dockerTools.buildLayeredImage`.
Here's an example of how you might use this:
```console
$ nix repl ./.
nix-repl> dockerTools.examples.nginx.passthru.stream
«derivation /nix/store/9zczmlp2kraszx4ssmh6fawnlnsa5a4n-stream-nginx-container.drv»
```
Jörg Thalheim