In the past I was very active with Python packaging.
For several years now I was hardly around as maintainer,
so it does not make sense I am listed as a maintainer for
these makes. Looking back, I should have removed myself
as maintainer already much longer ago. Anyway, better late
than never.
It's been a fun ride, and I do intend to occasionally contribute
to Nixpkgs, but not in the same way it once was.
The way we build python environments is subtly broken. A python
environment should be semantically identical to a vanilla Python
installation in, say, /usr/local. The current implementation, however,
differs in two important ways. The first is that it's impossible to use
python packages from the environment in python virtual environments. The
second is that the nix-generated environment appears to be a venv, but
it's not.
This commit changes the way python environments are built:
* When generating wrappers for python executables, we inherit argv[0]
from the wrapper. This causes python to initialize its configuration
in the environment with all the correct paths.
* We remove the sitecustomize.py file from the base python package.
This file was used tweak the python configuration after it was
incorrectly initialized. That's no longer necessary.
The end result is that python environments no longer appear to be venvs,
and behave more like a vanilla python installation. In addition it's
possible to create a venv using an environment and use packages from
both the environment and the venv.
The situation:
Python <3.11: under Linux the abi string is always -gnu*
Python 3.11-3.12: musl is treated as its own abi in the python build system, but when cross-compiling the build host's libc is used for the target abi string. Cross compiling from glibc to musl gives a -gnu* target abi string and vice versa.
Python >=3.13: musl is treated as its own abi, and when cross-compiling the target libc is used for the target abi string
We backport the fix for python 3.11-3.12, since the intermediate state is almost impossible to model in the nix expression
Python always uses "gnu" prefixed ABI names, and this patch handles this by
doing a musl->gnu string replacement.
With pkgsCross.armv7l-hf-multiplatform.pkgsStatic, the previous name was:
_sysconfigdata__linux_arm-linux-gnu
Now, the corrected name is:
_sysconfigdata__linux_arm-linux-gnueabihf
Since python documentation requires Sphinx to build, documentation is built in
separate derivation to avoid hard recursion/bootstrapping questions and plugged
into main derivation. Care is taken that "python3.doc" derivation feels and
looks as close as possible to as if it were main derivation were multi-output
one.
This change does not cause rebuild of python3 itself.
Add a marker file to the python outputs that tells pip and other tooling
following PEP 668 that they should not be installing things system-wide.
This provides better feedback to the user and also potentially avoids issues in
single-user installs where the /nix/store is owned by the user.
For more details, see <https://peps.python.org/pep-0668/>
Here is how it currently looks like:
$ pip install requests
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
The package set hasn't been working for a long time now, due to infinite
recursions, that nobody was going to fix.
The release is going to go EOL in 2023/06 and we don't want to ship it
in NixOS 23.05 anyway.
Frederik Rietdijk