rocmPackages needs a clang at runtime but has no need to include
clang-tidy.
The approach in this change avoids rebuilds and is messy, I will
make a followup in staging that changes this to more idiomatic
optionalString usage if this is merged
When Clang is statically linked against other programs they are unable to
find the headers in Clang's resource directory. Typically the resource
directory is found by searching a path relative to argv[0] but this
would only really work for Clang itself due to each binary having a
separate prefix (and not in Nix because of the full resource directory
being split between multiple derivations and assembled in the wrapper).
Because users of Clang as a library typically only need include in order
for parsing to succeed, let's set that as the resource directory. The LLVM
patch to make this work was sent upstream in llvm/llvm-project#145996,
I intend to land it upstream and drop it from this PR.
This increases the memory required to link by quite a bit, but it
should be fine for Hydra, and people building themselves with
insufficient RAM can either disable it, or just decrease --cores.
Mark these packages as big-parallel to match llvmPackages.llvm.
They are chosen because they are relatively expensive to build, as
opposed to others where I currently believe they are less of an issue
from a CPU time/memory standpoint.
I suspect most people don't care about big-parallel or it would have
been noticed earlier. I have a use case where I am building many
variants of clang all at once and I want to avoid OOM by constraining
the number of parallel builds of items marked big-parallel.
Signed-off-by: Peter Waller <p@pwaller.net>
Rationale: --replace is deprecated and emits warnings. If these
replacements fail it is probably better to know about it early and come
up with better fixes than the alternative of letting them silently fail
and discover some distal brokenness.
Signed-off-by: Peter Waller <p@pwaller.net>
Background:
LLVM has some tools that run at build time. In native builds, these are
built as a part of the usual build, but in cross builds they need to
come from buildPackages.
In many scenarios this is a small problem because LLVM from
buildPackages is already available as a build; but if cross building a
version of LLVM which is not available (e.g. a new git commit of LLVM)
this results in two builds of LLVM and clang, one native and one for the
cross.
Full builds of LLVM are expensive; and unnecessary in this scenario. We
don't need a native LLVM, only a native copy of the tools which run at
build time. This is only tablegen and related tooling, which are cheap
to build.
Implementation-wise, we introduce a derivation llvmPackages.tblgen,
which specifies the tablegen targets which need to be built and has a
custom installPhase to copy them to the output.
A previous attempt in https://github.com/NixOS/nixpkgs/pull/359967
dropped the use of LLVM_TABLEGEN_EXE and friends on the grounds that
llvm can already cross build these things, but that is false since it's
necessary in that case to wire in a cross compiler. This PR avoids that
problem by allowing access to buildPackages.tblgen.
Signed-off-by: Peter Waller <p@pwaller.net>
Seems like the patch we've written for LLVM 13 and above also works for
LLVM 12 which seems a little more robust.
(cherry picked from commit 37437849de)
(cherry picked from commit f30f8ee969e6a395da091e7a8053e779b39686ff)
As mentioned in #305146, keeping LLVM 12 is a source of pain because it
is the only version to be built from individual release tarball instead
of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo
as well, simplifying all common LLVM expressions in the process.
With #347887, some quirks in the expressions for LLVM <14 were ironed
out, so building LLVM through from the monorepo is quite simple now.
- Most expressions only required minor changes, mostly removing the
special casing for `sourceRoot`.
- The patch lists from llvm/12/default.nix were ported to
common/default.nix. This only required a few extra conditionals which
could be reduced via a rebuild also involving other LLVM versions.
Outstanding tasks of little urgency have been noted in TODO comments.
I have verified that the patch lists stay the same for all packages
except LLVM where merely the order changes. An extra set of eyes
is appreciated, of course.
- clang: The expression was reworked to use the same symlink location
for clang-tools-extra for all versions including LLVM 12. This
required adjusting the ad hoc patching of the clangd cmake files
slightly.
- libunwind: We no longer need to make the libcxx sources available
manually. We can rely on the monorepo source instead.
- lld: We no longer need to make the libunwind sources available manually.
- llvm: We no longer need to make the polly sources available manually
- On Darwin, we need to bypass CMake's C++ compiler for libcxx and
libunwind now. It isn't a 100% clear why, probably because we've
started to use Darwin's bootstrapStdenv for libcxx in the common
expression compared to LLVM 12 on master [1].
The reordering of flags for wasm causes a rebuild for some packages
like firefox, but this should be tolerable on staging.
[1]: 665ebfb253/pkgs/development/compilers/llvm/12/default.nix (L392-L430)
(cherry picked from commit ee9eacf23e)
(cherry picked from commit c4e9f17ef944bb7ce0d3d1226d7f922f28e46b02)
apply
5b77e752dc
to clang 14-16 and backport the patch to apply to clang 12-13
backported patch just drops the final change in Hunk 2 of Options.td as
it doesn't exist until clang-14
Remove a patch to decrease the divergence from upstream.
Motivation is a continuation of trying to reduce the number of clang
builds floating around. Wrapper builds are cheap, compiler builds are
not.
It's expected that this patch can be removed at some point and replaced
with some other cc-wrapper logic. This will be a little bit easier with
the logic already in the cc-wrapper.
Ongoing discussion in #191152. It would be nice to drop this but we
think it's probably needed at least on systems where /usr/include may be
available to prevent the possibility of picking up unwanted includes.
One potential method for eliminating nostdlibinc in the future is by
providing a non-existent -sysroot but that was reverted in #213185, and
remains a can of worms.
Signed-off-by: Peter Waller <p@pwaller.net>
Also exposes owner, repo, and other attributes available on the original
monorepoSrc. These attributes were previously hidden behind a
runCommand that consumes monorepoSrc and hides this information away.
```
$ nix eval --raw .#llvmPackages_18.llvm.src.rev
llvmorg-18.1.8
$ nix eval --raw .#llvmPackages_git.llvm.src.rev
0e8555d4dbfdfeddc01dc2ecf9a9b6e804f7b645
```
Signed-off-by: Peter Waller <p@pwaller.net>
As mentioned in #305146, keeping LLVM 12 is a source of pain because it
is the only version to be built from individual release tarball instead
of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo
as well, simplifying all common LLVM expressions in the process.
With #347887, some quirks in the expressions for LLVM <14 were ironed
out, so building LLVM through from the monorepo is quite simple now.
- Most expressions only required minor changes, mostly removing the
special casing for `sourceRoot`.
- The patch lists from llvm/12/default.nix were ported to
common/default.nix. This only required a few extra conditionals which
could be reduced via a rebuild also involving other LLVM versions.
Outstanding tasks of little urgency have been noted in TODO comments.
I have verified that the patch lists stay the same for all packages
except LLVM where merely the order changes. An extra set of eyes
is appreciated, of course.
- clang: The expression was reworked to use the same symlink location
for clang-tools-extra for all versions including LLVM 12. This
required adjusting the ad hoc patching of the clangd cmake files
slightly.
- libunwind: We no longer need to make the libcxx sources available
manually. We can rely on the monorepo source instead.
- lld: We no longer need to make the libunwind sources available manually.
- llvm: We no longer need to make the polly sources available manually
- On Darwin, we need to bypass CMake's C++ compiler for libcxx and
libunwind now. It isn't a 100% clear why, probably because we've
started to use Darwin's bootstrapStdenv for libcxx in the common
expression compared to LLVM 12 on master [1].
The reordering of flags for wasm causes a rebuild for some packages
like firefox, but this should be tolerable on staging.
[1]: 665ebfb253/pkgs/development/compilers/llvm/12/default.nix (L392-L430)
This is better handled in the cc-wrapper, and makes it possible to avoid
rebuilding clang in some scenarios.
It also appears to be unnecessary since the cc-wrapper already passes
-stdlib=libc++ where needed.
See: 8885a1e21a/pkgs/build-support/cc-wrapper/default.nix (L603)
Signed-off-by: Peter Waller <p@pwaller.net>
This change implements a leftover task from #307211, namely passing
monorepoSrc to the different llvmPackages_13 package expressions. Before
this change, all packages llvmPackages_13 would be built from a
subdirectory of the full LLVM monorepo tree. After this change only the
relevant directories are made available at build time. This
- reduces the size of the source that needs to be made available to the
builder.
- prevents LLVM from sidestepping our instructions and including extra
sources from other directories it shouldn't.
Since LLVM 12 and 13 don't have the `cmake` directory at the top level,
the runCommand expressions filtering the source need to be adjusted, but
this causes no rebuild for any other LLVM version (ofborg should confirm
this).
The only problem encountered was in lld:
- We need to make the patch to the inclusion of libunwind headers
unconditional now. lld needs this on non-darwin as well. In the
full monorepo, LLVM_MAIN_SRC_DIR would be set correctly, so the
patch wasn't necessary.
- The substitute mechanism for LLVM 12 and 13 can't be unified yet since
LLVM 12 still uses a non monorepo build, so we come up with a
different LLVM_MAIN_SRC_DIR.
Change was tested by building the following expression on x86_64-linux.
with import ./. {};
builtins.removeAttrs llvmPackages_13 [ "lldb" "lldbPlugins" ]'
lld was also tested on aarch64-darwin.
cmake flags have a 'last flag wins' logic, so by appending to the end of
the flags it is possible to override any cmake flag.
It also ignores (and warns) if a flag is unused, so passing flags across
all packages should be safe if you want to target one package.
In combination with #320261, this PR allows consistently overriding all
packages within LLVM with additional cmake arguments. Consistency here
means for example 'if you override LLVM, then all dependencies on it are
also see the overridden LLVM in their input'. Consistency is hard to
achieve with the other obvious way of implementing this as a user: if
you use overrideAttrs then you have to write a big mess of override code
in order to override all dependents, and this can be very difficult in a
cross-compilation scenario using crossSystem and useLLVM, for example.
With this PR it is possible to write an overlay which overlays
`llvmPackages` with `llvmPackage.override { devExtraCmakeFlags = [ ... ]; }`,
and then the toolchain used with useLLVM in effect should respect
these flags.
This is useful in development for experimenting with the effect of
various flags, hence the chosen name `devCmakeFlags`.
This won't work out of the box without #341855 applied, which fixes
override passthrough.
See-Also: #320261, #341855
Signed-off-by: Peter Waller <p@pwaller.net>
