pkgs-lib/formats: use yj instead of remarshal for toml.generate
`remarshal` is a Python application; evaluating its derivation forces a
large slice of `python3Packages` (rich, ruamel-yaml, cbor2, tomlkit and
their transitive build/check inputs — ~150 distinct python-modules,
~2700 derivations). 95 NixOS modules use `formats.toml`, so any system
enabling one of those services pays this on every `nixos-rebuild`.
`yj` is a small Go binary that does the same JSON→TOML conversion. For
all values accepted by `(formats.toml{}).type` the output is
semantically identical (verified via `tomllib`).
Reproducer (best of 3, `NIX_SHOW_STATS=1 nix-instantiate -E
'(pkgs.formats.toml{}).generate "x" {a.b=1;}'`):
cpuTime nrFunctionCalls nrThunks drv-closure
before (remarshal) 0.69s 634,117 1,184,499 2722 paths
after (yj) 0.36s 271,308 566,981 1481 paths
-49% -57% -52% -46%
Runtime closure of the converter: remarshal 202.3 MiB → yj 5.2 MiB.
Differences in output:
- yj emits intermediate table headers for deeply-nested keys (`[a]`
`[a.b]` `[a.b.c]` rather than just `[a.b.c]`). Valid TOML, parses
identically.
- yj renders very small/large floats as decimals rather than scientific
notation (`1e-300` → 300-digit literal). Valid TOML, parses
identically.
- Direct callers of `generate` that pass `null` (which the format type
already rejects) get key-dropped instead of a build-time error.
NixOS modules are unaffected since the option type filters this at
eval time; several modules already `filterAttrs (v: v != null)` for
exactly this reason.
Tested:
- `nix-build pkgs/pkgs-lib/tests -A formats` passes
- arrays of tables, nested tables, string escaping, int64, unicode
keys/values, mixed scalar+table siblings round-trip through `tomllib`
identically to remarshal output
This commit is contained in:
@@ -461,16 +461,16 @@ optionalAttrs allowAliases aliases
|
||||
generate =
|
||||
name: value:
|
||||
pkgs.callPackage (
|
||||
{ runCommand, remarshal }:
|
||||
{ runCommand, yj }:
|
||||
runCommand name
|
||||
{
|
||||
nativeBuildInputs = [ remarshal ];
|
||||
nativeBuildInputs = [ yj ];
|
||||
value = builtins.toJSON value;
|
||||
passAsFile = [ "value" ];
|
||||
preferLocalBuild = true;
|
||||
}
|
||||
''
|
||||
json2toml "$valuePath" "$out"
|
||||
yj -jt < "$valuePath" > "$out"
|
||||
''
|
||||
) { };
|
||||
|
||||
|
||||
@@ -701,6 +701,8 @@ runBuildTests {
|
||||
[attrs]
|
||||
foo = "foo"
|
||||
|
||||
[level1]
|
||||
[level1.level2]
|
||||
[level1.level2.level3]
|
||||
level4 = "deep"
|
||||
'';
|
||||
|
||||
Reference in New Issue
Block a user