pkgs-lib/formats: use yj instead of remarshal for toml.generate

`remarshal` is a Python application; evaluating its derivation forces a
large slice of `python3Packages` (rich, ruamel-yaml, cbor2, tomlkit and
their transitive build/check inputs — ~150 distinct python-modules,
~2700 derivations). 95 NixOS modules use `formats.toml`, so any system
enabling one of those services pays this on every `nixos-rebuild`.

`yj` is a small Go binary that does the same JSON→TOML conversion. For
all values accepted by `(formats.toml{}).type` the output is
semantically identical (verified via `tomllib`).

Reproducer (best of 3, `NIX_SHOW_STATS=1 nix-instantiate -E
'(pkgs.formats.toml{}).generate "x" {a.b=1;}'`):

                        cpuTime  nrFunctionCalls   nrThunks  drv-closure
    before (remarshal)    0.69s          634,117  1,184,499   2722 paths
    after  (yj)           0.36s          271,308    566,981   1481 paths
                           -49%             -57%       -52%         -46%

Runtime closure of the converter: remarshal 202.3 MiB → yj 5.2 MiB.

Differences in output:
- yj emits intermediate table headers for deeply-nested keys (`[a]`
  `[a.b]` `[a.b.c]` rather than just `[a.b.c]`). Valid TOML, parses
  identically.
- yj renders very small/large floats as decimals rather than scientific
  notation (`1e-300` → 300-digit literal). Valid TOML, parses
  identically.
- Direct callers of `generate` that pass `null` (which the format type
  already rejects) get key-dropped instead of a build-time error.
  NixOS modules are unaffected since the option type filters this at
  eval time; several modules already `filterAttrs (v: v != null)` for
  exactly this reason.

Tested:
- `nix-build pkgs/pkgs-lib/tests -A formats` passes
- arrays of tables, nested tables, string escaping, int64, unicode
  keys/values, mixed scalar+table siblings round-trip through `tomllib`
  identically to remarshal output
This commit is contained in:
zimbatm
2026-04-16 14:43:33 +02:00
parent 34cb0d6acb
commit 7b2c5ede9a
2 changed files with 5 additions and 3 deletions
+3 -3
View File
@@ -461,16 +461,16 @@ optionalAttrs allowAliases aliases
generate =
name: value:
pkgs.callPackage (
{ runCommand, remarshal }:
{ runCommand, yj }:
runCommand name
{
nativeBuildInputs = [ remarshal ];
nativeBuildInputs = [ yj ];
value = builtins.toJSON value;
passAsFile = [ "value" ];
preferLocalBuild = true;
}
''
json2toml "$valuePath" "$out"
yj -jt < "$valuePath" > "$out"
''
) { };
+2
View File
@@ -701,6 +701,8 @@ runBuildTests {
[attrs]
foo = "foo"
[level1]
[level1.level2]
[level1.level2.level3]
level4 = "deep"
'';