Before, we would run a mapAttrs on every attribute, checking listToValue
on each of them. But listToValue is known ahead of time. Instead, we can
only run that map if listToValue isn't null in the first place.
This expands on https://github.com/NixOS/nixpkgs/pull/498928 that
introduced __structuredAttrs here by actually using data in
`.attrs.json` when it makes sense, instead of relying on environment
variables. This leads to less temporary files, faster execution and
nicer code.
Add two shouldPass tests that catch known yj regressions:
- tomlHeterogeneousArray: arrays mixing scalars and attrsets crash yj
with 'unexpected primitive type' (NixOS/nixpkgs#511970)
- tomlCommaInKey: keys containing commas are silently truncated
because yj stores TOML keys in Go struct tags where commas are
option separators (sclevine/yj#52)
Expected values are taken from go-toml's jsontoml output.
Both tests fail under the current yj backend.
`remarshal` is a Python application; evaluating its derivation forces a
large slice of `python3Packages` (rich, ruamel-yaml, cbor2, tomlkit and
their transitive build/check inputs — ~150 distinct python-modules,
~2700 derivations). 95 NixOS modules use `formats.toml`, so any system
enabling one of those services pays this on every `nixos-rebuild`.
`yj` is a small Go binary that does the same JSON→TOML conversion. For
all values accepted by `(formats.toml{}).type` the output is
semantically identical (verified via `tomllib`).
Reproducer (best of 3, `NIX_SHOW_STATS=1 nix-instantiate -E
'(pkgs.formats.toml{}).generate "x" {a.b=1;}'`):
cpuTime nrFunctionCalls nrThunks drv-closure
before (remarshal) 0.69s 634,117 1,184,499 2722 paths
after (yj) 0.36s 271,308 566,981 1481 paths
-49% -57% -52% -46%
Runtime closure of the converter: remarshal 202.3 MiB → yj 5.2 MiB.
Differences in output:
- yj emits intermediate table headers for deeply-nested keys (`[a]`
`[a.b]` `[a.b.c]` rather than just `[a.b.c]`). Valid TOML, parses
identically.
- yj renders very small/large floats as decimals rather than scientific
notation (`1e-300` → 300-digit literal). Valid TOML, parses
identically.
- Direct callers of `generate` that pass `null` (which the format type
already rejects) get key-dropped instead of a build-time error.
NixOS modules are unaffected since the option type filters this at
eval time; several modules already `filterAttrs (v: v != null)` for
exactly this reason.
Tested:
- `nix-build pkgs/pkgs-lib/tests -A formats` passes
- arrays of tables, nested tables, string escaping, int64, unicode
keys/values, mixed scalar+table siblings round-trip through `tomllib`
identically to remarshal output
isabel