Commit Graph
1005305 Commits
Author SHA1 Message Date
zimbatmandadisbladis edb4b053f5 pkgs.formats.javaProperties: use lib.sources.sourceByGlobs
Uses the cleaner expression
2026-05-26 21:49:00 +12:00
zimbatmandadisbladis 01bdc186ae lib.sources.sourceByGlobs: address review comments 2026-05-26 21:49:00 +12:00
adisbladis 59d55cbaa3 lib.sources.sourceByGlobs: init function
Adds a source filtering function inspired by [doublestar](https://github.com/bmatcuk/doublestar).

This has been in used in a few private repositories since the last ~6 months with success.

- Testing

This was originally tested with the nix-unit testsuite:
```
let
  inherit (import ./internal.nix) mkSourceFilter mkMatcher;
in
{
  mkMatcher = {
    empty = {
      testMatch = {
        expr = mkMatcher "" "" "regular";
        expected = true;
      };

      testNoMatch = {
        expr = mkMatcher "" "foo" "regular";
        expected = false;
      };
    };

    simple = {
      testMatch = {
        expr = mkMatcher "foo" "foo" "regular";
        expected = true;
      };

      testNoMatch = {
        expr = mkMatcher "foo" "bar" "regular";
        expected = false;
      };
    };

    singleStar = {
      testMatch = {
        expr = mkMatcher "*.js" "foo.js" "regular";
        expected = true;
      };

      testNoMatch = {
        expr = mkMatcher "*.js" "foo.py" "regular";
        expected = false;
      };
    };

    doubleStar = {
      testMatch = {
        expr = mkMatcher "foo/**/bar" "foo/baz/bar" "regular";
        expected = true;
      };

      testNoMatch = {
        expr = mkMatcher "foo/**/bar" "foo/bar/baz" "regular";
        expected = false;
      };

      testMultiMatch = {
        expr = mkMatcher "foo/**/bar" "foo/baz/xyz/bar" "regular";
        expected = true;
      };

      testMultiMatchDoubleGlob = {
        expr = mkMatcher "foo/**/**/bar" "foo/baz/xyz/bar" "regular";
        expected = true;
      };

      testInfixMatch = {
        expr = mkMatcher "foo/**/qux/**/bar" "foo/baz/qux/baz/bar" "regular";
        expected = true;
      };

      testInfixNoMatch = {
        expr = mkMatcher "foo/**/xyz/**/bar" "foo/baz/qux/baz/bar" "regular";
        expected = false;
      };

      # Technically a partial match
      testInfixDirMatch = {
        expr = mkMatcher "foo/**/xyz/**/bar" "foo/baz/qux/baz/bar" "directory";
        expected = true;
      };
    };
  };

  mkSourceFilter = {
    testSourceFilter = {
      expr = mkSourceFilter ./fixtures [
        "bar/*.js"
      ] "bar/bar.js" "regular";
      expected = true;
    };
  };
}
```
but it was dropped in this nixpkgs contribution as the structure of nixpkgs lib testing is too primitive to incorp this without more extensive refactoring than I'd like at the momment.

- Performance

It's hard to benchmark this against anything else meaningful except [globsset](https://github.com/pdtpartners/globset), which has a very similar API.

`sourceByGlobs` avoids performance pitfalls by:

  - Using `builtins.filterSource`

      This is more performant than the fileset API.
      The downside compared to the fileset API is that any directory which matches the filter will be added to the build, even if it's empty.

  - Match paths component by component

      By splitting each pattern into a token per / separator.
      This is much faster in Nix than the doublestar algorithm.

- Globset source

```json
{
    "cpuTime": 0.8585879802703857,
    "envs": {
        "bytes": 148252864,
        "elements": 11899843,
        "number": 6631765
    },
    "gc": {
        "heapSize": 402915328,
        "totalBytes": 671288560
    },
    "list": {
        "bytes": 3358664,
        "concats": 28658,
        "elements": 419833
    },
    "nrAvoided": 11562713,
    "nrFunctionCalls": 4816963,
    "nrLookups": 4316209,
    "nrOpUpdateValuesCopied": 5686407,
    "nrOpUpdates": 464060,
    "nrPrimOpCalls": 2966970,
    "nrThunks": 7796186,
    "sets": {
        "bytes": 196404672,
        "elements": 10837802,
        "number": 1437490
    },
    "sizes": {
        "Attr": 16,
        "Bindings": 16,
        "Env": 8,
        "Value": 24
    },
    "symbols": {
        "bytes": 340652,
        "number": 32026
    },
    "values": {
        "bytes": 207367440,
        "number": 8640310
    }
}
```

- Glob-filter source

```json
{
    "cpuTime": 0.3904629945755005,
    "envs": {
        "bytes": 13263440,
        "elements": 1005877,
        "number": 652053
    },
    "gc": {
        "heapSize": 402915328,
        "totalBytes": 146914896
    },
    "list": {
        "bytes": 3032168,
        "concats": 5899,
        "elements": 379021
    },
    "nrAvoided": 1666598,
    "nrFunctionCalls": 484399,
    "nrLookups": 112698,
    "nrOpUpdateValuesCopied": 3432135,
    "nrOpUpdates": 13426,
    "nrPrimOpCalls": 1041954,
    "nrThunks": 1205792,
    "sets": {
        "bytes": 64304800,
        "elements": 3978167,
        "number": 40883
    },
    "sizes": {
        "Attr": 16,
        "Bindings": 16,
        "Env": 8,
        "Value": 24
    },
    "symbols": {
        "bytes": 285306,
        "number": 28864
    },
    "values": {
        "bytes": 42963240,
        "number": 1790135
    }
}
```
2026-05-26 21:48:59 +12:00
Rémi NICOLEandGitHub 60dd3b28f0 knewave: use installFonts (#523796) 2026-05-26 09:41:44 +00:00
Fabian AffolterandGitHub 8c7a7542c1 python3Packages.aiolichess: 1.2.0 -> 1.3.0 (#524375) 2026-05-26 09:39:15 +00:00
Weijia WangandGitHub 5a1bed9680 python3Packages.eigenpy: 3.12.0 -> 3.13.0 (#523923) 2026-05-26 09:39:00 +00:00
Bruno BELANYIandGitHub 71b6c3bf34 upsies: 2026.05.14 -> 2026.05.24 (#524282) 2026-05-26 09:33:10 +00:00
nixpkgs-ci[bot]andGitHub 9f7fdd3a43 gemini-cli: 0.42.0 -> 0.43.0 (#524359) 2026-05-26 09:28:33 +00:00
nixpkgs-ci[bot]andGitHub 4ddabe8496 vivaldi: 8.0.4033.26 -> 8.0.4033.34 (#524344) 2026-05-26 09:28:29 +00:00
nixpkgs-ci[bot]andGitHub df5303e7c0 netgen-vlsi: 1.5.319 -> 1.5.320 (#524313) 2026-05-26 09:28:23 +00:00
nixpkgs-ci[bot]andGitHub 7b62391d9c tsukimi: 0.21.0 -> 26.5.3 (#524300) 2026-05-26 09:28:22 +00:00
nixpkgs-ci[bot]andGitHub 8fc5885235 gefyra: 2.4.3 -> 2.4.4 (#523493) 2026-05-26 09:27:51 +00:00
K900andGitHub 99737c6d9b Revert "scx.cscheds: move env variable into env for structuredAttrs" (#524373) 2026-05-26 09:24:34 +00:00
R. Ryantm e7d163b4e6 python3Packages.aiolichess: 1.2.0 -> 1.3.0 2026-05-26 09:23:57 +00:00
K900andGitHub 520c7a1e49 Revert "scx.cscheds: move env variable into env for structuredAttrs" 2026-05-26 12:19:21 +03:00
Weijia WangandGitHub f44f7788c8 a2ps: 4.15.7 -> 4.15.8 (#523867) 2026-05-26 09:13:58 +00:00
Weijia WangandGitHub c707058ed4 nvc: 1.20.1 -> 1.21.0 (#523380) 2026-05-26 09:13:14 +00:00
Weijia WangandGitHub 6a5d17ec11 sarasa-gothic: 1.0.37 -> 1.0.39 (#523731) 2026-05-26 09:09:08 +00:00
Thiago Kenji OkadaandGitHub 12897ff37a libretro.mame2003-plus: 0-unstable-2026-05-15 -> 0-unstable-2026-05-23 (#524249) 2026-05-26 09:00:49 +00:00
zowoqandGitHub 0b62ee1217 terraform-providers.newrelic_newrelic: 3.87.3 -> 3.90.0 (#524365) 2026-05-26 08:58:45 +00:00
Rémi NICOLEandGitHub 0b336db6e5 gelly: 1.1.2 -> 1.3.0 (#522469) 2026-05-26 08:58:18 +00:00
R. Ryantm 9636984bdc terraform-providers.newrelic_newrelic: 3.87.3 -> 3.90.0 2026-05-26 08:42:32 +00:00
Martin WeineltandGitHub a9e6d52a04 home-assistant-custom-components.browser-mod: 2.13.3 -> 2.13.4 (#524349) 2026-05-26 08:31:34 +00:00
Martin WeineltandGitHub c058380123 gonic: 0.20.1 -> 0.21.0 (#524205) 2026-05-26 08:22:21 +00:00
Gaétan LepageandGitHub eb50837523 python3Packages.jaxtyping: 0.3.9 -> 0.3.10 (#524230) 2026-05-26 08:22:05 +00:00
R. Ryantm 1805404983 gemini-cli: 0.42.0 -> 0.43.0 2026-05-26 08:04:38 +00:00
nixpkgs-ci[bot]andGitHub 7a52b76f31 cargo-deny: 0.19.6 -> 0.19.7 (#524311) 2026-05-26 07:57:16 +00:00
nixpkgs-ci[bot]andGitHub 8bba3ef9b9 codebook: 0.3.39 -> 0.3.40 (#524286) 2026-05-26 07:57:13 +00:00
zowoqandGitHub 1497a40c57 terraform-providers.opentelekomcloud_opentelekomcloud: 1.36.65 -> 1.36.66 (#524351) 2026-05-26 07:55:29 +00:00
zowoqandGitHub d939abc42f fzf: 0.72.0 -> 0.73.1 (#524290) 2026-05-26 07:55:20 +00:00
Thomas GerbetandGitHub 92c689c930 avml: 0.17.0 -> 0.18.0 (#524343) 2026-05-26 07:45:11 +00:00
R. Ryantm bd783e582d terraform-providers.opentelekomcloud_opentelekomcloud: 1.36.65 -> 1.36.66 2026-05-26 07:36:32 +00:00
R. Ryantm 82588ddeba home-assistant-custom-components.browser-mod: 2.13.3 -> 2.13.4 2026-05-26 07:32:21 +00:00
André SilvaandGitHub 0719493e72 qbittorrent-nox: 5.2.0 -> 5.2.1 (#524284) 2026-05-26 07:30:04 +00:00
R. Ryantm 522ef4b01c vivaldi: 8.0.4033.26 -> 8.0.4033.34 2026-05-26 07:17:47 +00:00
R. Ryantm 190c510d99 avml: 0.17.0 -> 0.18.0 2026-05-26 07:09:04 +00:00
Arian van PuttenandGitHub 2e9190cc69 nitrotpm-tools: 1.1.1 -> 1.1.2 (#524273) 2026-05-26 06:50:57 +00:00
Martin WeineltandGitHub 45cf9b5c6d Firefox: 151.0.1 -> 151.0.2 (#524257) 2026-05-26 06:42:42 +00:00
@mjonesandGitHub aac1fdc5c2 mattermost: add .patch files for user limit and banner removal (#522925) 2026-05-26 06:33:12 +00:00
Jacek GalowiczandGitHub e72ed98bc9 various: remove vsock CID based instructions (#524287) 2026-05-26 06:32:07 +00:00
nixpkgs-ci[bot]andGitHub 2ae72abcf6 vacuum-go: 0.26.5 -> 0.26.7 (#524277) 2026-05-26 06:30:37 +00:00
nixpkgs-ci[bot]andGitHub 56a095b0cd cargo-public-api: 0.51.0 -> 0.52.0 (#524156) 2026-05-26 06:30:31 +00:00
nixpkgs-ci[bot]andGitHub 42122cb8da uclibc-ng: 1.0.57 -> 1.0.58 (#523890) 2026-05-26 06:30:19 +00:00
Jörg ThalheimandGitHub ec563b9896 nixos/console: fix tmpfile rule when imperativeLocale is enabled (#523974) 2026-05-26 06:27:00 +00:00
Morgan Jones 9ea1835918 nixos/mattermost: add v11 user limit changes to release notes 2026-05-25 23:26:45 -07:00
joseandMorgan Jones fc9f1c91b2 mattermost: add patches for user limit and banner removal 2026-05-25 23:20:44 -07:00
Vincent LaporteandGitHub bae78f0861 ocamlPackages.iri: 1.1.0 -> 1.2.0 (#498685) 2026-05-26 06:06:03 +00:00
Vincent LaporteandGitHub 84d837c27c cvc5: 1.3.3 → 1.3.4 (#521819) 2026-05-26 05:53:06 +00:00
nixpkgs-ci[bot]andGitHub 9278612b7a kubexporter: 0.8.4 -> 0.8.7 (#524213) 2026-05-26 05:00:46 +00:00
rewineandGitHub 8092af7e60 xmake: 3.0.8 -> 3.0.9 (#524295) 2026-05-26 05:00:44 +00:00