From ec7dc46bfc581569b24d1d871f01490a820b8568 Mon Sep 17 00:00:00 2001 From: Maximilian Bosch Date: Sun, 8 Mar 2026 18:58:45 +0100 Subject: [PATCH] nixos/grafana: update instructions on secret rotation Closes #495278 Turns out that the documentation about secrets[1] is actually a little misleading and I let myself fool by it: the rotation is about moving to envelope-based encryption, i.e. a way where a key is generated and stored in the DB that is encrypted with `secret_key` and used to decrypt actual secrets. In fact, there's no official way to rotate this on a running instance. The reason this wasn't noticed so far is because I'd expect most people to * use provisioning to set up their datasources (or) * talk to a local Prometheus (or else) API that is exposed on the machine only and thus no authentication is needed if you trust that machine enough. On top, the encryption is unauthenticated so only changing secret_key appears to work, but uses gibberish as password for authentication against data-sources. So effectively there are two ways to migrate (short of a fresh setup): * keep the old secret and call it a day. IMHO this is fine if the setup is e.g. a single-node machine where both the secret and the DB are on the same file-system, in the same data-dir and protected by the same permissions. * use a 3rd-party rotation tool[2]. I ensured by hand on a test-instance that following the instructions works and you end up with a freshly generated secret_key that you need to deploy. With that key deployed, the credentials still decrypt to the same plaintext. [1] https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-database-encryption/#envelope-encryption [2] https://github.com/erooke/grafana-secretkey-rotation-tool/tree/d9dc788902fa5185e15cb15ce6129f7237ab6138 --- nixos/doc/manual/release-notes/rl-2605.section.md | 8 +++++++- nixos/modules/services/monitoring/grafana.nix | 5 +++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/nixos/doc/manual/release-notes/rl-2605.section.md b/nixos/doc/manual/release-notes/rl-2605.section.md index c1bdf097f317..cb427673bbd0 100644 --- a/nixos/doc/manual/release-notes/rl-2605.section.md +++ b/nixos/doc/manual/release-notes/rl-2605.section.md @@ -121,11 +121,17 @@ of pulling the upstream container image from Docker Hub. If you want the old beh - `services.oauth2-proxy.clientSecret` and `services.oauth2-proxy.cookie.secret` have been replaced with `services.oauth2-proxy.clientSecretFile` and `services.oauth2-proxy.cookie.secretFile` respectively. This was done to ensure secrets don't get made world-readable. - [`services.grafana.settings.security.secret_key`](#opt-services.grafana.settings.security.secret_key) doesn't have a - default value anymore. Please generate your own key or hard-code the old one explicitly. + default value anymore. Please generate your own key or hard-code the old one ("SW2YcwTIb9zpOOhoPsMm") explicitly. See the [upstream docs](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#secret_key) and the [instructions on how to rotate](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-database-encryption/#re-encrypt-secrets) for further information. + Please do note that there's no official way to rotate. On a single-node instance with the database and the secret-key being + on the same filesystem with the same permissions for Grafana only to read it's most likely OK to keep using the old key. + + If you need to rotate, a [3rd-party tool, `grafana-secretkey-rotation-tool`](https://github.com/erooke/grafana-secretkey-rotation-tool/tree/d9dc788902fa5185e15cb15ce6129f7237ab6138) is a tested option. + When using a secret for this value, make sure to use [Grafana's variable expansion to inject secrets](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#variable-expansion). + - Ethercalc and its associated module have been removed, as the package is unmaintained and cannot be installed from source with npm now. - `services.cgit` before always had the git-http-backend and its "export all" setting enabled, which sidestepped any access control configured in cgit's settings. Now you have to make a decision and either enable or disable `services.cgit.gitHttpBackend.checkExportOkFiles` (or disable the git-http-backend). diff --git a/nixos/modules/services/monitoring/grafana.nix b/nixos/modules/services/monitoring/grafana.nix index 96a725b3a844..0d6c7eb055d4 100644 --- a/nixos/modules/services/monitoring/grafana.nix +++ b/nixos/modules/services/monitoring/grafana.nix @@ -2070,6 +2070,11 @@ in for more information. See https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-database-encryption/#re-encrypt-secrets on how to re-encrypt. + + As stated in the NixOS changelog for 26.05, there's no official way to rotate. + Either hard-code the old key ("SW2YcwTIb9zpOOhoPsMm") if your setup doesn't have any secrets in the DB that need + special protection or perform a rotation with a 3rd-party tool + (https://github.com/erooke/grafana-secretkey-rotation-tool/tree/d9dc788902fa5185e15cb15ce6129f7237ab6138). ''; } ];