From 16658f763475f33e1e4dce3489c5b30dc137cc79 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Mon, 8 May 2023 19:14:15 +0200 Subject: [PATCH] nixos/netdata: introduce `deadlineBeforeStopSec` Previously, we hardcoded a 60 second timer to stop netdata if we didn't have any answer back. This is wrong and can cause data loss because the SIGTERM sent by systemd can sometimes be not honored. Which in turn becomes a SIGKILL, causing potential data loss / corruption. Offer a flag to users and bump the deadline to 2 minutes. --- .../doc/manual/release-notes/rl-2305.section.md | 2 ++ nixos/modules/services/monitoring/netdata.nix | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/nixos/doc/manual/release-notes/rl-2305.section.md b/nixos/doc/manual/release-notes/rl-2305.section.md index 425b340b9eaf..7eafa6a9bef8 100644 --- a/nixos/doc/manual/release-notes/rl-2305.section.md +++ b/nixos/doc/manual/release-notes/rl-2305.section.md @@ -387,6 +387,8 @@ In addition to numerous new and upgraded packages, this release has the followin } ``` +- `services.netdata` offers a `deadlineBeforeStopSec` option which enable users who have netdata instance that takes time to initialize to not have systemd kill them for no reason. + - `services.dhcpcd` service now don't solicit or accept IPv6 Router Advertisements on interfaces that use static IPv6 addresses. If network uses both IPv6 Unique local addresses (ULA) and global IPv6 address auto-configuration with SLAAC, must add the parameter `networking.dhcpcd.IPv6rs = true;`. diff --git a/nixos/modules/services/monitoring/netdata.nix b/nixos/modules/services/monitoring/netdata.nix index 92c870bb23f1..bd0dea83e1a8 100644 --- a/nixos/modules/services/monitoring/netdata.nix +++ b/nixos/modules/services/monitoring/netdata.nix @@ -169,6 +169,20 @@ in { See: ''; }; + + deadlineBeforeStopSec = mkOption { + type = types.int; + default = 120; + description = lib.mdDoc '' + In order to detect when netdata is misbehaving, we run a concurrent task pinging netdata (wait-for-netdata-up) + in the systemd unit. + + If after a while, this task does not succeed, we stop the unit and mark it as failed. + + You can control this deadline in seconds with this option, it's useful to bump it + if you have (1) a lot of data (2) doing upgrades (3) have low IOPS/throughput. + ''; + }; }; }; @@ -205,7 +219,7 @@ in { while [ "$(${pkgs.netdata}/bin/netdatacli ping)" != pong ]; do sleep 0.5; done ''; - TimeoutStopSec = 60; + TimeoutStopSec = cfg.deadlineBeforeStopSec; Restart = "on-failure"; # User and group User = cfg.user;