gador/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Revert "nixos/doc: re-format"

Browse Source 
This reverts commit ea6e8775bd. The new
format is not an improvement.
This commit is contained in:
Eelco Dolstra
2019-09-19 19:17:30 +02:00
parent db3d31b903
commit b0ccd6dd16
124 changed files with 5709 additions and 1940 deletions
Download Patch File Download Diff File Expand all files Collapse all files

197
nixos/modules/services/databases/foundationdb.xml
View File

@@ -5,10 +5,12 @@
xml:id="module-services-foundationdb">
<title>FoundationDB</title>
<para>
<emphasis>Source:</emphasis> <filename>modules/services/databases/foundationdb.nix</filename>
<emphasis>Source:</emphasis>
<filename>modules/services/databases/foundationdb.nix</filename>
</para>
<para>
<emphasis>Upstream documentation:</emphasis> <link xlink:href="https://apple.github.io/foundationdb/"/>
<emphasis>Upstream documentation:</emphasis>
<link xlink:href="https://apple.github.io/foundationdb/"/>
</para>
<para>
<emphasis>Maintainer:</emphasis> Austin Seipp
@@ -17,13 +19,15 @@
<emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
</para>
<para>
FoundationDB (or "FDB") is an open source, distributed, transactional key-value store.
FoundationDB (or "FDB") is an open source, distributed, transactional
key-value store.
</para>
<section xml:id="module-services-foundationdb-configuring">
<title>Configuring and basic setup</title>
<para>
To enable FoundationDB, add the following to your <filename>configuration.nix</filename>:
To enable FoundationDB, add the following to your
<filename>configuration.nix</filename>:
<programlisting>
services.foundationdb.enable = true;
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
@@ -31,11 +35,18 @@ services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
</para>
<para>
The <option>services.foundationdb.package</option> option is required, and must always be specified. Due to the fact FoundationDB network protocols and on-disk storage formats may change between (major) versions, and upgrades must be explicitly handled by the user, you must always manually specify this yourself so that the NixOS module will use the proper version. Note that minor, bugfix releases are always compatible.
The <option>services.foundationdb.package</option> option is required, and
must always be specified. Due to the fact FoundationDB network protocols and
on-disk storage formats may change between (major) versions, and upgrades
must be explicitly handled by the user, you must always manually specify
this yourself so that the NixOS module will use the proper version. Note
that minor, bugfix releases are always compatible.
</para>
<para>
After running <command>nixos-rebuild</command>, you can verify whether FoundationDB is running by executing <command>fdbcli</command> (which is added to <option>environment.systemPackages</option>):
After running <command>nixos-rebuild</command>, you can verify whether
FoundationDB is running by executing <command>fdbcli</command> (which is
added to <option>environment.systemPackages</option>):
<screen>
<prompt>$ </prompt>sudo -u foundationdb fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.
@@ -66,7 +77,11 @@ Cluster:
</para>
<para>
You can also write programs using the available client libraries. For example, the following Python program can be run in order to grab the cluster status, as a quick example. (This example uses <command>nix-shell</command> shebang support to automatically supply the necessary Python modules).
You can also write programs using the available client libraries. For
example, the following Python program can be run in order to grab the
cluster status, as a quick example. (This example uses
<command>nix-shell</command> shebang support to automatically supply the
necessary Python modules).
<screen>
<prompt>a@link> </prompt>cat fdb-status.py
#! /usr/bin/env nix-shell
@@ -96,56 +111,91 @@ FoundationDB available: True
</para>
<para>
FoundationDB is run under the <command>foundationdb</command> user and group by default, but this may be changed in the NixOS configuration. The systemd unit <command>foundationdb.service</command> controls the <command>fdbmonitor</command> process.
FoundationDB is run under the <command>foundationdb</command> user and group
by default, but this may be changed in the NixOS configuration. The systemd
unit <command>foundationdb.service</command> controls the
<command>fdbmonitor</command> process.
</para>
<para>
By default, the NixOS module for FoundationDB creates a single SSD-storage based database for development and basic usage. This storage engine is designed for SSDs and will perform poorly on HDDs; however it can handle far more data than the alternative "memory" engine and is a better default choice for most deployments. (Note that you can change the storage backend on-the-fly for a given FoundationDB cluster using <command>fdbcli</command>.)
By default, the NixOS module for FoundationDB creates a single SSD-storage
based database for development and basic usage. This storage engine is
designed for SSDs and will perform poorly on HDDs; however it can handle far
more data than the alternative "memory" engine and is a better default
choice for most deployments. (Note that you can change the storage backend
on-the-fly for a given FoundationDB cluster using
<command>fdbcli</command>.)
</para>
<para>
Furthermore, only 1 server process and 1 backup agent are started in the default configuration. See below for more on scaling to increase this.
Furthermore, only 1 server process and 1 backup agent are started in the
default configuration. See below for more on scaling to increase this.
</para>
<para>
FoundationDB stores all data for all server processes under <filename>/var/lib/foundationdb</filename>. You can override this using <option>services.foundationdb.dataDir</option>, e.g.
FoundationDB stores all data for all server processes under
<filename>/var/lib/foundationdb</filename>. You can override this using
<option>services.foundationdb.dataDir</option>, e.g.
<programlisting>
services.foundationdb.dataDir = "/data/fdb";
</programlisting>
</para>
<para>
Similarly, logs are stored under <filename>/var/log/foundationdb</filename> by default, and there is a corresponding <option>services.foundationdb.logDir</option> as well.
Similarly, logs are stored under <filename>/var/log/foundationdb</filename>
by default, and there is a corresponding
<option>services.foundationdb.logDir</option> as well.
</para>
</section>
<section xml:id="module-services-foundationdb-scaling">
<title>Scaling processes and backup agents</title>
<para>
Scaling the number of server processes is quite easy; simply specify <option>services.foundationdb.serverProcesses</option> to be the number of FoundationDB worker processes that should be started on the machine.
Scaling the number of server processes is quite easy; simply specify
<option>services.foundationdb.serverProcesses</option> to be the number of
FoundationDB worker processes that should be started on the machine.
</para>
<para>
FoundationDB worker processes typically require 4GB of RAM per-process at minimum for good performance, so this option is set to 1 by default since the maximum amount of RAM is unknown. You're advised to abide by this restriction, so pick a number of processes so that each has 4GB or more.
FoundationDB worker processes typically require 4GB of RAM per-process at
minimum for good performance, so this option is set to 1 by default since
the maximum amount of RAM is unknown. You're advised to abide by this
restriction, so pick a number of processes so that each has 4GB or more.
</para>
<para>
A similar option exists in order to scale backup agent processes, <option>services.foundationdb.backupProcesses</option>. Backup agents are not as performance/RAM sensitive, so feel free to experiment with the number of available backup processes.
A similar option exists in order to scale backup agent processes,
<option>services.foundationdb.backupProcesses</option>. Backup agents are
not as performance/RAM sensitive, so feel free to experiment with the number
of available backup processes.
</para>
</section>
<section xml:id="module-services-foundationdb-clustering">
<title>Clustering</title>
<para>
FoundationDB on NixOS works similarly to other Linux systems, so this section will be brief. Please refer to the full FoundationDB documentation for more on clustering.
FoundationDB on NixOS works similarly to other Linux systems, so this
section will be brief. Please refer to the full FoundationDB documentation
for more on clustering.
</para>
<para>
FoundationDB organizes clusters using a set of <emphasis>coordinators</emphasis>, which are just specially-designated worker processes. By default, every installation of FoundationDB on NixOS will start as its own individual cluster, with a single coordinator: the first worker process on <command>localhost</command>.
FoundationDB organizes clusters using a set of
<emphasis>coordinators</emphasis>, which are just specially-designated
worker processes. By default, every installation of FoundationDB on NixOS
will start as its own individual cluster, with a single coordinator: the
first worker process on <command>localhost</command>.
</para>
<para>
Coordinators are specified globally using the <command>/etc/foundationdb/fdb.cluster</command> file, which all servers and client applications will use to find and join coordinators. Note that this file <emphasis>can not</emphasis> be managed by NixOS so easily: FoundationDB is designed so that it will rewrite the file at runtime for all clients and nodes when cluster coordinators change, with clients transparently handling this without intervention. It is fundamentally a mutable file, and you should not try to manage it in any way in NixOS.
Coordinators are specified globally using the
<command>/etc/foundationdb/fdb.cluster</command> file, which all servers and
client applications will use to find and join coordinators. Note that this
file <emphasis>can not</emphasis> be managed by NixOS so easily:
FoundationDB is designed so that it will rewrite the file at runtime for all
clients and nodes when cluster coordinators change, with clients
transparently handling this without intervention. It is fundamentally a
mutable file, and you should not try to manage it in any way in NixOS.
</para>
<para>
@@ -166,7 +216,9 @@ services.foundationdb.dataDir = "/data/fdb";
</itemizedlist>
<para>
A node must already be a member of the cluster in order to properly be promoted to a coordinator, so you must always add it first if you wish to promote it.
A node must already be a member of the cluster in order to properly be
promoted to a coordinator, so you must always add it first if you wish to
promote it.
</para>
<para>
@@ -181,26 +233,37 @@ services.foundationdb.dataDir = "/data/fdb";
</listitem>
<listitem>
<para>
Copy the <command>/etc/foundationdb/fdb.cluster</command> file from this server to all the other servers. Restart FoundationDB on all of these other servers, so they join the cluster.
Copy the <command>/etc/foundationdb/fdb.cluster</command> file from this
server to all the other servers. Restart FoundationDB on all of these
other servers, so they join the cluster.
</para>
</listitem>
<listitem>
<para>
All of these servers are now connected and working together in the cluster, under the chosen coordinator.
All of these servers are now connected and working together in the
cluster, under the chosen coordinator.
</para>
</listitem>
</itemizedlist>
<para>
At this point, you can add as many nodes as you want by just repeating the above steps. By default there will still be a single coordinator: you can use <command>fdbcli</command> to change this and add new coordinators.
At this point, you can add as many nodes as you want by just repeating the
above steps. By default there will still be a single coordinator: you can
use <command>fdbcli</command> to change this and add new coordinators.
</para>
<para>
As a convenience, FoundationDB can automatically assign coordinators based on the redundancy mode you wish to achieve for the cluster. Once all the nodes have been joined, simply set the replication policy, and then issue the <command>coordinators auto</command> command
As a convenience, FoundationDB can automatically assign coordinators based
on the redundancy mode you wish to achieve for the cluster. Once all the
nodes have been joined, simply set the replication policy, and then issue
the <command>coordinators auto</command> command
</para>
<para>
For example, assuming we have 3 nodes available, we can enable double redundancy mode, then auto-select coordinators. For double redundancy, 3 coordinators is ideal: therefore FoundationDB will make <emphasis>every</emphasis> node a coordinator automatically:
For example, assuming we have 3 nodes available, we can enable double
redundancy mode, then auto-select coordinators. For double redundancy, 3
coordinators is ideal: therefore FoundationDB will make
<emphasis>every</emphasis> node a coordinator automatically:
</para>
<screen>
@@ -209,33 +272,58 @@ services.foundationdb.dataDir = "/data/fdb";
</screen>
<para>
This will transparently update all the servers within seconds, and appropriately rewrite the <command>fdb.cluster</command> file, as well as informing all client processes to do the same.
This will transparently update all the servers within seconds, and
appropriately rewrite the <command>fdb.cluster</command> file, as well as
informing all client processes to do the same.
</para>
</section>
<section xml:id="module-services-foundationdb-connectivity">
<title>Client connectivity</title>
<para>
By default, all clients must use the current <command>fdb.cluster</command> file to access a given FoundationDB cluster. This file is located by default in <command>/etc/foundationdb/fdb.cluster</command> on all machines with the FoundationDB service enabled, so you may copy the active one from your cluster to a new node in order to connect, if it is not part of the cluster.
By default, all clients must use the current <command>fdb.cluster</command>
file to access a given FoundationDB cluster. This file is located by default
in <command>/etc/foundationdb/fdb.cluster</command> on all machines with the
FoundationDB service enabled, so you may copy the active one from your
cluster to a new node in order to connect, if it is not part of the cluster.
</para>
</section>
<section xml:id="module-services-foundationdb-authorization">
<title>Client authorization and TLS</title>
<para>
By default, any user who can connect to a FoundationDB process with the correct cluster configuration can access anything. FoundationDB uses a pluggable design to transport security, and out of the box it supports a LibreSSL-based plugin for TLS support. This plugin not only does in-flight encryption, but also performs client authorization based on the given endpoint's certificate chain. For example, a FoundationDB server may be configured to only accept client connections over TLS, where the client TLS certificate is from organization <emphasis>Acme Co</emphasis> in the <emphasis>Research and Development</emphasis> unit.
By default, any user who can connect to a FoundationDB process with the
correct cluster configuration can access anything. FoundationDB uses a
pluggable design to transport security, and out of the box it supports a
LibreSSL-based plugin for TLS support. This plugin not only does in-flight
encryption, but also performs client authorization based on the given
endpoint's certificate chain. For example, a FoundationDB server may be
configured to only accept client connections over TLS, where the client TLS
certificate is from organization <emphasis>Acme Co</emphasis> in the
<emphasis>Research and Development</emphasis> unit.
</para>
<para>
Configuring TLS with FoundationDB is done using the <option>services.foundationdb.tls</option> options in order to control the peer verification string, as well as the certificate and its private key.
Configuring TLS with FoundationDB is done using the
<option>services.foundationdb.tls</option> options in order to control the
peer verification string, as well as the certificate and its private key.
</para>
<para>
Note that the certificate and its private key must be accessible to the FoundationDB user account that the server runs under. These files are also NOT managed by NixOS, as putting them into the store may reveal private information.
Note that the certificate and its private key must be accessible to the
FoundationDB user account that the server runs under. These files are also
NOT managed by NixOS, as putting them into the store may reveal private
information.
</para>
<para>
After you have a key and certificate file in place, it is not enough to simply set the NixOS module options -- you must also configure the <command>fdb.cluster</command> file to specify that a given set of coordinators use TLS. This is as simple as adding the suffix <command>:tls</command> to your cluster coordinator configuration, after the port number. For example, assuming you have a coordinator on localhost with the default configuration, simply specifying:
After you have a key and certificate file in place, it is not enough to
simply set the NixOS module options -- you must also configure the
<command>fdb.cluster</command> file to specify that a given set of
coordinators use TLS. This is as simple as adding the suffix
<command>:tls</command> to your cluster coordinator configuration, after the
port number. For example, assuming you have a coordinator on localhost with
the default configuration, simply specifying:
</para>
<programlisting>
@@ -250,19 +338,36 @@ XXXXXX:XXXXXX@127.0.0.1:4500:tls
<title>Backups and Disaster Recovery</title>
<para>
The usual rules for doing FoundationDB backups apply on NixOS as written in the FoundationDB manual. However, one important difference is the security profile for NixOS: by default, the <command>foundationdb</command> systemd unit uses <emphasis>Linux namespaces</emphasis> to restrict write access to the system, except for the log directory, data directory, and the <command>/etc/foundationdb/</command> directory. This is enforced by default and cannot be disabled.
The usual rules for doing FoundationDB backups apply on NixOS as written in
the FoundationDB manual. However, one important difference is the security
profile for NixOS: by default, the <command>foundationdb</command> systemd
unit uses <emphasis>Linux namespaces</emphasis> to restrict write access to
the system, except for the log directory, data directory, and the
<command>/etc/foundationdb/</command> directory. This is enforced by default
and cannot be disabled.
</para>
<para>
However, a side effect of this is that the <command>fdbbackup</command> command doesn't work properly for local filesystem backups: FoundationDB uses a server process alongside the database processes to perform backups and copy the backups to the filesystem. As a result, this process is put under the restricted namespaces above: the backup process can only write to a limited number of paths.
However, a side effect of this is that the <command>fdbbackup</command>
command doesn't work properly for local filesystem backups: FoundationDB
uses a server process alongside the database processes to perform backups
and copy the backups to the filesystem. As a result, this process is put
under the restricted namespaces above: the backup process can only write to
a limited number of paths.
</para>
<para>
In order to allow flexible backup locations on local disks, the FoundationDB NixOS module supports a <option>services.foundationdb.extraReadWritePaths</option> option. This option takes a list of paths, and adds them to the systemd unit, allowing the processes inside the service to write (and read) the specified directories.
In order to allow flexible backup locations on local disks, the FoundationDB
NixOS module supports a
<option>services.foundationdb.extraReadWritePaths</option> option. This
option takes a list of paths, and adds them to the systemd unit, allowing
the processes inside the service to write (and read) the specified
directories.
</para>
<para>
For example, to create backups in <command>/opt/fdb-backups</command>, first set up the paths in the module options:
For example, to create backups in <command>/opt/fdb-backups</command>, first
set up the paths in the module options:
</para>
<programlisting>
@@ -270,7 +375,11 @@ services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
</programlisting>
<para>
Restart the FoundationDB service, and it will now be able to write to this directory (even if it does not yet exist.) Note: this path <emphasis>must</emphasis> exist before restarting the unit. Otherwise, systemd will not include it in the private FoundationDB namespace (and it will not add it dynamically at runtime).
Restart the FoundationDB service, and it will now be able to write to this
directory (even if it does not yet exist.) Note: this path
<emphasis>must</emphasis> exist before restarting the unit. Otherwise,
systemd will not include it in the private FoundationDB namespace (and it
will not add it dynamically at runtime).
</para>
<para>
@@ -286,13 +395,17 @@ services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
<title>Known limitations</title>
<para>
The FoundationDB setup for NixOS should currently be considered beta. FoundationDB is not new software, but the NixOS compilation and integration has only undergone fairly basic testing of all the available functionality.
The FoundationDB setup for NixOS should currently be considered beta.
FoundationDB is not new software, but the NixOS compilation and integration
has only undergone fairly basic testing of all the available functionality.
</para>
<itemizedlist>
<listitem>
<para>
There is no way to specify individual parameters for individual <command>fdbserver</command> processes. Currently, all server processes inherit all the global <command>fdbmonitor</command> settings.
There is no way to specify individual parameters for individual
<command>fdbserver</command> processes. Currently, all server processes
inherit all the global <command>fdbmonitor</command> settings.
</para>
</listitem>
<listitem>
@@ -311,14 +424,20 @@ services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
<title>Options</title>
<para>
NixOS's FoundationDB module allows you to configure all of the most relevant configuration options for <command>fdbmonitor</command>, matching it quite closely. A complete list of options for the FoundationDB module may be found <link linkend="opt-services.foundationdb.enable">here</link>. You should also read the FoundationDB documentation as well.
NixOS's FoundationDB module allows you to configure all of the most relevant
configuration options for <command>fdbmonitor</command>, matching it quite
closely. A complete list of options for the FoundationDB module may be found
<link linkend="opt-services.foundationdb.enable">here</link>. You should
also read the FoundationDB documentation as well.
</para>
</section>
<section xml:id="module-services-foundationdb-full-docs">
<title>Full documentation</title>
<para>
FoundationDB is a complex piece of software, and requires careful administration to properly use. Full documentation for administration can be found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.
FoundationDB is a complex piece of software, and requires careful
administration to properly use. Full documentation for administration can be
found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.
</para>
</section>
</chapter>

39
nixos/modules/services/databases/postgresql.xml
View File

@@ -7,10 +7,12 @@
<!-- FIXME: render nicely -->
<!-- FIXME: source can be added automatically -->
<para>
<emphasis>Source:</emphasis> <filename>modules/services/databases/postgresql.nix</filename>
<emphasis>Source:</emphasis>
<filename>modules/services/databases/postgresql.nix</filename>
</para>
<para>
<emphasis>Upstream documentation:</emphasis> <link xlink:href="http://www.postgresql.org/docs/"/>
<emphasis>Upstream documentation:</emphasis>
<link xlink:href="http://www.postgresql.org/docs/"/>
</para>
<!-- FIXME: more stuff, like maintainer? -->
<para>
@@ -21,12 +23,18 @@
<title>Configuring</title>
<para>
To enable PostgreSQL, add the following to your <filename>configuration.nix</filename>:
To enable PostgreSQL, add the following to your
<filename>configuration.nix</filename>:
<programlisting>
<xref linkend="opt-services.postgresql.enable"/> = true;
<xref linkend="opt-services.postgresql.package"/> = pkgs.postgresql_11;
</programlisting>
Note that you are required to specify the desired version of PostgreSQL (e.g. <literal>pkgs.postgresql_11</literal>). Since upgrading your PostgreSQL version requires a database dump and reload (see below), NixOS cannot provide a default value for <xref linkend="opt-services.postgresql.package"/> such as the most recent release of PostgreSQL.
Note that you are required to specify the desired version of PostgreSQL
(e.g. <literal>pkgs.postgresql_11</literal>). Since upgrading your
PostgreSQL version requires a database dump and reload (see below), NixOS
cannot provide a default value for
<xref linkend="opt-services.postgresql.package"/> such as the most recent
release of PostgreSQL.
</para>
<!--
@@ -43,7 +51,9 @@ Type "help" for help.
-->
<para>
By default, PostgreSQL stores its databases in <filename>/var/lib/postgresql/$psqlSchema</filename>. You can override this using <xref linkend="opt-services.postgresql.dataDir"/>, e.g.
By default, PostgreSQL stores its databases in
<filename>/var/lib/postgresql/$psqlSchema</filename>. You can override this using
<xref linkend="opt-services.postgresql.dataDir"/>, e.g.
<programlisting>
<xref linkend="opt-services.postgresql.dataDir"/> = "/data/postgresql";
</programlisting>
@@ -60,14 +70,18 @@ Type "help" for help.
<title>Options</title>
<para>
A complete list of options for the PostgreSQL module may be found <link linkend="opt-services.postgresql.enable">here</link>.
A complete list of options for the PostgreSQL module may be found
<link linkend="opt-services.postgresql.enable">here</link>.
</para>
</section>
<section xml:id="module-services-postgres-plugins">
<title>Plugins</title>
<para>
Plugins collection for each PostgreSQL version can be accessed with <literal>.pkgs</literal>. For example, for <literal>pkgs.postgresql_11</literal> package, its plugin collection is accessed by <literal>pkgs.postgresql_11.pkgs</literal>:
Plugins collection for each PostgreSQL version can be accessed with
<literal>.pkgs</literal>. For example, for
<literal>pkgs.postgresql_11</literal> package, its plugin collection is
accessed by <literal>pkgs.postgresql_11.pkgs</literal>:
<screen>
<prompt>$ </prompt>nix repl '&lt;nixpkgs&gt;'
@@ -84,9 +98,8 @@ postgresql_11.pkgs.pg_partman postgresql_11.pkgs.pgroonga
...
</screen>
</para>
<para>
To add plugins via NixOS configuration, set <literal>services.postgresql.extraPlugins</literal>:
To add plugins via NixOS configuration, set <literal>services.postgresql.extraPlugins</literal>:
<programlisting>
<xref linkend="opt-services.postgresql.package"/> = pkgs.postgresql_11;
<xref linkend="opt-services.postgresql.extraPlugins"/> = with pkgs.postgresql_11.pkgs; [
@@ -95,9 +108,10 @@ postgresql_11.pkgs.pg_partman postgresql_11.pkgs.pgroonga
];
</programlisting>
</para>
<para>
You can build custom PostgreSQL-with-plugins (to be used outside of NixOS) using function <literal>.withPackages</literal>. For example, creating a custom PostgreSQL package in an overlay can look like:
You can build custom PostgreSQL-with-plugins (to be used outside of NixOS) using
function <literal>.withPackages</literal>. For example, creating a custom
PostgreSQL package in an overlay can look like:
<programlisting>
self: super: {
postgresql_custom = self.postgresql_11.withPackages (ps: [
@@ -107,9 +121,8 @@ self: super: {
}
</programlisting>
</para>
<para>
Here's a recipe on how to override a particular plugin through an overlay:
Here's a recipe on how to override a particular plugin through an overlay:
<programlisting>
self: super: {
postgresql_11 = super.postgresql_11.override { this = self.postgresql_11; } // {