From 952781729fbc137349a043daf2aaa2aa0e2e023c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= Date: Wed, 23 Aug 2023 19:01:16 +0200 Subject: [PATCH 1/4] doc/lisp: minor changes to manual --- doc/languages-frameworks/lisp.section.md | 35 ++++++++++++------------ 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index 3c408eaa09da..c94ab6939a8d 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -7,9 +7,9 @@ libraries that use ASDF (Another System Definition Facility). It lives in ## Overview {#lisp-overview} The main entry point of the API are the Common Lisp implementation packages -(e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, `sbcl`) -themselves. They have the `pkgs` and `withPackages` attributes, which can be -used to discover available packages and to build wrappers, respectively. +themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, +`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used +to discover available packages and to build wrappers, respectively. The `pkgs` attribute set contains packages that were automatically imported from Quicklisp, and any other manually defined ones. Not every package works for all @@ -30,7 +30,7 @@ executables with `sb-ext:save-lisp-and-die`, are achieved via overriding the In addition, Lisps have the `withOverrides` function, which can be used to substitute any package in the scope of their `pkgs`. This will be useful together with `overrideLispAttrs` when dealing with slashy ASDF systems, because -they should stay in the main package and be build by specifying the `systems` +they should stay in the main package and be built by specifying the `systems` argument to `build-asdf-system`. ## The 90% use case example {#lisp-use-case-example} @@ -42,7 +42,7 @@ The most common way to use the library is to run ad-hoc wrappers like this: Then, in a shell: ``` -$ result/bin/sbcl +$ sbcl * (load (sb-ext:posix-getenv "ASDF")) * (asdf:load-system 'alexandria) ``` @@ -53,7 +53,7 @@ Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`: let sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]); in mkShell { - buildInputs = [ sbcl' ]; + packages = [ sbcl' ]; } ``` @@ -67,15 +67,16 @@ buildPhase = '' ## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp} -The library is able to very quickly import all the packages distributed by -Quicklisp by parsing its `releases.txt` and `systems.txt` files. These files are -available from [http://beta.quicklisp.org/dist/quicklisp.txt]. +The library is able to import all the packages distributed by Quicklisp by +parsing its `releases.txt` and `systems.txt` files. These files are available +from [quicklisp.org](http://beta.quicklisp.org/dist/quicklisp.txt). The import process is implemented in the `import` directory as Common Lisp -functions in the `org.lispbuilds.nix` ASDF system. To run the script, one can +code in the `org.lispbuilds.nix` ASDF system. To run the script, one can execute `ql-import.lisp`: ``` +cd pkgs/development/lisp-modules nix-shell --run 'sbcl --script ql-import.lisp' ``` @@ -92,7 +93,7 @@ The maintainer's job there is to: 3. For packages that still don't build, package them manually in `packages.nix` Also, the `imported.nix` file **must not be edited manually**! It should only be -generated as described in this section. +generated as described in this section (by running `ql-import.lisp`). ### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies} @@ -108,7 +109,7 @@ Packages defined in `packages.nix` contain these dependencies naturally. The previous implementation of `lisp-modules` didn't fully trust the Quicklisp data, because there were times where the dependencies specified were not -complete, and caused broken builds. It instead used a `nix-shell` environment to +complete and caused broken builds. It instead used a `nix-shell` environment to discover real dependencies by using the ASDF APIs. The current implementation has chosen to trust this data, because it's faster to @@ -126,8 +127,8 @@ replace the `systems` attribute of the affected packages. (See the definition of During Quicklisp import: -- `+` in names are converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus` -- `.` to `_dot_`: `iolib.base`->`iolib_dot_base` +- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus` +- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base` - names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`) - `_` in names is converted to `__` for reversibility @@ -148,11 +149,11 @@ The `build-asdf-system` function is documented with comments in ## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside} Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem` -function, which is the same as `build-asdf-system`, except for the `lisp` -argument which is set to the given CL implementation. +function, which is similar to `build-asdf-system` from `packages.nix`, but is +part of the public API. It can be used to define packages outside Nixpkgs, and, for example, add them -into the package scope with `withOverrides` which will be discussed later on. +into the package scope with `withOverrides` which will be discussed later. ### Including an external package in scope {#lisp-including-external-pkg-in-scope} From b991ea8385f053138eb2e68af69db7f6a52c4579 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= Date: Fri, 25 Aug 2023 20:38:44 +0200 Subject: [PATCH 2/4] doc/lisp: clarify section on importing from Quicklisp 1. Clarify what is the reason for importing and to where it saves 2. Clarify that packages.sqlite is a temporary file 3. Link to section about native dependencies from first mention of ql.nix --- doc/languages-frameworks/lisp.section.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index c94ab6939a8d..94d99b0392ad 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -67,9 +67,10 @@ buildPhase = '' ## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp} -The library is able to import all the packages distributed by Quicklisp by -parsing its `releases.txt` and `systems.txt` files. These files are available -from [quicklisp.org](http://beta.quicklisp.org/dist/quicklisp.txt). +To save some work of writing Nix expressions, there is a script that imports all +the packages distributed by Quicklisp into `imported.nix`. This works by parsing +its `releases.txt` and `systems.txt` files, which are published every couple of +months on [quicklisp.org](http://beta.quicklisp.org/dist/quicklisp.txt). The import process is implemented in the `import` directory as Common Lisp code in the `org.lispbuilds.nix` ASDF system. To run the script, one can @@ -83,13 +84,16 @@ nix-shell --run 'sbcl --script ql-import.lisp' The script will: 1. Download the latest Quicklisp `systems.txt` and `releases.txt` files -2. Generate an SQLite database of all QL systems in `packages.sqlite` +2. Generate a temporary SQLite database of all QL systems in `packages.sqlite` 3. Generate an `imported.nix` file from the database -The maintainer's job there is to: +(The `packages.sqlite` file can be deleted at will, because it is regenerated +each time the script runs.) -1. Re-run the `ql-import.lisp` script -2. Add missing native dependencies in `ql.nix` +The maintainer's job is to: + +1. Re-run the `ql-import.lisp` script when there is a new Quicklisp release +2. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix` 3. For packages that still don't build, package them manually in `packages.nix` Also, the `imported.nix` file **must not be edited manually**! It should only be From 9ce6e34ddf8be0fe467881fb25ed3c4d496ae133 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= Date: Fri, 25 Aug 2023 21:35:34 +0200 Subject: [PATCH 3/4] doc/lisp: add links to sections and upstream websites Also clean up complex paragraphs and fix section on building wrappers --- doc/languages-frameworks/lisp.section.md | 80 +++++++++--------------- 1 file changed, 29 insertions(+), 51 deletions(-) diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index 94d99b0392ad..53ddee837602 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -1,36 +1,32 @@ # lisp-modules {#lisp} This document describes the Nixpkgs infrastructure for building Common Lisp -libraries that use ASDF (Another System Definition Facility). It lives in -`pkgs/development/lisp-modules`. +systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System +Definition Facility). It lives in `pkgs/development/lisp-modules`. ## Overview {#lisp-overview} The main entry point of the API are the Common Lisp implementation packages -themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, +themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`, `sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used to discover available packages and to build wrappers, respectively. -The `pkgs` attribute set contains packages that were automatically imported from -Quicklisp, and any other manually defined ones. Not every package works for all -the CL implementations (e.g. `nyxt` only makes sense for `sbcl`). +The `pkgs` attribute set contains packages that were automatically +[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any +other [manually defined](#lisp-defining-packages-inside) ones. Not every package +works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`). -The `withPackages` function is of primary utility. It is used to build runnable -wrappers, with a pinned and pre-built ASDF FASL available in the `ASDF` -environment variable, and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` -configured to find the desired systems on runtime. - -With a few exceptions, the primary thing that the infrastructure does is to run -`asdf:load-system` for each system specified in the `systems` argument to -`build-asdf-system`, and save the FASLs to the Nix store. Then, it makes these -FASLs available to wrappers. Any other use-cases, such as producing SBCL -executables with `sb-ext:save-lisp-and-die`, are achieved via overriding the -`buildPhase` etc. +The `withPackages` function is of primary utility. It is used to build +[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built +[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable, +and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to +[find the desired systems on runtime](#lisp-loading-systems). In addition, Lisps have the `withOverrides` function, which can be used to -substitute any package in the scope of their `pkgs`. This will be useful -together with `overrideLispAttrs` when dealing with slashy ASDF systems, because -they should stay in the main package and be built by specifying the `systems` +[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of +their `pkgs`. This will also be useful together with `overrideLispAttrs` when +[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they +should stay in the main package and be built by specifying the `systems` argument to `build-asdf-system`. ## The 90% use case example {#lisp-use-case-example} @@ -94,7 +90,7 @@ The maintainer's job is to: 1. Re-run the `ql-import.lisp` script when there is a new Quicklisp release 2. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix` -3. For packages that still don't build, package them manually in `packages.nix` +3. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix` Also, the `imported.nix` file **must not be edited manually**! It should only be generated as described in this section (by running `ql-import.lisp`). @@ -139,13 +135,14 @@ During Quicklisp import: ## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside} -New packages, that for some reason are not in Quicklisp, and so cannot be -auto-imported, can be written in the `packages.nix` file. +Packages that for some reason are not in Quicklisp, and so cannot be +auto-imported, or don't work straight from the import, are defined in the +`packages.nix` file. In that file, use the `build-asdf-system` function, which is a wrapper around `mkDerivation` for building ASDF systems. Various other hacks are present, such as `build-with-compile-into-pwd` for systems which create files during -compilation. +compilation (such as cl-unicode). The `build-asdf-system` function is documented with comments in `nix-cl.nix`. Also, `packages.nix` is full of examples of how to use it. @@ -203,28 +200,6 @@ sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec { }) ``` -## Overriding packages in scope {#lisp-overriding-packages-in-scope} - -Packages can be woven into a new scope by using `withOverrides`: - -``` -let - sbcl' = sbcl.withOverrides (self: super: { - alexandria = super.alexandria.overrideLispAttrs (oldAttrs: rec { - pname = "alexandria"; - version = "1.4"; - src = fetchFromGitLab { - domain = "gitlab.common-lisp.net"; - owner = "alexandria"; - repo = "alexandria"; - rev = "v${version}"; - hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ="; - }; - }); - }); -in builtins.elemAt sbcl'.pkgs.bordeaux-threads.lispLibs 0 -``` - ### Dealing with slashy systems {#lisp-dealing-with-slashy-systems} Slashy (secondary) systems should not exist in their own packages! Instead, they @@ -245,8 +220,8 @@ ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: { }) ``` -See the respective section on using `withOverrides` for how to weave it back -into `ecl.pkgs`. +See the [respective section](#lisp-including-external-pkg-in-scope) on using +`withOverrides` for how to weave it back into `ecl.pkgs`. Note that sometimes the slashy systems might not only have more dependencies than the main one, but create a circular dependency between `.asd` @@ -258,13 +233,16 @@ Wrappers can be built using the `withPackages` function of Common Lisp implementations (`abcl`, `ecl`, `sbcl` etc.): ``` -sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ]) +nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])' ``` -Such a wrapper can then be executed like this: +Such a wrapper can then be used like this: ``` -result/bin/sbcl +$ sbcl +* (load (sb-ext:posix-getenv "ASDF")) +* (asdf:load-system 'alexandria) +* (asdf:load-system 'bordeaux-threads) ``` ### Loading ASDF {#lisp-loading-asdf} From 827b70a9b461d2d54af236da4d75f572391a765e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= Date: Fri, 25 Aug 2023 21:43:09 +0200 Subject: [PATCH 4/4] doc/lisp: document arguments of buildASDFSystem --- doc/languages-frameworks/lisp.section.md | 26 +++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/doc/languages-frameworks/lisp.section.md b/doc/languages-frameworks/lisp.section.md index 53ddee837602..8712c3412064 100644 --- a/doc/languages-frameworks/lisp.section.md +++ b/doc/languages-frameworks/lisp.section.md @@ -16,10 +16,10 @@ The `pkgs` attribute set contains packages that were automatically other [manually defined](#lisp-defining-packages-inside) ones. Not every package works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`). -The `withPackages` function is of primary utility. It is used to build -[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built -[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable, -and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to +The `withPackages` function is of primary utility. It is used to build +[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built +[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable, +and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to [find the desired systems on runtime](#lisp-loading-systems). In addition, Lisps have the `withOverrides` function, which can be used to @@ -144,8 +144,9 @@ In that file, use the `build-asdf-system` function, which is a wrapper around as `build-with-compile-into-pwd` for systems which create files during compilation (such as cl-unicode). -The `build-asdf-system` function is documented with comments in -`nix-cl.nix`. Also, `packages.nix` is full of examples of how to use it. +The `build-asdf-system` function is documented +[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of +examples of how to use it. ## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside} @@ -153,8 +154,19 @@ Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem` function, which is similar to `build-asdf-system` from `packages.nix`, but is part of the public API. +It takes the following arguments: + +- `pname`: the package name +- `version`: the package version +- `src`: the package source +- `patches`: patches to apply to the source before build +- `nativeLibs`: native libraries used by CFFI and grovelling +- `javaLibs`: Java libraries for ABCL +- `lispLibs`: dependencies on other packages build with `buildASDFSystem` +- `systems`: list of systems to build + It can be used to define packages outside Nixpkgs, and, for example, add them -into the package scope with `withOverrides` which will be discussed later. +into the package scope with `withOverrides`. ### Including an external package in scope {#lisp-including-external-pkg-in-scope}