Merge #243: Modularize Tests

1cc432a136 examples/deploy-container: use new extra-container features (Erik Arvstedt)
16b2783ae7 examples: enable running outside of working dir (Erik Arvstedt)
04075b108c test: use QEMU from stable nixpkgs (Erik Arvstedt)
03f8dbba47 test: add non-secure-node eval test (Erik Arvstedt)
bb763d6a26 run-tests: add 'eval' command (Erik Arvstedt)
36358066e4 spark-wallet: don't disable tor when onion-service is disabled (Erik Arvstedt)
c9251e72a1 README: add run-tests.sh to examples (Erik Arvstedt)
5a565dff66 netns test: use netns ips from config (Erik Arvstedt)
ac95fe7c82 netns test: don't test recurring-donations (Erik Arvstedt)
84744f38d7 netns test: disable backup test (Erik Arvstedt)
fcc67da9f4 test: add container support (Erik Arvstedt)
e99b7edb8e run-tests: pass script args verbatim to command without word splitting (Erik Arvstedt)
e7c397a485 run-tests: rename scriptDir -> testDir (Erik Arvstedt)
b552d17d55 run-tests: fix arg error messages (Erik Arvstedt)
572967d3ad extra-container: pre-release -> 0.5-pre (Erik Arvstedt)
ac6cee5c12 pkgs: add extra-container (Erik Arvstedt)
be2127ae5b test: fix noConnections configs (Erik Arvstedt)
1e18d3ea3b test: improve modularization (Erik Arvstedt)
14d2d97ba6 test: rename scenario withnetns -> netns (Erik Arvstedt)
9bf77ee3e8 backups test: simplify and speed up (Erik Arvstedt)
fcda69e8b6 netns test: connect from main netns (Erik Arvstedt)
24069aa2c6 electrs: add option 'monitoringPort' (Erik Arvstedt)
611cfe5a28 electrs: remove redundant daemonrpc option (Erik Arvstedt)
a19d3b07c2 electrs: add variable 'bitcoind' (Erik Arvstedt)
a6dde36b87 electrs: use consistent args formatting (Erik Arvstedt)
45bcbf683d test: rename test.nix -> tests.nix (Erik Arvstedt)
c92e85f707 test: rename base.py -> tests.py (Erik Arvstedt)

Pull request description:

ACKs for top commit:
  nixbitcoin:
    ACK 1cc432a136
  jonasnick:
    ACK 1cc432a136

Tree-SHA512: 388f195c85e740937f9e2cecbd672a1f1e64e6ce06a75ba8167aba4fc77b70e07d14282e9ca117b31e9085e37ed4cf286f90f1204cdaa4c0360141039b1cae95

.travis.yml

@@ -15,7 +15,8 @@ env:
   - secure: "xXCFZ7g+k5YmCGm8R8l3bZElVmt+RD1KscG3kGr5w4HyyDPTzFetPo+sT8bUpysDU0u3HWhfVhHtpog2mhNhwVl3tQwKXea3dHKC1i6ypBg3gjDngmJRR5wo++ocYDpK8qPaU7m/jHQTNFnTA4CbmMcc05GcYx/1Ai/ZGkNwWFjdIcVeOUoiol33gykMOXIGDg2qlXudt33wP53FHbX8L4fxzodWfAuxKK4AoGprxy5eSnU7LCaXxxJmu4HwuV+Ux2U1NfE/E33cvhlUvTQCswVSZFG06mg8rwhMG1ozsDvlL2itZlu/BeUQH5y3XMMlnJIUXUazkRBibf1w/ebVjpOF+anqkqmq8tcbFEa7T+RJeVTIsvP+L8rE8fcmuZtdg9hNmgRnLmaeT0vVwD1L2UqW9HdRyujdoS0jPYuoc1W7f1JQWfAPhBPQ1SrtKyNNqcbVJ34aN7b+4vCzRpQL1JTbmjzQIWhkiKN1qMo1v/wbIydW8yka4hc4JOfdQLaAJEPI1eAC1MLotSAegMnwKWE1dzm66MuPSipksYjZrvsB28cV4aCVUffIuRhrSr1i2afRHwTpNbK9U4/576hah15ftUdR79Sfkcoi1ekSQTFGRvkRIPYtkKLYwFa3jVA41qz7+IIZCf4TsApy3XDdFx91cRub7yPq9BeZ83A+qYQ="
   jobs:
   - TestModules=1 STABLE=1 SCENARIO=default
-  - TestModules=1 STABLE=1 SCENARIO=withnetns
+  - TestModules=1 STABLE=1 SCENARIO=netns
+  - EvalModules=1 STABLE=1
   - PKG=hwi STABLE=1
   - PKG=hwi STABLE=0
   - PKG=lightning-charge STABLE=1
@@ -34,6 +35,12 @@ env:
   - PKG=joinmarket STABLE=0
 script:
   - printf '%s (%s)\n' "$NIX_PATH" "$VER"
+  #
+  - |
+    if [[ $EvalModules ]]; then
+      test/run-tests.sh --scenario full eval
+      travis_terminate 0
+    fi
   - |
     getBuildExpr() {
         if [[ $TestModules ]]; then

README.md

@@ -24,11 +24,11 @@ The goal is to make it easy to deploy a reasonably secure Bitcoin node with a us
 It should allow managing bitcoin (the currency) effectively and providing public infrastructure.
 It should be a reproducible and extensible platform for applications building on Bitcoin.
 
-Example
+Examples
 ---
 The easiest way to try out nix-bitcoin is to use one of the provided examples.
 
-```
+```bash
 git clone https://github.com/fort-nix/nix-bitcoin
 cd nix-bitcoin/examples/
 nix-shell
@@ -39,7 +39,7 @@ shut down immediately. They leave no traces (outside of `/nix/store`) on the hos
 
 - [`./deploy-container.sh`](examples/deploy-container.sh) creates a [NixOS container](https://github.com/erikarvstedt/extra-container).\
   This is the fastest way to set up a node.\
-  Requires: [NixOS](https://nixos.org/)
+  Requires: [Nix](https://nixos.org/), a systemd-based Linux distro and root privileges
 
 - [`./deploy-qemu-vm.sh`](examples/deploy-qemu-vm.sh) creates a QEMU VM.\
   Requires: [Nix](https://nixos.org/nix/)
@@ -48,6 +48,28 @@ shut down immediately. They leave no traces (outside of `/nix/store`) on the hos
   NixOps can be used to deploy to various other backends like cloud providers.\
   Requires: [Nix](https://nixos.org/nix/), [VirtualBox](https://www.virtualbox.org)
 
+#### Tests
+The internal test suite is also useful for exploring features.  
+The following `run-tests.sh` commands leave no traces (outside of `/nix/store`) on
+the host system.
+
+```bash
+git clone https://github.com/fort-nix/nix-bitcoin
+cd nix-bitcoin/test
+
+# Run a Python test shell inside a VM node
+./run-tests.sh debug
+print(succeed("systemctl status bitcoind"))
+
+# Run a node in a container. Requires systemd and root privileges.
+./run-tests.sh container
+c systemctl status bitcoind
+
+# Explore a single feature
+./run-tests.sh --scenario electrs container
+```
+See [`run-tests.sh`](test/run-tests.sh) for a complete documentation.
+
 Available modules
 ---
 By default the `configuration.nix` provides:

examples/deploy-container.sh

@@ -9,67 +9,22 @@ set -euo pipefail
 # script in the interactive shell.
 
 if [[ $(sysctl -n net.ipv4.ip_forward) != 1 ]]; then
-    echo "Error: IP forwarding (net.ipv4.ip_forward) is not enabled"
+    echo "Error: IP forwarding (net.ipv4.ip_forward) is not enabled."
-    exit 1
+    echo "Needed for container WAN access."
-fi
-if [[ ! -e /run/current-system/nixos-version ]]; then
-    echo "Error: This script needs NixOS to run"
     exit 1
 fi
 
 if [[ ! -v IN_NIX_SHELL ]]; then
     echo "Running script in nix shell env..."
+    cd "${BASH_SOURCE[0]%/*}"
     exec nix-shell --run "${BASH_SOURCE[0]}"
 fi
 
-# Cleanup on exit
+# Uncomment to start a container shell session
-cleanup() {
+# interactive=1
-    echo
-    echo "Deleting container..."
-    sudo extra-container destroy demo-node
-}
-trap "cleanup" EXIT
-
-# Build container.
-# You can re-run this command with a changed container config.
-# The running container is then switched to the new config.
-# Learn more: https://github.com/erikarvstedt/extra-container
-#
-sudo extra-container create --start <<'EOF'
-{ pkgs, lib, ... }: let
-    containerName = "demo-node"; # container name length is limited to 11 chars
-    localAddress = "10.250.0.2"; # container address
-    hostAddress = "10.250.0.1";
-in {
-  containers.${containerName} = {
-    privateNetwork = true;
-    inherit localAddress hostAddress;
-    config = { pkgs, config, lib, ... }: {
-      imports = [
-        <nix-bitcoin/examples/configuration.nix>
-        <nix-bitcoin/modules/secrets/generate-secrets.nix>
-      ];
-      # Speed up evaluation
-      documentation.nixos.enable = false;
-    };
-  };
-  # Allow WAN access
-  systemd.services."container@${containerName}" = {
-    preStart = "${pkgs.iptables}/bin/iptables -w -t nat -A POSTROUTING -s ${localAddress} -j MASQUERADE";
-    # Delete rule
-    postStop = "${pkgs.iptables}/bin/iptables -w -t nat -D POSTROUTING -s ${localAddress} -j MASQUERADE || true";
-  };
-}
-EOF
-# Run command in container
-c() {
-    if [[ $# > 0 ]]; then
-        sudo extra-container run demo-node -- "$@" | cat;
-    else
-        sudo nixos-container root-login demo-node
-    fi
-}
 
+# These commands can also be executed interactively in a shell session
+demoCmds='
 echo
 echo "Bitcoind service:"
 c systemctl status bitcoind
@@ -85,8 +40,32 @@ c nodeinfo
 echo
 echo "Bitcoind data dir:"
 sudo ls -al /var/lib/containers/demo-node/var/lib/bitcoind
+'
 
-# Uncomment to start a shell session here
+if [[ ${interactive:-} ]]; then
-# . start-bash-session.sh
+    runCmd=
+else
+    runCmd=(--run bash -c "$demoCmds")
+fi
 
-# Cleanup happens at exit (see above)
+# Build container.
+# Learn more: https://github.com/erikarvstedt/extra-container
+#
+read -d '' src <<'EOF' || true
+{ pkgs, lib, ... }: {
+  containers.demo-node = {
+    extra.addressPrefix = "10.250.0";
+    extra.enableWAN = true;
+    config = { pkgs, config, lib, ... }: {
+      imports = [
+        <nix-bitcoin/examples/configuration.nix>
+        <nix-bitcoin/modules/secrets/generate-secrets.nix>
+      ];
+    };
+  };
+}
+EOF
+$([[ $EUID = 0 ]] || echo sudo "PATH=$PATH" "NIX_PATH=$NIX_PATH") \
+    $(type -P extra-container) shell -E "$src" "${runCmd[@]}"
+
+# The container is automatically deleted at exit

examples/deploy-nixops.sh

@@ -10,6 +10,7 @@ set -euo pipefail
 
 if [[ ! -v IN_NIX_SHELL ]]; then
     echo "Running script in nix shell env..."
+    cd "${BASH_SOURCE[0]%/*}"
     exec nix-shell --run "${BASH_SOURCE[0]}"
 fi
 

examples/deploy-qemu-vm.sh

@@ -13,6 +13,7 @@ set -euo pipefail
 
 if [[ ! -v IN_NIX_SHELL ]]; then
     echo "Running script in nix shell env..."
+    cd "${BASH_SOURCE[0]%/*}"
     exec nix-shell --run "${BASH_SOURCE[0]}"
 fi
 

examples/shell.nix

@@ -11,11 +11,6 @@ let
   nixpkgs = import nixpkgs-path {};
   nix-bitcoin = nixpkgs.callPackage nix-bitcoin-path {};
 
-  extraContainer = nixpkgs.callPackage (builtins.fetchTarball {
-    url = "https://github.com/erikarvstedt/extra-container/archive/6cced2c26212cc1c8cc7cac3547660642eb87e71.tar.gz";
-    sha256 = "0qr41mma2iwxckdhqfabw3vjcbp2ffvshnc3k11kwriwj14b766v";
-  }) {};
-
   nix-bitcoin-unpacked = (import <nixpkgs> {}).runCommand "nix-bitcoin-src" {} ''
     mkdir $out; tar xf ${builtins.fetchurl nix-bitcoin-release} -C $out
   '';
@@ -25,7 +20,7 @@ with nixpkgs;
 stdenv.mkDerivation rec {
   name = "nix-bitcoin-environment";
 
-  buildInputs = [ nix-bitcoin.nixops19_09 figlet extraContainer ];
+  buildInputs = [ nix-bitcoin.nixops19_09 nix-bitcoin.extra-container figlet ];
 
   shellHook = ''
     export NIX_PATH="nixpkgs=${nixpkgs-path}:nix-bitcoin=${toString nix-bitcoin-path}:."

modules/electrs.nix

@@ -5,6 +5,7 @@ let
   cfg = config.services.electrs;
   inherit (config) nix-bitcoin-services;
   secretsDir = config.nix-bitcoin.secretsDir;
+  bitcoind = config.services.bitcoind;
 in {
   options.services.electrs = {
     enable = mkEnableOption "electrs";
@@ -33,19 +34,17 @@ in {
     address = mkOption {
       type = types.str;
       default = "127.0.0.1";
-      description = "RPC listening address.";
+      description = "RPC and monitoring listening address.";
     };
     port = mkOption {
       type = types.port;
       default = 50001;
       description = "RPC port.";
     };
-    daemonrpc = mkOption {
+    monitoringPort = mkOption {
-      type = types.str;
+      type = types.port;
-      default = "127.0.0.1:8332";
+      default = 4224;
-      description = ''
+      description = "Prometheus monitoring port.";
-        Bitcoin daemon JSONRPC 'addr:port' to connect
-      '';
     };
     extraArgs = mkOption {
       type = types.separatedString " ";
@@ -57,7 +56,7 @@ in {
 
   config = mkIf cfg.enable {
     assertions = [
-      { assertion = config.services.bitcoind.prune == 0;
+      { assertion = bitcoind.prune == 0;
         message = "electrs does not support bitcoind pruning.";
       }
     ];
@@ -74,7 +73,7 @@ in {
       requires = [ "bitcoind.service" ];
       after = [ "bitcoind.service" ];
       preStart = ''
-        echo "cookie = \"${config.services.bitcoind.rpc.users.public.name}:$(cat ${secretsDir}/bitcoin-rpcpassword-public)\"" \
+        echo "cookie = \"${bitcoind.rpc.users.public.name}:$(cat ${secretsDir}/bitcoin-rpcpassword-public)\"" \
           > electrs.toml
         '';
       serviceConfig = nix-bitcoin-services.defaultHardening // {
@@ -84,22 +83,25 @@ in {
         ExecStart = ''
           ${pkgs.nix-bitcoin.electrs}/bin/electrs -vvv \
           ${if cfg.high-memory then
-              traceIf (!config.services.bitcoind.dataDirReadableByGroup) ''
+              traceIf (!bitcoind.dataDirReadableByGroup) ''
                 Warning: For optimal electrs syncing performance, enable services.bitcoind.dataDirReadableByGroup.
                 Note that this disables wallet support in bitcoind.
               '' ""
             else
               "--jsonrpc-import --index-batch-size=10"
           } \
-          --db-dir '${cfg.dataDir}' --daemon-dir '${config.services.bitcoind.dataDir}' \
+          --db-dir='${cfg.dataDir}' \
-          --electrum-rpc-addr=${toString cfg.address}:${toString cfg.port} \
+          --daemon-dir='${bitcoind.dataDir}' \
-          --daemon-rpc-addr=${toString cfg.daemonrpc} ${cfg.extraArgs}
+          --electrum-rpc-addr=${cfg.address}:${toString cfg.port} \
+          --monitoring-addr=${cfg.address}:${toString cfg.monitoringPort} \
+          --daemon-rpc-addr=${builtins.elemAt bitcoind.rpcbind 0}:${toString bitcoind.rpc.port} \
+          ${cfg.extraArgs}
         '';
         User = cfg.user;
         Group = cfg.group;
         Restart = "on-failure";
         RestartSec = "10s";
-        ReadWritePaths = "${cfg.dataDir} ${if cfg.high-memory then "${config.services.bitcoind.dataDir}" else ""}";
+        ReadWritePaths = "${cfg.dataDir} ${if cfg.high-memory then "${bitcoind.dataDir}" else ""}";
       } // (if cfg.enforceTor
           then nix-bitcoin-services.allowTor
           else nix-bitcoin-services.allowAnyIP

modules/netns-isolation.nix

@@ -298,10 +298,7 @@ in {
       cliExec = mkCliExec "liquidd";
     };
 
-    services.electrs = {
+    services.electrs.address = netns.electrs.address;
-      address = netns.electrs.address;
-      daemonrpc = "${netns.bitcoind.address}:${toString config.services.bitcoind.rpc.port}";
-    };
 
     services.spark-wallet = {
       host = netns.spark-wallet.address;

modules/spark-wallet.nix

@@ -71,9 +71,6 @@ in {
     };
     users.groups.spark-wallet = {};
 
-    services.tor.enable = cfg.onion-service;
-    # requires client functionality for Bitcoin rate lookup
-    services.tor.client.enable = true;
     services.tor.hiddenServices.spark-wallet = mkIf cfg.onion-service {
       map = [{
         port = 80; toPort = 9737; toHost = cfg.host;

pkgs/default.nix

@@ -13,6 +13,7 @@
   nixops19_09 = pkgs.callPackage ./nixops { };
   netns-exec = pkgs.callPackage ./netns-exec { };
   lightning-loop = pkgs.callPackage ./lightning-loop { };
+  extra-container = pkgs.callPackage ./extra-container { };
 
   pinned = import ./pinned.nix;
 

pkgs/extra-container/default.nix

@@ -0,0 +1,36 @@
+{ stdenv, lib, nixos-container, openssh
+, glibcLocales
+}:
+
+stdenv.mkDerivation rec {
+  name = "extra-container-${version}";
+  version = "0.5-pre";
+
+  src = builtins.fetchTarball {
+    url = "https://github.com/erikarvstedt/extra-container/archive/${version}.tar.gz";
+    sha256 = "0gdy2dpqrdv7f4kyqz88j34x1p2fpav04kznv41hwqq88hmzap90";
+  };
+
+  buildCommand = ''
+    install -D $src/extra-container $out/bin/extra-container
+    patchShebangs $out/bin
+    share=$out/share/extra-container
+    install $src/eval-config.nix -Dt $share
+
+    # Use existing PATH for systemctl and machinectl (for nixos-container)
+    scriptPath="export PATH=${lib.makeBinPath [ nixos-container openssh ]}:\$PATH"
+
+    sed -i \
+      -e "s|evalConfig=.*|evalConfig=$share/eval-config.nix|" \
+      -e "s|LOCALE_ARCHIVE=.*|LOCALE_ARCHIVE=${glibcLocales}/lib/locale/locale-archive|" \
+      -e "2i$scriptPath" \
+      $out/bin/extra-container
+  '';
+
+  meta = with lib; {
+    description = "Run declarative containers without full system rebuilds";
+    homepage = https://github.com/erikarvstedt/extra-container;
+    license = licenses.mit;
+    maintainers = [ maintainers.earvstedt ];
+  };
+}

test/base.py

@@ -1,193 +0,0 @@
-is_interactive = "is_interactive" in vars()
-
-
-def succeed(*cmds):
-    """Returns the concatenated output of all cmds"""
-    return machine.succeed(*cmds)
-
-
-def assert_matches(cmd, regexp):
-    out = succeed(cmd)
-    if not re.search(regexp, out):
-        raise Exception(f"Pattern '{regexp}' not found in '{out}'")
-
-
-def assert_full_match(cmd, regexp):
-    out = succeed(cmd)
-    if not re.fullmatch(regexp, out):
-        raise Exception(f"Pattern '{regexp}' doesn't match '{out}'")
-
-
-def log_has_string(unit, str):
-    return f"journalctl -b --output=cat -u {unit} --grep='{str}'"
-
-
-def assert_no_failure(unit):
-    """Unit should not have failed since the system is running"""
-    machine.fail(log_has_string(unit, "Failed with result"))
-
-
-def assert_running(unit):
-    machine.wait_for_unit(unit)
-    assert_no_failure(unit)
-
-
-def run_tests(extra_tests):
-    """
-    :param extra_tests: Test functions that hook into the testing code below
-    :type extra_tests: Dict[str, Callable[]]
-    """
-    # Don't execute the following test suite when this script is running in interactive mode
-    if is_interactive:
-        raise Exception()
-
-    test_security()
-
-    assert_running("bitcoind")
-    machine.wait_until_succeeds("bitcoin-cli getnetworkinfo")
-    assert_matches("su operator -c 'bitcoin-cli getnetworkinfo' | jq", '"version"')
-    # RPC access for user 'public' should be restricted
-    machine.fail(
-        "bitcoin-cli -rpcuser=public -rpcpassword=$(cat /secrets/bitcoin-rpcpassword-public) stop"
-    )
-    machine.wait_until_succeeds(
-        log_has_string("bitcoind", "RPC User public not allowed to call method stop")
-    )
-
-    assert_running("electrs")
-    extra_tests.pop("electrs")()
-    # Check RPC connection to bitcoind
-    machine.wait_until_succeeds(log_has_string("electrs", "NetworkInfo"))
-    # Stop electrs from spamming the test log with 'wait for bitcoind sync' messages
-    succeed("systemctl stop electrs")
-
-    assert_running("liquidd")
-    machine.wait_until_succeeds("elements-cli getnetworkinfo")
-    assert_matches("su operator -c 'elements-cli getnetworkinfo' | jq", '"version"')
-    succeed("su operator -c 'liquidswap-cli --help'")
-
-    assert_running("clightning")
-    assert_matches("su operator -c 'lightning-cli getinfo' | jq", '"id"')
-
-    assert_running("lnd")
-    assert_matches("su operator -c 'lncli getinfo' | jq", '"version"')
-    assert_no_failure("lnd")
-
-    assert_running("lightning-loop")
-    assert_matches("su operator -c 'loop --version'", "version")
-    # Check that lightning-loop fails with the right error, making sure
-    # lightning-loop can connect to lnd
-    machine.wait_until_succeeds(
-        log_has_string(
-            "lightning-loop",
-            "Waiting for lnd to be fully synced to its chain backend, this might take a while",
-        )
-    )
-
-    assert_running("nbxplorer")
-    machine.wait_until_succeeds(log_has_string("nbxplorer", "BTC: RPC connection successful"))
-    extra_tests.pop("nbxplorer")()
-    assert_running("btcpayserver")
-    machine.wait_until_succeeds(log_has_string("btcpayserver", "Listening on"))
-    extra_tests.pop("btcpayserver")()
-
-    assert_running("spark-wallet")
-    extra_tests.pop("spark-wallet")()
-
-    assert_running("lightning-charge")
-    extra_tests.pop("lightning-charge")()
-
-    assert_running("nanopos")
-    extra_tests.pop("nanopos")()
-
-    assert_running("onion-chef")
-
-    assert_running("joinmarket")
-    machine.wait_until_succeeds(
-        log_has_string("joinmarket", "P2EPDaemonServerProtocolFactory starting on 27184")
-    )
-    machine.wait_until_succeeds(
-        log_has_string("joinmarket-yieldgenerator", "Failure to get blockheight",)
-    )
-
-    # FIXME: use 'wait_for_unit' because 'create-web-index' always fails during startup due
-    # to incomplete unit dependencies.
-    # 'create-web-index' implicitly tests 'nodeinfo'.
-    machine.wait_for_unit("create-web-index")
-    assert_running("nginx")
-    extra_tests.pop("web-index")()
-
-    machine.wait_until_succeeds(log_has_string("bitcoind-import-banlist", "Importing node banlist"))
-    assert_no_failure("bitcoind-import-banlist")
-
-    ### Additional tests
-
-    # Current time in µs
-    pre_restart = succeed("date +%s.%6N").rstrip()
-
-    # Sanity-check system by restarting all services
-    succeed(
-        "systemctl restart bitcoind clightning lnd lightning-loop spark-wallet lightning-charge nanopos liquidd"
-    )
-
-    # Now that the bitcoind restart triggered a banlist import restart, check that
-    # re-importing already banned addresses works
-    machine.wait_until_succeeds(
-        log_has_string(f"bitcoind-import-banlist --since=@{pre_restart}", "Importing node banlist")
-    )
-    assert_no_failure("bitcoind-import-banlist")
-
-    extra_tests.pop("prestop")()
-
-    ### Test duplicity
-
-    succeed("systemctl stop bitcoind")
-    succeed("systemctl start duplicity")
-    machine.wait_until_succeeds(log_has_string("duplicity", "duplicity.service: Succeeded."))
-    # Make sure files in duplicity backup and /var/lib are identical
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity verify '--archive-dir' '/var/lib/duplicity' 'file:///var/lib/localBackups' '/var/lib'",
-        "0 differences found",
-    )
-    # Make sure duplicity backup includes important files
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity list-current-files 'file:///var/lib/localBackups'",
-        "var/lib/clightning/bitcoin/hsm_secret",
-    )
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity list-current-files 'file:///var/lib/localBackups'",
-        "secrets/lnd-seed-mnemonic",
-    )
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity list-current-files 'file:///var/lib/localBackups'",
-        "secrets/jm-wallet-seed",
-    )
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity list-current-files 'file:///var/lib/localBackups'",
-        "var/lib/bitcoind/wallet.dat",
-    )
-    assert_matches(
-        "export $(cat /secrets/backup-encryption-env); duplicity list-current-files 'file:///var/lib/localBackups'",
-        "var/backup/postgresql/btcpaydb.sql.gz",
-    )
-
-    ### Check that all extra_tests have been run
-    assert len(extra_tests) == 0
-
-
-def test_security():
-    assert_running("setup-secrets")
-    # Unused secrets should be inaccessible
-    succeed('[[ $(stat -c "%U:%G %a" /secrets/dummy) = "root:root 440" ]]')
-
-    # Access to '/proc' should be restricted
-    machine.succeed("grep -Fq hidepid=2 /proc/mounts")
-
-    machine.wait_for_unit("bitcoind")
-    # `systemctl status` run by unprivileged users shouldn't leak cgroup info
-    assert_matches(
-        "sudo -u electrs systemctl status bitcoind 2>&1 >/dev/null",
-        "Failed to dump process list for 'bitcoind.service', ignoring: Access denied",
-    )
-    # The 'operator' with group 'proc' has full access
-    assert_full_match("sudo -u operator systemctl status bitcoind 2>&1 >/dev/null", "")

test/lib/make-container.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+
+# Usage:
+#
+#   run-tests.sh [--scenario|-s <scenario>] container
+#
+#     Start container and start a shell session with helper commands
+#     for accessing the container.
+#     A short command documentation is printed at the start of the session.
+#     The container is destroyed after exiting the shell.
+#     An existing container is destroyed before starting.
+#
+#     Supported arguments:
+#
+#       --destroy|-d to destroy
+#
+#         When `run-tests.sh container` from inside an existing shell session,
+#         the current container is updated without restarting by switching
+#         its NixOS configuration.
+#         Use this arg to destroy and restart the container instead.
+#
+#       --no-destroy|-n
+#
+#         By default, all commands destroy an existing container before starting and,
+#         when appropriate, before exiting.
+#         This ensures that containers start with no leftover filesystem state from
+#         previous runs and that containers don't consume system resources after use.
+#         This args disables auto-destructing containers.
+#
+#
+#   run-tests.sh container --run|-r c systemctl status bitcoind
+#
+#     Run a command in the shell session environmentand exit.
+#     Destroy the container afterwards.
+#     All arguments following `--run` are used as a command.
+#     Supports argument '--no-destroy|-n' (see above for an explanation).
+#
+#     Example: Start shell inside container
+#     run-tests.sh container --run c
+#
+#
+#   run-tests.sh [--scenario|-s <scenario>] container --command|--c
+#
+#     Provide a custom extra-container command.
+#
+#     Example:
+#       run-tests.sh container --command create -s
+#       Create and start a container without a shell.
+#
+#
+#   All extra args are passed to extra-container (unless --command is used):
+#   run-tests.sh container --build-args --builders 'ssh://worker - - 8'
+
+set -euo pipefail
+
+if [[ $EUID != 0 ]]; then
+    # NixOS containers require root permissions.
+    # By using sudo here and not at the user's call-site extra-container can detect if it is running
+    # inside an existing shell session (by checking an internal environment variable).
+    exec sudo scenario="$scenario" testDir="$testDir" NIX_PATH="$NIX_PATH" PATH="$PATH" \
+         scenarioOverridesFile="${scenarioOverridesFile:-}" "$testDir/lib/make-container.sh" "$@"
+fi
+
+export containerName=nb-test
+containerCommand=shell
+
+while [[ $# > 0 ]]; do
+    case $1 in
+        --command|-c)
+            shift
+            containerCommand=$1
+            shift
+            ;;
+        *)
+            break
+    esac
+done
+
+containerBin=$(type -P extra-container) || true
+if [[ ! ($containerBin && $(realpath $containerBin) == *extra-container-0.5*) ]]; then
+    echo "Building extra-container. Skip this step by adding extra-container 0.5 to PATH."
+    nix-build --out-link /tmp/extra-container "$testDir"/../pkgs -A extra-container >/dev/null
+    export PATH="/tmp/extra-container/bin${PATH:+:}$PATH"
+fi
+
+read -d '' src <<EOF || true
+(import "$testDir/tests.nix" { scenario = "$scenario"; }).container
+EOF
+exec extra-container $containerCommand -E "$src" "$@"

test/lib/make-test-vm.nix

@@ -3,11 +3,6 @@ testArgs:
 let
   pkgs = import <nixpkgs> { config = {}; overlays = []; };
 
-  pkgs19_09 = import (pkgs.fetchzip {
-    url = "https://github.com/NixOS/nixpkgs-channels/archive/a7ceb2536ab11973c59750c4c48994e3064a75fa.tar.gz";
-    sha256 = "0hka65f31njqpq7i07l22z5rs7lkdfcl4pbqlmlsvnysb74ynyg1";
-  }) { config = {}; overlays = []; };
-
   test = (import "${pkgs.path}/nixos/tests/make-test-python.nix") testArgs;
 
   fixedTest = { system ? builtins.currentSystem, ... }@args:
@@ -23,11 +18,6 @@ let
              exec ${pkgs.python3Packages.black}/bin/black $extraArgs "$@"
            '';
          };
-
-         # QEMU 4.20 from unstable fails on Travis build nodes with message
-         # "error: failed to set MSR 0x48b to 0x159ff00000000"
-         # Use version 4.0.1 instead.
-         inherit (pkgs19_09) qemu_test;
        };
     in
       test (args // { pkgs = pkgsFixed; });

test/lib/make-test.nix

@@ -0,0 +1,51 @@
+scenario: testConfig:
+
+{
+  vm = import ./make-test-vm.nix {
+    name = "nix-bitcoin-${scenario}";
+
+    machine = {
+      imports = [ testConfig ];
+      # Needed because duplicity requires 270 MB of free temp space, regardless of backup size
+      virtualisation.diskSize = 1024;
+    };
+
+    testScript = nodes: let
+      cfg = nodes.nodes.machine.config;
+      data = {
+        data = cfg.test.data;
+        tests = cfg.tests;
+      };
+      dataFile = builtins.toFile "test-data" (builtins.toJSON data);
+      initData = ''
+        import json
+
+        with open("${dataFile}") as f:
+            data = json.load(f)
+
+        enabled_tests = set(test for (test, enabled) in data["tests"].items() if enabled)
+        test_data = data["data"]
+      '';
+    in
+      builtins.concatStringsSep "\n\n" [
+        initData
+        (builtins.readFile ./../tests.py)
+        # Don't run tests in interactive mode.
+        # is_interactive is set in ../run-tests.sh
+        ''
+          if not "is_interactive" in vars():
+              run_tests()
+        ''
+      ];
+  };
+
+  container = {
+    # The container name has a 11 char length limit
+    containers.nb-test = { config, ...}: {
+      config = {
+        extra = config.config.test.container;
+        config = testConfig;
+      };
+    };
+  };
+}

test/lib/test-lib.nix

@@ -0,0 +1,39 @@
+{ config, lib, ... }:
+with lib;
+{
+  options = {
+    test = {
+      noConnections = mkOption {
+        type = types.bool;
+        default = !config.test.container.enableWAN;
+        description = ''
+          Whether services should be configured to not connect to external hosts.
+          This can silence some warnings while running the test in an offline environment.
+        '';
+      };
+      data = mkOption {
+        type = types.attrs;
+        default = {};
+        description = ''
+          Attrs that are available in the Python test script under the global
+          dictionary variable 'test_data'. The data is exported via JSON.
+        '';
+      };
+
+      container = {
+        # Forwarded to extra-container. For descriptions, see
+        # https://github.com/erikarvstedt/extra-container/blob/master/eval-config.nix
+        addressPrefix = mkOption { default = "10.225.255"; };
+        enableWAN = mkOption { default = false; };
+        firewallAllowHost = mkOption { default = true; };
+        exposeLocalhost = mkOption { default = false; };
+      };
+    };
+
+    tests = mkOption {
+      type = with types; attrsOf bool;
+      default = {};
+      description = "Python tests that should be run.";
+    };
+  };
+}

test/run-tests.sh

@@ -1,14 +1,20 @@
 #!/usr/bin/env bash
 
 # Modules integration test runner.
-# The test (./test.nix) uses the NixOS testing framework and is executed in a VM.
+# The tests (./tests.nix) use the NixOS testing framework and are executed in a VM.
 #
 # Usage:
 #   Run all tests
 #   ./run-tests.sh
 #
 #   Test specific scenario
-#   ./run-tests.sh --scenario <scenario>
+#   ./run-tests.sh --scenario|-s <scenario>
+#
+#     When <scenario> is undefined, the test is run with an adhoc scenario
+#     where services.<scenario> is enabled.
+#
+#     Example:
+#     ./run-tests.sh -s electrs
 #
 #   Run test and link results to avoid garbage collection
 #   ./run-tests.sh [--scenario <scenario>] --out-link-prefix /tmp/nix-bitcoin-test build
@@ -19,8 +25,18 @@
 #   Run interactive test debugging
 #   ./run-tests.sh [--scenario <scenario>] debug
 #
-#   This starts the testing VM and drops you into a Python REPL where you can
+#     This starts the testing VM and drops you into a Python REPL where you can
-#   manually execute the tests from ./test-script.py
+#     manually execute the tests from ./tests.py
+#
+#   Run a test scenario in a container
+#   sudo ./run-tests.sh [--scenario <scenario>] container
+#
+#     This is useful for quick experiments; containers start much faster than VMs.
+#     Running the Python test suite in containers is not yet supported.
+#     For now, creating NixOS containers requires root permissions.
+#     See ./lib/make-container.sh for a complete documentation.
+#
+#   To add custom scenarios, set the environment variable `scenarioOverridesFile`.
 
 set -eo pipefail
 
@@ -34,7 +50,7 @@ while :; do
                 shift
                 shift
             else
-                >&2 echo 'Error: "$1" requires an argument.'
+                >&2 echo "Error: $1 requires an argument."
                 exit 1
             fi
             ;;
@@ -44,7 +60,7 @@ while :; do
                 shift
                 shift
             else
-                >&2 echo 'Error: "$1" requires an argument.'
+                >&2 echo "Error: $1 requires an argument."
                 exit 1
             fi
             ;;
@@ -57,9 +73,9 @@ numCPUs=${numCPUs:-$(nproc)}
 # Min. 800 MiB needed to avoid 'out of memory' errors
 memoryMiB=${memoryMiB:-2048}
 
-scriptDir=$(cd "${BASH_SOURCE[0]%/*}" && pwd)
+testDir=$(cd "${BASH_SOURCE[0]%/*}" && pwd)
 
-export NIX_PATH=nixpkgs=$(nix eval --raw -f "$scriptDir/../pkgs/nixpkgs-pinned.nix" nixpkgs)
+export NIX_PATH=nixpkgs=$(nix eval --raw -f "$testDir/../pkgs/nixpkgs-pinned.nix" nixpkgs)
 
 # Run the test. No temporary files are left on the host system.
 run() {
@@ -67,20 +83,14 @@ run() {
     export TMPDIR=$(mktemp -d /tmp/nix-bitcoin-test.XXX)
     trap "rm -rf $TMPDIR" EXIT
 
-    nix-build --out-link $TMPDIR/driver -E "import \"$scriptDir/test.nix\" { scenario = \"$scenario\"; }" -A driver
+    nix-build --out-link $TMPDIR/driver -E "(import \"$testDir/tests.nix\" { scenario = \"$scenario\"; }).vm" -A driver
 
     # Variable 'tests' contains the Python code that is executed by the driver on startup
     if [[ $1 == --interactive ]]; then
         echo "Running interactive testing environment"
         tests=$(
             echo 'is_interactive = True'
-            # The test script raises an error when 'is_interactive' is defined so
+            echo 'exec(os.environ["testScript"])'
-            # that it just loads the initial helper functions and stops before
-            # executing the actual tests
-            echo 'try:'
-            echo '    exec(os.environ["testScript"])'
-            echo 'except:'
-            echo '    pass'
             # Start VM
             echo 'start_all()'
             # Start REPL
@@ -109,6 +119,15 @@ debug() {
     run --interactive
 }
 
+evalTest() {
+    nix eval --raw "($(vmTestNixExpr)).outPath"
+    echo # nix eval doesn't print a newline
+}
+
+container() {
+  . "$testDir/lib/make-container.sh" "$@"
+}
+
 # Run the test by building the test derivation
 buildTest() {
     if [[ $outLinkPrefix ]]; then
@@ -130,14 +149,19 @@ exprForCI() {
     ((memAvailableMiB < memoryMiB)) && memoryMiB=$memAvailableMiB
     >&2 echo "VM stats: CPUs: $numCPUs, memory: $memoryMiB MiB"
     >&2 echo "Host memory total: $((memTotalKiB / 1024)) MiB, available: $memAvailableMiB MiB"
-    vmTestNixExpr
+
+    # VMX is usually not available on CI nodes due to recursive virtualisation.
+    # Explicitly disable VMX, otherwise QEMU 4.20 fails with message
+    # "error: failed to set MSR 0x48b to 0x159ff00000000"
+    vmTestNixExpr "-cpu host,-vmx"
 }
 
 vmTestNixExpr() {
+  extraQEMUOpts="$1"
   cat <<EOF
-    (import "$scriptDir/test.nix" { scenario = "$scenario"; } {}).overrideAttrs (old: rec {
+    ((import "$testDir/tests.nix" { scenario = "$scenario"; }).vm {}).overrideAttrs (old: rec {
       buildCommand = ''
-        export QEMU_OPTS="-smp $numCPUs -m $memoryMiB"
+        export QEMU_OPTS="-smp $numCPUs -m $memoryMiB $extraQEMUOpts"
         echo "VM stats: CPUs: $numCPUs, memory: $memoryMiB MiB"
       '' + old.buildCommand;
     })
@@ -149,7 +173,8 @@ build() {
         buildTest "$@"
     else
         scenario=default buildTest "$@"
-        scenario=withnetns buildTest "$@"
+        scenario=netns buildTest "$@"
+        scenario=full evalTest "$@"
     fi
 }
 
@@ -158,4 +183,9 @@ if [[ $1 && $1 != build ]]; then
     : ${scenario:=default}
 fi
 
-eval "${@:-build}"
+command="${1:-build}"
+shift || true
+if [[ $command == eval ]]; then
+    command=evalTest
+fi
+$command "$@"

test/scenarios/default.py

@@ -1,56 +0,0 @@
-def electrs():
-    machine.wait_for_open_port(4224)  # prometeus metrics provider
-
-
-def nbxplorer():
-    machine.wait_for_open_port(24444)
-
-
-def btcpayserver():
-    machine.wait_for_open_port(23000)
-    # test lnd custom macaroon
-    assert_matches(
-        'sudo -u btcpayserver curl -s --cacert /secrets/lnd-cert --header "Grpc-Metadata-macaroon: $(xxd -ps -u -c 1000 /run/lnd/btcpayserver.macaroon)" -X GET https://127.0.0.1:8080/v1/getinfo | jq',
-        '"version"',
-    )
-
-
-def spark_wallet():
-    machine.wait_for_open_port(9737)
-    spark_auth = re.search("login=(.*)", succeed("cat /secrets/spark-wallet-login"))[1]
-    assert_matches(f"curl -s {spark_auth}@localhost:9737", "Spark")
-
-
-def lightning_charge():
-    machine.wait_for_open_port(9112)
-    charge_auth = re.search("API_TOKEN=(.*)", succeed("cat /secrets/lightning-charge-env"))[1]
-    assert_matches(f"curl -s api-token:{charge_auth}@localhost:9112/info | jq", '"id"')
-
-
-def nanopos():
-    machine.wait_for_open_port(9116)
-    assert_matches("curl localhost:9116", "tshirt")
-
-
-def web_index():
-    machine.wait_for_open_port(80)
-    assert_matches("curl localhost", "nix-bitcoin")
-    assert_matches("curl -L localhost/store", "tshirt")
-
-
-def prestop():
-    pass
-
-
-extra_tests = {
-    "electrs": electrs,
-    "nbxplorer": nbxplorer,
-    "btcpayserver": btcpayserver,
-    "spark-wallet": spark_wallet,
-    "lightning-charge": lightning_charge,
-    "nanopos": nanopos,
-    "web-index": web_index,
-    "prestop": prestop,
-}
-
-run_tests(extra_tests)

test/scenarios/withnetns.py

@@ -1,126 +0,0 @@
-# netns IP addresses
-bitcoind_ip = "169.254.1.12"
-clightning_ip = "169.254.1.13"
-lnd_ip = "169.254.1.14"
-liquidd_ip = "169.254.1.15"
-electrs_ip = "169.254.1.16"
-sparkwallet_ip = "169.254.1.17"
-lightningcharge_ip = "169.254.1.18"
-nanopos_ip = "169.254.1.19"
-recurringdonations_ip = "169.254.1.20"
-nginx_ip = "169.254.1.21"
-lightningloop_ip = "169.254.1.22"
-nbxplorer_ip = "169.254.1.23"
-btcpayserver_ip = "169.254.1.24"
-
-
-def electrs():
-    machine.wait_until_succeeds(
-        "ip netns exec nb-electrs nc -z localhost 4224"
-    )  # prometeus metrics provider
-
-
-def nbxplorer():
-    machine.wait_until_succeeds("ip netns exec nb-nbxplorer nc -z %s 24444" % nbxplorer_ip)
-
-
-def btcpayserver():
-    machine.wait_until_succeeds("ip netns exec nb-btcpayserver nc -z %s 23000" % btcpayserver_ip)
-    # test lnd custom macaroon
-    assert_matches(
-        'ip netns exec nb-btcpayserver sudo -u btcpayserver curl -s --cacert /secrets/lnd-cert --header "Grpc-Metadata-macaroon: $(xxd -ps -u -c 1000 /run/lnd/btcpayserver.macaroon)" -X GET https://%s:8080/v1/getinfo | jq'
-        % lnd_ip,
-        '"version"',
-    )
-
-
-def spark_wallet():
-    machine.wait_until_succeeds("ip netns exec nb-spark-wallet nc -z %s 9737" % sparkwallet_ip)
-    spark_auth = re.search("login=(.*)", succeed("cat /secrets/spark-wallet-login"))[1]
-    assert_matches(
-        f"ip netns exec nb-spark-wallet curl -s {spark_auth}@%s:9737" % sparkwallet_ip, "Spark"
-    )
-
-
-def lightning_charge():
-    machine.wait_until_succeeds("ip netns exec nb-nanopos nc -z %s 9112" % lightningcharge_ip)
-    charge_auth = re.search("API_TOKEN=(.*)", succeed("cat /secrets/lightning-charge-env"))[1]
-    assert_matches(
-        f"ip netns exec nb-nanopos curl -s api-token:{charge_auth}@%s:9112/info | jq"
-        % lightningcharge_ip,
-        '"id"',
-    )
-
-
-def nanopos():
-    machine.wait_until_succeeds("ip netns exec nb-lightning-charge nc -z %s 9116" % nanopos_ip)
-    assert_matches("ip netns exec nb-lightning-charge curl %s:9116" % nanopos_ip, "tshirt")
-
-
-def web_index():
-    machine.wait_until_succeeds("ip netns exec nb-nginx nc -z localhost 80")
-    assert_matches("ip netns exec nb-nginx curl localhost", "nix-bitcoin")
-    assert_matches("ip netns exec nb-nginx curl -L localhost/store", "tshirt")
-
-
-def prestop():
-    ping_bitcoind = "ip netns exec nb-bitcoind ping -c 1 -w 1"
-    ping_nanopos = "ip netns exec nb-nanopos ping -c 1 -w 1"
-    ping_nbxplorer = "ip netns exec nb-nbxplorer ping -c 1 -w 1"
-
-    # Positive ping tests (non-exhaustive)
-    machine.succeed(
-        "%s %s &&" % (ping_bitcoind, bitcoind_ip)
-        + "%s %s &&" % (ping_bitcoind, clightning_ip)
-        + "%s %s &&" % (ping_bitcoind, lnd_ip)
-        + "%s %s &&" % (ping_bitcoind, liquidd_ip)
-        + "%s %s &&" % (ping_bitcoind, nbxplorer_ip)
-        + "%s %s &&" % (ping_nbxplorer, btcpayserver_ip)
-        + "%s %s &&" % (ping_nanopos, lightningcharge_ip)
-        + "%s %s &&" % (ping_nanopos, nanopos_ip)
-        + "%s %s" % (ping_nanopos, nginx_ip)
-    )
-
-    # Negative ping tests (non-exhaustive)
-    machine.fail(
-        "%s %s ||" % (ping_bitcoind, sparkwallet_ip)
-        + "%s %s ||" % (ping_bitcoind, lightningloop_ip)
-        + "%s %s ||" % (ping_bitcoind, lightningcharge_ip)
-        + "%s %s ||" % (ping_bitcoind, nanopos_ip)
-        + "%s %s ||" % (ping_bitcoind, recurringdonations_ip)
-        + "%s %s ||" % (ping_bitcoind, nginx_ip)
-        + "%s %s ||" % (ping_nanopos, bitcoind_ip)
-        + "%s %s ||" % (ping_nanopos, clightning_ip)
-        + "%s %s ||" % (ping_nanopos, lnd_ip)
-        + "%s %s ||" % (ping_nanopos, lightningloop_ip)
-        + "%s %s ||" % (ping_nanopos, liquidd_ip)
-        + "%s %s ||" % (ping_nanopos, electrs_ip)
-        + "%s %s ||" % (ping_nanopos, sparkwallet_ip)
-        + "%s %s ||" % (ping_nanopos, recurringdonations_ip)
-        + "%s %s" % (ping_nanopos, btcpayserver_ip)
-    )
-
-    # test that netns-exec can't be run for unauthorized namespace
-    machine.fail("netns-exec nb-electrs ip a")
-
-    # test that netns-exec drops capabilities
-    assert_full_match(
-        "su operator -c 'netns-exec nb-bitcoind capsh --print | grep Current '", "Current: =\n"
-    )
-
-    # test that netns-exec can not be executed by users that are not operator
-    machine.fail("sudo -u clightning netns-exec nb-bitcoind ip a")
-
-
-extra_tests = {
-    "electrs": electrs,
-    "nbxplorer": nbxplorer,
-    "btcpayserver": btcpayserver,
-    "spark-wallet": spark_wallet,
-    "lightning-charge": lightning_charge,
-    "nanopos": nanopos,
-    "web-index": web_index,
-    "prestop": prestop,
-}
-
-run_tests(extra_tests)

test/test.nix

@@ -1,76 +0,0 @@
-# Integration test, can be run without internet access.
-
-# Make sure to update build() in ./run-tests.sh when adding new scenarios
-{ scenario ? "default" }:
-
-import ./make-test.nix rec {
-  name = "nix-bitcoin-${scenario}";
-
-  hardened = {
-    imports = [ <nixpkgs/nixos/modules/profiles/hardened.nix> ];
-    security.allowUserNamespaces = true; # re-enable disabled option
-  };
-
-  machine = { pkgs, lib, ... }: with lib; {
-    imports = [
-      ../modules/presets/secure-node.nix
-      ../modules/secrets/generate-secrets.nix
-      # using the hardened profile increases total test duration by ~50%, so disable it for now
-      # hardened
-    ];
-
-    # needed because duplicity requires 270 MB of free temp space, regardless of backup size.
-    virtualisation.diskSize = 1024;
-
-    nix-bitcoin.netns-isolation.enable = (scenario == "withnetns");
-
-    services.bitcoind.extraConfig = mkForce "connect=0";
-
-    services.clightning.enable = true;
-    services.spark-wallet.enable = true;
-    services.lightning-charge.enable = true;
-    services.nanopos.enable = true;
-
-    services.lnd.enable = true;
-    services.lnd.listenPort = 9736;
-    services.lightning-loop.enable = true;
-
-    services.electrs.enable = true;
-
-    services.liquidd = {
-      enable = true;
-      listen = mkForce false;
-      extraConfig = "noconnect=1";
-    };
-
-    services.nix-bitcoin-webindex.enable = true;
-
-    services.hardware-wallets = {
-      trezor = true;
-      ledger = true;
-    };
-
-    services.backups.enable = true;
-
-    services.btcpayserver.enable = true;
-    services.btcpayserver.lightningBackend = "lnd";
-    # needed to test macaroon creation
-    environment.systemPackages = with pkgs; [ openssl xxd ];
-    
-    services.joinmarket.enable = true;
-    services.joinmarket.yieldgenerator = {
-      enable = true;
-      customParameters = ''
-        txfee = 200
-        cjfee_a = 300
-      '';
-    };
-
-    # to test that unused secrets are made inaccessible by 'setup-secrets'
-    systemd.services.generate-secrets.postStart = ''
-      install -o nobody -g nogroup -m777 <(:) /secrets/dummy
-    '';
-  };
-  testScript =
-    builtins.readFile ./base.py + "\n\n" + builtins.readFile "${./.}/scenarios/${scenario}.py";
-}

test/tests.nix

@@ -0,0 +1,151 @@
+# Integration tests, can be run without internet access.
+
+{ scenario ? "default" }:
+
+import ./lib/make-test.nix scenario (
+{ config, pkgs, lib, ... }: with lib;
+let testEnv = rec {
+  cfg = config.services;
+  mkIfTest = test: mkIf (config.tests.${test} or false);
+
+  baseConfig = {
+    imports = [
+      ./lib/test-lib.nix
+      ../modules/modules.nix
+      ../modules/secrets/generate-secrets.nix
+    ];
+
+    config = {
+      tests.bitcoind = cfg.bitcoind.enable;
+      services.bitcoind = {
+        enable = true;
+        extraConfig = mkIf config.test.noConnections "connect=0";
+      };
+
+      tests.clightning = cfg.clightning.enable;
+
+      tests.spark-wallet = cfg.spark-wallet.enable;
+
+      tests.nanopos = cfg.nanopos.enable;
+
+      tests.lnd = cfg.lnd.enable;
+      services.lnd.listenPort = 9736;
+
+      tests.lightning-loop = cfg.lightning-loop.enable;
+
+      tests.electrs = cfg.electrs.enable;
+
+      tests.liquidd = cfg.liquidd.enable;
+      services.liquidd.extraConfig = mkIf config.test.noConnections "connect=0";
+
+      tests.btcpayserver = cfg.btcpayserver.enable;
+      services.btcpayserver.lightningBackend = "lnd";
+      # Needed to test macaroon creation
+      environment.systemPackages = mkIfTest "btcpayserver" (with pkgs; [ openssl xxd ]);
+
+      tests.joinmarket = cfg.joinmarket.enable;
+      services.joinmarket.yieldgenerator = {
+        enable = config.services.joinmarket.enable;
+        customParameters = ''
+          txfee = 200
+          cjfee_a = 300
+        '';
+      };
+
+      tests.backups = cfg.backups.enable;
+
+      # To test that unused secrets are made inaccessible by 'setup-secrets'
+      systemd.services.generate-secrets.postStart = mkIfTest "security" ''
+        install -o nobody -g nogroup -m777 <(:) /secrets/dummy
+      '';
+    };
+  };
+
+  scenarios = {
+    base = baseConfig; # Included in all scenarios
+
+    default = scenarios.secureNode;
+
+    # All available basic services and tests
+    full = {
+      tests.security = true;
+
+      services.clightning.enable = true;
+      services.spark-wallet.enable = true;
+      services.lightning-charge.enable = true;
+      services.nanopos.enable = true;
+      services.lnd.enable = true;
+      services.lightning-loop.enable = true;
+      services.electrs.enable = true;
+      services.liquidd.enable = true;
+      services.btcpayserver.enable = true;
+      services.joinmarket.enable = true;
+      services.backups.enable = true;
+
+      services.hardware-wallets = {
+        trezor = true;
+        ledger = true;
+      };
+    };
+
+    secureNode = {
+      imports = [
+        scenarios.full
+        ../modules/presets/secure-node.nix
+      ];
+      services.nix-bitcoin-webindex.enable = true;
+      tests.secure-node = true;
+      tests.banlist-and-restart = true;
+    };
+
+    netns = {
+      imports = [ scenarios.secureNode ];
+      nix-bitcoin.netns-isolation.enable = true;
+      test.data.netns = config.nix-bitcoin.netns-isolation.netns;
+      tests.netns-isolation = true;
+
+      # This test is rather slow and unaffected by netns settings
+      tests.backups = mkForce false;
+    };
+
+    ## Examples / debug helper
+
+    # Run a selection of tests in scenario 'netns'
+    selectedTests = {
+      imports = [ scenarios.netns ];
+      tests = mkForce {
+        btcpayserver = true;
+        netns-isolation = true;
+      };
+    };
+
+    # Container-specific features
+    containerFeatures = {
+      # Container has WAN access and bitcoind connects to external nodes
+      test.container.enableWAN = true;
+      # See ./lib/test-lib.nix for a description
+      test.container.exposeLocalhost = true;
+    };
+
+    adhoc = {
+      # <Add your config here>
+      # You can also set the env var `scenarioOverridesFile` (used below) to define custom scenarios.
+    };
+  };
+};
+in
+  let
+    overrides = builtins.getEnv "scenarioOverridesFile";
+    scenarios = testEnv.scenarios // (optionalAttrs (overrides != "") (import overrides {
+      inherit testEnv config pkgs lib;
+    }));
+    autoScenario = {
+      services.${scenario}.enable = true;
+    };
+  in {
+    imports = [
+      scenarios.base
+      (scenarios.${scenario} or autoScenario)
+    ];
+  }
+)

test/tests.py

@@ -0,0 +1,330 @@
+from collections import OrderedDict
+
+
+def succeed(*cmds):
+    """Returns the concatenated output of all cmds"""
+    return machine.succeed(*cmds)
+
+
+def assert_matches(cmd, regexp):
+    out = succeed(cmd)
+    if not re.search(regexp, out):
+        raise Exception(f"Pattern '{regexp}' not found in '{out}'")
+
+
+def assert_full_match(cmd, regexp):
+    out = succeed(cmd)
+    if not re.fullmatch(regexp, out):
+        raise Exception(f"Pattern '{regexp}' doesn't match '{out}'")
+
+
+def log_has_string(unit, str):
+    return f"journalctl -b --output=cat -u {unit} --grep='{str}'"
+
+
+def assert_no_failure(unit):
+    """Unit should not have failed since the system is running"""
+    machine.fail(log_has_string(unit, "Failed with result"))
+
+
+def assert_running(unit):
+    with machine.nested(f"waiting for unit: {unit}"):
+        machine.wait_for_unit(unit)
+    assert_no_failure(unit)
+
+
+def wait_for_open_port(address, port):
+    def is_port_open(_):
+        status, _ = machine.execute(f"nc -z {address} {port}")
+        return status == 0
+
+    with log.nested(f"Waiting for TCP port {address}:{port}"):
+        retry(is_port_open)
+
+
+### Test runner
+
+tests = OrderedDict()
+
+
+def test(name):
+    def x(fn):
+        tests[name] = fn
+
+    return x
+
+
+def run_tests():
+    enabled = enabled_tests.copy()
+    to_run = []
+    for test in tests:
+        if test in enabled:
+            enabled.remove(test)
+            to_run.append(test)
+    if enabled:
+        raise RuntimeError(f"The following tests are enabled but not defined: {enabled}")
+    machine.connect()  # Visually separate boot output from the test output
+    for test in to_run:
+        with log.nested(f"test: {test}"):
+            tests[test]()
+
+
+def run_test(test):
+    tests[test]()
+
+
+### Tests
+# All tests are executed in the order they are defined here
+
+
+@test("security")
+def _():
+    assert_running("setup-secrets")
+    # Unused secrets should be inaccessible
+    succeed('[[ $(stat -c "%U:%G %a" /secrets/dummy) = "root:root 440" ]]')
+
+    if "secure-node" in enabled_tests:
+        # Access to '/proc' should be restricted
+        machine.succeed("grep -Fq hidepid=2 /proc/mounts")
+
+        machine.wait_for_unit("bitcoind")
+        # `systemctl status` run by unprivileged users shouldn't leak cgroup info
+        assert_matches(
+            "sudo -u electrs systemctl status bitcoind 2>&1 >/dev/null",
+            "Failed to dump process list for 'bitcoind.service', ignoring: Access denied",
+        )
+        # The 'operator' with group 'proc' has full access
+        assert_full_match("sudo -u operator systemctl status bitcoind 2>&1 >/dev/null", "")
+
+
+@test("bitcoind")
+def _():
+    assert_running("bitcoind")
+    machine.wait_until_succeeds("bitcoin-cli getnetworkinfo")
+    assert_matches("su operator -c 'bitcoin-cli getnetworkinfo' | jq", '"version"')
+    # RPC access for user 'public' should be restricted
+    machine.fail(
+        "bitcoin-cli -rpcuser=public -rpcpassword=$(cat /secrets/bitcoin-rpcpassword-public) stop"
+    )
+    machine.wait_until_succeeds(
+        log_has_string("bitcoind", "RPC User public not allowed to call method stop")
+    )
+
+
+# Impure: Stops electrs
+@test("electrs")
+def _():
+    assert_running("electrs")
+    wait_for_open_port(ip("electrs"), 4224)  # prometeus metrics provider
+    # Check RPC connection to bitcoind
+    machine.wait_until_succeeds(log_has_string("electrs", "NetworkInfo"))
+    # Stop electrs from spamming the test log with 'wait for bitcoind sync' messages
+    succeed("systemctl stop electrs")
+
+
+@test("liquidd")
+def _():
+    assert_running("liquidd")
+    machine.wait_until_succeeds("elements-cli getnetworkinfo")
+    assert_matches("su operator -c 'elements-cli getnetworkinfo' | jq", '"version"')
+    succeed("su operator -c 'liquidswap-cli --help'")
+
+
+@test("clightning")
+def _():
+    assert_running("clightning")
+    assert_matches("su operator -c 'lightning-cli getinfo' | jq", '"id"')
+
+
+@test("lnd")
+def _():
+    assert_running("lnd")
+    assert_matches("su operator -c 'lncli getinfo' | jq", '"version"')
+    assert_no_failure("lnd")
+
+
+@test("lightning-loop")
+def _():
+    assert_running("lightning-loop")
+    assert_matches("su operator -c 'loop --version'", "version")
+    # Check that lightning-loop fails with the right error, making sure
+    # lightning-loop can connect to lnd
+    machine.wait_until_succeeds(
+        log_has_string(
+            "lightning-loop",
+            "Waiting for lnd to be fully synced to its chain backend, this might take a while",
+        )
+    )
+
+
+@test("btcpayserver")
+def _():
+    assert_running("nbxplorer")
+    machine.wait_until_succeeds(log_has_string("nbxplorer", "BTC: RPC connection successful"))
+    wait_for_open_port(ip("nbxplorer"), 24444)
+    assert_running("btcpayserver")
+    machine.wait_until_succeeds(log_has_string("btcpayserver", "Listening on"))
+    wait_for_open_port(ip("btcpayserver"), 23000)
+    # test lnd custom macaroon
+    assert_matches(
+        "sudo -u btcpayserver curl -s --cacert /secrets/lnd-cert "
+        '--header "Grpc-Metadata-macaroon: $(xxd -ps -u -c 1000 /run/lnd/btcpayserver.macaroon)" '
+        f"-X GET https://{ip('lnd')}:8080/v1/getinfo | jq",
+        '"version"',
+    )
+
+
+@test("spark-wallet")
+def _():
+    assert_running("spark-wallet")
+    wait_for_open_port(ip("spark-wallet"), 9737)
+    spark_auth = re.search("login=(.*)", succeed("cat /secrets/spark-wallet-login"))[1]
+    assert_matches(f"curl -s {spark_auth}@{ip('spark-wallet')}:9737", "Spark")
+
+
+@test("lightning-charge")
+def _():
+    assert_running("lightning-charge")
+    wait_for_open_port(ip("lightning-charge"), 9112)
+    machine.wait_until_succeeds(f"nc -z {ip('lightning-charge')} 9112")
+    charge_auth = re.search("API_TOKEN=(.*)", succeed("cat /secrets/lightning-charge-env"))[1]
+    assert_matches(
+        f"curl -s api-token:{charge_auth}@{ip('lightning-charge')}:9112/info | jq", '"id"'
+    )
+
+
+@test("nanopos")
+def _():
+    assert_running("nanopos")
+    wait_for_open_port(ip("nanopos"), 9116)
+    assert_matches(f"curl {ip('nanopos')}:9116", "tshirt")
+
+
+@test("joinmarket")
+def _():
+    assert_running("joinmarket")
+    machine.wait_until_succeeds(
+        log_has_string("joinmarket", "P2EPDaemonServerProtocolFactory starting on 27184")
+    )
+    machine.wait_until_succeeds(
+        log_has_string("joinmarket-yieldgenerator", "Failure to get blockheight",)
+    )
+
+
+@test("secure-node")
+def _():
+    assert_running("onion-chef")
+
+    # FIXME: use 'wait_for_unit' because 'create-web-index' always fails during startup due
+    # to incomplete unit dependencies.
+    # 'create-web-index' implicitly tests 'nodeinfo'.
+    machine.wait_for_unit("create-web-index")
+    assert_running("nginx")
+    wait_for_open_port(ip("nginx"), 80)
+    assert_matches(f"curl {ip('nginx')}", "nix-bitcoin")
+    assert_matches(f"curl -L {ip('nginx')}/store", "tshirt")
+
+
+# Run this test before the following tests that shut down services
+# (and their corresponding network namespaces).
+@test("netns-isolation")
+def _():
+    ping_bitcoind = "ip netns exec nb-bitcoind ping -c 1 -w 1"
+    ping_nanopos = "ip netns exec nb-nanopos ping -c 1 -w 1"
+    ping_nbxplorer = "ip netns exec nb-nbxplorer ping -c 1 -w 1"
+
+    # Positive ping tests (non-exhaustive)
+    machine.succeed(
+        "%s %s &&" % (ping_bitcoind, ip("bitcoind"))
+        + "%s %s &&" % (ping_bitcoind, ip("clightning"))
+        + "%s %s &&" % (ping_bitcoind, ip("lnd"))
+        + "%s %s &&" % (ping_bitcoind, ip("liquidd"))
+        + "%s %s &&" % (ping_bitcoind, ip("nbxplorer"))
+        + "%s %s &&" % (ping_nbxplorer, ip("btcpayserver"))
+        + "%s %s &&" % (ping_nanopos, ip("lightning-charge"))
+        + "%s %s &&" % (ping_nanopos, ip("nanopos"))
+        + "%s %s" % (ping_nanopos, ip("nginx"))
+    )
+
+    # Negative ping tests (non-exhaustive)
+    machine.fail(
+        "%s %s ||" % (ping_bitcoind, ip("spark-wallet"))
+        + "%s %s ||" % (ping_bitcoind, ip("lightning-loop"))
+        + "%s %s ||" % (ping_bitcoind, ip("lightning-charge"))
+        + "%s %s ||" % (ping_bitcoind, ip("nanopos"))
+        + "%s %s ||" % (ping_bitcoind, ip("nginx"))
+        + "%s %s ||" % (ping_nanopos, ip("bitcoind"))
+        + "%s %s ||" % (ping_nanopos, ip("clightning"))
+        + "%s %s ||" % (ping_nanopos, ip("lnd"))
+        + "%s %s ||" % (ping_nanopos, ip("lightning-loop"))
+        + "%s %s ||" % (ping_nanopos, ip("liquidd"))
+        + "%s %s ||" % (ping_nanopos, ip("electrs"))
+        + "%s %s ||" % (ping_nanopos, ip("spark-wallet"))
+        + "%s %s" % (ping_nanopos, ip("btcpayserver"))
+    )
+
+    # test that netns-exec can't be run for unauthorized namespace
+    machine.fail("netns-exec nb-electrs ip a")
+
+    # test that netns-exec drops capabilities
+    assert_full_match(
+        "su operator -c 'netns-exec nb-bitcoind capsh --print | grep Current '", "Current: =\n"
+    )
+
+    # test that netns-exec can not be executed by users that are not operator
+    machine.fail("sudo -u clightning netns-exec nb-bitcoind ip a")
+
+
+# Impure: stops bitcoind (and dependent services)
+@test("backups")
+def _():
+    succeed("systemctl stop bitcoind")
+    succeed("systemctl start duplicity")
+    machine.wait_until_succeeds(log_has_string("duplicity", "duplicity.service: Succeeded."))
+    run_duplicity = "export $(cat /secrets/backup-encryption-env); duplicity"
+    # Files in backup and /var/lib should be identical
+    assert_matches(
+        f"{run_duplicity} verify --archive-dir /var/lib/duplicity file:///var/lib/localBackups /var/lib",
+        "0 differences found",
+    )
+    # Backup should include important files
+    files = succeed(f"{run_duplicity} list-current-files file:///var/lib/localBackups")
+    assert "var/lib/clightning/bitcoin/hsm_secret" in files
+    assert "secrets/lnd-seed-mnemonic" in files
+    assert "secrets/jm-wallet-seed" in files
+    assert "var/lib/bitcoind/wallet.dat" in files
+    assert "var/backup/postgresql/btcpaydb.sql.gz" in files
+
+
+# Impure: restarts services
+@test("banlist-and-restart")
+def _():
+    machine.wait_until_succeeds(log_has_string("bitcoind-import-banlist", "Importing node banlist"))
+    assert_no_failure("bitcoind-import-banlist")
+
+    # Current time in µs
+    pre_restart = succeed("date +%s.%6N").rstrip()
+
+    # Sanity-check system by restarting all services
+    succeed(
+        "systemctl restart bitcoind clightning lnd lightning-loop spark-wallet lightning-charge nanopos liquidd"
+    )
+
+    # Now that the bitcoind restart triggered a banlist import restart, check that
+    # re-importing already banned addresses works
+    machine.wait_until_succeeds(
+        log_has_string(f"bitcoind-import-banlist --since=@{pre_restart}", "Importing node banlist")
+    )
+    assert_no_failure("bitcoind-import-banlist")
+
+
+if "netns-isolation" in enabled_tests:
+
+    def ip(name):
+        return test_data["netns"][name]["address"]
+
+
+else:
+
+    def ip(_):
+        return "127.0.0.1"