Commit Graph

66 Commits

Author SHA1 Message Date
Erik Arvstedt
a51f7b419e
run-tests: use arg instead of env var for scenario overrides
This removes a source of implicit state and guarantees that regular
calls to `run-tests.sh` always run the builtin tests.
2023-01-06 23:46:43 +01:00
Erik Arvstedt
edbaeb9813
tests: define tests via flake
Advantages:
- Pure test evaluations
- The test framework can now be used by flakes that extend nix-bitcoin
- Most features of `run-tests.sh` are now accessible via `nix build`/`nix run`.
  We keep `run-tests.sh` for advanced features like `scenarioOverridesFile` and adhoc scenarios.

Other changes:
- `run-tests.sh` now builds aggregate VM tests like `basic` or
  `buildable` by creating all VMs in a single evaluation.
  This speeds up the tests and eases debugging by separating the eval and build steps.
- Use the new `nix` CLI which has improved build output logging
  by prefixing output lines with the origin drv name.
2022-11-03 23:08:06 +01:00
Erik Arvstedt
450de19803
tests/run-tests.sh: print examples before running
This eases debugging example failures.
2022-11-03 23:08:05 +01:00
Erik Arvstedt
5f1bb2a8fc
tests/copy-src: always copy .git dir
This is required by a later commit that introduces flakes-based test
evaluation. Evaluating local flakes needs a repo dir.
2022-11-03 23:08:05 +01:00
Jonas Nick
7c16fc5865
Merge fort-nix/nix-bitcoin#563: lnd: fix missing RPC permissions when bitcoind is pruned
67949a002a lnd: fix missing RPC permissions when bitcoind is pruned (Erik Arvstedt)
49303be2e0 test/shellcheck-services: fix error by excluding unavailable services (Erik Arvstedt)
46f17fe313 test/shellcheck-services: simplify accessing service definitions (Erik Arvstedt)

Pull request description:

ACKs for top commit:
  jonasnick:
    ACK 67949a002a

Tree-SHA512: 28652d8ec67a164aef068f3df32d1ae8df4e0920cafedc6e3d568b631333b29e57f7370e54a82e7cde9710a3df0a1494ed94272af101d31dd7859a08bb363e4b
2022-10-28 08:38:28 +00:00
Erik Arvstedt
67949a002a
lnd: fix missing RPC permissions when bitcoind is pruned 2022-10-25 22:56:51 +02:00
Erik Arvstedt
277510c7ee
tests: run flake-info in sandbox
Don't use sandboxing in Cirrus CI where namespace support is missing.
2022-10-25 22:04:17 +02:00
Erik Arvstedt
d3b7e8c432
revert "tests: disable nixosSearch" 2022-09-23 09:04:57 +02:00
Erik Arvstedt
c63da4734e
tests: disable nixosSearch 2022-09-05 21:11:15 +02:00
Erik Arvstedt
af115d746b
internal scripts: use pinned, cached pkgs
Instead of setting up the script PATH via nix-shell, use
`nix shell` with inputs from the nix-bitcoin flake.
Advantages:
- Uses the nixpkgs version from the nix-bitcoin flake instead of
  `<nixpkgs>` from the user env (NIX_PATH), so the script runtime
  env is reproducible.
- The pkg derivations for the runtime env are cached, which greatly
  increases script startup speed.

This commit was generated by running the following script inside the
repo root dir:

def transform(path, src)
  if src =~ /#! *nix-shell +-i +bash +-p +(.*)/
    pkgs = $1
    if src =~ /^.*?(set -e.*?pipefail)\n/
      set_statement = $1
      src.sub!($&, '')
    end
    src.sub!(/\A.*?#! *nix-shell.*?\n/m, '')

    parents = ([ '..' ] * (path.split('/').count - 1)).join('/')

    [
      '#!/usr/bin/env bash',
      *set_statement,
      %(. "${BASH_SOURCE[0]%/*}/#{parents}/helper/run-in-nix-env" "#{pkgs}" "$@"),
      nil,
      src
    ].join("\n")
  end
end

Dir['**/*.sh'].each do |f|
  src = File.read(f)
  if new_src = transform(f, src)
    puts "Changed file #{f}"
    File.write(f, new_src)
  end
end
2022-08-28 23:49:12 +02:00
Otto Sabart
f184bb34e6
shellcheck: fix lint of scripts in tests 2022-08-28 18:25:37 +02:00
Otto Sabart
9a92d29111
tests: add shellcheck 2022-08-28 18:25:37 +02:00
Erik Arvstedt
952cd3773f
run-tests: support tests with multiple VMs in run 2022-08-05 13:48:36 +00:00
nixbitcoin
17507835fc
clightning: native database replication
Don't put `clightning.replication` options in
`examples/configuration.nix` until it is more "battle-tested."
2022-08-05 13:48:24 +00:00
Erik Arvstedt
2ae2eea54e
run-tests: fix command debug on NixOS 22.05 2022-07-05 10:13:55 +02:00
Erik Arvstedt
98bc0917d7
run-tests: fix vm cmd
Previously, qemu failed with error
`restrict=on : Parameter 'restrict' expects 'on' or 'off'`
due to the trailing space after `on`.
2022-01-04 22:30:35 +01:00
Erik Arvstedt
f13ffe0146
run-tests: make nixosSearch compatible with --copy-src 2021-12-15 10:58:04 +01:00
Erik Arvstedt
0fd3be5343
move ci/ to test/ 2021-12-12 16:21:11 +01:00
Erik Arvstedt
eaf98f92db
test: add test nixosSearch
This checks that creating module docs for search.nixos.org succceeds.
Errors can happen when complex `default` values can't be evaluated
or on malformed Docbook XML in descriptions.
2021-12-12 16:21:02 +01:00
Erik Arvstedt
77d8c00c53
run-tests.sh: extract checkFlakeSupport
Required by the following commit.
2021-12-12 16:20:40 +01:00
Erik Arvstedt
96df81a4d0
run-tests: move flake next to other tests 2021-12-12 16:20:40 +01:00
Jonas Nick
daeea4012e
tests: remove pkgsUnstable
The latest nixpkgs-unstable update would require node-based packages to
distinguish between stable and unstable when building. Instead of dealing with
that complexity, we will only guarantee compatability of our packages with
stable.
2021-11-28 19:16:21 +00:00
Erik Arvstedt
32ce9d0ff4
run-tests: fix 'eval' command for newer versions of nix
There's no common `nix` command argument syntax for eval'ing a nix
expression that supports both older and newer (flake support) versions of nix.
So fall back to nix-instantiate.
2021-09-04 08:17:38 +02:00
Erik Arvstedt
f7c2133250
add flake support
This change is fully backwards compatible.

We continue to use the standard non-flake evaluation mode in our
examples and internal tooling until the flakes design has stabilized.

'clightning-plugins = pkgs.recurseIntoAttrs' in pkgs/default.nix is
needed by flake-utils.lib.flattenTree in flake.nix.
It transforms the packages in `clightning-plugins` to top-level packages
named like `clightning-plugins/summary`. (The flake attr `packages`
must be a non-nested attrset of derivations.)
2021-08-26 12:45:10 +02:00
Erik Arvstedt
b0c66c41e1
tests: add container-minimal example 2021-08-15 22:42:03 +02:00
Erik Arvstedt
6de9aba854
run-tests: quote scriptDir
scriptDir may contain spaces.
2021-08-15 11:29:35 +02:00
Erik Arvstedt
3cce822c64
run-tests: add 'vm' command 2021-03-26 23:23:22 +01:00
Erik Arvstedt
44439e2a81
tests: optimize building multiple tests at once
The result of `import tests.nix {}` is now an attrset of tests.
This makes it easier and more efficient to evaluate or build multiple
tests in one call to `nix build`.

Simplify tests.nix by removing the large module args scope in favor of
self-contained scenario module definitions.

Add CPU core and memory size defaults to the test configuration to
enable building tests without `run-tests.sh`.

Add the following top-level args to tests.nix:
- `extraScenarios` to provide a nix-level way to define extra scenarios.
- `pkgs` to allow building tests with custom pkgs or systems.
2021-03-22 14:35:29 +01:00
Erik Arvstedt
9ca52af523
tests: improve make-test-vm.nix
1. fixedTest: Remove some unneeded layers of function calls.

2. Don't add a modified version of `black` to the global pkgs set.
   Tests should not affect the pkgs of the tested system modules.
   Fix the driver build script instead by adding an extra arg to the
   call to `black`.
2021-03-22 14:35:29 +01:00
Erik Arvstedt
e8b47f099c
examples: add deploy-krops.sh 2021-03-15 19:02:58 +01:00
Erik Arvstedt
0a2c8e4864
run-tests: add option --copy-src 2021-02-08 12:20:20 +01:00
Erik Arvstedt
7533f12ef1
bitcoind, clightning, run-tests: minor refactoring
bitcoind: use builtins.toFile
clightning: use boolToString
run-tests: remove leftover var
2021-02-07 22:39:05 +01:00
Erik Arvstedt
44546561fc
run-tests: allow defining scenarios via cmdline args
This simplifies running self-contained scenarios for testing and debugging.
2021-01-30 11:38:47 +01:00
nixbitcoin
bcedf69549
readme: update and split into various parts 2020-12-30 15:59:22 +00:00
Erik Arvstedt
3403795c86
tests: add example scripts 2020-12-18 19:56:56 +01:00
Erik Arvstedt
ff94985b8b
tests: add test 'hardened' 2020-12-18 19:56:56 +01:00
Erik Arvstedt
9977fa69af
ci: use run-tests.sh 2020-12-11 13:27:06 +01:00
Erik Arvstedt
a82f0f5f48
add test 'pkgsUnstable'
Included in 'basic' tests.
Function 'doBuild' is needed by the following commit.
2020-12-11 13:27:05 +01:00
Erik Arvstedt
95bc1237e2
run-tests: rename testDir -> scriptDir 2020-12-11 13:27:05 +01:00
Erik Arvstedt
7265742655
run-tests: add 'instantiate' command
Useful for diffing test derivations.
2020-12-11 13:27:04 +01:00
Erik Arvstedt
8cbdef8bf6
run-tests: fix CLI
Restore the original behavior that was accidentally changed:
When no args are given, run the basic test suite.
Otherwise, run the given command with default scenario 'default'.
Previously, `run-tests.sh build` ran the basic test suite instead of
building the default scenario.
2020-12-11 13:27:04 +01:00
Erik Arvstedt
1c0233c0a8
use Cirrus CI
- Make more economic use of the free CI resources by removing redundant build tasks:
  - Build unstable pkgs in a single separate task ("pkgs_unstable").
  - All stable pkgs are implicitly built by the modules tests.
- The build script (ci/build.sh) can now be executed locally for easier
  debugging.
- Use an explicit 'cachix push' command instead of helper/wait-for-network-idle.rb.
  This is simpler and more reliable.
2020-12-06 19:07:54 +01:00
Erik Arvstedt
0e2ff948d3
test: add scenario 'netnsRegtest'
The 'basic' test command now cover regtest mode and using nix-bitcoin without
the secure-node preset.
2020-10-29 21:21:27 +01:00
Erik Arvstedt
9951f10e74
test: add scenario 'regtest' 2020-10-16 23:55:13 +02:00
Erik Arvstedt
bfed10b2fa
run-tests: add command 'all'
'all' includes test 'full', which now succeeds.
2020-10-16 16:46:56 +02:00
Erik Arvstedt
0a6b9beda5
run-tests: simplify setting default scenario 2020-10-16 16:46:56 +02:00
Erik Arvstedt
04075b108c
test: use QEMU from stable nixpkgs 2020-10-16 15:53:34 +02:00
Erik Arvstedt
03f8dbba47
test: add non-secure-node eval test
This tests that the modules work without the secure-node template.

The test currently fails at runtime, but evaluating already helps
catching module-related errors.
2020-10-16 15:53:34 +02:00
Erik Arvstedt
bb763d6a26
run-tests: add 'eval' command 2020-10-16 15:53:34 +02:00
Erik Arvstedt
c9251e72a1
README: add run-tests.sh to examples 2020-10-16 15:53:33 +02:00