The result of `import tests.nix {}` is now an attrset of tests.
This makes it easier and more efficient to evaluate or build multiple
tests in one call to `nix build`.
Simplify tests.nix by removing the large module args scope in favor of
self-contained scenario module definitions.
Add CPU core and memory size defaults to the test configuration to
enable building tests without `run-tests.sh`.
Add the following top-level args to tests.nix:
- `extraScenarios` to provide a nix-level way to define extra scenarios.
- `pkgs` to allow building tests with custom pkgs or systems.
1. fixedTest: Remove some unneeded layers of function calls.
2. Don't add a modified version of `black` to the global pkgs set.
Tests should not affect the pkgs of the tested system modules.
Fix the driver build script instead by adding an extra arg to the
call to `black`.
Keeping the secrets dir read-only is more simple and robust.
- lnd seed mnemonic creation and joinmarket wallet creation can be
run as the regular service user instead of root.
- It is easier to switch to a third-party secrets deployment
method in the future.
Don't create a seed mnemonic for lnd when a wallet exists.
This avoids creating unused mnemonics and helps simplifying
the migration command in `versioning.nix`.
Move this feature from a module preset to a regular option, so that it's
easily discoverable and accessible.
Simplify the implementation of `generateSecrets` by adding it to the
existing `setup-secrets` service script.
Also rename option setup-secrets -> setupSecrets.
When running as root, use runuser instead of sudo.
As opposed to sudo or doas, runuser is a standalone
binary that needs no external configuration.
Also, it's a bit faster.
- bitcoind 0.20.1 -> 0.21.0
Manually create a wallet in the backup test because bitcoind
does not create a default wallet anymore
- disable the failing elementsd build on unstable
Only check enabled services. This allows running the backup test with a custom subset of other
tests.
Also, show a meaningful error on test failure.
Previously, just an AssertionError without a message was shown.
Because the test code is evaluated from a string and not a file, there
was also no backtrace to the tests file.
- enable usage outside of secure-node.nix
- use json as the output format
- show ports
- also show local addresses, which is particularly useful when
netns-isolation is enabled
- only show enabled services
Restore the original behavior that was accidentally changed:
When no args are given, run the basic test suite.
Otherwise, run the given command with default scenario 'default'.
Previously, `run-tests.sh build` ran the basic test suite instead of
building the default scenario.
- Make more economic use of the free CI resources by removing redundant build tasks:
- Build unstable pkgs in a single separate task ("pkgs_unstable").
- All stable pkgs are implicitly built by the modules tests.
- The build script (ci/build.sh) can now be executed locally for easier
debugging.
- Use an explicit 'cachix push' command instead of helper/wait-for-network-idle.rb.
This is simpler and more reliable.
Since nixpks 20.09, the test output is just an empty directory.
Restore saving the log output and linking to the driver.
Without linking to the driver, the driver is eligible for
garbage collection after running a test via `run-tests.sh --out-link-prefix`,
which implies lengthy driver rebuilds.
When 'loop getparams' fails, jq gets no stdin and exits with code 0.
Because -o pipefail is not enabled in the testing shell, the whole test
command succeeds, although it should fail.
Just test "loop getparams" instead and ignore its output.
- Use fping for pinging multiple hosts in parallel.
Significantly improves test runtime:
>13 s -> ~200 ms for the negative ping tests.
- Only test network namespaces that are enabled.
This allows running the netns test with a reduced service set for debugging.
- Remove deprecated services, instead add btcpayserver, spark-wallet
This tests that the modules work without the secure-node template.
The test currently fails at runtime, but evaluating already helps
catching module-related errors.
- bitcoind: remove mkForce because otherwise the whole extraConfig is replaced
by the value of mkForce.
- liquidd: don't disable 'listen' because it is entirely benign in offline
mode, we also allow it for bitcoind.
This improves debugging and experimenting by making it easy to compose fine-grained
scenarios that have specific tests and features enabled.
The VM test output now includes the subtest name and duration.
Remove the 'raise Exception()' hack for interactive mode.
Run 'banlist-and-restart' test before 'backups'. This speeds up the test
by avoiding an extra shutdown of all bitcoin-related services.
a89a3e934f test: increase diskSize (nixbitcoin)
24b506ff8a tests: simplify lightning-loop test (nixbitcoin)
e7c5f956ea lightning-loop: update module (nixbitcoin)
4a503f57bd lightning-loop: v0.8.1 -> v0.9.0 (nixbitcoin)
Pull request description:
ACKs for top commit:
jonasnick:
reACK a89a3e934f
erikarvstedt:
I think it's okay if you would just merge 24b506ff8a, which is the direct parent of the ACK'd a89a3e934f, and removing a89a3e934f itself is totally uncontroversial.
Tree-SHA512: cee2a2714c714a22c35cea0fa829b42a371540983609cda6609f4d063d849f2e725643bd77cfe78eb71665725164d63f83b6c2589be9e72ba30aaecd7c8dee6c
Test all scenarios by default when running 'build' (which happens
when the script is called without arguments).
Default to scenario 'default' in other test commands like 'debug'.
