31ad56950f CI build: print number of VM CPUs (Erik Arvstedt)
Pull request description:
ACKs for top commit:
jonasnick:
ACK 31ad56950f
Tree-SHA512: ff6c43d24f4121d0f1a89e82bb58e182ba044662cb00e7fc879168e7f697d171bc6fbed5d22d983fc6ded11fadbbb4e433133709bf4512882412ac8908e15156
187ff884db add modules test (Erik Arvstedt)
826245484e make secrets dir location configurable (Erik Arvstedt)
b1e13e9415 simplify secrets file format (Erik Arvstedt)
314272a228 lnd, nanopos: move user and group definitions to the bottom (Erik Arvstedt)
766fa4f300 travis: cache all build outputs with cachix (Erik Arvstedt)
b0e759160d travis: set NIX_PATH as early as possible (Erik Arvstedt)
c51bbcf104 travis: move comment (Erik Arvstedt)
7092dce0c7 travis: remove use of deprecated statements (Erik Arvstedt)
190a92507c travis: split up scripts into statements (Erik Arvstedt)
10d6b04ac8 support enabling clightning and lnd simultaneously (Erik Arvstedt)
ad7a519284 bitcoind: wait until RPC port is open (Erik Arvstedt)
5536b64fb3 lnd: wait until wallet is created (Erik Arvstedt)
6f2a55d63c lnd: wait until RPC port is open (Erik Arvstedt)
1868bef462 lnd: add option 'rpcPort' (Erik Arvstedt)
120e3e8cfe lnd postStart: suppress curl response output (Erik Arvstedt)
3e86637327 lnd postStart: poll for REST service availability (Erik Arvstedt)
795c51dc01 lnd postStart: make more idiomatic (Erik Arvstedt)
6e58beae8a lnd: use postStart option for script (Erik Arvstedt)
86167c6e6d clightning: wait until the RPC socket appears (Erik Arvstedt)
60c732a6a1 onion-chef: set RemainAfterExit, fix tor dependency (Erik Arvstedt)
2b9b3ba1c5 systemPackages: improve readability with shorter service references (Erik Arvstedt)
14ecb5511a liquid: add cli option (Erik Arvstedt)
cd5ed39b9c lnd: add cli option (Erik Arvstedt)
1833b15888 clightning: add cli option (Erik Arvstedt)
b90bf6691b add generate-secrets.service (Erik Arvstedt)
6447694214 add generate-secrets pkg (Erik Arvstedt)
e34093a8ac generate_secrets.sh: add opensslConf option (Erik Arvstedt)
9d14d5ba64 generate_secrets.sh: write secrets to working directory (Erik Arvstedt)
51fb054001 generate_secrets.sh: extract makepw command (Erik Arvstedt)
e3b47ce18a add setup-secrets.service (Erik Arvstedt)
437b268433 extract make-secrets.nix (Erik Arvstedt)
f9c29b9318 simplify secret definitions (Erik Arvstedt)
cd0fd6926b don't copy secret files to store during nixops deployment (Erik Arvstedt)
f0a36fe0c7 add 'nix-bitcoin-services' option (Erik Arvstedt)
7aaf30501c nix-bitcoin-services: simplify formatting (Erik Arvstedt)
760da232e0 add nix-bitcoin pkgs namespace (Erik Arvstedt)
6def181dbc add modules.nix (Erik Arvstedt)
3b842e5fe7 add nix-bitcoin-secrets.target (Erik Arvstedt)
bbf2bbc04a network.nix: simplify import of main config (Erik Arvstedt)
7e021a2629 simplify overlay.nix (Erik Arvstedt)
07dc3e04ac move bitcoinrpc group definition to bitcoind (Erik Arvstedt)
d61b185c3a simplify user and group definitions (Erik Arvstedt)
Pull request description:
The nix-bitcoin modules consist of three fundamental components:
1. a set of bitcoin-related modules for general use.
2. an opinionated configuration of these modules (`nix-bitcoin.nix`), to be deployed on a
dedicated machine.
3. machinery for nixops deployment.
This PR removes dependencies that reach from top to bottom in the list.
This means that 1. is now usable on its own and that 2. can be used without 3.
Besides improving nix-bitcoin's general usefulness, this
- simplifies testing. This PR includes a Travis-enabled modules test using the NixOS testing framework.
- paves the way for krops deployment.
- unlocks direct deployment in NixOS containers which allows for super fast experimentation.
### Details
Here are the unnecessary inter-component dependencies and how they're resolved by the commits. I'm using the numbering from the list above.
- `1. -> 3.` The modules (1.) use the nixops-specific (3.) `keys` group.
Resolved by `add nix-bitcoin-secrets.target`.
- `1. -> 3.` 1. requires nixops-specific key services.
Resolved by `add nix-bitcoin-secrets.target`.
- `1. -> 2.` bitcoind needs the bitcoinrpc group which is defined in `nix-bitcoin.nix` (2.).
Resolved by `move bitcoinrpc group definition to bitcoind`.
Further obstacles for standalone usage of 1.:
- We can't easily import 1. as a standalone module set.
Resolved by `add modules.nix`.
- Users of 1. shouldn't be forced to import nix-bitcoin's packages as top-level items in the pkgs namespace.
Resolved by `add nix-bitcoin pkgs namespace`.
### Non-nixops deployments
Commit `add setup-secrets.service` simplifies non-nixops deployment methods like containers, NixOS VMs or krops.
Secrets can now deployed as follows:
1. create local secrets.
2. transfer secrets to machine.
3. on the machine, `setup-secrets.service` creates extra secrets from `secrets.nix` and sets owner and
permissions for all secrets.
As krops integrates step 2. we now have all ingredients for automatic krops deployment.
The service is complicated by the creation of secrets like `bitcoin-rpcpassword` that are composed of attrs from `secrets.nix` instead of being simply backed by a file like `lnd_key`. We could simplify this by creating all secret files locally.
Running nix-bitcoin in NixOS containers gives you faster rebuild cycles when developing. [Here's](https://gist.github.com/5db4fa7dd3f1137920b58e39647116f6) an example.
### Test
The last commits starting with `clightning: add cli option` are testing-related and mostly fix non-critical bugs that were exposed by the test.
All `STABLE=1` builds from the Travis build matrix are implicit in the modules test.
Should we remove these individual builds?
Regarding commit `travis: cache all build outputs with cachix`:
To replace my cache with a cache that's owned by you (maybe named `nix-bitcoin-ci`), run
```
nix-shell -p travis --run 'travis encrypt CACHIX_SIGNING_KEY=... -r fort-nix/nix-bitcoin'
```
where `...` is the value of `secretKey` in `~/.config/cachix/cachix.dhall`. Let me know the travis secret and I'll fixup the commit.
### Docs
If you like the proposed changes, I'll add another PR with updates to the docs regarding the project layout, non-nixops deployment, and how to use nix-bitcoin within a larger NixOS config.
ACKs for top commit:
jonasnick:
ACK 187ff884db
Tree-SHA512: f4be65215c592a4f41bb7fa991a6d8d7c463cf631b88bf53051ca57ba280e7a60b8b09d0d1521345d5b656f844daa2166fff5d00a3105077c9e263465eacfb0a
Each secret file to be deployed is now backed by one local file.
This simplifies 'setup-secrets' and the secret definitions.
Also, with the old format it was not possible to add new secrets
to secrets.nix in a simple way.
Old secrets are automatically converted to the new format when running
nix-shell.
Using the new option 'nix-bitcoin.secrets', secrets are now directly
defined by the services that use them.
This further speeds up builds, in particular the modules test in the
next commit. By checking if the expected final build output has already
been cached, we can even skip the download of cached builds.
Advantages:
- The build fails if a statement fails.
Previously, errors in all but the last statement were silently
ignored.
- The Travis log gives a fine-grained view of the output of each
statement.
Add if statements because the '[…] &&' prefix results in YAML syntax
errors.
- set -e is implicit
- coreutils are in PATH and don't have to be explicitly referenced (echo is a shell builtin anyways)
- exit 0 is unneeded ('if' statements never fail)
This better fits the semantics of this unit and allows for easier
automated testing whether the service is active.
wantedBy = bindsTo = after = tor.service is the simplest way to ensure
that this unit is always running/restarted in lockstep with tor.
Previously, onion-chef would have stayed inactive in the case
that tor was stopped and then later restarted.
An executable is more robust to use than shell aliases.
This is also a preparation for commit 'add module test' because the
NixOS testing framework makes interactive aliases hard to use: It
unsets 'PS1' which is used by programs/bash/bash.nix to detect
interactive shells.
43507a7ce5 Update assumevalid to block 605181 (Jonas Nick)
Pull request description:
Top commit has no ACKs.
Tree-SHA512: 717b4885aad72b5ac86e1d7c9644732e6559055cef138f1a8adad0414760886ef58593fe24717cd070ce86728df9e738613f26c7f08a7c711088226f04f394f3
Not polluting the main pkgs namespace with internal pkgs makes it
easier to integrate the nix-bitcoin modules into a larger config.
Also, by overriding the nix-bitcoin namespace, users can now easily set the
packages used by services that offer no explicit `package` option, like `clightning`.
Remove use of nixops-specific 'keys' group and key services.
Instead:
- Add nix-bitcoin-secrets.target, which should be required by all
units that depend on secrets. (To keep it simple, it's okay to meet
the secrets dependency indirectly by e.g. depending on bitcoind.)
Various secret deployment methods can use this target by
setting up the secrets before activating the target.
In case of nixops we just specify that nixops' keys.target comes
before nix-bitcoin-secrets.target.
If the target is left undefined in the case of manual secrets
deployment, systemd will simply ignore unit dependencies on
the target.
- Allow all users to access the secrets dir.
The access protection for the individual secret files is unchanged.
This allows us to drop the unit dependency on the nixops 'keys' group.
