Commit Graph

1475 Commits

Author SHA1 Message Date
Erik Arvstedt
fa3455d01f
electrs: don't leak bitcoinrpc secret through process ARGV
Supply secret via private config file instead.
2020-03-04 18:09:51 +01:00
Erik Arvstedt
f30aadbef2
electrs: enable unstable build, pin pkg to unstable
stable's buildRustPackage is painfully slow when evaluating (adds >1 sec
even on a fast machine).

This is fixed (https://github.com/NixOS/nixpkgs/pull/69274) in unstable.
2020-03-04 18:09:50 +01:00
Erik Arvstedt
5c6571654e
electrs: 0.7.1 -> 0.8.3
Use buildRustPackage instead of buildRustCrate (via crate2nix).

buildRustPackage builds the whole executable and its libraries in a
single `cargo build` process.

With the create2nix approach each library is built in a separate derivation,
directly using rustc instead of the cargo wrapper.

Benefits of buildRustPackage:
- Much simpler to maintain
- Package derivation evaluates much faster

Benefits of crate2nix:
- Build can be distributed over multiple build hosts
- Better sharing of common dependencies between different builds
- More fine-grained rebuilding on build failures

In nixpkgs buildRustPackage is used for almost all Rust pkgs, it's
also a better fit for our use case.
2020-03-04 18:09:50 +01:00
Erik Arvstedt
47481b2642
electrs: quote dataDir in shell cmd 2020-03-04 18:09:50 +01:00
Erik Arvstedt
8fb33d1099
electrs: use bitcoind.dataDir option 2020-03-04 18:09:50 +01:00
Erik Arvstedt
45ba1f1fb3
electrs: don't print timestamps to log
Already provided by journald.
2020-03-04 18:09:49 +01:00
Erik Arvstedt
88080a58bf
electrs: wrap long lines in preStart 2020-03-04 18:09:49 +01:00
Erik Arvstedt
301bb91ae5
simplify setting high-memory options 2020-03-04 18:09:49 +01:00
Erik Arvstedt
93fd2329b8
electrs: make nginx TLS proxy optional
Electrs users shouldn't be forced to run a TLS proxy.
2020-03-04 18:09:48 +01:00
Erik Arvstedt
acde24ce43
electrs: move user/group definitions to bottom
Consistent with other service defs.
2020-03-04 18:09:48 +01:00
Erik Arvstedt
148327326b
electrs: formatting 2020-03-04 18:09:48 +01:00
Erik Arvstedt
cce9932b62
make pinned pkgs accessible through pkgs/default.nix
Useful for developing and for importing pinned pkgs via config.nix.
2020-03-04 18:09:48 +01:00
Jonas Nick
aca23197d3
Merge #142: fix qemu error on travis
4d8590f8ed fix qemu error on travis (Erik Arvstedt)

Pull request description:

ACKs for top commit:
  jonasnick:
    ACK 4d8590f8ed

Tree-SHA512: 0576e4dc0a77261760614d95dbfb4b706543f2a7d56d41f218d7b811944651f3450a7ba57fbbb373d735e48b6ae03b10a3fb30d71d0533bc76361b5ec6a6e081
2020-02-27 10:17:30 +00:00
Erik Arvstedt
4d8590f8ed
fix qemu error on travis 2020-02-27 01:28:20 +01:00
Jonas Nick
ea8d29d96f
Merge #141: Fix secrets setup
ad23b508e3 {generate,setup}-secrets: remove process hardening (Erik Arvstedt)
89f9bedb9d generate-secrets.nix: fix indentation (Erik Arvstedt)

Pull request description:

ACKs for top commit:
  jonasnick:
    ACK ad23b508e3

Tree-SHA512: 1cb031f9dbfd3150316e4d4f365d37cb7f591910412ee3c70e01beda3498dbf514d4b620f257f32f64c6dcc4845659f45f69f5839e0b7401997320140530d2a0
2020-02-26 21:40:14 +00:00
Jonas Nick
a26a82ab90
Merge #138: Update nixpkgs (lnd 0.8.1 -> lnd 0.9.0, clightning 0.8.0 -> 0.8.1)
b6d7e81c31 Update nixpkgs (lnd 0.8.1 -> lnd 0.9.0, clightning 0.8.0 -> 0.8.1) (Jonas Nick)

Pull request description:

ACKs for top commit:
  erikarvstedt:
    ACK b6d7e81c31

Tree-SHA512: 82e672a22543770fb08f9199f7c626451a8f112b6b981db39a21643133c85c46dff8ad23b07166633cd5efbf8f5063fe65e41cd0847f9c0f2dc4ec5a4c083750
2020-02-26 21:36:30 +00:00
Jonas Nick
56c69b5253
Merge #139: Convert nix-bitcoin extraConfig options to regular options
323b2a7f17 Allow adding multiple nodes to bitcoind with the addnodes option and improve bitcoin module option descriptions (Jonas Nick)
ed6511c96e Document how to override attributes in configuration.nix (Jonas Nick)
9d3588e1de Convert nix-bitcoin extraConfig options to regular options (Jonas Nick)

Pull request description:

Top commit has no ACKs.

Tree-SHA512: 02d7a38e41742f76979e2e12ae2195304a11a86c6547f5e1f3ff82b6031ad36b80d006a78cb4ec03fdfc4227ffdd60c5cc15bf898c32a3f213acaf2598da8eaf
2020-02-26 21:35:42 +00:00
Jonas Nick
323b2a7f17
Allow adding multiple nodes to bitcoind with the addnodes option and improve bitcoin module option descriptions 2020-02-26 21:34:18 +00:00
Erik Arvstedt
ad23b508e3
{generate,setup}-secrets: remove process hardening
ProtectSystem=full disables writing to /etc which is the default
secrets location.

Besides that, hardening is pointless for {generate,setup}-secrets which
don't read external input and are fully under our control.
2020-02-26 20:38:46 +01:00
Erik Arvstedt
89f9bedb9d
generate-secrets.nix: fix indentation 2020-02-26 20:38:46 +01:00
Jonas Nick
87129b10c1
Merge #140: CI build: print number of VM CPUs
31ad56950f CI build: print number of VM CPUs (Erik Arvstedt)

Pull request description:

ACKs for top commit:
  jonasnick:
    ACK 31ad56950f

Tree-SHA512: ff6c43d24f4121d0f1a89e82bb58e182ba044662cb00e7fc879168e7f697d171bc6fbed5d22d983fc6ded11fadbbb4e433133709bf4512882412ac8908e15156
2020-02-25 22:53:37 +00:00
Erik Arvstedt
31ad56950f
CI build: print number of VM CPUs 2020-02-25 22:51:57 +01:00
Jonas Nick
ed6511c96e
Document how to override attributes in configuration.nix 2020-02-23 19:30:32 +00:00
Jonas Nick
9d3588e1de
Convert nix-bitcoin extraConfig options to regular options 2020-02-23 19:22:07 +00:00
Jonas Nick
b6d7e81c31
Update nixpkgs (lnd 0.8.1 -> lnd 0.9.0, clightning 0.8.0 -> 0.8.1) 2020-02-23 18:30:39 +00:00
Jonas Nick
f78ccbac34
Merge #134: Update nixpkgs, spark-wallet and elementsd
83863c03e5 use pinned nixpkgs-unstable for the test driver (Erik Arvstedt)
6fe647ecc4 spark-wallet, lightning-charge: specify mainchain network in clightning data dir (Erik Arvstedt)
eaaa6b8701 clightning: update to v0.8.0 (Erik Arvstedt)
7dd5f01527 lightning-charge: 0.4.7 -> 0.4.14 (Erik Arvstedt)
8eb2346358 elementsd 0.18.1.1 -> 0.18.1.3 (Jonas Nick)
d68b77c0ca spark-wallet 0.2.9 -> 0.2.12 (Jonas Nick)
4e7b67d5e7 Update nixpkgs (Jonas Nick)

Pull request description:

  nixpkgs updates include
  ```
  bitcoin: 0.18.1 -> 0.19.0.1
  clightning: 0.7.3 -> 0.8.0
  ```

Top commit has no ACKs.

Tree-SHA512: c5a681bdcd3b1e7180ef21f3e6e5c14b78e5cbbf183b1dd221acdd70a810fb23e07d8dee2a1860ac110a78c4830af718794d34a0600367939cc90cba3c9619ac
2020-01-16 09:06:56 +00:00
Erik Arvstedt
83863c03e5
use pinned nixpkgs-unstable for the test driver 2020-01-15 23:14:04 +00:00
Erik Arvstedt
6fe647ecc4
spark-wallet, lightning-charge: specify mainchain network in clightning data dir
This fixes warnings in each service.
2020-01-15 23:13:48 +00:00
Erik Arvstedt
eaaa6b8701
clightning: update to v0.8.0
To continue using inotify would be too complicated because we would also need to
monitor the creation of the 'bitcoin' subdirectory.
2020-01-15 23:13:19 +00:00
Erik Arvstedt
7dd5f01527
lightning-charge: 0.4.7 -> 0.4.14 2020-01-15 23:09:08 +00:00
Jonas Nick
8eb2346358
elementsd 0.18.1.1 -> 0.18.1.3 2020-01-13 08:23:19 +00:00
Jonas Nick
d68b77c0ca
spark-wallet 0.2.9 -> 0.2.12 2020-01-13 08:23:19 +00:00
Jonas Nick
4e7b67d5e7
Update nixpkgs 2020-01-13 08:23:11 +00:00
Jonas Nick
a985abcd21
Merge #133: Improve modularity, remove dependency on nixops, add modules test
187ff884db add modules test (Erik Arvstedt)
826245484e make secrets dir location configurable (Erik Arvstedt)
b1e13e9415 simplify secrets file format (Erik Arvstedt)
314272a228 lnd, nanopos: move user and group definitions to the bottom (Erik Arvstedt)
766fa4f300 travis: cache all build outputs with cachix (Erik Arvstedt)
b0e759160d travis: set NIX_PATH as early as possible (Erik Arvstedt)
c51bbcf104 travis: move comment (Erik Arvstedt)
7092dce0c7 travis: remove use of deprecated statements (Erik Arvstedt)
190a92507c travis: split up scripts into statements (Erik Arvstedt)
10d6b04ac8 support enabling clightning and lnd simultaneously (Erik Arvstedt)
ad7a519284 bitcoind: wait until RPC port is open (Erik Arvstedt)
5536b64fb3 lnd: wait until wallet is created (Erik Arvstedt)
6f2a55d63c lnd: wait until RPC port is open (Erik Arvstedt)
1868bef462 lnd: add option 'rpcPort' (Erik Arvstedt)
120e3e8cfe lnd postStart: suppress curl response output (Erik Arvstedt)
3e86637327 lnd postStart: poll for REST service availability (Erik Arvstedt)
795c51dc01 lnd postStart: make more idiomatic (Erik Arvstedt)
6e58beae8a lnd: use postStart option for script (Erik Arvstedt)
86167c6e6d clightning: wait until the RPC socket appears (Erik Arvstedt)
60c732a6a1 onion-chef: set RemainAfterExit, fix tor dependency (Erik Arvstedt)
2b9b3ba1c5 systemPackages: improve readability with shorter service references (Erik Arvstedt)
14ecb5511a liquid: add cli option (Erik Arvstedt)
cd5ed39b9c lnd: add cli option (Erik Arvstedt)
1833b15888 clightning: add cli option (Erik Arvstedt)
b90bf6691b add generate-secrets.service (Erik Arvstedt)
6447694214 add generate-secrets pkg (Erik Arvstedt)
e34093a8ac generate_secrets.sh: add opensslConf option (Erik Arvstedt)
9d14d5ba64 generate_secrets.sh: write secrets to working directory (Erik Arvstedt)
51fb054001 generate_secrets.sh: extract makepw command (Erik Arvstedt)
e3b47ce18a add setup-secrets.service (Erik Arvstedt)
437b268433 extract make-secrets.nix (Erik Arvstedt)
f9c29b9318 simplify secret definitions (Erik Arvstedt)
cd0fd6926b don't copy secret files to store during nixops deployment (Erik Arvstedt)
f0a36fe0c7 add 'nix-bitcoin-services' option (Erik Arvstedt)
7aaf30501c nix-bitcoin-services: simplify formatting (Erik Arvstedt)
760da232e0 add nix-bitcoin pkgs namespace (Erik Arvstedt)
6def181dbc add modules.nix (Erik Arvstedt)
3b842e5fe7 add nix-bitcoin-secrets.target (Erik Arvstedt)
bbf2bbc04a network.nix: simplify import of main config (Erik Arvstedt)
7e021a2629 simplify overlay.nix (Erik Arvstedt)
07dc3e04ac move bitcoinrpc group definition to bitcoind (Erik Arvstedt)
d61b185c3a simplify user and group definitions (Erik Arvstedt)

Pull request description:

  The nix-bitcoin modules consist of three fundamental components:
  1. a set of bitcoin-related modules for general use.
  2. an opinionated configuration of these modules (`nix-bitcoin.nix`), to be deployed on a
     dedicated machine.
  3. machinery for nixops deployment.

  This PR removes dependencies that reach from top to bottom in the list.
  This means that 1. is now usable on its own and that 2. can be used without 3.

  Besides improving nix-bitcoin's general usefulness, this
  - simplifies testing. This PR includes a Travis-enabled modules test using the NixOS testing framework.
  - paves the way for krops deployment.
  - unlocks direct deployment in NixOS containers which allows for super fast experimentation.

  ### Details
  Here are the unnecessary inter-component dependencies and how they're resolved by the commits. I'm using the numbering from the list above.

  - `1. -> 3.` The modules (1.) use the nixops-specific (3.) `keys` group.
    Resolved by `add nix-bitcoin-secrets.target`.

  - `1. -> 3.` 1. requires nixops-specific key services.
    Resolved by `add nix-bitcoin-secrets.target`.

  - `1. -> 2.` bitcoind needs the bitcoinrpc group which is defined in `nix-bitcoin.nix` (2.).
    Resolved by `move bitcoinrpc group definition to bitcoind`.

  Further obstacles for standalone usage of 1.:

  - We can't easily import 1. as a standalone module set.
    Resolved by `add modules.nix`.

  - Users of 1. shouldn't be forced to import nix-bitcoin's packages as top-level items in the pkgs namespace.
    Resolved by `add nix-bitcoin pkgs namespace`.

  ### Non-nixops deployments
  Commit `add setup-secrets.service` simplifies non-nixops deployment methods like containers, NixOS VMs or krops.

  Secrets can now deployed as follows:
  1. create local secrets.
  2. transfer secrets to machine.
  3. on the machine, `setup-secrets.service` creates extra secrets from `secrets.nix` and sets owner and
     permissions for all secrets.

  As krops integrates step 2. we now have all ingredients for automatic krops deployment.

  The service is complicated by the creation of secrets like `bitcoin-rpcpassword` that are composed of attrs from `secrets.nix` instead of being simply backed by a file like `lnd_key`. We could simplify this by creating all secret files locally.

  Running nix-bitcoin in NixOS containers gives you faster rebuild cycles when developing. [Here's](https://gist.github.com/5db4fa7dd3f1137920b58e39647116f6) an example.

  ### Test
  The last commits starting with `clightning: add cli option` are testing-related and mostly fix non-critical bugs that were exposed by the test.

  All `STABLE=1` builds from the Travis build matrix are implicit in the modules test.
  Should we remove these individual builds?

  Regarding commit `travis: cache all build outputs with cachix`:
  To replace my cache with a cache that's owned by you (maybe named `nix-bitcoin-ci`), run
  ```
  nix-shell -p travis --run 'travis encrypt CACHIX_SIGNING_KEY=... -r fort-nix/nix-bitcoin'
  ```
  where `...` is the value of `secretKey` in `~/.config/cachix/cachix.dhall`. Let me know the travis secret and I'll fixup the commit.

  ### Docs
  If you like the proposed changes, I'll add another PR with updates to the docs regarding the project layout, non-nixops deployment, and how to use nix-bitcoin within a larger NixOS config.

ACKs for top commit:
  jonasnick:
    ACK 187ff884db

Tree-SHA512: f4be65215c592a4f41bb7fa991a6d8d7c463cf631b88bf53051ca57ba280e7a60b8b09d0d1521345d5b656f844daa2166fff5d00a3105077c9e263465eacfb0a
2020-01-13 08:22:17 +00:00
Erik Arvstedt
187ff884db
add modules test
The build of package 'nodeinfo' is implicitly tested by the modules test.
2020-01-13 00:25:12 +01:00
Erik Arvstedt
826245484e
make secrets dir location configurable
Users of the nix-bitcoin modules shouldn't be forced to add an extra
dir under root.
The secrets location is unchanged for the default node config.
2020-01-13 00:25:12 +01:00
Erik Arvstedt
b1e13e9415
simplify secrets file format
Each secret file to be deployed is now backed by one local file.
This simplifies 'setup-secrets' and the secret definitions.
Also, with the old format it was not possible to add new secrets
to secrets.nix in a simple way.

Old secrets are automatically converted to the new format when running
nix-shell.

Using the new option 'nix-bitcoin.secrets', secrets are now directly
defined by the services that use them.
2020-01-13 00:25:11 +01:00
Erik Arvstedt
314272a228
lnd, nanopos: move user and group definitions to the bottom
This is the default service formatting style in nixpkgs.
2020-01-13 00:25:11 +01:00
Erik Arvstedt
766fa4f300
travis: cache all build outputs with cachix
This further speeds up builds, in particular the modules test in the
next commit. By checking if the expected final build output has already
been cached, we can even skip the download of cached builds.
2020-01-13 00:25:11 +01:00
Erik Arvstedt
b0e759160d
travis: set NIX_PATH as early as possible
This avoids some unneded downloads when installing cachix.
2020-01-12 20:53:35 +01:00
Erik Arvstedt
c51bbcf104
travis: move comment 2020-01-12 20:53:35 +01:00
Erik Arvstedt
7092dce0c7
travis: remove use of deprecated statements 2020-01-12 20:53:34 +01:00
Erik Arvstedt
190a92507c
travis: split up scripts into statements
Advantages:
- The build fails if a statement fails.
  Previously, errors in all but the last statement were silently
  ignored.
- The Travis log gives a fine-grained view of the output of each
  statement.

Add if statements because the '[…] &&' prefix results in YAML syntax
errors.
2020-01-12 20:02:04 +01:00
Erik Arvstedt
10d6b04ac8
support enabling clightning and lnd simultaneously
Needed for testing.
2020-01-12 20:02:04 +01:00
Erik Arvstedt
ad7a519284
bitcoind: wait until RPC port is open
This fixes rare failures in clightning which requires an open bitcoind
RPC port
2020-01-12 20:02:04 +01:00
Erik Arvstedt
5536b64fb3
lnd: wait until wallet is created 2020-01-12 20:02:04 +01:00
Erik Arvstedt
6f2a55d63c
lnd: wait until RPC port is open 2020-01-12 20:02:03 +01:00
Erik Arvstedt
1868bef462
lnd: add option 'rpcPort'
10009 is lnd's default port.
Needed for the following commit.
2020-01-12 20:02:03 +01:00
Erik Arvstedt
120e3e8cfe
lnd postStart: suppress curl response output
Errors are still shown
2020-01-12 20:02:03 +01:00
Erik Arvstedt
3e86637327
lnd postStart: poll for REST service availability
Improves service startup time compared to just sleeping
2020-01-12 20:02:03 +01:00