Utility to run a regtest bitcoind process, useful in integration testing environment

Overview

Bitcoind

Utility to run a regtest bitcoind process, useful in integration testing environment.

use bitcoincore_rpc::RpcApi;
let bitcoind = bitcoind::BitcoinD::new("/usr/local/bin/bitcoind").unwrap();
assert_eq!(0, bitcoind.client.get_blockchain_info().unwrap().blocks);

Features

  • It waits until bitcoind daemon become ready to accept RPC commands
  • bitcoind use a temporary directory as datadir
  • Free ports are asked to the OS (a low probability race condition is still possible)
  • the process is killed when the struct goes out of scope no matter how the test finishes
  • allows easy spawning of dependent process like https://github.com/RCasatta/electrsd

Cargo features

When a feature like 0_21_1 is selected, the build script will automatically download the bitcoin core version 0.21.1 verifying the hashes and placing it in ${CARGO_HOME}. Use utility function downloaded_exe_path() to have the downloaded executable path.

Example

Cargo.toml
[dev-dependencies]
bitcoind = { version = "0.12.0", features = "0_21_1" }

In your tests

let bitcoind = bitcoind::BitcoinD::new(bitcoind::downloaded_exe_path().unwrap()).unwrap();
Comments
  • Use OUT_DIR when building

    Use OUT_DIR when building

    In build.rs you should only ever write to OUT_DIR. Currently we are writing to CARGO_HOME. This is annoying because using cache in CI seems to break this for me because that part of CARGO_HOME is not saved.

    Probably same treatment needs to be applied to electrum crate too.

    opened by LLFourn 9
  • Update node initiation to handle Persistent DB

    Update node initiation to handle Persistent DB

    In usage of recently added #52 , the expected behaviour for Persistent type datadir is, the new instance of core will inherent all the existing data. Which is working as expected for other data (blocks, chainstate etc), except the wallet directory.

    The current behavior tries to always create a new default wallet at initiation. In case of persistent DB that gives rpc error "wallet file already exists".

    This PR updates the init process to handle persistent type database properly..

    A test to check the behavior is also added.

    opened by rajarshimaitra 7
  • Allow optional persistent datadir

    Allow optional persistent datadir

    The intention here is to open possibility to have persistent data directory option for the node along with tempdir option too..

    There were many ways of doing this.. I choose this one as it seemed the one with least changes..

    Looking for approach acks..

    opened by rajarshimaitra 7
  • Cannot use `-rpcuser` and `-rpcpassword` as arguments, does not return Err and enters infinite loop

    Cannot use `-rpcuser` and `-rpcpassword` as arguments, does not return Err and enters infinite loop

    Describe the bug
    Cannot use -rpcuser=<user> and -rpcpassword=<password> args with bitcoind, although it's indicated to be deprecated when running bitcoind from the binaries I think it's useful and would be great to maintain compatibility.

    The -rpcuser and -rpcpassword does not create a .cookie file, therefore it produces an error when trying to read from the file and get auth parameters.

    It enters in an infinite loop because it cannot create and connect a RPC client in step: let client_result = Client::new(&rpc_url, Auth::CookieFile(cookie_file.clone())); when calling auth.get_user_pass it returns an io error because the file does not exists, but the error is not treated and it skip to another pass in the loop considering it's just because the process is not up yet.

    It would be better to not consider the .cookie file when receiving -rpcuser and -rpcpassword arguments or raise an error recommending to use -rpcauth, which works because it uses user and password and creates a .cookie file, and I'm using as a workaround.

    To Reproduce You can simply try to start a process passing the -rpcauth and -rpcpassword as arguments in conf, it won't event start the process. I've created two tests here: https://github.com/oleonardolima/bitcoind/commit/b522d5ec1c42584b81c3b6595e36b9df0b2517f6

    • one for reproducing the error for -rpcuser and -rpcpassword, recommended to run with debug logs: https://github.com/RCasatta/bitcoind/commit/b522d5ec1c42584b81c3b6595e36b9df0b2517f6#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R621-R643
    • one to show that for -rpcauth works successfully: https://github.com/RCasatta/bitcoind/commit/b522d5ec1c42584b81c3b6595e36b9df0b2517f6#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R645-R668

    Expected behavior
    I expect that it would work successfully or return an error indicating that it's indicated to use -rpcauth instead.

    Build environment

    • bitcoind tag/commit: v0.13.0 | bd952c186ffa30e328a7b12c350569856b43c7c1
    • OS+version: macOS 12.4
    • Rust/Cargo version: rustc 1.60.0 (7737e0b5c 2022-04-04) | cargo 1.60.0 (d1fd9fe2c 2022-03-01)
    • Rust/Cargo target: aarch64-apple-darwin
    opened by oleonardolima 6
  • Upgrade bitcoincore-rpc depency

    Upgrade bitcoincore-rpc depency

    Soon as https://github.com/rust-bitcoin/rust-bitcoincore-rpc/pull/234 merges we can upgrade the bitcoincore-rpc dependency allowing rust-miniscript to upgrade rust-bitcoin.

    opened by tcharding 5
  • feat: Add specific error type for -rpcuser and -rpcpassword usage

    feat: Add specific error type for -rpcuser and -rpcpassword usage

    Description

    As discussed in #68 trying to use the -rpcuser and -rpcpassword leads to an infinite loop, because this authentication method does not work alongside cookie authentication leading to an empty file, and the BitcoinD keeps retrying indefinitely.

    This PR adds:

    • a new Error variant for this RpcUserAndPasswordUse
    • a new function to validate the args specified
    • two new tests, one for -rpcuser and -rpcpassword usage, which asserts a failure, and another one for the usage of -rpcauth which works fine alongside cookie authentication.

    Fixes #68

    opened by oleonardolima 4
  • Use home crate to determine cargo home

    Use home crate to determine cargo home

    According to my understanding of cargo docs $CARGO_HOME should be set up by cargo, but it appears it's not always the case -> #13

    This uses the suggested home crate to determine cargo_home during the build process, since it was used dirs_next to do the same things in the binary we used home also in the binary so that only one dep is used

    opened by RCasatta 4
  • Fails to compile on new macs?

    Fails to compile on new macs?

    A user reported this on one of the new m1 macs:

    error[E0277]: the size for values of type `str` cannot be known at compilation time
    --> /Users/s/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.17.0/build.rs:50:9
    |
    50 | / println!(
    51 | | "filename:{} version:{} hash:{}",
    52 | | download_filename, VERSION, expected_hash
    53 | | );
    | |__________^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `str`
    = note: this error originates in the macro `$crate::format_args_nl` (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0277]: the size for values of type `str` cannot be known at compilation time
    --> /Users/s/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.17.0/build.rs:55:19
    |
    55 | let url = format!(
    | ___________________^
    56 | | "https://bitcoincore.org/bin/bitcoin-core-{}/{}",
    57 | | VERSION, download_filename
    58 | | );
    | |_________^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `str`
    = note: this error originates in the macro `$crate::__export::format_args` (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error: aborting due to 3 previous errors
    
    Some errors have detailed explanations: E0277, E0425.
    For more information about an error, try `rustc --explain E0277`.
    error: could not compile `bitcoind`
    
    To learn more, run the command again with --verbose.
    warning: build failed, waiting for other jobs to finish...
    error: build failed
    

    Can anyone confirm or deny?

    opened by LLFourn 3
  • Doesn't work with Alpine linux

    Doesn't work with Alpine linux

    The binary that this crate downloads from bitcoincore.org doesn't run on an Alpine Docker container with FROM alpine:3.15

    The shared library ld-linux-x86-64.so.2 is missing and no compatible version available from the Alpine repo: `/app # /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind --version sh: /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: not found

    /app # ldd /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind /lib64/ld-linux-x86-64.so.2 (0x7f53f39aa000) libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f53f39aa000) librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f53f39aa000) libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f53f39aa000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f53f2d2e000) libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f53f39aa000) Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind) Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __strdup: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __rawmemchr: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __snprintf_chk: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __fread_chk: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __strftime_l: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __vsnprintf_chk: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: pthread_yield: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: malloc_info: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __fprintf_chk: symbol not found Error relocating /app/target/release/build/bitcoind-0886d6cbe0ab5cd4/out/bitcoin/bitcoin-22.0/bin/bitcoind: __memcpy_chk: symbol not found `

    Also using the packaged version of bitcoind from the alpine repository doesn't work, because it was compiled without wallet functionality. https://git.alpinelinux.org/aports/tree/community/bitcoin/APKBUILD line 51

    I'm not sure what is the best way to make it work with Alpine.

    opened by ulrichard 2
  • Use the `home` crate to determine $CARGO_HOME

    Use the `home` crate to determine $CARGO_HOME

    It looks like the recommended way to get the $CARGO_HOME value is from the home crate:

    https://crates.io/crates/home

    Also @ulrichard ran into an error that could be related (he's on a Debian Bullseye system) and got this error:

    When I execute "cargo test" on the current master, I get a compiler error:
    Compiling bitcoincore-rpc v0.13.0
    error: failed to run custom build command for bitcoind v0.13.0
    
    Caused by:
      process didn't exit successfully: /home/richi/src/github/bdk/target/debug/build/bitcoind-4c60e35c276999bb/build-script-build (exit status: 101)
      --- stderr
      thread 'main' panicked at 'called Result::unwrap() on an Err value: NotPresent', /home/richi/.cargo/registry/src/github.com-1ecc6299db9ec823/bitcoind-0.13.0/build.rs:39:78
      note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
    warning: build failed, waiting for other jobs to finish...
    error: build failed
    

    it was fixed when he manually set his $CARGO_HOME env variable.

    opened by notmandatory 2
  • Can't locate downloaded binary after repository is moved

    Can't locate downloaded binary after repository is moved

    For some reason, after I moved the location of my repository which uses bitcoind in its dev-dependencies, downloaded_exe_path() continues returning the old path. Everything from target/... onwards is correct - I've verified that there is indeed a bitcoind binary in that location - but the path to the repo seems to remain constant even across system restarts.

    How to reproduce

    Platform: macOS Rust version: nightly-2022-06-27-x86_64-apple-darwin bitcoind crate version: 0.26.1

    Have this in Cargo.toml:

    [dev-dependencies]
    bitcoind = { version = "0.26", features = [ "22_0" ] }
    

    Ensure that OUT_DIR is not set in env vars

    echo $OUT_DIR
    

    Have this test:

    #[cfg(test)]
    mod test {
        fn bitcoind_fails() {
            let conf = Conf::default();
            let exe_path = bitcoind::downloaded_exe_path()
                .expect("Didn't specify bitcoind version in feature flags");
    
            // This prints
            // [node/src/command/test/mod.rs:72] &exe_path = "<path_to_OLD_repo_location>/target/debug/build/bitcoind-65c3b20abafd4893/out/bitcoin/bitcoin-22.0/bin/bitcoind"
            // Aside from the path to the repo, everything else is correct
            dbg!(&exe_path);
    
            // This causes the code not to compile
            // let out_dir = std::env!("OUT_DIR");
            // This panics
            // let out_dir = std::env::var_os("OUT_DIR").unwrap();
    
            // This panics when running from new repository location
            let bitcoind = BitcoinD::with_conf(exe_path, &conf)
                .expect("Failed to init bitcoind");
        }
    }
    

    Run the test (should pass)

    Move the repository, e.g. from $HOME/dev/myrepo to $HOME/dev/bitcoin/myrepo

    Rerun the test (should fail)

    What I've tried, but which still produce the same error

    • cargo clean
    • rm -rf <path_to_repo>/target
    • rm -rf ~/.cargo/registry
    • rm -rf ~/.cargo/git
    • cargo cache -e
    • Cloning the repo to a new location on the same computer
    • Even restarting my computer

    Workarounds which prevent the panic (but which don't solve the problem)

    • Downgrading to 0_21_0 in Cargo.toml: bitcoind = { version = "0.26", features = [ "0_21_0" ] }
    • Specifying bitcoind using a relative path bitcoind = { path = "../../../github/bitcoind", version = "0.26", features = [ "22_0" ] }
    • Moving the repo back to its original location

    Any idea what could be going on?

    opened by MaxFangX 1
  • Address the race condition

    Address the race condition

    If my understanding is correct this crate launches a separate instance per test, which is nice but because of the race condition and tests being run in parallel this can cause flaky tests which is quite bad. It'd be great to fix the race completely.

    I took a look at the obvious "what happens if I set -port 0 -rpcport 0" The answer is `bitcoind seems to attempt to bind the default port anyway (WTF?) and doesn't log the actual port so if we do this we have no way of figuring it out.

    I see only two solutions that include existing bitcoind instances:

    1. On platforms that support it we could LD_PRELOAD a library that overrides bind() to pass in port 0 and then reports the mapping over a pipe. We could read that report and learn the actual port number.
    2. Create private network namespace which has a custom IP and is somehow allowed to communicate with the parent. This should be possible on all platforms which support Docker because it presumably does this but it may require root which is not nice.

    Of course the long term solution is to add direct support to bitcoind itself.

    opened by Kixunil 2
  • Remove the `time =

    Remove the `time = "=0.3.10"` hack once MSRV reaches 1.59

    I'd just like to request that the time = "=0.3.10" hack is removed once bitcoind's MSRV reaches 1.59. I have confirmed that the bitcoind build succeeds with time = "0.3" if compiling with version 1.59.

    More information

    The latest 0.28.1 release which introduced the time = "=0.3.10" hack to support MSRV 1.57 broke our build, which contains serde_with.

    Steps to reproduce

    cargo new --lib repro-build
    

    Cargo.toml:

    [package]
    name = "repro-build"
    version = "0.1.0"
    edition = "2021"
    
    [dependencies]
    serde_with = "=2.1.0"
    bitcoind = "=0.28.1"
    
    $ cargo build
    
    error: failed to select a version for `time`.
        ... required by package `serde_with v2.1.0`
        ... which satisfies dependency `serde_with = "=2.1.0"` of package `repro-build v0.1.0 (/Users/fang/temp/repro-build)`
    versions that meet the requirements `~0.3.11` are: 0.3.17, 0.3.16, 0.3.15, 0.3.14, 0.3.13, 0.3.12, 0.3.11
    
    all possible versions conflict with previously selected packages.
    
      previously selected package `time v0.3.10`
        ... which satisfies dependency `time = "=0.3.10"` of package `bitcoind v0.28.1`
        ... which satisfies dependency `bitcoind = "=0.28.1"` of package `repro-build v0.1.0 (/Users/fang/temp/repro-build)`
    
    failed to select a version for `time` which could resolve this conflict
    

    Solution

    Downgrading our serde_with caused us other problems (specifically, serde_with 2.0.0 is broken), so we had to downgrade bitcoind to 0.28.0 to fix our build. We would like to switch back to the latest version of bitcoind at some point, but to do so requires removing the time = "=0.3.10" hack currently in the Cargo.toml.

    Possible workaround to remove the hack while keeping MSRV at 1.57

    I spent some time playing around with bitcoind and rust 1.57 and was able to reproduce the 1.57 build problem. Interestingly, after building the crate once with time = "=0.3.10" and generating the Cargo.lock, cargo build still succeeds if you change the dependency back to time = "0.3". If there is a way to specify that time should be version 0.3.10 in the cargo invocation, bitcoind's CI can be fixed, then the hack can be removed, allowing libraries and binaries downstream of bitcoind to avoid this dependency resolution problem. In other words, the CI script seems a better place to put a hack than the Cargo.toml, which has ripple effects on downstream crates.

    opened by MaxFangX 3
  • v27.1 is NOT 1.41.1 MSRV because of ureq

    v27.1 is NOT 1.41.1 MSRV because of ureq

    ureq = "1.0" allows ureq 1.6 to be installed. 1.6 depends on once_cell v1.16.0 which is not msrv 1.41.1. once_cell 1.9.0 is te latest version of at msrv 1.41.1. once_cell 1.10.0 breaks it.

    Unfortunately, ureq does not adhere to any MSRV, and only specifies once_cell = "1"

    it seems this crate only has 1 ureq call, here https://github.com/RCasatta/bitcoind/blob/2c1758a2d5d638a388906c97bc011802b12e498b/build.rs#L106

    so maybe the whole dependency could be replaced with something that maintains an MSRV

    This is causing problems downstream. https://github.com/Kixunil/payjoin/pull/36

    bug 
    opened by DanGould 2
Owner
Riccardo Casatta
Riccardo Casatta
A CLI tool which can help you automatically kill process of your choice. Useful for freeing up memory and CPU usage!

Quickiller There are always programs such as chrome that keep eating up your resources even when closed! The only way to prevent this is to kill all o

Codingsquirrel 1 Dec 8, 2021
Rust command line utility to quickly display useful secrets in a Kubernetes namespace

kube-secrets This is a command line utility for quickly looking at secrets in a Kubernetes namespace that are typically looked at by humans. It specif

Frank Wiles 8 Feb 10, 2022
Run the right version of python, in the right environment, for your project

rpy Do you deal with lots of virtual python environments? rpy is for you! Before rpy: ~/dev/prj$ env PYTHONPATH=src/py path/to/my/interpreter src/py/m

Aquatic Capital Management 2 Dec 8, 2022
secmem-proc is a crate designed to harden a process against low-privileged attackers running on the same system trying to obtain secret memory contents of the current process.

secmem-proc is a crate designed to harden a process against low-privileged attackers running on the same system trying to obtain secret memory contents of the current process. More specifically, the crate disables core dumps and tries to disable tracing on unix-like OSes.

null 3 Dec 19, 2022
A Rust on-site channel benchmarking helper. Inter-Process (async / busy) & Intra-Process (async single threaded / async multi threaded)

On-Site Rust Channel Benchmarking Helper Deploy on server to determine which public crates are the fastest for communicating in different architecture

null 23 Jul 9, 2024
Lupus is a utility to administer backups with future integration with rsync

Lupus is a utility to administer backups with future integration with rsync. Many other features are either included or planned such as chat bridges using rcon and or parsing the pipe output from programs/games.

null 3 Sep 19, 2022
🔔 CLI utility to send notifications to Slack via integration webhooks

Slack notifier Just a simple CLI tool to send notifications to Slack. Please note that this project is just a playground to start learning Rust, it is

Green.Mod 2 May 21, 2022
Bolt is a desktop application that is designed to make the process of developing and testing APIs easier and more efficient.

Bolt ⚡ Bolt is a desktop application that is designed to make the process of developing and testing APIs easier and more efficient. Quick start ??‍??

0xHiro 6 Mar 26, 2023
A WIP property-based testing library in Rust, built with generalized targeted-property testing in mind.

Crabcheck A WIP property-based testing library in Rust, built with generalized targeted-property testing in mind. What is property-based testing? TODO

Alperen Keleş 9 Mar 27, 2024
Distrod is a meta-distro for WSL 2 which runs Systemd as init process. You can run Ubuntu, Arch, and et.c with systemd in a minute. Distrod also has built-in auto-start feature on Windows startup and port forwarding abiltiy.

Distrod - WSL2 Distros with Systemd! Distrod is a systemd-based meta-distro for WSL2 that allows you to install Ubuntu, Arch Linux, Gentoo and many ot

Takaya Saeki 1.6k Jan 9, 2023
Run a program with a modified process name.

tagrun Run a program with a modified process name. You can add a tag for a process with tagrun, and use the tag to locate the process with ps, pgrep o

Li 2 Dec 21, 2022
This is a tiny (but delightful!) utility library for exhaustive testing.

Exhaustigen This is a tiny (but delightful!) utility library for exhaustive testing. It is based (directly) on the idea and code in the following blog

Graydon Hoare 34 Dec 14, 2022
A series of utility macros for outputting testing results.

test-results A series of utility macros for outputting testing results. Getting Started Simply add the test-results crate to your project's Cargo.toml

EYHN 1 Jan 27, 2022
Unified directories for different use cases of an application, providing standard directories for local development, when run as service or when run by a user.

UniDirs Unified directories for different use cases of an application, providing standard directories for local development, when run as service or wh

Dominik Nakamura 3 Sep 30, 2022
Docker images for compiling static Rust binaries using musl-libc and musl-gcc, with static versions of useful C libraries. Supports openssl and diesel crates.

rust-musl-builder: Docker container for easily building static Rust binaries Source on GitHub Changelog UPDATED: Major updates in this release which m

Eric Kidd 1.3k Jan 1, 2023
A Rust library to extract useful data from HTML documents, suitable for web scraping.

select.rs A library to extract useful data from HTML documents, suitable for web scraping. NOTE: The following example only works in the upcoming rele

Utkarsh Kukreti 829 Dec 28, 2022
`ls` alternative with useful info and a splash of color 🎨

?? Natls ?? Why Natls? Showing file permissions Showing file size Showing the date that the file was modified last Showing the user that the file belo

Will 1.2k Dec 19, 2022
Parsing and inspecting Rust literals (particularly useful for proc macros)

litrs: parsing and inspecting Rust literals litrs offers functionality to parse Rust literals, i.e. tokens in the Rust programming language that repre

Lukas Kalbertodt 31 Dec 26, 2022
Structopt derived ethers-rs types, useful for building Ethereum CLIs

ethers-structopt Provides ethers-compatible Structopt derives, useful for building Ethereum CLIs. Contributing Pull requests are welcome. For major ch

Georgios Konstantopoulos 6 Dec 27, 2022
Automatically updates your Cloudflare DNS records for specific zones. Especially useful if you have dynamic IP address

Cloudflare DNS updater What does it do? Cloudflare DNS updater updates specified dns records for specified zones effortlessly and automatically. It wa

Niko Huuskonen 8 Aug 30, 2022