High Performance Blockchain Deserializer

Overview

bitcoin-explorer

rust test publish Crates.io Downloads

bitcoin_explorer is an efficient library for reading bitcoin-core binary blockchain file as a database (utilising multi-threading).

Documentation

Go to Rust Documentation

Compatibility Note

This package deals with the binary file of another software Bitcoin Core. It might not be compatible with older Bitcoin Core versions.

Tested on Bitcoin Core version v0.21.1.0-g194b9b8792d9b0798fdb570b79fa51f1d1f5ebaf Copyright (C) 2009-2020 The Bitcoin Core developers.

Performance

SSD allows faster performance.

Iterating through all 700000 blocks (non-connected, in sequential order) takes about 10 minutes (Windows 10, CPU Core i7-9700, Block chain data on external SSD drive connected through USB 3.1).

Examples

get a block (i.e., see doc for what is full/simple format)

use bitcoin_explorer::{BitcoinDB, FBlock, SBlock, Block};
use std::path::Path;

let path = Path::new("/Users/me/bitcoin");

// launch without reading txindex
let db = BitcoinDB::new(path, false).unwrap();

// get block of height 600000 (in different formats)
let block: Block = db.get_block(600000).unwrap();
let block: FBlock = db.get_block(600000).unwrap();
let block: SBlock = db.get_block(600000).unwrap();

get a particular transaction (in different formats)

use bitcoin_explorer::{BitcoinDB, Transaction, FTransaction, STransaction, Txid, FromHex};
use std::path::Path;

let path = Path::new("/Users/me/bitcoin");

// !!must launch with txindex=true!!
let db = BitcoinDB::new(path, true).unwrap();

// get transaction
// e3bf3d07d4b0375638d5f1db5255fe07ba2c4cb067cd81b84ee974b6585fb468
let txid_str = "e3bf3d07d4b0375638d5f1db5255fe07ba2c4cb067cd81b84ee974b6585fb468";
let txid = Txid::from_hex(txid_str).unwrap();

// get transactions in different formats
let tx: Transaction = db.get_transaction(&txid).unwrap();
let tx: FTransaction = db.get_transaction(&txid).unwrap();
let tx: STransaction = db.get_transaction(&txid).unwrap();

Iterate through blocks (in different formats)

use bitcoin_explorer::{BitcoinDB, Block, SBlock, FBlock};
use std::path::Path;

let path = Path::new("/Users/me/bitcoin");

// launch without reading txindex
let db = BitcoinDB::new(path, false).unwrap();

// iterate over block from 600000 to 700000
for block in db.iter_block::<Block>(600000, 700000) {
    for tx in block.txdata {
        println!("do something for this transaction");
    }
}

// iterate over block from 600000 to 700000
for block in db.iter_block::<FBlock>(600000, 700000) {
    for tx in block.txdata {
        println!("do something for this transaction");
    }
}

// iterate over block from 600000 to 700000
for block in db.iter_block::<SBlock>(600000, 700000) {
    for tx in block.txdata {
        println!("do something for this transaction");
    }
}

Iterate through blocks (in different format) with outpoints connected to outputs

Iterating to 700000 blocks requires a minimal amount of 32GB memory.

use bitcoin_explorer::{BitcoinDB, FConnectedBlock, SConnectedBlock};
use std::path::Path;

let path = Path::new("/Users/me/bitcoin");

// launch without reading txindex
let db = BitcoinDB::new(path, false).unwrap();

// iterate over block from 0 to 700000, (full format)
for block in db.iter_connected_block::<FConnectedBlock>(700000) {
    for tx in block.txdata {
        println!("do something for this transaction");
    }
}

// iterate over block from 0 to 700000, (simple format)
for block in db.iter_connected_block::<SConnectedBlock>(700000) {
    for tx in block.txdata {
        println!("do something for this transaction");
    }
}
You might also like...
The Nervos CKB is a public permissionless blockchain, and the layer 1 of Nervos network.

Nervos CKB - The Common Knowledge Base master develop About CKB CKB is the layer 1 of Nervos Network, a public/permissionless blockchain. CKB uses Pro

The Phala Network Blockchain, pRuntime and the bridge.
The Phala Network Blockchain, pRuntime and the bridge.

Phala Blockchain Phala Network is a TEE-Blockchain hybrid architecture implementing Confidential Contract. This repo includes: node/: the main blockch

Substrate: The platform for blockchain innovators
Substrate: The platform for blockchain innovators

Substrate · Substrate is a next-generation framework for blockchain innovation 🚀 . Trying it out Simply go to substrate.dev and follow the installati

An extensible open-source framework for creating private/permissioned blockchain applications

Exonum Status: Project info: Community: Exonum is an extensible open-source framework for creating blockchain applications. Exonum can be used to crea

Local blockchain for Free TON DApp development and testing.

TON OS Startup Edition Local blockchain for Free TON DApp development and testing. Have a question? Get quick help in our channel: TON OS Startup Edit

A value transfer bridge between the Monero blockchain and the Secret Network.

Secret-Monero-Bridge A value transfer bridge between the Monero blockchain and the Secret Network. Proof-of-Concept Video Demonstration: https://ipfs.

C++ `std::unique_ptr` that represents each object as an NFT on the Ethereum blockchain
C++ `std::unique_ptr` that represents each object as an NFT on the Ethereum blockchain

C++ `std::unique_ptr` that represents each object as an NFT on the Ethereum blockchain

HyperCube is a free and open source blockchain project for everyone to use.

XPZ Public Chain HyperCube is a free and open source blockchain project for everyone to use. 日本語 简体中文 正體中文 HyperCube Wiki Wha is HyperCube HyperCube i

🌍 The Earth Blockchain on Polkadot (archived)

Social Network Blockchain · The Social Network blockchain is a next-generation governance, economic, and social system for humanity built on Polkadot

Comments
  • Failure to run (secp256k1-sys)

    Failure to run (secp256k1-sys)

    When I try to run the code i get following error: error: failed to run custom build command for secp256k1-sys v0.4.1

    Code: use bitcoin_explorer::{BitcoinDB, FConnectedBlock, SConnectedBlock}; use std::path::Path;

    fn main() { let path = Path::new("D:/BTC/Chain"); let db = BitcoinDB::new(path, false).unwrap(); }

    Cargo.toml: [package] name = "BTC_explorer_rust" version = "0.1.0" edition = "2018" [dependencies] bitcoin-explorer = { git = "https://github.com/Congyuwang/Rusty-Bitcoin-Explorer" }

    I have gcc also installed. System Win10 x64.

    opened by TomaszKot2 7
  • Bug: File is too large for PlainTableReader!

    Bug: File is too large for PlainTableReader!

    2021-10-31 06:37:15,869 INFO [analyzer] start clustering address and compute daily balance
    [01:28:03] [============>--------]  392110185/671514283  (74211/s, 1h)
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [189, 217, 242, 73, 158, 167, 48, 9, 147, 207, 176, 165, 15, 209, 230, 117, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to write UTXO to cache, error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [161, 178, 36, 41, 180, 74, 84, 111, 198, 119, 132, 240, 21, 205, 33, 251, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [208, 65, 152, 196, 90, 232, 71, 148, 132, 118, 3, 219, 131, 117, 154, 103, 1, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [244, 101, 198, 100, 219, 40, 48, 32, 206, 32, 118, 27, 130, 241, 134, 131, 9, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [114, 208, 117, 122, 72, 17, 27, 198, 121, 13, 142, 186, 245, 130, 47, 30, 7, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [213, 133, 207, 51, 207, 42, 147, 251, 208, 53, 213, 156, 206, 88, 231, 62, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [45, 178, 98, 57, 152, 174, 36, 32, 238, 163, 50, 37, 112, 136, 19, 56, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [56, 247, 109, 153, 67, 176, 173, 93, 52, 171, 77, 136, 39, 94, 135, 127, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [136, 32, 11, 160, 196, 191, 225, 44, 11, 55, 130, 114, 216, 245, 16, 20, 1, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [52, 144, 192, 150, 186, 236, 123, 136, 135, 231, 216, 187, 192, 143, 91, 6, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [57, 91, 200, 157, 181, 39, 237, 184, 3, 13, 71, 142, 92, 199, 96, 86, 1, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [122, 171, 64, 27, 20, 193, 235, 98, 50, 67, 23, 187, 148, 17, 154, 106, 21, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [139, 109, 99, 231, 180, 219, 68, 93, 20, 234, 170, 149, 109, 72, 13, 206, 12, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [95, 158, 55, 153, 191, 184, 192, 123, 134, 231, 153, 73, 163, 7, 91, 43, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,276 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [139, 118, 110, 114, 84, 4, 142, 17, 7, 205, 142, 248, 46, 96, 247, 151, 0, 0, 0, 0], error: Not implemented: File is too large for PlainTableReader!
    2021-10-31 08:05:20,390 ERROR [bitcoin_explorer::iter::fetch_connected_async] failed to remove key [143, 140, 237, 159, 198, 107, 247, 38, 6, 128, 116, 238, 100, 122, 90, 133, 0, 0, 0, 0], error: Not implemented: File is too large for Plain[01:28:19] [=====================]  671514283/671514283  (76318/s, 0s)
    
    opened by Congyuwang 1
  • Compiling x86_64-unknown-linux-musl failed

    Compiling x86_64-unknown-linux-musl failed

    compile command:

    cross build --target x86_64-unknown-linux-musl
    

    error info:

    error: could not find native static library `stdc++`, perhaps an -L flag is missing?
    
    error: could not compile `leveldb-sys` due to previous error
    warning: build failed, waiting for other jobs to finish...
    error: build failed
    

    or

    compile command:

    cargo build --release --target=x86_64-unknown-linux-musl
    

    error info:

    error: failed to run custom build command for `leveldb-sys v2.0.9`
    Caused by:
      process didn't exit successfully: `/home/o/CLionProjects/block_info/target/release/build/leveldb-sys-e10747659cf234ec/build-script-build` (exit status: 101)
      --- stdout
      [build] Started
      [snappy] Building
      CMAKE_TOOLCHAIN_FILE_x86_64-unknown-linux-musl = None
      CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_musl = None
      TARGET_CMAKE_TOOLCHAIN_FILE = None
      CMAKE_TOOLCHAIN_FILE = None
      CMAKE_GENERATOR_x86_64-unknown-linux-musl = None
      CMAKE_GENERATOR_x86_64_unknown_linux_musl = None
      TARGET_CMAKE_GENERATOR = None
      CMAKE_GENERATOR = None
      CMAKE_PREFIX_PATH_x86_64-unknown-linux-musl = None
      CMAKE_PREFIX_PATH_x86_64_unknown_linux_musl = None
      TARGET_CMAKE_PREFIX_PATH = None
      CMAKE_PREFIX_PATH = None
      CMAKE_x86_64-unknown-linux-musl = None
      CMAKE_x86_64_unknown_linux_musl = None
      TARGET_CMAKE = None
      CMAKE = None
      running: "cmake" "/mnt/work/env/rust/data/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/snappy-1.1.7" "-DBUILD_SHARED_LIBS=OFF" "-DSNAPPY_BUILD_TESTS=OFF" "-DHAVE_LIBZ=OFF" "-DCMAKE_INSTALL_LIBDIR=/home/o/CLionProjects/block_info/target/x86_64-unknown-linux-musl/release/build/leveldb-sys-2837f3d0bd4b5bb0/out/lib" "-DCMAKE_INSTALL_PREFIX=/home/o/CLionProjects/block_info/target/x86_64-unknown-linux-musl/release/build/leveldb-sys-2837f3d0bd4b5bb0/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_COMPILER=musl-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_BUILD_TYPE=Release"
      -- The CXX compiler identification is unknown
      -- Configuring incomplete, errors occurred!
      See also "/home/o/CLionProjects/block_info/target/x86_64-unknown-linux-musl/release/build/leveldb-sys-2837f3d0bd4b5bb0/out/build/CMakeFiles/CMakeOutput.log".
      See also "/home/o/CLionProjects/block_info/target/x86_64-unknown-linux-musl/release/build/leveldb-sys-2837f3d0bd4b5bb0/out/build/CMakeFiles/CMakeError.log".
      --- stderr
      CMake Error at CMakeLists.txt:2 (project):
        The CMAKE_CXX_COMPILER:
          musl-g++
        is not a full path and was not found in the PATH.
        Tell CMake where to find the compiler by setting either the environment
        variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
        to the compiler, or to the compiler name if it is in the PATH.
      thread 'main' panicked at '
      command did not execute successfully, got: exit status: 1
      build script failed, must exit now', /home/o/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.48/src/lib.rs:975:5
      stack backtrace:
         0: rust_begin_unwind
                   at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
         1: core::panicking::panic_fmt
                   at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
         2: cmake::fail
         3: cmake::run
         4: cmake::Config::build
         5: build_script_build::build_snappy
         6: build_script_build::main
         7: core::ops::function::FnOnce::call_once
      note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    warning: build failed, waiting for other jobs to finish...
    error: build failed
    
    opened by litcc 0
  • Release v1.2.0: on-disk-utxo

    Release v1.2.0: on-disk-utxo

    • add on-disk UTXO cache option for db.iter_connected_block()
    • by default use in-memory UTXO cache for best performance, allowing on-disk-utxo feature.
    opened by Congyuwang 0
Releases(v1.2.18)
A high performance blockchain kernel for enterprise users.

English | 简体中文 What is CITA CITA is a fast and scalable blockchain kernel for enterprises. CITA supports both native contract and EVM contract, by whi

CITAHub 1.3k Dec 22, 2022
A high performance blockchain kernel for enterprise users.

English | 简体中文 What is CITA CITA is a fast and scalable blockchain kernel for enterprises. CITA supports both native contract and EVM contract, by whi

CITAHub 1.3k Dec 22, 2022
An extremely high performance matching engine written in Rust.

Galois Introduction Galois is an extremely high performance matching engine written in Rust, typically used for the crypto currency exchange service.

UINB Tech 66 Jan 7, 2023
An easy-to-use, high-performance Interledger implementation written in Rust

Interledger implementation in Rust ?? Requirements All crates require Rust 2018 edition and are tested on the following channels: stable Connecting to

Interledger.rs 184 Dec 13, 2022
An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and language wrappers. written in rust(🦀) with ❤️

Les.rs - Rust Cryptocurrency Exchange Library An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and

Crabby AI 4 Jan 9, 2023
Glommio Messaging Framework (GMF) is a high-performance RPC system designed to work with the Glommio framework.

Glommio Messaging Framework (GMF) The GMF library is a powerful and innovative framework developed for facilitating Remote Procedure Calls (RPCs) in R

Mohsen Zainalpour 29 Jun 13, 2023
An open source, high performance limit order book for the Seaport smart contracts. Implemented in Rust using ethers-rs, this offers a turnkey option for digital asset marketplaces.

Quay Quay is an open source, high performance backend for the Seaport smart contracts. The project is implemented in Rust, using Postgres as a storage

Valorem Labs Inc. 169 Jun 23, 2023
A high performance Remote Procedure Call system.

A high performance Remote Procedure Call (RPC) system. Usage Add this to your Cargo.toml file. [dependencies] frpc = { git = "https://github.com/nurmo

Nur 5 Jul 28, 2023
Y-Octo is a high-performance CRDT implementation compatible with yjs

Y-Octo Y-Octo is a high-performance CRDT implementation compatible with yjs. Introduction Y-Octo is a tiny, ultra-fast CRDT collaboration library buil

null 79 Oct 5, 2023
A high-performance, highly compatible EVM Inscriptions Indexer

Insdexer A high-performance, highly compatible EVM Inscriptions Indexer by Rust. An accessible and complete version of the documentation is available

null 105 Mar 17, 2024