In read_detect it seems like you already have a deserialized block, and then you throw it away and return File and offsets outside to have to read and deserialize it again later. Why not just return that block out?

Use the Iterator trait instead of the model with the channel.

Also extract code from verify example to something like par_iter( Fn(BlockExtra) -> Vec<T>, Fn(T) ) with the second function running in parallel

This is to show the pattern I always use for threads in Rust. Feel free to ignore if you don't like it.

ReadDetect represents a thread/worker. Creating it starts it, eliminating the need for the caller to bother handling anything about it. (There's really no point in creating the ReadDetect without starting it).

The returned value is a real handle. It will join automatically on a drop (methods could be added to change that behavior if desired).

There's a little bit of right drift as the start was inlined, but it could be refactored out into a static function if desired.

blocks_iterator uses a truncated representation of the utxo set, saving a lot of memory, but if we want to serialize the UTXO set with this structure is impossible.

The best approach is probably to have a UTXO Map in the end, at the cost of duplicating information and using more RAM it doesn't change too much the library. The alternative is to optionally use a non truncated representation of the utxo map, with callbacks to operate on it

As shown in perf report here https://github.com/rust-bitcoin/bitcoin_hashes/issues/160

in the memory case a lot of time is spent in hashing bitcoin tx, consider having a pipeline stage creating the hash of tx and storing it in the BlockExtra so that following stages could simply use the cached value

The first read of the blocks*dat detects the position of the block inside the file, this shouldn't change across runs apart from the last dat. Cache the position of the blocks in a binary file so that reading for detecting is necessary only for the latest blocks. (use file length in the key so that dat files half full are read again)

https://github.com/RCasatta/blocks_iterator/blob/0785ebfc652a91e026ccf6d211f1d75c33b49aaa/src/truncmap.rs#L12

Without checking I guess very few outputs use more than 4 bytes, so an idea could be to use the full map for values greater than u32::MAX so that every value in the truncmap saves 4 bytes

PipeIterator should have the option to not write to stdout, so that re-serialization is skipped. (like for example changing Stdout to Option<Stdout>) This is not only useful for last job in a pipechain but you can avoid re-serialization altogether using named pipes, for example running two blocks consumer:

mkfifo second.fifo
$ blocks_iterator .... | tee second.fifo | first_job &
cat second.fifo | second_job

the pipe option will disable logging on standard out (move to stderr?) and prints BlockExtra serialized blocks into stdout, allowing Unix style composition

Since many TXO are spent in the same block they are created, a possible optimization may be to calculate the UTXO per block before inserting in the global UTXO map (which is bigger and due to the space optimization is doing 2 lookup for every get)

~~In draft because it requires updates to rust_bitcoin to handle verify flags~~ released

~~also, I am using ENABLE_ENDOMORPHISM in rust_bitcoinconsensus branch~~ not anymore to use released version

attempt to fix #58

:~/git/blocks_iterator(cfg_bench)$ RUSTFLAGS='--cfg=bench' cargo +nightly bench
   Compiling blocks_iterator v1.0.0 (/home/casatta/git/blocks_iterator)
error[E0432]: unresolved import `sha2`
   --> src/lib.rs:287:9
    |
287 |     use sha2::{Digest, Sha256};
    |         ^^^^ use of undeclared crate or module `sha2`

error: unused imports: `DB`, `Options`, `WriteBatch`
   --> src/utxo/db.rs:184:19
    |
184 |     use rocksdb::{Options, WriteBatch, DB};
    |                   ^^^^^^^  ^^^^^^^^^^  ^^
    |
note: the lint level is defined here
   --> src/lib.rs:18:9
    |
18  | #![deny(unused_imports)]
    |         ^^^^^^^^^^^^^^

error: unused import: `test::Bencher`
   --> src/utxo/db.rs:185:9
    |
185 |     use test::Bencher;
    |         ^^^^^^^^^^^^^

error: unused imports: `HashEngine`, `Hash`, `sha256`
   --> src/lib.rs:284:34
    |
284 |     use crate::bitcoin::hashes::{sha256, Hash, HashEngine};
    |                                  ^^^^^^  ^^^^  ^^^^^^^^^^

error: unused import: `crate::bitcoin::OutPoint`
   --> src/lib.rs:285:9
    |
285 |     use crate::bitcoin::OutPoint;
    |         ^^^^^^^^^^^^^^^^^^^^^^^^

error: unused import: `bitcoin::hashes::sha256::Midstate`
   --> src/lib.rs:286:9
    |
286 |     use bitcoin::hashes::sha256::Midstate;
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: unused imports: `Bencher`, `black_box`
   --> src/lib.rs:288:16
    |
288 |     use test::{black_box, Bencher};
    |                ^^^^^^^^^  ^^^^^^^

For more information about this error, try `rustc --explain E0432`.

After https://github.com/RCasatta/blocks_iterator/commit/86da242ba8ac34c079e6002ae6ad6f59b70b9012 the database contains data to compute the UTXO set at the height of the last run.

Create a binary bin/utxo_set.rs dumping the utxo set in a format like

Vec<OutPoint, TxOut>

plus height and a hash matching the one returned from bitcoin-cli gettxoutsetinfo

That's why I had benching as a standalone iterator step.

https://github.com/dpc/block-iter/blob/40578bb5eca918e7570b8ab82370ee88cebb7a9f/libs/main/src/bench.rs#L6

so one can insert .bench_x() at the end of a middle of an iterator chain and get stats. Though I now see that I forgot to make the bench_x be an iterator that returns the items it's benching, so right now it only works at the end, as a sort of of a .for_each() or .count().

Originally posted by @dpc in https://github.com/RCasatta/blocks_iterator/issues/42#issuecomment-1107527823

I'm moving discussion from https://github.com/rust-bitcoin/rust-bitcoin/issues/595 here since I think this might be an ongoing discussion, and people might find it useful, and feels like we're off topic for a long time now.