Intro

This is a helper library and cli tool to work with https://github.com/flashbots/mempool-dumpster/, provider of historical mempool data for Ethereum.

There are 3 kind of files that are provided for each day:

sourcelog - has timestamp when transactions arrived with the source of transactions
transaction-data - has the earliest timestamp when transaction was seen and transaction data (from, to, nonce, etc)
transactions - has the same data as transaction-data with a full raw transaction

CLI

CLI help with downloading files and converting all of them to the parquet format.

Set directory to store downloaded files with MEMPOOL_DATADIR or cli arg --datadir.

mempool-dumpster list-months # list available months
mempool-dumpster list-days 2023-09 # list available days in a month


mempool-dumpster get 2023-09-01 # download sourcelog and transactions files for a day
mempool-dumpster get 2023-09-01 --sourcelog 
mempool-dumpster get 2023-09-01 --sourcelog --transaction-data 
mempool-dumpster get 2023-09-01 --transactions 

mempool-dumpster --datadir ./data get 2023-09-01 --transactions 

mempool-dumpster get 2023-09 # download sourcelog and transactions files for a month

Install

cargo install mempool-dumpster

# from local repo
cargo install --path .

Data Format

Files will be downloaded from mempool-dumpster website, converted to the compressed parquet files if needed and stored in the data directory. There would be one file per file type per day and each type would be in separate directory.

├── sourcelog
│   └── 2023-08-31_sourcelog.parquet
├── transaction-data
│   └── 2023-08-31_transaction-data.parquet
└── transactions
    └── 2023-09-08.parquet

Sourcelog

Path:

$DATA_DIR/sourcelog/

Columns:

INT64 timestamp (TIMESTAMP(MILLIS,false)) timestamp with millisecond precision
BYTE_ARRAY hash (STRING) 0x-prefixed hex-encoded tx hash
BYTE_ARRAY source (STRING) source of transactions

Example:

┌─────────────────────────┬────────────────────────────────────────────────────────────────────┬─────────┐
│        timestamp        │                                hash                                │ source  │
│        timestamp        │                              varchar                               │ varchar │
├─────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────┤
│ 2023-09-07 00:00:00.11  │ 0x8d7fbf19e135fe78cf27cd44865679eedb1b15b1816d97babfaa6a55a06b9261 │ infura  │
│ 2023-09-07 00:00:00.121 │ 0xc12ac4752baf7cebd9fe81d9c82bc1b2cc263f70d0099e4969ed32b2a7d55557 │ alchemy │
│ 2023-09-07 00:00:00.13  │ 0xce024c2799b5e47011e68b62daa610c54dabdd8327e3bcce97642bf416eb4842 │ blx     │
│ 2023-09-07 00:00:00.159 │ 0xae47341cce1913e794548330759fc8a99e3740112365c4ba49fbe8bcbafcddd6 │ local   │
│ 2023-09-07 00:00:00.16  │ 0xe45826dd9f37236f05a3fb4ff8fd5d04f8460cae1ea6eaccbdbd93337229155f │ local   │
└─────────────────────────┴────────────────────────────────────────────────────────────────────┴─────────┘

Transaction Data

Path:

$DATADIR/transaction-data/

Columns:

INT64 timestamp (TIMESTAMP(MILLIS,false)) earliest timestamp when transaction appeared
BYTE_ARRAY hash (STRING) 0x-prefixed hex-encoded tx hash
BYTE_ARRAY chainId (STRING) string of chain id (base 10)
BYTE_ARRAY from (STRING) 0x-prefixed hex-encoded from address
BYTE_ARRAY to (STRING) 0x-prefixed hex-encoded to address(empty for contract creation)
BYTE_ARRAY value (STRING) string of value in wei (base 10)
BYTE_ARRAY nonce (STRING) string of nonce (base 10)
BYTE_ARRAY gas (STRING) string of gas limit (base 10)
BYTE_ARRAY gasPrice (STRING) string of gas price in wei (base 10)
BYTE_ARRAY gasTipCap (STRING) string of gas tip cap in wei (base 10)
BYTE_ARRAY gasFeeCap (STRING) string of gas fee cap in wei (base 10)
INT64 dataSize size of data field in bytes
BYTE_ARRAY data4Bytes (STRING) first 4 bytes of data field (0x-prefixed hex-encoded)

Example:

┌──────────────────────┬──────────────────────┬─────────┬──────────────────────┬──────────────────────┬───┬─────────────┬───────────┬─────────────┬──────────┬────────────┐
│      timestamp       │         hash         │ chainId │         from         │          to          │ … │  gasPrice   │ gasTipCap │  gasFeeCap  │ dataSize │ data4Bytes │
│      timestamp       │       varchar        │ varchar │       varchar        │       varchar        │   │   varchar   │  varchar  │   varchar   │  int64   │  varchar   │
├──────────────────────┼──────────────────────┼─────────┼──────────────────────┼──────────────────────┼───┼─────────────┼───────────┼─────────────┼──────────┼────────────┤
│ 2023-08-08 00:00:0…  │ 0xfd357043a6dca490…  │ 1       │ 0x9bd59d85cdd41693…  │ 0xb2d513b9a54a9999…  │ … │ 10000000000 │ 500000000 │ 10000000000 │        0 │            │
│ 2023-08-08 00:00:0…  │ 0x8c334ff637d8df9f…  │ 1       │ 0x329b4a3343da7b5b…  │ 0xb2d513b9a54a9999…  │ … │ 10000000000 │ 500000000 │ 10000000000 │        0 │            │
│ 2023-08-08 00:00:0…  │ 0x5cf6f66511451fa3…  │ 1       │ 0xa0915532179371db…  │ 0xb2d513b9a54a9999…  │ … │ 10000000000 │ 500000000 │ 10000000000 │        0 │            │
│ 2023-08-08 00:00:0…  │ 0x91c3e7d726f5834a…  │ 1       │ 0xd8aa8f3be2fb0c79…  │ 0x0ed1bcc400acd345…  │ … │ 23385350014 │ 444240358 │ 23385350014 │        4 │ 0x98e5b12a │
│ 2023-08-08 00:00:0…  │ 0xdbbf976af37eb599…  │ 1       │ 0xd8aa8f3be2fb0c79…  │ 0x2659dbe2d2e6f880…  │ … │ 23385350014 │ 444240358 │ 23385350014 │        4 │ 0x98e5b12a │
├──────────────────────┴──────────────────────┴─────────┴──────────────────────┴──────────────────────┴───┴─────────────┴───────────┴─────────────┴──────────┴────────────┤

Transactions

Path: $DATADIR/transactions/

Columns:

INT64 timestamp (TIMESTAMP(MILLIS,false)) earliest timestamp when transaction appeared
BYTE_ARRAY hash (STRING) 0x-prefixed hex-encoded tx hash
BYTE_ARRAY chainId (STRING) string of chain id (base 10)
BYTE_ARRAY from (STRING) 0x-prefixed hex-encoded from address
BYTE_ARRAY to (STRING) 0x-prefixed hex-encoded to address(empty for contract creation)
BYTE_ARRAY value (STRING) string of value in wei (base 10)
BYTE_ARRAY nonce (STRING) string of nonce (base 10)
BYTE_ARRAY gas (STRING) string of gas limit (base 10)
BYTE_ARRAY gasPrice (STRING) string of gas price in wei (base 10)
BYTE_ARRAY gasTipCap (STRING) string of gas tip cap in wei (base 10)
BYTE_ARRAY gasFeeCap (STRING) string of gas fee cap in wei (base 10)
INT64 dataSize size of data field in bytes
BYTE_ARRAY data4Bytes (STRING) first 4 bytes of data field (0x-prefixed hex-encoded)
BYTE_ARRAY rawTx raw transaction bytes

Example:

┌──────────────────────┬──────────────────────┬─────────┬──────────────────────┬──────────────────────┬───┬─────────────┬──────────┬────────────┬──────────────────────┐
│      timestamp       │         hash         │ chainId │         from         │          to          │ … │  gasFeeCap  │ dataSize │ data4Bytes │        rawTx         │
│      timestamp       │       varchar        │ varchar │       varchar        │       varchar        │   │   varchar   │  int64   │  varchar   │         blob         │
├──────────────────────┼──────────────────────┼─────────┼──────────────────────┼──────────────────────┼───┼─────────────┼──────────┼────────────┼──────────────────────┤
│ 2023-09-08 00:00:0…  │ 0x07017d7bc566bdb9…  │ 1       │ 0x08dc8ffc2db71ea0…  │ 0x6982508145454ce3…  │ … │ 24000000000 │       68 │ 0xa9059cbb │ \x02\xF8\xB2\x01\x…  │
│ 2023-09-08 00:00:0…  │ 0x28ea098d9dfd59e3…  │ 1       │ 0x19777274b0e613bd…  │ 0x28c6c06298d514db…  │ … │ 24000000000 │        0 │            │ \x02\xF8s\x01\x02\…  │
│ 2023-09-08 00:00:0…  │ 0xb03b8cff6dc46893…  │ 1       │ 0xe67fc443fa1d4927…  │ 0x514910771af9ca65…  │ … │ 24000000000 │       68 │ 0xa9059cbb │ \x02\xF8\xB2\x01\x…  │
│ 2023-09-08 00:00:0…  │ 0x272c5580ed4ba3f4…  │ 1       │ 0xf6492394b174f678…  │ 0x95ad61b0a150d792…  │ … │ 24000000000 │       68 │ 0xa9059cbb │ \x02\xF8\xB0\x01\x…  │
│ 2023-09-08 00:00:0…  │ 0x13fc9ac90f6eede4…  │ 1       │ 0x61b7e18ba8ba0413…  │ 0xdac17f958d2ee523…  │ … │ 24000000000 │       68 │ 0xa9059cbb │ \x02\xF8\xB2\x01\x…  │
├──────────────────────┴──────────────────────┴─────────┴──────────────────────┴──────────────────────┴───┴─────────────┴──────────┴────────────┴──────────────────────┤

PDF command-line utils written in Rust

rpdf PDF command-line utils written in Rust. rpdf makes working with PDF annotions super easy! It can merge annotations from multiple files, some show