The powerful analysis platform to explore and visualize data from blockchain.

Related tags

Cryptography rust schema bitcoin etl ethereum blockchain web3
Overview

Mars: The powerful analysis platform to explore and visualize data from Web3

Features

  • Blazing Fast Create from scratch with Rust.

  • Pipeline Processor Export Ethereum chain-data to structured-data in hours.

  • Low Cost Store structured-data to AWS S3, Azure Blob.

  • Easy to Use Web3 visualization and analysis at your fingertips.

Tools

  • ethetl Lets you export Ethereum data into CSV/Parquet/JSON file format and databases, blazing fast.

How to Use

$ make build

time ./target/release/ethetl --start-block 50000 --end-block 100005 --provider-uri http://127.0.0.1:8848 --max-worker 16
[2022-07-05T07:14:41Z INFO  mars] Config: Config { provider_uri: "http://127.0.0.1:8848", start_block: 50000, end_block: 100005, batch_size: 1000, max_worker: 16 }
[2022-07-05T07:14:43Z INFO  mars::workers::progress] 11500 blocks processed, 4464 transactions processed, 0 receipts processed. Progress is 22%
[2022-07-05T07:14:45Z INFO  mars::workers::progress] 21000 blocks processed, 7732 transactions processed, 5631 receipts processed. Progress is 41%
[2022-07-05T07:14:47Z INFO  mars::workers::progress] 31000 blocks processed, 15699 transactions processed, 10389 receipts processed. Progress is 61%
[2022-07-05T07:14:49Z INFO  mars::workers::progress] 44000 blocks processed, 22021 transactions processed, 17280 receipts processed. Progress is 87%
[2022-07-05T07:14:51Z INFO  mars::workers::progress] 50006 blocks processed, 25100 transactions processed, 25100 receipts processed. Progress is 100%

real	0m9.605s

License

Mars is licensed under Apache 2.0.

You might also like...
reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no extra setup alongside exposing a API ready to query the data.
reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no extra setup alongside exposing a API ready to query the data.

reth-indexer reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no e

Josh Stevens 306 Jul 12, 2023
Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Ahmad Abdullahi Adamu 7 Jan 9, 2023
Selendra is a multichains interoperable nominated Proof-of-Stake network for developing and running Substrate-based and EVM compatible blockchain applications.

Selendra An interoperable nominated Proof-of-Stake network for developing and running Substrate-based and EVM compatible blockchain applications. Read

Selendra 16 Nov 29, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Dec 30, 2022
The Nervos CKB is a public permissionless blockchain, and the layer 1 of Nervos network.

Nervos CKB - The Common Knowledge Base master develop About CKB CKB is the layer 1 of Nervos Network, a public/permissionless blockchain. CKB uses Pro

Nervos Network 1k Dec 30, 2022
The Phala Network Blockchain, pRuntime and the bridge.
The Phala Network Blockchain, pRuntime and the bridge.

Phala Blockchain Phala Network is a TEE-Blockchain hybrid architecture implementing Confidential Contract. This repo includes: node/: the main blockch

Phala Network 314 Jan 6, 2023
Local blockchain for Free TON DApp development and testing.

TON OS Startup Edition Local blockchain for Free TON DApp development and testing. Have a question? Get quick help in our channel: TON OS Startup Edit

TON Labs 35 Jan 2, 2023
A value transfer bridge between the Monero blockchain and the Secret Network.

Secret-Monero-Bridge A value transfer bridge between the Monero blockchain and the Secret Network. Proof-of-Concept Video Demonstration: https://ipfs.

null 28 Dec 7, 2022
HyperCube is a free and open source blockchain project for everyone to use.

XPZ Public Chain HyperCube is a free and open source blockchain project for everyone to use. 日本語 简体中文 正體中文 HyperCube Wiki Wha is HyperCube HyperCube i

null 949 Dec 31, 2022
Comments
  • feat:Add the contract address of the event source

    feat:Add the contract address of the event source

    Summary

    The contract_address field in the logs table stores the contract address if one is created in that transaction. I needed to get the contract address that was the source of the event for analysis. I'm not sure of the best naming of the fields, but didn't want the change to break anything downstream so haven't renamed contract_address.

    I've also corrected a small typo in a variable name (contracet -> contract).

    This is my first rust contribution, so hopefully this was the right way of doing it. Mars looks extremely useful, thanks.

    pr-feature 
    opened by IanCal 2
  • chore(deps): bump lz4-sys from 1.9.3 to 1.9.4

    chore(deps): bump lz4-sys from 1.9.3 to 1.9.4

    Bumps lz4-sys from 1.9.3 to 1.9.4.

    Changelog

    Sourced from lz4-sys's changelog.

    1.24.0:

    • Update to lz4 1.9.4 (lz4-sys 1.9.4) - this fixes CVE-2021-3520, which was a security vulnerability in the core lz4 library
    • export the include directory of lz4 from build.rs

    1.23.3 (March 5, 2022):

    • Update lz4 to 1.9.3
    • Add [de]compress_to_buffer to block API to allow reusing buffers (#16)
    • Windows static lib support
    • Support favor_dec_speed
    • Misc small fixes

    1.23.2:

    • Update lz4 to 1.9.2
    • Remove dependency on skeptic (replace with build-dependency docmatic for README testing)
    • Move to Rust 2018 edition

    1.23.0:

    • Update lz4 to v1.8.2
    • Add lz4 block mode api

    1.22.0:

    • Update lz4 to v1.8.0
    • Remove lz4 redundant dependency to gcc #22 (thanks to Xidorn Quan)

    1.21.1:

    • Fix always rebuild issue #21

    1.21.0:

    • Fix smallest 11-byte stream decoding (thanks to Niklas Hambüchen)
    • Update lz4 to v1.7.5

    1.20.0:

    • Split out separate sys package #16 (thanks to Thijs Cadier)

    1.19.173:

    • Update lz4 to v1.7.3

    1.19.131:

    • Update dependencies for correct work with change build environmet via rustup override

    1.18.131:

    • Implemented Send for Encoder/Decoder #15 (thanks to Maxime Lenoir)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    pr-chore dependencies 
    opened by dependabot[bot] 0
  • Add ens table

    Add ens table

    Schema:

    tx_hash
label
owner
name
cost
expires

    txn: https://etherscan.io/tx/0xc2ed0f5d895348382000056463b9b819b02b8d39cc784a137406b7311113ca24#eventlog

    sig: NameRegistered (string name, index_topic_1 bytes32 label, index_topic_2 address owner, uint256 cost, uint256 expires)

    C-feature 
    opened by BohuTANG 0
  • Roadmap Q3

    Roadmap Q3

    Main tasks

    | Task | Status | Release Target | Comments | | ----------------------------------------------- | --------- | -------------- | --------------- | | Export chain data to CSV/Paruquet | DONE | | | | Analytic with Databend Cloud | PROGRESS | | |

    opened by BohuTANG 0
Releases(v0.1.147-nightly)
Owner
DeepETH
Focuses on building the world's most powerful Web3.0 data infrastructure for modern developers.
DeepETH
GitHub
A fast, simple and powerful open-source cross platform utility tool for generating strong, unique and random passwords

password-generator-pro A fast, simple and powerful open-source cross platform utility tool for generating strong, unique and random passwords. Feature

Sebastien Rousseau 3 Dec 16, 2022
A guide for Mozilla's developers and data scientists to analyze and interpret the data gathered by our data collection systems.

Mozilla Data Documentation This documentation was written to help Mozillians analyze and interpret data collected by our products, such as Firefox and

Mozilla 75 Dec 1, 2022
Substrate: The platform for blockchain innovators

Substrate · Substrate is a next-generation framework for blockchain innovation ?? . Trying it out Simply go to substrate.dev and follow the installati

Parity Technologies 7.7k Dec 30, 2022
Substrate: The platform for blockchain innovators

Substrate · Substrate is a next-generation framework for blockchain innovation ?? . Trying it out Simply go to docs.substrate.io and follow the instal

Viktor Karazhov 1 Jan 8, 2022
The Data Highway Substrate-based blockchain node.

DataHighway-Parachain, a parachain on the Polkadot network. Planned features include a decentralized LPWAN roaming hub for LoRaWAN IoT devices and network operator roaming agreements, participative mining, an inter-chain data market, and DAO governance.

DataHighway 11 Dec 2, 2022
cryo is the easiest way to extract blockchain data to parquet, csv, or json

❄️ ?? cryo ?? ❄️ cryo is the easiest way to extract blockchain data to parquet, csv, or json cryo is also extremely flexible, with many different opti

Paradigm 287 Jul 12, 2023
Heimdall is an advanced Ethereum smart contract toolkit for forensic and heuristic analysis.

Heimdall is an advanced EVM toolkit which aims to make dealing with smart contracts on EVM based chains easier. Installation & Usage Heimdall's update

Jonathan Becker 489 Jan 2, 2023
`llm-chain` is a powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks

llm-chain ?? llm-chain is a collection of Rust crates designed to help you work with Large Language Models (LLMs) more effectively. Our primary focus

Sobel IO 36 Apr 6, 2023
Torii ⛩️ is a simple, powerful and extensible open-source Internal Developer Portal

Torii ⛩️ Torii is a simple, powerful and extensible open-source Internal Developer Portal where developers can find all the tools and services they ne

Qovery 64 Dec 17, 2023
The Zero Knowledge Whitelist Tool is a powerful utility for managing an address whitelist using Zero-Knowledge (ZK) proofs.

zk_whitelist: A Zero Knowledge Whitelist Tool The Zero Knowledge Whitelist Tool is a powerful utility for managing an address whitelist using Zero-Kno

Nikos Koumbakis 4 Nov 21, 2023