A tool for bulk downloading and exporting the account data inside Solana snapshots.

Related tags

Cryptography memento
Overview

Memento

Memento is a tool used to load and save accounts from old Solana snapshots in Google Cloud Storage.

Why should I use Memento?

  • No fighting solana versioning between your geyser plugin and the validator.
  • No fighting the rust compiler between your geyser plugin and the validator.
  • Easy programmatic access to bulk snapshot ingestion.

Developing

Build

$ cargo b --release

Run

Worker

$ ./target/release/jito-workers --help

Backfill

The following is a backfill client that backfills to clickhouse.

$ ./target/release/jito-replayer-backfill --help

How does it work

  • There's a server and client.
  • When starting up the server, it will load in all snapshots available across all GCP buckets you provide on the CLI.
  • The clients can connect to the server over GRPC to start backfills and request information on the snapshots available.

System Setup

Note: During the development a machine with the following specs was used:

  • CPU: AMD Epyc 7313P
  • Cores/Threads: 16c / 32t
  • RAM: 128 GB
  • Drive 0: NVME0N1: 15.36 TB
  • Drive 1: NVME1N1: 15.36 TB
  • Drive 2: NVME2N1: 15.36 TB

Drive 1 and 2 were mounted as an NVME in RAID0 (directions below).

You can get away with NVMEs if you're dumping to an external database. Also, you should tune the number of workers based on your system memory.

Steps

Kernel modifications for validator. (Some might be unnecessary.)

$ sudo bash -c "cat >/etc/sysctl.d/21-solana-validator.conf <<EOF
# Increase UDP buffer sizes
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728

# Increase memory mapped files limit
vm.max_map_count = 10000000
EOF"

$ sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
# Increase process file descriptor count limit
* - nofile 1000000
EOF"

$ sudo sysctl -p /etc/sysctl.d/21-solana-validator.conf

Install packages

$ sudo apt update
$ sudo apt install -y \
      unzip \
      mdadm \
      libssl-dev \
      libudev-dev \
      pkg-config \
      zlib1g-dev \
      llvm \
      clang \
      cmake \
      make \
      libprotobuf-dev \
      protobuf-compiler \
      libpq-dev \
      nload \
      iotop \
      apt-transport-https \
      ca-certificates \
      dirmngr \
      --no-install-recommends

Install protobufs

export PROTOC_VERSION=21.12 && \
        export PROTOC_ZIP=protoc-$PROTOC_VERSION-linux-x86_64.zip && \
        curl -Ss -OL https://github.com/google/protobuf/releases/download/v$PROTOC_VERSION/$PROTOC_ZIP \
        && sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
        && sudo unzip -o $PROTOC_ZIP -d /usr/local include/* \
        && rm -f $PROTOC_ZIP

Install rust:

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ source "$HOME/.cargo/env"

Mount NVMEs and setup swap (your system may differ)

$ sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 \
 /dev/nvme1n1 \
 /dev/nvme2n1
$ sudo mkfs.ext4 -F /dev/md0
$ sudo mkdir -p /mnt/disks/ledger
$ sudo mount /dev/md0 /mnt/disks/ledger
$ sudo chmod a+w /mnt/disks/ledger

$ sudo fallocate -l 512G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ sudo swapon --show

Setup and install clickhouse.

$ GNUPGHOME=$(mktemp -d)
$ sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754
$ sudo rm -r "$GNUPGHOME"
$ sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
$ sudo apt-get update
$ sudo apt-get install -y clickhouse-server clickhouse-client
$ sudo systemctl start clickhouse-server
You might also like...
reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no extra setup alongside exposing a API ready to query the data.
reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no extra setup alongside exposing a API ready to query the data.

reth-indexer reth-indexer reads directly from the reth db and indexes the data into a postgres database all decoded with a simple config file and no e

Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

A framework for creating PoC's for Solana Smart Contracts in a painless and intuitive way

Solana PoC Framework DISCLAIMER: any illegal usage of this framework is heavily discouraged. Most projects on Solana offer a more than generous bug bo

The Voting example based on MoonZoon and Solana + Anchor framework.
The Voting example based on MoonZoon and Solana + Anchor framework.

The Voting example based on MoonZoon and Solana + Anchor framework.

A suite of programs for Solana key management and security.
A suite of programs for Solana key management and security.

๐Ÿ”‘ goki Goki is a suite of programs for Solana key management and security. It currently features: Goki Smart Wallet: A wallet loosely based on the Se

Simple template for building smart contract(Rust) and RPC Client(web3.js) on Solana (WIP) โ›๐Ÿ‘ท๐Ÿšงโš ๏ธ
Simple template for building smart contract(Rust) and RPC Client(web3.js) on Solana (WIP) โ›๐Ÿ‘ท๐Ÿšงโš ๏ธ

Solana BPF Boilerplate Simple template for building smart contract(Rust) and RPC Client(web3.js) on Solana This boilerplate provides the following. Si

๐Ÿง‘โ€โœˆ Version control and key management for Solana programs.

captain ๐Ÿง‘โ€โœˆ๏ธ Version control and key management for Solana programs. Automatic versioning of program binaries based on Cargo Separation of deployer a

Making Token Exchange program with Solana(Rust), Web3, and Vue
Making Token Exchange program with Solana(Rust), Web3, and Vue

Escrow program for Solana blockchain Tech stack Program (Smart Contract) Rust crates: solana-program, spl-token Solana CLI for test validator UI Types

Galleries of NFTs using Solana and Rust for contracts

About this Package created to simplify the process of parsing NFTs on Solana. It consists of: Package basic things like fetch all NFTs for specific wa

Owner
null
a Solana program for granting friends permissions on your account without revealing your private key.

Delegatooooor Granting Permission: An account holder (the delegator) decides to grant permission to a delegate. They create and sign a transaction tha

0xGhostMac 3 Apr 3, 2024
An easy-to-use CLI tool to recover files from zfs snapshots

zfs-undelete an easy-to-use cli tool to recover files from zfs snapshots Usage Use zfs-undelete <file-to-restore>. Works for file and folders. By defa

null 9 Dec 15, 2022
My attempt at learning Solana program (smart contract) development through RareSkill's Solana course.

60-days-of-solana My attempt at learning Solana program (smart contract) development through RareSkill's Solana course. Originally, I was trying to cr

Jasper 3 Feb 25, 2024
An all-in-one IBC protocol providing fungible token transfer, interchain account, and async query functionalities

ICS-999 An all-in-one IBC protocol providing fungible token transfer, interchain account (ICA), and query (ICQ) functionalities, implemented in CosmWa

larry 9 Apr 1, 2023
Automated Solana tool for quick arbitrage, customizable, with real-time data and wallet integration. Trade responsibly.

Solana Arbitrage Trading Tool The Solana Arbitrage Trading Tool is an automated solution crafted to spot and capitalize on arbitrage opportunities wit

null 43 Mar 12, 2024
A template to build smart contracts in Rust to run inside a Cosmos SDK module on all chains that enable it.

CosmWasm Starter Pack This is a template to build smart contracts in Rust to run inside a Cosmos SDK module on all chains that enable it. To understan

null 1 Mar 7, 2022
Smart Contract built in Rust to run inside Cosmos SDK module on all chains that enable it

CoinSwap is a Smart Contract that is built on the terra blockchain and can be used to swap cryptocurrencies such as LUNA, UST, TerraUSD, Anchor, Mirror Protocol, LUNI and other CW20 tokens. The Project also contains a smart contract which works as a analysis tool for the gas fees on the Terra Blockchain.

Prajjwal Chittori 9 Oct 11, 2022
L2 validity rollup combined with blind signatures over elliptic curves inside zkSNARK, to provide offchain anonymous voting with onchain binding execution on Ethereum

blind-ovote Blind-OVOTE is a L2 voting solution which combines the validity rollup ideas with blind signatures over elliptic curves inside zkSNARK, to

Aragon ZK Research 3 Nov 18, 2022
A guide for Mozilla's developers and data scientists to analyze and interpret the data gathered by our data collection systems.

Mozilla Data Documentation This documentation was written to help Mozillians analyze and interpret data collected by our products, such as Firefox and

Mozilla 75 Dec 1, 2022
Extract data from helium-programs via Solana RPC and serves it via HTTP

hnt-explorer This application extracts data from helium-programs via Solana RPC and serves it via HTTP. There are CLI commands meant to run and test t

Louis Thiery 3 May 4, 2023