Synchronized shadow state of Solana programs available for off-chain processing.

Overview

Solana Shadow

The Solana Shadow crate adds shadows to solana on-chain accounts for off-chain processing. This create synchronises all accounts and their data related to a program in real time and allows off-chain bots to act upon changes to those accounts.

Apache 2.0 licensed

Usage

Add this in your Cargo.toml:

[dependencies]
solana-shadow = "*"

Take a look at the examples/ directory for usage examples.

Mirroring a program id and all its owned accounts:

// this is the prog id that owns all pyth oracles on mainnet
let prog = "FsJ3A3u2vn5cTVofAjvy6y5kwABJAqYWpe4975bi2epH".parse()?;
let network = Network::Mainnet;
let local = BlockchainShadow::new_for_program(&prog, network).await?;

loop {
  local.for_each_account(|pubkey, account| {
    println!(" - [{}]: {:?}", pubkey, account);
  });

  sleep(Duration::from_secs(3)).await;
}

local.worker().await?;

Mirroring few random accounts

  // https://pyth.network/developers/accounts/
  let ethusd = "JBu1AL4obBcCMqKBBxhpWCNUt136ijcuMZLFvTP7iWdB".parse()?;
  let btcusd = "GVXRSBjFk6e6J3NbVPXohDJetcTjaeeuykUpbQF8UoMU".parse()?;

  let local = BlockchainShadow::new_for_accounts(&vec![ethusd, btcusd], Network::Mainnet).await?;

  loop {
    let ethacc = shadow.get_account(&ethusd).unwrap();
    let ethprice = cast::<Price>(&ethacc.data).agg.price;

    let btcacc = shadow.get_account(&btcusd).unwrap();
    let btcprice = cast::<Price>(&btcacc.data).agg.price;

    println!("ETH/USD: {}", ethprice);
    println!("BTC/USD: {}", btcprice);

    sleep(Duration::from_secs(3)).await;
  }

  local.worker().await?;

Listening on changes to accounts:

  // https://pyth.network/developers/accounts/
  let ethusd = "JBu1AL4obBcCMqKBBxhpWCNUt136ijcuMZLFvTP7iWdB".parse()?;
  let btcusd = "GVXRSBjFk6e6J3NbVPXohDJetcTjaeeuykUpbQF8UoMU".parse()?;
  let solusd = "H6ARHf6YXhGYeQfUzQNGk6rDNnLBQKrenN712K4AQJEG".parse()?;

  // create an offline shadow of the on-chain data.
  // whenever the data change on-chain those changes
  // will be reflected immediately in this type.
  let shadow = BlockchainShadow::new_for_accounts(
    &vec![ethusd, btcusd, solusd],
    Network::Mainnet,
  )
  .await?;

  tokio::spawn(async move {
    // start printing updates only after 5 seconds
    tokio::time::sleep(Duration::from_secs(5)).await;

    // now everytime an account changes, its pubkey will be
    // broadcasted to all receivers that are waiting on updates.
    while let Ok((pubkey, account)) = updates_channel.recv().await {
      let price = cast::<Price>(&account.data).agg.price;
      println!("account updated: {}: {}", &pubkey, price);
    }
  });

  shadow.worker().await?;
Comments
  • use base64+zstd for account data encoding by default, add ping support and make recv cancellation safe

    use base64+zstd for account data encoding by default, add ping support and make recv cancellation safe

    Sorry I squeezed the fixes for #17 #19 and a base64+zstd fix into the same PR.

    The first commit in the PR fixes the following problem. The jsonParsed encoding won't work if the account data can be parsed by the RPC node. For example, the spl-token account will be actually parsed! The notification from WS becomes

    {
        "jsonrpc": "2.0",
        "method": "accountNotification",
        "params": {
            "result": {
                "context": {
                    "slot": 114915064
                },
                "value": {
                    "lamports": 2039280,
                    "data": {
                        "program": "spl-token",
                        "parsed": {
                            "info": {
                                "isNative": false,
                                "mint": "USDhTjkUXFfigLELiFpbBnpLmEm4aXHvdY2kDSadJDH",
                                "owner": "8dyR4YwFHzh2Pxmgs9TPst5Mu4kkES5tu8w6SuhdEyUG",
                                "state": "initialized",
                                "tokenAmount": {
                                    "amount": "5771086231612606",
                                    "decimals": 9,
                                    "uiAmount": 5771086.231612606,
                                    "uiAmountString": "5771086.231612606"
                                }
                            },
                            "type": "account"
                        },
                        "space": 165
                    },
                    "owner": "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
                    "executable": false,
                    "rentEpoch": 266
                }
            },
            "subscription": 10447
        }
    }
    

    and cannot be deserialized into the AccountRepresentation struct.

    Choosing base64+zstd because it is compressed and should save network bandwidth theoretically (which is the bottleneck).

    The rest two commits fixes #17 and #19.

    opened by dovahcrow 5
  • Feature/allow mutation

    Feature/allow mutation

    Need it for this usecase:

    let mut users = vec![];
    shadow.for_each_account(|_, account| {
        if let Ok(user) = deserialize::<UserMetadata>(account) {
            if user.debt > 0 {
                users.push(user);
            }
        }
    });
    
    opened by y2kappa 1
  • SolanaChangeListener::recv is not cancellation safe!

    SolanaChangeListener::recv is not cancellation safe!

    SolanaChangeListener::recv is used in https://github.com/hubbleprotocol/solana-shadow/blob/master/src/blockchain.rs#L225. However, SolanaChangeListener::recv is not cancellation safe means the internal state can be dropped unnoticed! e.g. When someone calls add_account and then during SolanaChangeListener's handling of the confirmation response and awaited in the middle, select! will cancel the half executed recv causes add_account to return AsyncWrap error.

    opened by dovahcrow 0
  • Consider do json rpc ping instead of reconnect every now and then?

    Consider do json rpc ping instead of reconnect every now and then?

    Reconnecting every now and then might make the listener lose some updates. Maybe we can ping the RPC node every 5 secs to make the ws live?

    One solution maybe use select! in the recv function of SolanaChangeListener and add a timer branch into select!.

    opened by dovahcrow 0
  • stubborn connect_async

    stubborn connect_async

    While testing I noticed a stalling due to the connect_async hanging in the underlying tungstenite. src/stubborn implements a timeout and retry mechanism to ensure active reconnecting.

    opened by boymaas 0
  • Commitment Config Option

    Commitment Config Option

    Quick Fix to test the Commitment level, changed default level from finalised. As such, if any consumers depend on the default to be finalised this will have to change.

    opened by boymaas 0
  • Auto-reconnect on connection drop

    Auto-reconnect on connection drop

    Solana RPC endpoints automatically drop websocket connections when there is no activity for around 30 seconds. This change automatically reconnects to the Solana WS endpoint and recreates all subscriptions when the socket is closed for whatever reason.

    opened by karim-agha 0
Owner
null
Examples of Solana on-chain programs

spl-examples List of Solana on-chain programs which demonstrate different aspects of Solana architecture. 01__state It's a counter program. Each user

Max Block 51 Dec 6, 2022
A collection of Solana-maintained on-chain programs

Solana Program Library The Solana Program Library (SPL) is a collection of on-chain programs targeting the Sealevel parallel runtime. These programs a

Solana Foundation 2.2k Jan 6, 2023
CLI tool for deterministically building and verifying executable against on-chain programs or buffer accounts

Solana Verify CLI A command line tool to build and verify solana programs. Users can ensure that the hash of the on-chain program matches the hash of

Ellipsis Labs 5 Jan 30, 2023
Gum Program Library, a collection of on chain programs maintained by Gum

Gum, at its core, is a decentralized social media protocol on Solana. It unbundles traditional social media into Social Legos similar to how Defi unbu

Gum 4 Feb 22, 2023
⛏ An open protocol for launching liquidity mining programs on Solana.

⛏ Quarry An open protocol for launching liquidity mining programs on Solana. Background Quarry was built with the intention of helping more Solana pro

Quarry Protocol 207 Dec 19, 2022
A suite of programs for Solana key management and security.

?? goki Goki is a suite of programs for Solana key management and security. It currently features: Goki Smart Wallet: A wallet loosely based on the Se

Goki Protocol 157 Dec 8, 2022
🧑‍✈ Version control and key management for Solana programs.

captain ??‍✈️ Version control and key management for Solana programs. Automatic versioning of program binaries based on Cargo Separation of deployer a

Saber 35 Mar 1, 2022
Deploy your Solana programs during high load.

solana-deployer IMPORTANT: There is a known bug with the current program that will be fixed soon. In the meantime you should deploy from Solana Playgr

acheron 34 Nov 7, 2022
Write Anchor-compatible Solana programs in TypeScript

Axolotl Write Achor-compatible Solana programs using TypeScript. Writing Rust is hard, but safe. It's also the go-to language for writing Solana progr

Anthony Morris 17 Nov 27, 2022
Extract data from helium-programs via Solana RPC and serves it via HTTP

hnt-explorer This application extracts data from helium-programs via Solana RPC and serves it via HTTP. There are CLI commands meant to run and test t

Louis Thiery 3 May 4, 2023
NFT mixing program on solana chain

Mixture_machine Core logic of composing NFT Compose of NFT As you invoke compose_nft function with 2 or more child NFTs, child NFTs are locked in to M

Jay Soh 7 Dec 12, 2022
Demonstrates Solana data account versioning used in supporting the Solana Cookbook article: Account Data Versioning

versioning-solana This repo demonstrates ONE rudimentary way to upgrade/migrate account data changes with solana program changes. What is data version

Frank V. Castellucci 6 Sep 30, 2022
My attempt at learning Solana program (smart contract) development through RareSkill's Solana course.

60-days-of-solana My attempt at learning Solana program (smart contract) development through RareSkill's Solana course. Originally, I was trying to cr

Jasper 3 Feb 25, 2024
A safe implementation of the secure remote password authentication and key-exchange protocol (SRP), SRP6a and legacy are as features available.

Secure Remote Password (SRP 6 / 6a) A safe implementation of the secure remote password authentication and key-exchange protocol (SRP version 6a). Ver

Sven Assmann 10 Nov 3, 2022
Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic

Nethoscope Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic. Check video on how it

Vesa Vilhonen 86 Dec 5, 2022
Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Ahmad Abdullahi Adamu 7 Jan 9, 2023
Rust API Client for ImageKit.io a file storage and image processing service

Rust API Client for ImageKit.io a file storage and image processing service Usage You must retrieve your Public and Private Keys from the ImageKit Dev

Esteban Borai 4 Jul 31, 2022
A vertically scalable stream processing framework focusing on low latency, helping you scale and consume financial data feeds.

DragonflyBot A vertically scalable stream processing framework focusing on low latency, helping you scale and consume financial data feeds. Design The

null 17 Jul 12, 2023
⬆ A program for deploying and upgrading programs.

DeployDAO Migrator WARNING: This code is a work in progress. Please do not use it as is. A program for deploying and upgrading programs. About The Mig

Deploy DAO 28 May 28, 2022