Solana Foundation stake bot used on the Solana Testnet and Mainnet-Beta

Problem

In order to have stake pools run nicely next to not-a-stake-pool, let's reuse a lot of the plumbing.

Solution

Add testnet-stake-pool as a cluster option for use with buildkite. Even though the variable RESERVE_ACCOUNT_ADDRESS is poorly named for stake pools, since it actually represents the address of the pool, it seems like it's best to maintain the same name to work with the current setup.

We'll just need to add another step to run the testnet-stake-pool version of stake-o-matic to buildkite.

While running the stake-o-matic for testnet stake pools, I kept running into the same issue often reported on buildkite:

[2021-09-10T01:47:00.354078368Z WARN  solana_stake_o_matic::rpc_client_utils] Failed to send transaction: ClientError { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: Blockhash not found", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(BlockhashNotFound), logs: Some([]), accounts: None }) }) }

So as a little patch / Ugly Thing :tm: , this force refreshes the blockhash.

This is not intended as a replacement for the work happening in #143, but as a band-aid. Note that the number of 100 was chosen based on my local machine. 200 still resulted in some blockhashes expiring, so I went with this safe value, which consistently manages to push all transactions out. We may be able to bump it up for the actual runs.

Problem Logic for making first-in-the-epoch classifications is lumped in with whether or not to push notifications. This prevents follower builds from pushing notifications from existing classifications.

Solution Break out the booleans.

Issues were:

• Bot was looking at the wrong testnet epochs (if MB epoch was 200, it checked TN epoch 200 instead of the current TN epoch)
• If a validator had None stake in all epochs where the bot checked for participation, the validator would not be destaked
• The most recent epoch was ignore when looking at participation

Adds an "export" subcommand that exports validator info for an epoch/cluster to a postgres database. The database will be used by the solana.foundation website.

Historical information is pulled from the yml files in the stake-o-matic.wiki files, and where the info is not available in those files it is pulled using RPC calls.

I currently plan on updating the stake-o-matic bot to export all needed data to the wiki yml files to make the more time-consuming RPC calls unnecessary when running the "export" command.

An actually-a-stake-pool stake-o-matic implementation!

This mostly follows the lead of stake_pool_v0, and adds a similar this_test_is_too_long implementation, all specialized for stake pools. It's possible to refactor a lot of the math to calculate how much stake to distribute to each validator, but I wanted to keep that separate for now.

A few differences to note design-wise for this stake pool implementation:

• the lamports for validator stake accounts come directly from the authorized staker's lamports, which functions as the staker's reserve
• separately, the stake pool also has a reserve for increasing / decreasing stake on the validators

Questions / comments / concerns appreciated! This shares a similar weakness to stake_pool_v0, that you can run out of stake from your reserve while trying to increase the validator stake levels. Have there been any issues with that in the wild?

Problem

Markdown and epoch classification output are tied together, but the stake pool's mainnet run needs the stake pool testnet classification. At the same time, we don't want to clobber the markdown generated by the not-a-stake-pool run.

Solution

Add an --epoch-classification flag, which works exactly the same way as the markdown flag, allowing for more fine-grained control over epoch classification output.

Problem

The mainnet job 429'ed on creation of the TpuClient, which we should be able to recover from.

Solution

Recover from any connection error on creation of the TpuClient!

Little note: I had this match in the loop originally as:

error @ Err(TpuSenderError::PubsubError(PubsubClientError::ConnectionError(_)))

which then lets us return error; on the last round, but it felt easier to read to just add another attempt at the end after the retries. Happy to be overruled though!

I register testnet and mainnet key the same. Now I can't withdraw and re-register new key for mainnet (Error: "Registration already exists") When I try withdraw registration [2021-06-17T12:40:41.908291066Z DEBUG solana_client::rpc_client] -32002 Transaction simulation failed: Error processing Instruction 0: invalid account data for instruction [2021-06-17T12:40:41.908303953Z DEBUG solana_client::rpc_client] 1: Program reg8X1V65CSdmrtEjMgnXZk96b9SUSQrJ8n1rP1ZMg7 invoke [1] [2021-06-17T12:40:41.908305990Z DEBUG solana_client::rpc_client] 2: Program log: Withdraw [2021-06-17T12:40:41.908307218Z DEBUG solana_client::rpc_client] 3: Program log: Error: mainnet and testnet identities must be unique [2021-06-17T12:40:41.908308479Z DEBUG solana_client::rpc_client] 4: Program reg8X1V65CSdmrtEjMgnXZk96b9SUSQrJ8n1rP1ZMg7 consumed 2160 of 200000 compute units [2021-06-17T12:40:41.908309831Z DEBUG solana_client::rpc_client] 5: Program reg8X1V65CSdmrtEjMgnXZk96b9SUSQrJ8n1rP1ZMg7 failed: invalid account data for instruction [2021-06-17T12:40:41.908311228Z DEBUG solana_client::rpc_client] Error: "error: send transaction: RPC response error -32002: Transaction simulation failed: Error processing Instruction 0: invalid account data for instruction [5 log messages]"

Bumps serde from 1.0.137 to 1.0.141.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Choose your own adventure! #163 or this.

Problem

Blockhashes expire too often while running the bot.

Solution

Use the TpuClient to send transactions instead of RpcClient. This adapts the work from spl-token-cli to send and check transactions: https://github.com/solana-labs/solana-program-library/blob/21ae18dbd0458f9e5f18b1407a626670f6475438/token/cli/src/rpc_client_utils.rs#L36

There's a bit of churn to update parameters all over the place, but otherwise the code isn't too different.

I'll be running tests over the next couple of days to see this performs compared to the other PR, and I'll move it out of draft once the testing is complete. If everything looks good, I'll probably opt for this.