PolkaBTC Clients | Vault, Staked Relayer, Oracle, Faucet

Generates a PID file named {spec_name}_{account_id}.pid, to the default temp directory of the OS.

On startup, if such a file exists and the PID in it is indeed a live process, the vault throws an error.

The PID file is removed after a termination signal is received, by implementing Drop on the PidFile struct.

Signed-off-by: Gregory Hill [email protected]

See the comment on this PR: https://github.com/interlay/interbtc-clients/pull/389

On a side note, because we removed the checks in service to always restart on any error we will be stuck in a loop now if the wrong Bitcoin network is configured. We should distinguish between recoverable and unrecoverable errors for which this is the latter.

Specifies Abort and Retry variants on the service::Error enum to allow the consumer to propagate a hard-error to the ConnectionManager.

On a related note, I'm not sure there is any justification for the service logic to have its own crate since it is only ever consumed by the vault binary. I wonder if we should fold it in?

Is your feature request related to a problem? Please describe. The vault client is released with breaking changes on a regular basis, forcing operators to check the docs website and manually update. This is both error-prone and inconvenient. Providing operators with an opt-in updating solution can ensure higher system availability.

This problem multiplies when an operator runs vaults on multiple networks, having to keep track of multiple releases.

Describe alternatives you've considered

• Cosmos releases can be run within a process manager that handles auto-updates, called Cosmovisor. It listens for on-chain version upgrade events to download and run new releases. Positional arguments are passed through to the node binary. Cosmovisor is implemented in Go, so it's not possible to just swap the cosmos sdk event listener with subxt.
• Google open-sourced Omaha, the updater used by Chrome and Google Earth. Similarly, Omaha checks an update server and then downloads both the new binary and updater logic (e.g. migrations). It seems to only target Windows, and the update server implies trusting the host.

Describe the solution you'd like Implement Vaultvisor, a wrapper software similar to the Cosmovisor that listens for on-chain version release events and updates the vault client. Given a JSON file with input arguments, the Vaultvisor can run and auto-update binaries across the Kintsugi, Interlay, and their respective testnet networks.

• Vaultvisor spawns a separate process to run the vault binary. Positional arguments are specified in a JSON file that can have migrations applied to it.
• Add a new extrinsic in the Vault Registry: set_vault_release(version, binary_checksum), which emits a VaultClientUpdate(version, binary_checksum) event. The extrinsic writes to a storage item which is read when the Vaultvisor is first started up.
• Vaultvisor uses subxt to listen for the VaultClientUpdate event. Based on the connected parachain network and the emitted version, it can derive the GitHub URI of the new release. Then, it:
  • Downloads the binary and ensures its checksum matches the one on-chain.
  • SIGINTs the current vault process.
  • Runs the new binary. If successful, removes the old binary. Optional: if unsuccessful, runs the old binary and periodically retries the new one.
• Once the vault starts persisting local state with rocksdb, migrations for this would also need to be handled by Vaultvisor.

Additional Context This solution requires releasing the vault client before a runtime upgrade is merged. JSON argument and rocksdb migrations would require their own script, with its own checksum stored on-chain.

Assumption: Sending a SIGINT signal to the vault process does not cause errors or side-effects.

Signed-off-by: Gregory Hill [email protected]

Errors which occur in one of the "tasks" are not propagated to the ConnectionManager so we should read this reason from the shutdown channel.

The experimental features #![feature(array_zip, int_log)] make some parts of the code a lot cleaner (array_zip lets me iterate over fixed-size arrays maintaining size info, int_log makes it trivial to do the ordering calculation without casting to floats). What's our stance on adding these? I can also just rewrite those bits without them.

Closes #294

Add/improve Prometheus rules for:

• vault startup doom loop (if bitcoind can't service RPC requests in time)
• new redeem not serviced yet
• below secure threshold

In particular, set a minimum (0.01) for some values to avoid false negative alerts.

Handles termination signals, using a similar approach to https://github.com/interlay/interbtc-clients/pull/330.

This feature makes way for cleaning up the PID file on shutdown, in https://github.com/interlay/interbtc-clients/pull/347

I tried simplifying the scaffolding required for the mocks but still couldn't find a way around using trait methods only as an indirection to the actual implementation.

This feature is opt-in in the code (call catch_signals() instead of start()) and could be moved to the service crate instead.

Panics

If the given signal is [forbidden][crate::FORBIDDEN].
If the signal number is negative or larger than internal limit. The limit should be larger than any supported signal the OS supports.
If the relevant Exfiltrator does not support this particular signal. The default [SignalOnly] one supports all signals. <- we're using the default

• Uses signal-hook crate to subscribe to termination signals from the OS. The subscription sets a flag stored in the Runner struct, which has to be checked periodically.
• As a result, there are now two types of actions occurring at intervals: checking the termination flag and checking for a new release.
• Adds unit test that asserts the run function: 1) successfully terminates if the flag is set; 2) loops if the flag is not set

Description

The vault logs uses prefix 42 instead of whichever chain it is connected to e.g. 2092 for kintsugi.

Tested against: kintsugi main net, vault is successfully integrated to the network though, so this is logging defect only.

Steps to recreate

1. Run setup for kintsugi mode and start vault as normal

⚠️ Account format used is not for kintsugi

Apr 27 18:29:10 timbo-ubun2 systemd[1]: Started Vault servicing issue and redeem requests for Kintsugi.
Apr 27 18:29:10 timbo-ubun2 vault[7384]: Apr 27 18:29:10.824  INFO vault: Starting Prometheus exporter at http://127.0.0.1:9615
Apr 27 18:29:10 timbo-ubun2 vault[7384]: Apr 27 18:29:10.824  INFO service: Version: 1.9.6
Apr 27 18:29:10 timbo-ubun2 vault[7384]: Apr 27 18:29:10.824  INFO service: AccountId: 5CFUfGrWKPdTJQ5bM4co11cYijfHXnp96PBtYj3K2dk2KSpd

Signed-off-by: Gregory Hill [email protected]

We could fetch this parameter from the metadata but then it would become very involved to pass the value.

Signed-off-by: Gregory Hill [email protected]

For whatever reason impl std::fmt::Display for DispatchError does not work so this seems to be the only way to capture and interpret the runtime error.

Related: https://github.com/interlay/interbtc-clients/pull/381

When starting a vault while the bitcoin node is still syncing, I got this behavior:

Dec 20 11:12:44.359  INFO bitcoin: Waiting for bitcoin-core to sync...    
Dec 20 11:26:32.005  INFO bitcoin: Synced!    // NOTE: this took a long time
Dec 20 11:26:32.029  INFO bitcoin: Loading wallet vault-live-polkadot-master...    
Dec 20 11:26:47.234  INFO bitcoin: Call timed out - retrying...    
Dec 20 11:26:47.235  INFO bitcoin: Loading wallet vault-live-polkadot-master...    
Dec 20 11:26:47.265  INFO vault::process: Removing PID file at: /tmp/interlay-parachain_5HCEA2NynWi1J7xfup8dvjV6tVYGN5GkVYMLhddTn8ujFTur.pid
Dec 20 11:26:47.273 ERROR vault: Exiting: BitcoinError: BitcoinError: JSON-RPC error: RPC error response: RpcError { code: -4, message: "Wallet already loading.", data: None }

Is your feature request related to a problem? Please describe. Vault registration will fail when using the faucet if the amount exceeds the free balance or currency ceiling.

Describe the solution you'd like The Vault should use the maximum possible value (i.e. max(ceiling - total_collateral, max(amount, free_balance))) to prevent exiting early.

Additional context This may actually be preferable behavior since otherwise the operator will not know that the full amount was not locked.

As reported by Quarkslabs:

Description: If a user makes multiple issue requests and pays them in the same Bitcoin trans-action, then the second payment output will be missed by the vaults. This is due to the use of find_map in the process_transaction_and_execute_issue function which only returns the first matching element as stated in its documentation. Recommandation: A possible fix for this issue would be turning the if let construct into a while let construct. However this alone should not be enough as the core of the function can return and needs to be changed accordingly.

As reported by Quarkslab:

Description: When the relayer tries to send multiple Bitcoin blocks to the parachain, it makes use of the collect_headers function which could potentially panic due to the use of unwrap(). When the relayer queries the Bitcoin node using the get_block_header function, the collect_-headers function could receive a Ok(None) result which would trigger a panic. Recommandation: This issue can be fixed by changing the line pushing the header using a if let construct