Tasks

• [x] Keep track of lowest and highest priorities found in recent epochs (using a circular queue with a fixed capacity)
• [x] Estimate priority values for different priority tiers (stinky, low, medium, high and opulent)
• [x] Estimate time-to-block for the same priority tiers
• [x] Logging current estimation for each epoch when in debug level
• [x] Add tests
• [x] Make this feature available through JSONRPC
• [x] Make this feature available through the CLI
• [x] Make this feature available through the wallet component
• [x] Update send, split and join CLI methods for interactive fee choosing with --suggest-fee flag
• [x] Maybe do the same as above with sendRequest
• [x] Disallow estimation methods until synced
• [x] Try to get rid of the big warning below by using some proper fixed point type
• [x] Move PriorityEngine from ChainState to ChainManager
• [x] Linting

~~ALL PRIORITY VALUES USED IN THIS FEATURE ARE MULTIPLIED BY 1000 FOR SUB-NANOWIT PRECISION, PROVIDED THAT THE SMALLEST POSSIBLE FEE IS 1 NANOWIT BUT THE LIGHTEST TRANSACTION IS 169 WEIGHT UNITS.~~

~~FOR THE SAKE OF UX, THESE VALUES STILL NEED TO BE PRESENTED TO WALLET USERS IN NANOWIT, IDEALLY WITH 1 DECIMAL DIGIT IF THAT DIGIT IS DIFFERENT FROM 0.~~

fix #2273

As Rust 1.34 is now on the stable channel, TryFrom has been removed the unstable flag and we don't need our own TryFrom any more.

Contributing

• All your commits need to be verified with your PGP signature.
• Commit messages need to abide by the conventional commits specification, and references to the original issue are not needed as we use use ZenHub for that. A example for this issue could be:

  refactor(data_structures): remove our own try_from
  

Btw, feel free to drop by the Witnet community on Discord at any time to discuss any future contributions and learn more about the project :D

According to https://github.com/witnet/WIPs/blob/master/wip-0010.md, we discuss that a Rescue Committee could help supporting the network during long rollbacks.

The Rescue Committe would always send their Superblock votes, that they would be valid for the network, but they would only be taken into account in specific moments. The Normal Committee and the Rescue Committee would be handle separately and the decision to consolidate or not a Superblock would depend of the result of both of them.

We could describe as:

• n1: Normal Committee has a consensus > 2/3
• n2: Normal Committee has a consensus > 1/3, but < 2/3
• n3: Normal Committee has a consensus < 1/3
• r1: Rescue Committee has a consensus > 2/3
• r2: Rescue Committee has a consensus < 2/3

With those possibilities we could describe different situations and how should work:

• n1 & (r1 | r2) -> n1 (If Normal Committee has a consensus > 2/3, Rescue Committee is not needed and it will be ignored)
• (n2 | n3) & r2 -> Rollback (If Normal Committee and Rescue Committee have not a consensus > 2/3, it will be a Rollback equals than current flow)
• n3 & r1 -> r1 (If Normal Committee has a consensus < 1/3, we will use the consensus achieved by the Rescue Committee)
• n2 & r1 if n2=r1 -> n2 | r1 (If Normal Committee has a consensus < 2/3 but it is the same than the Rescue Committee consensus, that consensus will be applied)
• n2 & r1 if n2!=r1 -> Rollback (If Normal Committee has a consensus < 2/3 and it is different than the Rescue Committee consensus, it will be a Rollback)

With this logic, the Rescue Committee only be taken into account when the Normal Committee has not a proper consensus (n2) offering support to take a decision or using their own consensus in case of Normal Committee is in a bad state (n3), until it would be recovered to a healthy state (n1)

Moreover, the logic of reducing the committee size is not longer require, so it would reduce the possibilities that several forks could be produced when it achieved the size of 1.

This PR introduces a JSON RPC method to calculate the current supply of WIT tokens. This method takes the utxo-pool to calculate current supply, locked supply and a hardcoded total supply. The latter is apparently not a constant I can take from somewhere, but I assume it is implicitly defined through the block reward calculation function block_reward in validations.rs.

The supply that is calculated is a bit of an approximation. I assume the discrepancies in the below examples can be explained by in-flight transactions that mark UTXO's as in use (and thus not a UTXO anymore).

For example between screenshots 1 and 2, there are 32 epochs, which should result in a supply increase of 8000, but it is 8200.

The difference in supply between screenshots 3 and 4 is even negative (despite a difference of 18 epochs).

Maybe this could be improved by checking the transaction pool, but I guess I first need to know if my suspicion is actually correct?

Let me know what you think should be improved.

I purposely did not rebase this on master because I guess it's easier to test the PR on the latest release.

to make it lighter than using Docker and downloading at every start it should make possible to run dozens of nodes natively on the host without requiring much

This is a proposal to limit spamming the network with free VTT's (like we saw today). It could also be extended to DR's, but I did not include that yet because I'm not sure that is something we ultimately want.

This PR would allow a node operator to set a minimum transaction fee he wants to receive before including a VTT into a block whenever he's eligible. I would set this to 1 nanoWIT (though I did not set that as default), but obviously everyone can choose a setting.

Let me know what you think about this and if this unexpectedly breaks something. For example, I am not sure if this would let the VTT pool grow unlimited in the case of spam (I thought there was some sort of periodic cleanup).

Additionally, now a LastBeacon message is sent to all inbound peers as soon as they consolidate their connection, which greatly reduces the synchronization delay.

The problem

• The wallet component was initially created as a backend to the Sheikah wallet app.
• It was an assumption that Sheikah users wouldn't create 2 transactions in a row in the lapse of 1 block. And even if they did, they would be ok retrying the second one after a minute if it failed.
• The Witnet protocol doesn't allow creating and spending an UTXO inside the same block.
• The node component implements logic that caches out recently used UTXOs to prevent any attempt of double spending when 2 transactions are created in a row.
• As far as I can tell, the wallet doesn't include such protection.
• This may be causing accidental double spend attempts for "programmatic users" (e.g. exchanges). In case that a withdrawal .transaction is issued while a previous one is still pending, there's a possibility for both transactions to use one or more UTXOs in common. When broadcasting, there will be an error saying that the UTXO is no longer available for spend.

The solution

• Implement caching logic in the wallet that prevents a UTXO from being "mounted" as input into a transaction if it was recently used in another one.
• This shall use a timeout to "unmark" used UTXOs if the transaction that used it was not confirmed in a while.
• The node already implements all that logic, so we can replicate, or even better, unify.

Immediate mitigations for users before a proper fix is implemented

1. Split the balance in your wallet across multiple UTXOs of roughly the same size. If you have 10 of those, you will be able to process up to 10 withdrawals concurrently without facing this issue. This is probably the most effective and effortless mitigation.
2. Bundle up all withdrawal requests received in the last 5 minutes or so and issue them at once in a single transaction with multiple recipients.This is a normal procedure of most exchanges to save transaction fees.
3. If a transaction fails at creation, retry after a minute. Well, just in case anything else didn't work.

Recently there have been many blocks in the network that do not include any TX apart from the tally and the mint transactions (mandatory). After investigating these block producers, we have observed that all of them come from newcomers. The proportion of blocks coming from newcomers is bigger than we expected, so after reviewing our equations we have found that the probability for newcomers is wrongly calculated in accordance to our DoS protections. Currently, the eligibility probability is being calculated utilizing the active nodes as a denominator, but then only those blocks produced by reputed nodes are given priority over the rest. With current numbers this yields:

Active Nodes: ~6000

Reputed Nodes: ~2000

RF=3

P(no Reputed node produces block) = P(one reputed node does not produce block)^(number of reputed nodes) = (1 - (3/6000))^2000 = 0.37

In other words, currently there is a 37% chance that a newcomer can insert its node in the ARS, and thus also censor TXs by creating empty blocks. This is far from the numbers we analyzed, and in accordance the following preemptive measures have been discussed:

• Utilize as a denominator for the eligibility the Reputed number of nodes, instead of the ARS length. This would increases the probability for reputed nodes to become eligible, and thus would make it more difficult for an outsider to censor TXs. The only caveat is that it increases the number of candidates.

• Utilize as a tie-breaker the fact that a node IS IN THE ARS, instead of a node having reputation. This also decreases the number of odds for an outsider to censor TXs, as usually the ARS is larger than the number of reputed nodes.

• Increase the replication factor while removing the back up factor. We are not even near of being epochs without block candidates, so the back up factor removal is something we should start considering. Further, an incerase on the replication factor gives more probability for the reputed/ARS nodes to find a valid candidate and thus prevent an outisder from continously proposing blocks.

• Take into account the block weight in terms of data requests, giving of course a lower priority to empty candidates.

This should follow a proper discussion, and most likely yield a hard fork. I see option 1 and option 2 as exclusive, while 3 and 4can be added on top any of the first two

Resources: https://github.com/witnet/research/blob/master/reputation/docs/randpoe.md

Close #1029

Add optional field prefill to unlock_wallet

{
  "wallet_id": "...",
  "password": "...",
  "prefill": [1000, 2000, 3000]
}

This will generate 3 new addresses and send 1000 nanowits to the first address, 2000 to the second and 3000 to the third.

In case of error (if the node is not synced or it does not have any balance), the addresses are generated but will not receive any wits.

This is a rehash of #1523

Enables using proxies (HTTP, HTTPS, SOCKS4 and SOCKS5) when performing data retrieval. This allows retrieving data sources through different transports so as to ensure that the data sources are consistent and we are taking as small of a risk as possible when committing to specially crafted data requests that may be potentially ill-intended.

Just add this to the [connections] section of witnet.toml to proxy your retrievals through 3 different open proxies:

retrieval_proxies = [
        "socks5://181.102.35.195:1080",
        "socks5://8.210.60.118:1080",
        "socks5://47.114.8.133:1080"
]

These addresses have been taken from this public list of proxies.

Close #1274 Close #1511

Bumps tokio from 1.20.1 to 1.20.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

When trying to prototype #2331, we have noticed that the surf client does not properly support HTTP HEAD requests. That issue has been reported on the surf repo more than two years ago, but it has not been fixed yet, even though it looks like a simple fix. And there are similar issues and FIXMEs in witnet-rust regarding surf issues, such as url parsing for example. Also, the last commit on the surf repo is from May 10, 2022, and it doesn't look very active anymore. Therefore it seems reasonable to stop using surf as a dependency.

Fortunately, we now have a witnet http client implemented here: https://github.com/witnet/witnet-rust/blob/11d37d32c6caa3965d716854beae728867c5ce70/net/src/client/http/mod.rs#L1

That client is a wrapper over the isahc http client, of which surf is also a wrapper. So it should be straightforward to add the missing API there, and drop the dependency on surf. And since we will still be using isahc under the hood, this should be a minor change, and it shouldn't break any requests.

Data requests with RADType::Unknown sources are wrongly accepted into blocks since WIP0020.

Example transaction: 38d8cf497622082fa8fd4e89831641834918a99472b722310d60664046a29883

{"jsonrpc":"2.0","result":{"blockEpoch":1544550,"blockHash":"3a077d6b4064524afa01e1b8781cfb059693653d5138734b0ffef27253a24bc9","confirmed":true,"transaction":{"DataRequest":{"body":{"dr_output":{"collateral":10000000000,"commit_and_reveal_fee":1000,"data_request":{"aggregate":{"filters":[{"args":[251,63,241,153,153,153,153,153,154],"op":5}],"reducer":3},"retrieve":[{"kind":"Unknown","script":[130,24,119,130,24,101,101,108,105,107,101,115],"url":"https://api-middlewares.vercel.app/api/twitter/tweets/0"}],"tally":{"filters":[{"args":[251,63,241,153,153,153,153,153,154],"op":5}],"reducer":3},"time_lock":1630669134},"min_consensus_percentage":51,"witness_reward":1000000,"witnesses":100},"inputs":[{"output_pointer":"7cc0415818cd633916c4a3d3ea097d13e04973a8be010baea39244d83fa6d6bc:0"}],"outputs":[{"pkh":"wit1c0ucz2u9hwgqflkn4xjeyxtla9tqp0sderzkjr","time_lock":0,"value":2166708072}]},"signatures":[{"public_key":{"bytes":[176,193,72,220,165,255,123,112,10,173,38,116,83,157,103,237,213,54,147,62,72,34,100,153,202,212,153,241,200,151,13,32],"compressed":2},"signature":{"Secp256k1":{"der":[48,69,2,33,0,144,199,29,249,135,183,74,33,30,61,204,36,30,87,98,167,21,127,197,46,46,14,71,113,28,140,180,187,140,181,170,26,2,32,52,178,190,191,36,97,187,28,26,178,118,166,244,232,234,177,76,247,204,102,75,70,54,251,121,21,52,208,151,229,70,15]}}}]}},"weight":64009},"id":"1"}

Summary of the behavior:

• Before WIP0019: RADType::Unknown are accepted, and treated as HttpGet.
• WIP0019: RADType::Unknown are rejected.
• WIP0020: RADType::Unknown are accepted, and resolved with RadonTypes::RadonError(UnhandledIntercept { inner: None, message: Some("inner: UnknownRetrieval") }).

This is implemented in the validate_rad_request function: https://github.com/witnet/witnet-rust/blob/11d37d32c6caa3965d716854beae728867c5ce70/validations/src/validations.rs#L239

So this looks like a bug introduced when implementing WIP0020. I believe that the intended behavior was to reject RADType::Unknown sources, because they were already being rejected after WIP0019, so that should be fixed.

However, this issue shouldn't have any negative effects on the network, because we used to accept this kind of transactions in the past, and all the other validations are working correctly so it is not possible to abuse this bug to perform double spends or anything like that.

So only the metadata from a web resource is actually fetched in the form of HTTP response headers.

For instance, these HTTP response headers would be quite handfull for implementing content-based use cases with Witnet:

$ curl --head https://witnet.io/_nuxt/img/dragon_reading.a37f8cb.png
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 498219
etag: "632067ee-79a2b"
...

$ curl --head https://ipfs.io/ipfs/QmQqzMTavQgT4f4T5v6PWBp7XNKtoPmC9jvn12WPT3gkSE
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 38376
Etag: "QmQqzMTavQgT4f4T5v6PWBp7XNKtoPmC9jvn12WPT3gkSE"
X-Ipfs-Path: /ipfs/QmQqzMTavQgT4f4T5v6PWBp7XNKtoPmC9jvn12WPT3gkSE
X-Ipfs-Roots: QmQqzMTavQgT4f4T5v6PWBp7XNKtoPmC9jvn12WPT3gkSE
...

The implementation of HTTP/HEAD requests should:

• Support addition of HTTP request headers.
• Transform response header lines into a key/value map

Also, it would be nice to support a new String operator in Radon:

• decodeHash(<encoding_type>), as to transform etag literals into properly decoded array of bytes, at least supporting SHA-256 as possible encoding type.

Possible new use-cases for Witnet:

• Verifying a web binary content (i.e. an image, pdf document, ...) is available at the moment the data request is executed, without the nodes actually needing to download the whole file.
• Verifying the content type, length and etag of any given URL.

Javascript DSL usage example:

const token_image_digest = new Witnet.HttpHeadSource(
  "https://api.game.art/images/1.png",  {
    "Transfer-Encoding": "identity"
  }
)
  .parseJSONMap() // perhaps not necessary, as response to HttpHeadSource should always be a key/value map
  .mapGetString("Etag")
  .stringDecode(Witnet.HASHES.SHA256)

There's a small caveat with this PR

• both the bridge workflow & master workflow trigger the second a tag is released
• the bridge docker image in docker/bridge/Dockerfile uses witnet-rust:latest as a base
• naturally this will end up making the bridge use the pre-last witnet-rust image

Courses of Action:

• if it doesn't matter, approve PR and we can leave it as is
• if it does matter, we incorporate the workflow into the master workflow after the image deployment so that we can use that stage as a dependency
• we use/build a different image base (wouldn't recommend it!!!)