Reference client for NEAR Protocol

NEAR

Last update: Jan 3, 2023

Related tags

Overview

Reference implementation of NEAR Protocol

About NEAR

NEAR's purpose is to enable community-driven innovation to benefit people around the world.

To achieve this purpose, NEAR provides a developer platform where developers and entrepreneurs can create apps that put users back in control of their data and assets, which is the foundation of "Open Web" movement.

One of the components of NEAR is the NEAR Protocol, an infrastructure for server-less applications and smart contracts powered by a blockchain. NEAR Protocol is built to deliver usability and scalability of modern PaaS like Firebase at fraction of the prices that blockchains like Ethereum charge.

Overall, NEAR provides a wide range of tools for developers to easily build applications:

JS Client library to connect to NEAR Protocol from your applications.
Rust and AssemblyScript SDKs to write smart contracts and stateful server-less functions.
Numerous examples with links to hack on them right inside your browser.
Lots of documentation, with Tutorials and API docs.

Join the Network

The easiest way to join the network, is by using the nearup command, which you can install as follows:

pip3 install --user nearup

You can join all the active networks:

TestNet: nearup run testnet
BetaNet: nearup run betanet

Check the nearup repository for more details how to run with or without docker.

To learn how to become validator, checkout documentation.

Development Status

We are using ZenHub to manage the development process. You can either login on their website or install ZenHub extension to see additional information right on Github.

For the high-level roadmap, checkout Chain&Middleware's roadmap.

Contributing

The workflow and details of setup to contribute are described in CONTRIBUTING.md, and security policy is described in SECURITY.md. To propose new protocol changes or standards use Specification & Standards repository.

Comments

Runtime hits unknown perfomance cliff in the estimator
See

https://github.com/matklad/nearcore/commit/80b44a877a1d2de76b4adcf63567984d615e7c36

In that commit, I add a benchmark which sends 150 blocks, where each block contains 100 deploy code transaction.

EDIT: https://github.com/near/nearcore/issues/4771#issuecomment-933347496 is modernized version.

The benchmark is implemented in two ways:

on top of parameter estimator testbed

on top of runtime tester

In the first case, I observe a very stage behavior, where at some point block processsing times severely degrades.

To reproduce, checkout this commit and run

$ cargo test --release --package runtime-params-estimator --lib --features required – v2::fast --exact --nocapture $ cargo test --release --package runtime-params-estimator --lib --features required – v2::slow --exact --nocapture

Look for Runtime::apply lines in the output, they they tell you how long it takes to process one apply.

In the fast case, every apply takes about 16ms for me. In the slow case, apply start at about 30ms, but then abruptly go to 600ms.

Would be great to understand why the time suddenly balloons!
A-transaction-runtime
opened by matklad 43
NumBlocks, NumShards, NumSeats, HeightDelta and _seats
Using NumBlocks and BlockHeightDelta in a proper way.

Correct naming in Epoch Manager.

Fixed https://github.com/nearprotocol/nearcore/issues/1855.

Unification of BlockHeight naming.

Set of apples in not an apple. Size of set of apples is not an apple either.

Size of string is not a string. Number of strings is not a string either.

num_shards() shouldn't return a ShardId. Number of validators is not a ValidatorId. Vector of number of validators shouldn't be Vec<ValidatorId>. Applying .iter().sum() to Vec<ValidatorId> shouldn't make any sense.

I expect mixing types leads to errors like https://github.com/nearprotocol/nearcore/issues/1855.
S-automerge
opened by Kouprin 40
State dump and verification
Currently when we dump the state from a node, we always dump state as of the last block that the node has. @ailisp points out that this block might be on a fork that is not canonical. I think there are two related problems here:

From which block should we dump state?

How do we set the genesis height of the new network?

I think before we figure out upgradability, it is fine that when we do hard fork, the NEAR team proposes a new state from the current network that other validators can verify. In order to do this, I think we should dump the state as of the last final block. Since for a block to be final, it needs to go through pre-vote and pre-commit with 2/3 stake approval, it is safe to assume that this block is known by the majority of the validators and they can run the same state dump & migration script to reach the same state as proposed.

Since we would like to avoid potential double signing, I think it is best to not reusable a height for the new network that has appeared in the old network. However, when the state dump happens, there might still be blocks that are inflight in the network, which makes it hard to determine the latest height. I think for the genesis height of the new network, it might be reasonable to use latest height from current node + some constant (like 10 for example) and it is fine if some heights are skipped.

To summarize, I propose that we:

dump state as of the last final block and publish the hash of the block from which we dump the state, together with the new state itself.

set the genesis height of the new network to be the latest height from the node + some constant.

A-build
opened by bowenwang1996 36
Investigate too many open files error

Sometimes after running for a while, a node would stops working because of some too many files open error. We need to investigate the cause and see whether we need to increase the limit for the number of file descriptors allowed.

opened by bowenwang1996 36
Error messages refactoring
Fixes: #978 and #1813

Overview

As discussed with @vgrichina @evgenykuzyakov @frol @fckt @janedegtiareva , we want the node RPC to return structured error messages, e.g. like these:

{"InvalidTx": {"NotEnoughBalance": {"signer_id": "alice_near", "signer_balance": 1000000000000000000000000, "total_cost": 1000000000000924119500000}} }

And we want the front-end and the Applayer to implement the conversion of the structured error messages to human-readable error messages, like these:

Sender alice_near does not have enough balance 1000000000000000000000000 for operation costing 1000000000000924119500000

More specifically we want nearlib to implement this conversion, which is predicated on having a detailed and exhaustive error specification.

Properties

There are two major differences with how error messages are currently returned by RPC:

We are decoupling a human-readable error message from the node code. This is not expected to have a downside since the software developer that will be interacting with the node directly and not through nearshell/nearlib is expected to be qualified enough to understand the JSON format. The upsides are:

We can iterate on error message description independently from the node releases;

Localization;

We are motivating dApp developers to have dApp-specific error descriptions instead of relying on the system errors, promoting better UX. E.g. "The character cannot be created because there is not enough gold" is better than "Sender alice_near does not have enough balance ... ".

We are using JSON (it seems like people are either indifferent on whether we use JSON vs our own format or are leaning towards JSON) for structured errors returned by the node. The upsides:

We can get rid this code by decorating error types with serde: https://github.com/nearprotocol/nearcore/blob/d869465f0400678b5df9e29d3bd926e8887d2337/core/primitives/src/views.rs#L677

We can now have error parameters, see signer_id, signer_balance, total_cost in the above example.

Implementation steps

[x] Get rid of the places where we are prematurely converting error to string, instead of using error hierarchy, e.g. here: https://github.com/nearprotocol/nearcore/blob/f93e13c0be52b8069a98de4792b861f07787f003/core/primitives/src/errors.rs#L67

[x] Also make sure we are not using boxed errors;

[x] Add a detailed rustdoc for every error message exposed through RPC. Generate the rustdoc and refer to it as error message spec (maybe even extract all errors into near-vm-errors);

[x] Replace error_type: String in ExecutionErrorView with error_type: ExecutionError and use simultaneously switch to using JSON in nearlib in places that rely on old format of error_type, e.g.: https://github.com/nearprotocol/nearlib/pull/87/files?file-filters%5B%5D=.js&file-filters%5B%5D=.json&file-filters%5B%5D=.ts#diff-52776b8b9355ae3e46add4ae6ac70c2eR47

[x] Implement the above error spec in the nearlib to produce human-readable error messages from structured error messages;

[x] Replace ExecutionErrorView with ExecutionError in nearcore, and simultaneously remove usage of .type field in nearlib, e.g.: https://github.com/nearprotocol/nearlib/pull/87/files?file-filters%5B%5D=.js&file-filters%5B%5D=.json&file-filters%5B%5D=.ts#diff-52776b8b9355ae3e46add4ae6ac70c2eR47

P-critical A-transaction-runtime C-epic A-RPC
opened by nearmax 35

nearcore doesn't return structured errors for most use cases of API

Structured errors work (https://github.com/nearprotocol/nearcore/issues/1839) is not complete and it's hard to know what to expect.

Example of error from nearcore:

    response.error {
      code: -32000,
      message: 'Server error',
      data: 'Other Error: block DtdfWgnt3QDEjoeARUK25vg8qh31kc3Lrg6qREsoJsST is ahead of head block 34eT8dRWfDPhZBiRGYvkHDj65ThaXCBTrnM4LQjn2Zab \n' +
        ' Cause: Unknown \n' +
        ' Backtrace:    0: failure::backtrace::Backtrace::new\n' +
        '   1: <actix::address::envelope::SyncEnvelopeProxy<A,M> as actix::address::envelope::EnvelopeProxy>::handle\n' +
        '   2: <actix::contextimpl::ContextFut<A,C> as core::future::future::Future>::poll\n' +
        '   3: tokio::runtime::task::raw::poll\n' +
        '   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\n' +
        '   5: actix_rt::runtime::Runtime::block_on\n' +
        '   6: near::main\n' +
        '   7: std::rt::lang_start_internal::{{closure}}::{{closure}}\n' +
        '             at rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/rt.rs:52\n' +
        '      std::sys_common::backtrace::__rust_begin_short_backtrace\n' +
        '             at rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sys_common/backtrace.rs:130\n' +
        '   8: main\n' +
        '   9: __libc_start_main\n' +
        '  10: _start\n'
    }

Note that even actual error message Other Error: block DtdfWgnt3QDEjoeARUK25vg8qh31kc3Lrg6qREsoJsST is ahead of head block 34eT8dRWfDPhZBiRGYvkHDj65ThaXCBTrnM4LQjn2Zab doesn't go into proper place in message.

This effectively makes current error message strings part of API, which is very suboptimal.

Here's how we have to handle errors because of this:

near-api-js

src/account.ts
120:                if (!error.message.match(/Transaction \w+ doesn't exist/)) {
205:            if (e.message.includes('does not exist while viewing')) {

src/wallet-account.ts
193:                if (e.message.includes('does not have enough balance')) {

near-wallet

src/actions/account.js
78:            if (error.message.indexOf('does not exist while viewing') !== -1) {

src/reducers/allAccounts.js
16:        if (payload.message.match('does not exist')) {

Note that we also handle errors explicitly only in few cases (degrading user experience) because we want to avoid depending on error messages in this way. This problem also degrades quality of every app built on NEAR, as everybody has same problem.

P-high A-RPC

opened by vgrichina 31

Fully switch to wasmer 1 by default

In https://github.com/near/nearcore/pull/3799, @ailisp implementing support for wasmer 1.0 :tada:

However, we are still using wasmer 0.17 by default -- major upgrade of vm version is not a laughing matter, we must make sure that the two behave identically. This issue exists to track our progress on switching to wasmer 1.0 by default, and eventual removal of support for wasemer 0.17 from the tree.

The next step in transition is to deploy a fraction of nodes with wasmer1 to the beta net.
T-SRE

opened by matklad 28
Extend the Math API with EVM precompiles
To speed up performance and lower the cost of EVM-as-a-contract, this adds several functions to the contract runtime Math API in order to support implementing the standard EVM precompiled contracts:

#3922 ripemd160() (RIPEMD-160 hash function)

#3953 blake2b() (BLAKE2b hash function)

#3921 ecrecover() (ECRecover ECDSA signer recovery)

References:

https://ethereum.stackexchange.com/questions/15479/list-of-pre-compiled-contracts/15483#15483

https://docs.soliditylang.org/en/latest/units-and-global-variables.html#mathematical-and-cryptographic-functions

https://en.wikipedia.org/wiki/RIPEMD

https://en.wikipedia.org/wiki/BLAKE_(hash_function)

https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/cryptography/ECDSA.sol

C-enhancement A-transaction-runtime A-cryptography A-EVM T-public-interfaces
opened by artob 27
Incompatible with Macs based on the Apple M1 chip

I'm recording this as a current known limitation: nearcore currently requires an x86 host which supports the AVX extensions.

https://github.com/near/nearcore/blob/daad871d042a838220ce5823d12844e94ce3af1f/runtime/near-vm-runner/src/wasmer_runner.rs#L191-L196

Apple's Rosetta does not support AVX as of macOS 11.1 (Big Sur):

Rosetta translates all x86_64 instructions, but it doesn’t support the execution of some newer instruction sets and processor features, such as AVX, AVX2, and AVX512 vector instructions. If you include these newer instructions in your code, execute them only after verifying that they are available.

This effectively precludes running nearcore on Apple Silicon, such as the initial Apple M1 chip found in the new 2020 MacBook Pro, MacBook Air, and Mac Mini. While that's unlikely to be an issue for any production deployments, it will increasingly hamper our development as developers switch to new Macs.

The future is NEAR and the future is ARM (on Macs, at least), so we better figure out how to reconcile the two before too long.
C-bug C-discussion A-contract-runtime

opened by artob 26
rpc requests interfere with block production

We have seen multiple times now that on testnet even a moderate amount of rpc requests can cause block production to slow down, which is quite undesirable. We need to understand the root cause and whether it is easily solvable. The last resort would be to disable rpc on block producing nodes.
C-enhancement P-low A-network A-RPC T-node

opened by bowenwang1996 25

refactor(neard): switch to derive clap

Refactors CLI to use the derive flavour of clap (which is basically just structopt)
Separates subcommand into separate modules (to keep main concise and allow cleaner expansion on the CLI)
- Commit 8d822b8 (#4356) which I'm happy to revert if wanted

I will decide now whether or not to

Move some of the validation logic into these new structures, to avoid passing strings and parsing later

Any input would be appreciated what is ideal :)

I wrote a script to run on all sub-commands, and although testing functionality would be tedious, here is their outputs if anyone wants to diff:

Before

NEAR Protocol Node 1.2.0 (build 07b4fd45) (protocol 45) (db 23)

USAGE:
    neard [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --home <home>          Directory for config and data (default "~/.near") [default: /home/austinabell/.near]
        --verbose <verbose>    Verbose logging

SUBCOMMANDS:
    help                 Prints this message or the help of the given subcommand(s)
    init                 Initializes NEAR configuration
    run                  Runs NEAR node
    testnet              Setups testnet configuration with all necessary files (validator key, node key, genesis and
                         config)
    unsafe_reset_all     (unsafe) Remove all the config, keys, data and effectively removing all information about
                         the network
    unsafe_reset_data    (unsafe) Remove all the data, effectively resetting node to genesis state (keeps genesis
                         and config)
neard-init 
Initializes NEAR configuration

USAGE:
    neard init [FLAGS] [OPTIONS]

FLAGS:
        --download-genesis    Download the verified NEAR genesis file automatically.
        --fast                Makes block production fast (TESTING ONLY)
    -h, --help                Prints help information
    -V, --version             Prints version information

OPTIONS:
        --account-id <account-id>                        Account ID for the validator key
        --chain-id <chain-id>                            Chain ID, by default creates new random
        --download-genesis-url <download-genesis-url>    Specify a custom download URL for the genesis-file.
        --genesis <genesis>
            Genesis file to use when initialize testnet (including downloading)

        --num-shards <num-shards>                        Number of shards to initialize the chain with
        --test-seed <test-seed>                          Specify private key generated from seed (TESTING ONLY)
neard-testnet 
Setups testnet configuration with all necessary files (validator key, node key, genesis and config)

USAGE:
    neard testnet [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --n <n>              Number of non-validators to initialize the testnet with (default 0)
        --prefix <prefix>    Prefix the directory name for each node with (node results in node0, node1, ...) (default
                             "node")
        --shards <s>         Number of shards to initialize the testnet with (default 4)
        --v <v>              Number of validators to initialize the testnet with (default 4)
neard-run 
Runs NEAR node

USAGE:
    neard run [FLAGS] [OPTIONS]

FLAGS:
        --archive    Keep old blocks in the storage (default false)
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --boot-nodes <boot-nodes>                        Set the boot nodes to bootstrap network from
        --min-peers <min-peers>                          Minimum number of peers to start syncing / producing blocks
        --network-addr <network-addr>
            Customize network listening address (useful for running multiple nodes on the same machine)

        --produce-empty-blocks <produce-empty-blocks>
            Set this to false to only produce blocks when there are txs or receipts (default true)

        --rpc-addr <rpc-addr>
            Customize RPC listening address (useful for running multiple nodes on the same machine)

        --telemetry-url <telemetry-url>                  Customize telemetry url
neard-unsafe_reset_data 
(unsafe) Remove all the data, effectively resetting node to genesis state (keeps genesis and config)

USAGE:
    neard unsafe_reset_data

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information
neard-unsafe_reset_all 
(unsafe) Remove all the config, keys, data and effectively removing all information about the network

USAGE:
    neard unsafe_reset_all

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

After

neard 1.2.0 (build 6ee4308a) (protocol 45) (db 23)
NEAR Protocol Node

USAGE:
    neard [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --home <home>          Directory for config and data (default "~/.near")
        --verbose <verbose>    Verbose logging

SUBCOMMANDS:
    help                 Prints this message or the help of the given subcommand(s)
    init                 Initializes NEAR configuration
    run                  Runs NEAR node
    testnet              Sets up testnet configuration with all necessary files (validator key,
                         node key, genesis and config)
    unsafe_reset_all     (unsafe) Remove all the config, keys, data and effectively removing all
                         information about the network
    unsafe_reset_data    (unsafe) Remove all the data, effectively resetting node to the genesis
                         state (keeps genesis and config)
neard-init 
Initializes NEAR configuration

USAGE:
    neard init [FLAGS] [OPTIONS]

FLAGS:
        --download-genesis    Download the verified NEAR genesis file automatically
        --fast                Makes block production fast (TESTING ONLY)
    -h, --help                Prints help information
    -V, --version             Prints version information

OPTIONS:
        --account-id <account-id>                        Account ID for the validator key
        --chain-id <chain-id>                            Chain ID, by default creates new random
        --download-genesis-url <download-genesis-url>
            Specify a custom download URL for the genesis-file

        --genesis <genesis>
            Genesis file to use when initializing testnet (including downloading)

        --num-shards <num-shards>
            Number of shards to initialize the chain with [default: 1]

        --test-seed <test-seed>
            Specify private key generated from seed (TESTING ONLY)

neard-testnet 
Sets up testnet configuration with all necessary files (validator key, node key, genesis and config)

USAGE:
    neard testnet [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --n <non-validators>    Number of non-validators to initialize the testnet with [default: 0]
        --prefix <prefix>       Prefix the directory name for each node with (node results in node0,
                                node1, ...) [default: node]
        --shards <shards>       Number of shards to initialize the testnet with [default: 4]
        --v <validators>        Number of validators to initialize the testnet with [default: 4]
neard-run 
Runs NEAR node

USAGE:
    neard run [FLAGS] [OPTIONS]

FLAGS:
        --archive    Keep old blocks in the storage (default false)
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --boot-nodes <boot-nodes>
            Set the boot nodes to bootstrap network from

        --min-peers <min-peers>
            Minimum number of peers to start syncing/producing blocks

        --network-addr <network-addr>
            Customize network listening address (useful for running multiple nodes on the same
            machine)

        --produce-empty-blocks <produce-empty-blocks>
            Set this to false to only produce blocks when there are txs or receipts (default true)

        --rpc-addr <rpc-addr>
            Customize RPC listening address (useful for running multiple nodes on the same machine)

        --telemetry-url <telemetry-url>                  Customize telemetry url
neard-unsafe_reset_data 
(unsafe) Remove all the data, effectively resetting node to the genesis state (keeps genesis and
config)

USAGE:
    neard unsafe_reset_data

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information
neard-unsafe_reset_all 
(unsafe) Remove all the config, keys, data and effectively removing all information about the
network

USAGE:
    neard unsafe_reset_all

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

Closes #4347

S-automerge

opened by austinabell 24

1.31.0-rc.1 - SIGTERM bug

Saw this while upgrading testnet to 1.31.0-rc.1 The node restarted and recovered. It says "bug detected", so letting the team know about it.

2023-01-04T17:06:45.517603Z  INFO neard: version="1.31.0-rc.1" build="1.31.0-rc.1" latest_protocol=57
2023-01-04T17:08:37.275282Z  INFO stats: #111558947 Downloading blocks 52.59% (339 left; at 111558947) 24 peers ⬇ 4.08 MB/s ⬆ 2.82 MB/s 12.11 bps 111 Tgas/s CPU: 279%, Mem: 2.16 GB
2023-01-04T17:08:38.339374Z  WARN neard: SIGTERM, stopping... this may take a few minutes.
thread '<unnamed>' panicked at 'bug detected: OutboundHandshakePermit dropped before calling Pool.insert_ready()', chain/network/src/peer_manager/connection/mod.rs:323:25
2023-01-04T17:08:38.344593Z  INFO neard: Waiting for RocksDB to gracefully shutdown
2023-01-04T17:08:38.344602Z  INFO db: Waiting for remaining RocksDB instances to shut down num_instances=1
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at ./rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: near_network::peer_manager::connection::Pool::insert_ready
   3: tokio::runtime::task::harness::poll_future
   4: tokio::runtime::task::raw::poll
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

when I first deployed the upgrade, I got an error message saying my ulimit was too low and it needed to be updated to 65535. I did that and restarted the node and then I got the above message.

I also run 2 other nodes, I applied the ulimit changes to those first and never saw the panic above.

opened by nick-u410 0

chore: remove outdated constant ATTO_NEAR
This constant is:

wrong: it should be yocto near (10^-24) and not atto (10^-18)

outdated: we haven't used atto near since 2019 (#1827)

unused: there are no code references to it

Better we get rid of it.
opened by jakmeier 0
Regression for Tier1 public addresses in 1.31.0-rc.1
Describe the bug

In neard 1.30 we were able to specify in addition to our ip address also an legacy ipv4 address like this:

"network": { "public_addrs": [ "ed25519:AtHDrCYckKmhJKqzX94mKHtshWC3VSkEsDa3tBVvEkj@[2001:41d0:800:3f9a::1]:24567", "ed25519:[email protected]:24567" ], }

With neard 1.31.0-rc.1 however we get the following error message:

thread 'main' panicked at 'Error loading config: public_addrs: found multiple entries with peer_id ed25519:ydgzeXHJ5Xyt7M1gXLxqLBW1Ejx6scNV5Nx2pxFM8su. Only 1 entry per peer_id is supported.', neard/src/cli.rs:397:33

Expected behavior

neard accepts at least two addresses so that we can also support ipv4 in addition to ipv6. Otherwise people that do not have ipv6 yet, won't be able to connect to our nodes.
opened by Mic92 0
chore: upgrade rust-version to 1.66.0
https://blog.rust-lang.org/2022/12/15/Rust-1.66.0.html

Updated the same files as in the previous version bump: https://github.com/near/nearcore/pull/7993

This enables https://github.com/near/nearcore/issues/7650 (see this comment).

Also includes fixes for the following warning:

warning: for loop over an `Option`. This is more readably written as an `if let` statement ... = note: `#[warn(for_loops_over_fallibles)]` on by default
opened by pugachAG 1
Dynamic Chunk Gas Limit (only in discussion)
There have been discussions around adjusting gas limits per chunk dynamically, in an effort to automatically resolve undercharging that negatively impacts block production time. Let's track the facts around that here so it does not get lost.

How it Works Without a Protocol Upgrade

The chunk header has a field called gas_limit which is initialized to the constant 1000 Tgas in the genesis configuration. When producing a new chunk, we always set it to the new limit to the old value. (See pub fn produce_chunk)

However, it seems that adjusting the limit by less than 1 / GAS_LIMIT_ADJUSTMENT_FACTOR = 0.1% will pass chunk header validation. This means, AFAIU, we could implement a change where a chunk producer (single-handedly) decides to increase or decrease the value based on its local "measurement of undercharging" without violating the protocol. This adjustment does not need to be deterministic, the chunk producer has the freedom to decide this value.

What we could do

We could continuously measure the chunk production time in nearcore clients and compare it to the gas burnt. If the chunk producer observes undercharging, it decreases the limit. If there is overcharging, the limit can be increased up to a limit of at most 1000 Tgas.

To avoid volatility, I suggest to use a moving average of observed under/overcharging. And we might want to limit how much a single validator adjusts the value, especially if it is the only one observing the undercharging. For example, if a chunk producer has decreased the limit already in the previous chunk it produced, it would only decrease again if another chunk producer has also decreased the limit at least once in the meantime.

Pros and Cons

Good:

Prevent moderate undercharging from stalling the network.

No protocol change necessary, we could easily experiment and revert if it does not work well.

Bad:

Very broad granularity. (Undercharging in one parameter affects all users, even those that never use the undercharged parts.)

Dependence on validator hardware. (Someone running overspecced hardware will continuously want to increase the limit, others might run with underspecced hardware and continuously want to decrease the limit.)

Malicious undercharging attacks are unlikely to be prevented by this. (A single 10x undercharged receipt still needs to be processed using the old limit. Adjusting 0.1% per block means 100 chunks can only change by a maximum of 1.1x and 1000 chunks could do a factor of 2.7).

Conflict with transaction & receipt limit. A transaction or receipt can (today) use up to 300Tgas. The effective limit per chunk is gas_limit + 300Tgas since receipts are added to a chunk until one exceeds the limit and the last receipt is not removed. Thus a gas limit of 0gas only reduces the effective limit from 1300Tgas to 300Tgas, which means undercharging of > 4.33x is not covered.

C-discussion A-transaction-runtime
opened by jakmeier 1

Releases(1.31.0-rc.1)

1.31.0-rc.1(Jan 3, 2023)
Non-protocol changes

Enable TIER1 network. Participants of the BFT consensus (block & chunk producers) now can establish direct TIER1 connections between each other, which will optimize the communication latency and minimize the number of dropped chunks. To configure this feature, see advanced_configuration/networking. #8141, #8085, #7759

/status response has now two more fields: node_public_key and validator_public_key. The node_key field is now deprecated and should not be used since it confusingly holds validator key. #7828

Added near_node_protocol_upgrade_voting_start Prometheus metric whose value is timestamp when voting for the next protocol version starts. #7877

neard cmd can now verify proofs from JSON files. #7840

In storage configuration, the value trie_cache_capacities now is no longer a hard limit but instead sets a memory consumption limit. For large trie nodes, the limits are close to equivalent. For small values, there can now fit more in the cache than previously. #7749

New options store.trie_cache and store.view_trie_cache in config.json to set limits on the trie cache. Deprecates the never announced store.trie_cache_capacities option which was mentioned in previous change. #7578

New option store.background_migration_threads in config.json. Defines number of threads to execute background migrations of storage. Currently used for flat storage migration. Set to 8 by default, can be reduced if it slows down block processing too much or increased if you want to speed up migration. #8088,

Tracing of work across actix workers within a process: #7866, #7819, #7773.

Scope of collected tracing information can be configured at run-time: #7701.

Attach node's chain_id, node_id, and account_id values to tracing information: #7711.

Change exporter of tracing information from opentelemetry-jaeger to opentelemetry-otlp: #7563.

Tracing of requests across processes: #8004.

CODE_COLOR: CODE_GREEN_TESTNET RELEASE_VERSION: 1.31.0-rc.1 PROTOCOL_UPGRADE: FALSE DATABASE_UPGRADE: TRUE SECURITY_UPGRADE: FALSE
Source code(tar.gz)
Source code(zip)
1.30.0(Dec 12, 2022)
Features:

account_id_in_function_call_permission enforces that account id in function call permission is indeed a valid account id https://github.com/near/nearcore/pull/7569

Add proof to ViewState response and include_proof argument to ViewState request https://github.com/near/nearcore/pull/7603 https://github.com/near/nearcore/pull/7593

Move the check for is_height_processed forward before process_block_header (https://github.com/near/nearcore/pull/7831)

[Debug UI] Show github link for version on debug page

[Debug UI] Improve last-blocks debug page

Add hash to PeerChainInfo

Change debug page to display error messages in block processing

Fixes

do not enter sync mode if peer is at an invalid hash mark block as invalid in preprocess_block

Deprecate GCCount column https://github.com/near/nearcore/pull/7632

near-vm-runner: fix m1 build https://github.com/near/nearcore/pull/7587

CODE_COLOR: CODE_YELLOW_MAINNET RELEASE_VERSION: 1.30.0 PROTOCOL_UPGRADE: TRUE DATABASE_UPGRADE: FALSE SECURITY_UPGRADE: FALSE
Source code(tar.gz)
Source code(zip)