Computational Component of Polkadot Network

Resolves #900

• enable output of JUnit XML for nextest in pallet tests
• add execution time measurement in gear-runtime-test-cli
• add output of similiar JUnit XML in gear-runtime-test-cli
• add utility to compare execution times for provided junit.xml
• adjust CI to collect data regularly
• add script to collect measurements locally and compare results
• add bot commenting PR with compare results

@gear-tech/dev

• Resolves #1624
• Removed gear-node crate.
• Changed gear-cli crate to produce a binary.
• Replaced gear-node binary with gear and gear-node crate with gear-cli.
• Changed default --base-path to gear-node instead of executable name (gear[.exe]) for backward compatibility.

@gear-tech/dev

Resolves #1131.

gear-tech.github.io/performance-charts

To do:

• [x] Instruction weight charts
• [x] 3 charts in a row in GH pages
• [x] Make unit as ms
• [x] Check SSH deploy keys work instead of personal access token https://github.com/marketplace/actions/push-directory-to-another-repository
• [x] #1230

in order to optimise lazy pages work - move logic which calc keys for pages in signal handler. Also completely remove lazy_pages set because it can be very costly to support it when memory size is big

Perf

I have run gr_read benchmarks to test perf increase:

perf record --all-user --call-graph=dwarf --freq 997 ./target/release/gear-node benchmark pallet --chain=dev --steps=50 --repeat=100 --pallet=pallet_gear --execution=native --extrinsic="gr_read"

gear_core_processor::executor::prepare_memory (where lazy pages init info is called) takes

master       :    24.9% from all samples
this branch  :    0.01% from all samples

time for gr_read:

master       :    42.93 µs
this branch  :    32.65 µs

time for gr_reply_commit:

master       :    68.2 µs
this branch  :    49.92 µs

Testing

Run node with this patch over all test-net chain - all OK

Important

This patch requires node binary update after runtime update.

Problem to Solve

Sometimes we need to send the message after some delay instead of doing it immediately. It might be useful for time scheduled routines, e.g. in DAO, auctions, lotteries.

Possible Solution

Introduce new functions in gstd::msg (all names, parameters, and return types are to be discussed!):

/// `msg` module

/// Add the message to the shedule.
fn schedule(dest, payload, delay, value) -> Result<MessageId, ...>;
// fn schedule_with_gas(...)
// async fn schedule_for_reply(...)

/// Deny early sending.
fn deny_triggering(msg_id) -> bool;

/// Cancel sending scheduled message.
fn cancel_scheduled(msg_id) -> Result<(), ...>;

/// Send scheduled message immediately.
fn trigger(msg_id) -> Result<(), ...>;
// fn trigger_for_reply(...)

Also need to add correspondent functionality to Gear core.

Notes

Using example (DAO):

let msg_id = msg::schedule(
    exec::program_id(),
    DaoAction::ProcessProposal(self.proposal_id)),
    exec::block_timestamp() + self.voting_period,
    0,
)
.expect("Error while deferring message");
msg::deny_triggering(msg_id);

let reply: DaoEvent = 
    exec::trigger_as(msg_id).expect("Error while shoving scheduled message");

exec::cancel_scheduled(msg_id).expect("Error while cancelling scheduled message");

let reply: DaoEvent = msg::schedule_for_reply_as(
    exec::program_id(),
    DaoAction::ProcessProposal(self.proposal_id),
    exec::block_timestamp() + self.voting_period,
    0,
)
.expect("Error while scheduling message")
.await
.expect("Error while processing result");

File Location(s)

core-processor/src/executor.rs

Proposal

We have the following loop at the moment:

for (page, new_data) in info.pages_data {
    // ...

    if let Some(initial_data) = pages_data.get(&page) {
            if new_data != *initial_data {
                // ...
            }
        // ...
    }
}

Instead the pages could be marked as dirty during the execution it there were some changes to them. To sum up the beforementioned loop should look like this:

for (page, new_data) in info.pages_data {
    // ...

    if let Some(initial_data) = pages_data.get(&page) {
            if new_data.modified {
                // ...
            }
        // ...
    }
}

Resolves #18. Resolves #515. Resolves #579.

• Split examples/init-wait/lib.rs into two files: one with crate-level no_std attribute and the other with the program code. This trick allows to include code.rs in other modules/crates.
• Add tests/init-wait that includes code.rs and exports WASM_BINARY_BLOATY.
• Add three tests to pallet-gear/tests. They are ignored at the moment since node doesn't support calling wait ini init method.

@gear-tech/dev

Release Notes: Programs lifecycle has been changed so now their initialization can be deferred with wait/wake syscalls. This is the next step towards the asynchronous initialization of programs.

File Location(s)

https://github.com/gear-tech/gear/blob/9bd56a0aa17931985c80ec3fa2377518eacc25bb/pallets/gear/src/lib.rs#L576

Proposal

We can just get the max gas spent via JournalNote::GasBurned { amount, .. }

in case of fixing the node key issue in https://github.com/gear-tech/gear/pull/975, we can not get remaining_gas via T::GasHandler::get_limit(message_id) after the message has been consumed ( in some cases it would be None )

Finally resolves #587.

• Add functions (aka syscalls) blacklist.
• Generate a trap when trying to call a forbidden function in a program.
• Forbid to call gr_gas_available when executing WASM from get_gas_spent function.

TODO:

• [x] Fix tests

Use case

Now it is possible to pass a list of forbidden syscalls when calling the gear_core_processor::process() function. If the WASM program contains a call to that forbidden function, the execution will return a correspondent error.

For example when we calculate the amount of gas needed for processing messages in the get_gas_spent() function, we forbid calling the gr_gas_available syscall in the program. get_gas_spent() will return Program terminated with a trap: Unable to call a forbidden function error if the program calls exec:gas_available() function during message processing.

Up to now, the main end-user application is that he will receive an error when trying to calculate the gas amount to be spent for handling a message by the program that uses the exec::gas_available() function inside.

Also, there are potential use cases in the future.

@gear-tech/dev

Problem

gtest/spec/test_vec.yaml "test-vec (20000 * size_of(i32) = 80 KiB = 2 pages)"

Hello, the test creates vec which have size 0x13880 bytes (page is 0x10000). But as I can see in my allocator logs in the end of fn handle - destructor for vector is called, so the check for two pages is incorrect.

Problem

First I found this problem when run gtest for compiled with debug_assertions wasm tests - some of them failed on execution.

$ sh scripts/test.sh async 

Fixture async-await (step: 1): Ok
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 2): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 3): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 4): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 5): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 6): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 7): Messages check [expected: 1, actual: 2]
DEBUG: 1 total message(s) stored: 
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 8): Messages check [expected: 1, actual: 3]
DEBUG: 1 total message(s) stored: 
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 9): Messages check [expected: 1, actual: 3]
DEBUG: 1 total message(s) stored: 
DEBUG: 1 total message(s) stored: 
Fixture async-await (step: 10): Log check [expected: 1, actual: 0]
Error: 9 tests failed

Investigating problem I found that debug_assertion is not a reason and something fails tests in checks code. So I tried to localize problem - remove all unneeded checks and unneeded code. Only one check function is left and inside only one small loop is left:

check_malloc_state(&mut self) {
        if !DL_CHECKS { return; }
        for i in 0..11 {
            let head_chunk : *mut Chunk = self.smallbin_at(i as u32);
        }
}

#[inline(never)]
 unsafe fn smallbin_at(&mut self, idx: u32) -> *mut Chunk {
        let idx = (idx * 2) as usize;
        dlassert!(idx < self.smallbins.len());

        let smallbins_ptr = &self.smallbins as *const *mut Chunk;
        let idx_ptr = smallbins_ptr.offset(idx as isize ) as *mut Chunk;
        return idx_ptr;
}

Here we just take few times mutable pointer and do nothing else. This function is called sometimes in allocator.

So, if DL_CHECKS is true then gtest is failed, else all goes right. We can also tries to make function smallbin_at as inline - in that case tests goes well. If we change iterations number in loop, for example 0..10, then tests also goes well.

I noticed that if iterations number >= 11 then we fails in other case goes well: 20..25 and 20..30 - goes well; 20..31 - fails

Steps

Linux kernel: 5.4.0-86-generic OS: 20.04.3 LTS (Focal Fossa) Proc: x86_64

1. git fetch && git checkout gsobol_alloc_fail
2. sh scripts/test.sh async // this must fail
3. remove in galloc/Cargo.toml feature "gsobol_check" which is for dlallocator
4. sh scripts/test.sh async // this runs well

Possible Solution

I think that problem may be in wasmtime compiler or when we compile from rust to wasm code. No other ideas.

Notes

No response

Relevant Log Output

No response

Problem to Solve

gprogram new generates template from https://github.com/gear-tech/gear/blob/master/program/src/template.rs without tests, need to add tests for it.

Possible Solution

• add tests for example ping
• generate template from ping with build script

Notes

No response

Problem to Solve

Sometimes it is helpful to have some not initial state of the program. E.g., you can perform some long during operation once and use its results in tests multiple times.

Possible Solution

Add to gtest/gclient:

• Function for saving the whole program's memory to file: save_memory(program_id, file_path)
• Function for restoring the program's memory from a file: restore_memory(program_id, file_path)

Also:

• Consider using a text format for the file to keep it in the version control system.
• Don't keep empty memory blocks.
• File format should be interchangeable between gtest and gclient.

Notes

No response

Resolves the issue with a flaky machinery of deciding wether the message queue should be processed in the end of each block depending on the outcome of previous runs.

The old mechanism relied on the inherent-like extrinsic signalling a successful completion of the message queue processing by setting some storage value to a specific status. If not set, this was deemed as a panic having occurred during one of the previous runs so that a new attempt wouldn't (by default) take place. This creates an attack vector of a malicious validator opting for deliberately not including this inherent into a block thereby blocking the queue processing in all the blocks thereafter, until the issue would have been investigated and fixed via a runtime upgrade which would render the network useless in the meantime.

The proposed solution changes this approach so that the inherent-like extrinsic is always included in a block; however, there is another storage value - ExecuteInherent, that is checked inside this extrinsic: if set to false, an error is returned thereby causing the block builder to drop this extrinsic. This value can only be altered by a root origin. This is useful to manually disable queue processing in case it panics and re-enable once the issue has been fixed.

Resolves #2050

Add gear stack end to wasm-gen to test all cases depending on this

~1% of cases with invalid stack size not a multiple of the page size ~1% of cases with invalid stack size that is bigger than import memory ~1% of cases stack size is not generated at all all other cases should be valid