Diem’s mission is to build a trusted and innovative financial network that empowers people and businesses around the world.

Bumps ini from 1.3.5 to 1.3.8.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I've discovered an alarming vulnerability, but fortunately there's a really simple fix so I've sent a pull request to address it.

In the current implementation, trusted 'validator nodes' are core to the security model. This means that hard power is centralised around these few entities. The protocol itself depends on these entities to (as the name suggests) validate the protocol. This means the protocol is whatever they decide it is. These entities can change the rules whenever they want. This means they can freeze your coins, take your coins, issue new coins, or really whatever they want - the sky is the limit.

This problem can easily be solved by using a permissionless system where the hard power is decentralised across a very large number of participants in such a way that making changes to the protocol is impossible without near unanimous agreement by everyone involved.

This pull request contains a patch to the existing codebase to resolve this issue.

Edit: related issue: https://github.com/binance-chain/node-binary/issues/36

Summary

• We generate load on the cluster and pick one validator and run perf against it to generate the FlameGraph
• This FlameGraph is then copied locally to cluster-test.
• The next step would be to upload this local file to S3

Test Plan

Tested on my cluster

Temp change to kube.py for testing- will revert back

Motivation

Migrate forge to new cloud- using this pr to test the backend. Currently set up to run in parallel to the jobs done on Novi servers.

Upgraded the following: regex, zeroize, nix, thread_local, tokio

Compiled with no issues (on linux vm- can't compile on M1 mac until rocksdb is updated per https://github.com/diem/diem/pull/10295) and fixes their respective RUSTSEC vulnerabilities thrown by cargo audit

Left out time crate because it is only used as a sub-dependency of chrono, which is already updated to latest version and seems to have no plans to upgrade the crate. However, per https://github.com/chronotope/chrono/pull/578#issuecomment-1075907329, it seems chrono is unaffected by the vulnerability.

Also left out scratchpad because it is a custom diem crate which cargo audit confuses with a different crate with the same name

Motivation

(Write your motivation for proposed changes here.)

Have you read the Contributing Guidelines on pull requests?

(Write your answer here.)

Test Plan

(Share your test plan here. If you changed code, please provide us with clear instructions for verifying that your changes work.)

Related PRs

(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/diem/diem/tree/main/developers.diem.com, and link to your PR here.)

If targeting a release branch, please fill the below out as well

• Justification and breaking nature (who does it affect? validators, full nodes, tooling, operators, AOS, etc.)
• Comprehensive test results that demonstrate the fix working and not breaking existing workflows.
• Why we must have it for V1 launch.
• What workarounds and alternative we have if we do not push the PR.

Bumps ssri from 6.0.1 to 6.0.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Motivation

(Write your motivation for proposed changes here.)

Have you read the Contributing Guidelines on pull requests?

(Write your answer here.)

Test Plan

(Share your test plan here. If you changed code, please provide us with clear instructions for verifying that your changes work.)

Related PRs

(If this PR adds or changes functionality, please take some time to update or suggest changes to the docs at https://developers.diem.com, and link to your PR here.)

If targeting a release branch, please fill the below out as well

• Justification and breaking nature (who does it affect? validators, full nodes, tooling, operators, AOS, etc.)
• Comprehensive test results that demonstrate the fix working and not breaking existing workflows.
• Why we must have it for V1 launch.
• What workarounds and alternative we have if we do not push the PR.

Here are the changes to the binary format. We have few things missing around Scripts vs Modules and around functions (native function flags and function signatures). However this works and it's the bulk of the changes. There is more to clean up and correct but this needs to go in asap and we can all iterate over

This PR implements phantom type parameters in the bytecode. A type argument instantiating a phantom type parameter is not considered when deriving the abilities for the struct instantiation. For this to be sound, in a struct declaration a phantom type parameter can only appear in phantom position (or not appear at all), i.e., as an argument to a phantom type parameter.

Motivation

Avoid spurious ability annotations for marker types like XUS.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

Tests will be coming in the next PR.

Renames all occurrences of "libra" with "diem" (with casing preserved). Renames all occurrences of "lbr" with "XDX" (with casing preserved). Renames all occurences of "coin1" to "XUS" (with casing preserved).

This PR is meant to be the bulk renaming, there will need to be smaller cleanup passes that we will want to do afterwards, but that can be done by folks in each area.

Motivation

Today, state sync is responsible for notifying mempool whenever new transactions are committed. However, the interface between the two components is very messy, e.g., the code leaks implementation details between the components and forces a very tight coupling. This makes it difficult to update (or swap out) component implementations.

To address this, the PR introduces an explicit interface and set of components between state sync and mempool:

MempoolNotificationSender is the interface allowing state sync to send notifications to mempool (MempoolNotifier is the component that implements this interface). MempoolNotificationListener is the component that mempool uses to listen for notifications and respond accordingly. This structure avoids leaking implementation details (e.g., the fact that we use a one-shot callback to respond to notifications). In addition, the PR adds explicit (previously missing) tests for the implementation.

Note: at a fundamental level, this new interface implementation still uses channels, but wraps the interface-specific details of the communication (e.g., message formats, channel timeouts, etc.) to offer better implementation encapsulation.

Test Plan

• CI/CD testcases were covered.

[State Sync] Small cleanups to MempoolNotifier. #9047=

Motivation

This (mini) PR offers two small cleanups for the newly added MempoolNotifier (https://github.com/diem/diem/pull/9006), each in their own commit:

First, remove the redundant Result wrapper when the MempoolNotificationListener acks a commit notification. The result is not used. Second, it removes the need for mutability when methods of the MempoolNotifierSender are invoked. Mutability is also not required. Have you read the Contributing Guidelines on pull requests? Yes.

Test Plan

• CI/CD testcases were covered.

[State Sync] Add ConsensusNotifier interface and implementation. #9057

Motivation

Today, consensus is responsible for notifying state sync of two events: (i) when a set of transactions has just been committed; and (ii) when state sync should synchronize to a specified target. However, the interface between these components is very messy, leaks implementation details and is not thoroughly tested. As a result, it becomes difficult to update (or swap out) implementations.

To address this, the PR introduces an explicit interface and set of components between consensus and state sync:

ConsensusNotificationSender is the interface allowing consensus to send notifications to state sync (ConsesusNotifier is the component that implements this interface). ConsensusNotificationListener is the component that state sync uses to listen for notifications and respond accordingly. This structure avoids leaking implementation details. In addition, the PR adds explicit (previously missing) tests for the implementation. To achieve this, the PR offers the following commits:

First, we add the new ConsesusNotifier, ConsensusNotificationSender and ConsensusNotificationListener components and interfaces (alongside tests for these). Second, we update consensus to use the ConsensusNotifier (instead of the current StateSyncClient) to notify state sync. Third, we update state sync to use the ConsensusNotificationListener directly and update the StateSyncClient to remove support for the old consensus APIs. Fourth, we update Diem node to wire consensus and state sync together using the ConsensusNotifier. Finally, we remove the (newly) redundant code from mempool. Note: This PR takes a very similar approach to: https://github.com/diem/diem/pull/9006

Have you read the Contributing Guidelines on pull requests? Yes.

Test Plan

• cargo test --package state-sync-v1 --tests

[forge] refactory Forge CLI interface #9061

Forge Cli is increasingly complicate since it supports more and more operations. Refactory CLI to different operation categories for better user experience.

Test Plan

For cluster Opertions:

• cargo run -p forge-cli -- operator clean-up
• cargo run -p forge-cli -- operator resize
• cargo run -p forge-cli -- operator set-validator
• for Testing:

cargo run -p forge-cli -- test local-swarm

• ./scripts/fgi/run --local-swarm
• ./script/fgi/run --pr xxx
• ./scripts/fgi/run -T dev_czh_pull_9061 --suite land_blocking_compat

smoke-test: finish conversion to using local forge backend #9045

This PR finishes out the conversion of smoke-tests from using the old DiemSwarm to using the local forge backend as well as removing the now unused DiemSwarm infrastructure.

Test Plan

• CI/CD tests are covered

[CI] update the min_tps to 1100 to prevent regressions #9063

• CI/CD tests are covered

Bumps archlinux from base-devel-20210725.0.29770 to base-devel-20230101.0.115167.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

[backup] replay-verify CLI tool #9012

Motivation

For use by VM compatilibity tests. Restores proper state snapshot, on top of which replays desired range of transactions, to comfirm compatibilities with the latest software.

Test Plan

CI/CD test covers the test cases.

[diem_vm] Implemented a standalone parallel transaction executor #8923

Motivation

This PR aims to implement a standalone version of parallel executor that we will use in the diem network. The newly introduced parallel_executor crate will serve as the abstract scheduler and executor for parallel and have zero dependencies to diem/move components. The structure of the package will look like following:

• src/task.rs This is the file that defines the type of transaction that could be parallelly processed by this executor and the single threaded executor for running such transaction.

• src/error.rs Defines the error and result type of parallel execution.

• src/scheduler.rs The naive scheduler that tracks transaction dependencies and task dispatcher for all threads.

• src/executor.rs The actual orchestration of the computation: spawn threads to parallel process transactions that are passed in.

• src/proptest_types Defines the proptest based testing and bencher strategy

  • [ ] types.rs Defines a naive transaction type that implements all traits required by parallel executor and its sequential computation baseline.
  • [ ] tests.rs Defines a number of proptest to test the correctness of the scheduling algorithm
  • [ ] bench.rs Generates random transactions to be executed and test the throughput of the system under trivial payloads. Executing each transaction could yield one of the following results:

1. Success: The transaction finished its execution with output.
2. SkipRest: The transaction finished its execution with output, but the executor will drop the execution result of any transaction that comes after it.
3. Abort: The transaction execution failed with non recoverable error. The executor will need to propagate such error to its caller
4. Retry: The transaction reads from a key that is not yet computed. The executor will need to re-execute this transaction when that key is assigned.

Note that for either SkipRest or Abort state, the executor guarantees to return the first transaction in the list that causes such status, regardless of the scheduling order.

Test Plan

A proptest strategy is implemented for the executor. The baic setup is following:

We randomly generate 100 distinct keys as a key universe. Then we randomly generate 3000 transactions where each of the transaction will randomly pick 10 keys they are estimated to write to, and will actually write to a half of those keys. Each transaction will also read a few keys and test if the value they see is the same as the sequential execution.

Based on the basic setup, we then randomly tell some of those transactions to return SkipRest or Abort signal to force the early termination of the algorithm. We then check if the termination is indeed at the right index (only the first of those transactions should matter for the scheduler).

We also tested the scenario when read set estimation is imprecise: the read inferencer will randomly drop a key that a transaction will read from, and make sure the system won't halt in such scenario.

Other than the proptest, we also implemented a bencher to test the system's raw throughput on those naive transaction, here's the result:

random_benches time: [73.161 ms 74.892 ms 76.656 ms] The number reflects the time spent on processing 10k transactions generated in the previous setup described.

[transactional-tests] More migrated tests #9042

Motivation

• Migrated linker_tests/, method_decorators/, module_member_types/, modules/, mutate_tests/, and mutation/
• Most tests were bytecode verifier tests, with a few VM tests
• Nearly done migrating bytecode verifier/VM related IR tests to transactional tests.

Test Plan

CI/CD testcases covered.

[move-prover] minor improvements to the verification analysis pass #9053

Motivation

Two improvements are made in the new verification analysis pass:

• [move-prover] only verify functions that directly modifies a target invariant the prior behavior is to verify functions that directly accesses a target invariant, which unnecessarily leaves many functions to be marked as verified. NOTE: only memory modifications cause global invariants to be asserted; memory accesses without modification leads to assumption of global invariants only, but not assertions.
• [move-prover] improve error reporting on invariant suspension pramgas. The prior errors are too verbose.

Test Plan

• CI/CD
• This new pass is not turned on as of now. It will be made as default in a later PR.

[move-prover] algorithm for progressive instantiation #9056

Motivation

This algorithm deals with finding a complete set of instantiation combinations for all type parameters when unifying two types.

// problem definition The algorithm is encapsulated in struct TypeInstantiationDerivation and is not conceptually hard to understand.

// algorithm description The algorithm works by finding all instantiations for X0 first, and then progress to X1, X2, ..., until finishing Xn.

// other notes

• The implementation has a bit of fine-tuning rooted by the fact that sometimes we want to treat a type parameter as a variable (i.e., participate in type unification) while in other cases, we want to treat a type parameter as a concrete type (i.e., do not participate in type unification).
• We also have a fine-tuning on whether we treat a type parameter that does not have any valid instantiations as an error or remains as a concrete type parameter. This is rooted by the differentation of type parameters in function vs type parameters in a global invariant. Essentially, all type parameters in a global invariant must be instantiated in order for the invariant to be instrumented. But not all function type paramters need to be instantiated.
• This is not the most efficient algorithm, especially when we have a large number of type parameters. But a vast majority of Move code we have seen so far have at most one type parameter, so in this commit, we trade-off efficiency with simplicity.

Test Plan

CI/CD testcases were covered.

[move-prover] dump bytecode and result in output dir #9052

Motivation

Previous dumping location is at the first Move source location, which may pollute the source directories. Furthermore, if we pass a directory as the first source location, the process will panic.

This commit changes the default output location for --dump-bytecode to the parent directory of the output.bpl file, and format the bytecode dumps with bytecode_{step_number}_{step_name}.bytecode.

Test Plan

• CI
• cargo run -p move-prover -- <dir-name>/<file-name>.move --dump-bytecode. Note that the bytecode dumps are generated at the current work directory instead of under <dir-name>.

[move prover] Arithmetic mutations #8980

Motivation

This change updates two components of the mutation tester. First, it adds the sub-add, mul-div, and div-mul mutation operators. Second, it provides a fix to needing to provide addresses when working on the Diem framework.

Test Plan

CI/CD testcases were covered.

[move-prover] an analysis pass for the global invariants #9072

Motivation

This pass collects information on how to inject global invariants into the bytecode of a function. The end result of this analysis is summarized in two structs:

/// A named struct for holding the information on how an invariant is
/// relevant to a bytecode location.
struct PerBytecodeRelevance {
    /// for each `inst_fun` (instantiation of function type parameters) in the key set, the
    /// associated value is a set of `inst_inv` (instantiation of invariant type parameters) that
    /// are applicable to the concrete function instance `F<inst_fun>`.
    insts: BTreeMap<Vec<Type>, BTreeSet<Vec<Type>>>,
}

/// A named struct for holding the information on how invariants are relevant to a function.
struct PerFunctionRelevance {
    /// Invariants that needs to be assumed at function entrypoint
    /// - Key: global invariants that needs to be assumed before the first instruction,
    /// - Value: the instantiation information per each related invariant.
    entrypoint_assumptions: BTreeMap<GlobalId, PerBytecodeRelevance>,

    /// For each bytecode at given code offset, the associated value is a map of
    /// - Key: global invariants that needs to be asserted after the bytecode instruction and
    /// - Value: the instantiation information per each related invariant.
    per_bytecode_assertions: BTreeMap<CodeOffset, BTreeMap<GlobalId, PerBytecodeRelevance>>,
}

A note about PerBytecodeRelevance : in fact, in this phase, we don't intend to instantiation the function nor do we want to collect information on how this function (or this bytecode) needs to be instantiated. All we care is how the invariant should be instantiated in order to be instrumented at this code point, with a generic function (generic bytecode).

But unfortunately, based on how the type unification logic is written now, this two-step instantiation is needed in order to find all possible instantiations of the invariant. I won't deny that there might be a way to collect invariant instantiation combinations without instantiating the function type parameters, but I haven't iron out one so far.

Test Plan

CI/CD testcases were covered.

Add support for CRSNs #8528

Motivation

Bottom Commit

Implements CRSNs and the CRSN logic according to DIP-168. These changes have been approved in #8403.

Middle Commit

Adds support for CRSNs to mempool. In particular, updates needed to be performed in order to support:

• Processing of transactions in a non-blocking manner
• Eviction of transactions based on the LHS of the sliding nonce of the account, and not on the committed transactions sequence nonce (since this method does not work in the CRSN case).

I'm not that familiar with mempool, so there might be something I've forgotten that I need to change -- please let me know if that's the case. Also feel free to add reviewers if I've forgotten to add someone.

The general methodology for implementation here is:

• When a transaction hits mempool we determine if it is a CRSN or seqno transaction
• Validation will return both the LHS of the CRSN window and k
• Transactions will be removed from mempool if the transaction's seqnonce < the current LHS when a new transaction is added
• A transaction will be accepted to mempool if the transaction's seqnonce >= LHS
• A transaction will be considered for processing as long as it is >= LHS and < LHS + k
• On commit of a transaction, other transactions may be evicted from mempool based off of the LHS of the account at the time the transaction entered mempool. This adds a number of tests to verify the expected behavior for transactions with CRSNs.

Top Commit

Adds support for CRSNs to the prologue/epilogue.

Test Plan

CI/CD tetscases were covered.

[move] type layout generation from modules #9073

Motivation

This is the clone of #8968 just because I don't know how to commandar that PR.

Add code in the move-binary-format crate to create a MoveTypeLayout from a CompiledModule In the cli, add new generate struct-layout command that leverages this functionality to dump layouts in YAML format The eventual goal is to have the CLI spit out YAML that can be used directly by serde-generate to create typed struct bindings in any language supported by serde-generate Added the error propagation comparing to ([move] type layout generation from modules #8968).

Test Plan

CI/CD testcases were covered.

[diem-framework] Port DiemTimestamp to unit tests #9079

Motivation

Ports the tests for DiemTimestamp to use unit tests. Adds additional tests for full coverage of the module.

TestPlan

CI/CD testcases were covered.

[move-prover] Rewrite the access control spec as two state invariants #9025

Motivation

This PR defines is_txn_signer, and replaced is_signer with it because the previous implementation of is_signer didn't fit for purpose (see the issue [Bug] issue with is_signer #9018).

The existing access control spec uses the schema application which is verbose. To simplify the access control spec and make it easier to read, this PR rewrites the existing access control spec as two state invariants with the new spec construct is_txn_signer_addr (and is_txn_signer).

This PR is mostly about cleaning-up the spec for the role-based access control in the Diem Framework. The next step is to look into the capability-based one (e.g., MintCapability, ...)

Test Plan

cargo test

[move-prover] an instrumentation pass for the global invariants #9077

Motivation

An instrumentation pass that consumes the information produced by the global invariant analysis pass and instrument the global invariants into the function.

The instrumenter supports two mode of operations, depending on whether the prover backend supports monomorphization or not:

• With the option --boogie-poly, the instrumenter will instrument instantiated invariants in the generic function (and the generic instance is the only function instance).
• Without the --boogie-poly option, the instrumenter will instrument instantiated invariants per each instantiation of a generic function (this is the traditional workflow). And this means that a function will have multiple instances for verification.

This is not exactly the plan we had before (and does not clearly adhere to the paper). The original plan is to go for option 1 first and defer the instantiation of functions to the mono pass. Therefore, the option 2 here is essentially a combination of

• instrumentation,
• monomorphization, and
• optimization (eliminating redundant expressions)

But option 2 does not completely solve the "type-dependent" code problem because the (move_to<T>, move_to<u64>) case still requires a second step of function instantiation and still requires the mono pass to perform such instantiation.

The main reason why we still have option 2 (and not only that, we made option 2 as default as of now) is three-fold: 1.I am uncertain of Boogie's monomorphization implementation matches the complexity of what we have done here. 2.I want to get at least the whole Diem Framework verifying again to test out the whole transformation pipeline, in order to boost our confidence. 3.Solving the move_to<T>; move_to<u64>; problem requires more than instantiation; we need spec language support as well to express the fact that the function will surely abort if T == u64 and may not abort otherwise.

With this final piece, we complete the new invariant instrumentation pipeline (modulus the misaligned implementation plan mentioned above) and the next PR will switch the pipeline into the new one and fix the specs in the Diem Framework.

Test Plan

CI/CD tetscases were covered.

[transactional-tests] Migrate remaining VM/Bytecode verifier IR tests #9078

Motivation

• migrated remaining tests
• tests left over are either for the DF or the IR compiler itself
• Check for tokens after module or script in IR. Caused silent test failures

Test Plan

CI/CD testcases were covered.