Bug report

Expected behavior

The project should build on Mac

Actual behavior

The project fails to build

Steps to reproduce

Clone repository. Run 'cargo build'

Crash log? Screenshots? Videos? Sample project?

error: suffixes on a tuple index are invalid
  --> primitives/block/src/target.rs:12:64
   |
12 | #[derive(Default, Clone, Copy, PartialEq, PartialOrd, Eq, Ord, Serialize, Deserialize)]
   |                                                                ^^^^^^^^^ invalid suffix `i32`

error: proc-macro derive produced unparseable tokens
  --> primitives/block/src/target.rs:12:64
   |
12 | #[derive(Default, Clone, Copy, PartialEq, PartialOrd, Eq, Ord, Serialize, Deserialize)]
   |                                                                ^^^^^^^^^

error: aborting due to 2 previous errors

error: Could not compile `nimiq-block`.
warning: build failed, waiting for other jobs to finish...
error: build failed

New issue checklist

• [x] I have read all of the README
• [x] I have searched existing issues and this is not a duplicate.

Tried to build core-rs from source by cloning from Github. I switched to nightly build with rustup default nightly. As soon as I started building with cargo +nightly build the compiling stopped with the following error: error: linking with `cc` failed: exit code: 1

The README of core-rs states that besides rust nightly you need gcc, pkg-config and libssl-dev, which I installed through apt.

Solution

Install gcc-multilib as well and it started compiling with no issues.

Currently, it is very cumbersome to use/debug only parts of the implementation. The main reason for that is almost all code residing in the main project source.

I propose to make use of more sub-crates (similar to parity-bitcoin). This also helps cleaning up unwanted dependencies. Currently, for example, our consensus module has dependencies on the network module. It also speeds up compilation and testing, when only some parts were changed. Finally, it allows us to compile relevant parts of it to WebAssembly.

As there are many ways to divide the current code into sub-crates, this issue is meant for discussion about how to structure those.

My proposal would be something like:

• db (database layer) ✅
• hash (hashes, non-key related cryptography) ✅
• keys (private keys, signatures, address,...) ✅
• primitives (Target, Coin, Compact, everything related to blocks and transactions, contracts and basic accounts,...) ✅
• datastructures (Accounts, Blockchain, ChainStore and other concrete db stores) ✅
• consensus (Consensus, ConsensusAgent, Mempool) ✅
• mnemonic (mnemonics) ✅
• key-derivation (key derivation) ✅
• network (all networking) ✅
• network-primitives (primitives used in multiple locations) ✅
• core (putting things together, but no binary) (currently in main folder ✅)
• rpc (JSON-RPC) (to be merged)
• utils (most utils, configurable via feature flags) ✅
• Separate binary crate ✅ (by @jgraef)

Note: I just quickly wrote down this as a starting point. Looking forward to your ideas. :) Those that I already created are marked with ✅. Those that are currently in progress are marked with ☑️.

Pull request checklist

• [X] All tests pass. Demo project builds and runs.
• [X] I have resolved any merge conflicts.

What's in this pull request?

On block 488308 a fork happened that was resolved at block 488309. After rebranching a transaction was pushed to the mempool which crashed the node. Somehow the sender balance was out of funds. It is possible for a transaction to be verified, but later when all transactions for the sender address are being rechecked, that the sender is out of funds. It is unclear if this caused the crash. The log looks like the transaction was received after the block was processed, but maybe the other transactions with the same sender address weren't removed from the mempool yet. I will check this later. I fixed this, by rejecting a transaction during rechecking if the sender is out of funds, as this should be the intended behavior. Another possible fix is to synchronize the whole Mempool::push_transaction over the Blockchain or Accounts.

After the crash the node didn't restart, because the rebranching code didn't update the head hash in the ChainStore, which caused the head's accounts tree hash to be incorrect. I fixed this by updating the ChainStore's head hash before commiting the rebranching transaction.

• Branch terorie/albatross devnet
• Sometimes, when crate validator produces a block, it gets rejected by crate blockchain
• Failing check:

if !intended_slot_owner.verify(&micro_block.header, &justification) {
                warn!("Rejecting block - invalid justification for intended slot owner");

• Log:

 INFO nimiq_validator::validator                 > Produced block: block_number=73529, view_number=0, hash=73e7f9e...                                                                                                                   
 INFO nimiq_consensus::consensus                 > Now at block #73529                                                                                         
 TRACE nimiq_validator::validator                 > Next block producer: CompressedPublicKey(ae88f18...)
 TRACE nimiq_validator::validator                 > Push result: Ok(Extended)                                                                                  
 INFO nimiq_validator::validator                 > Produced block: block_number=73530, view_number=0, hash=186435...
 WARN nimiq_blockchain_albatross::blockchain     > Rejecting block - invalid justification for intended slot owner                                             
 DEBUG nimiq_blockchain_albatross::blockchain     > Intended slot owner: CompressedPublicKey(ae88f18...)
 ERROR nimiq_validator::validator                 > Failed to push produced micro block to blockchain: InvalidBlock(InvalidJustification)                                                                                                                                                     

I ran into this multiple times, but never got the time to fix it. So here is the issue open for anyone to fix. Or I will fix it, once I got some spare time.

error[E0277]: `keys::AddressParseError` doesn't implement `std::fmt::Display`
  --> src/config.rs:52:53
   |
52 |     Address::from_user_friendly_address(&s).map_err(Error::custom)
   |                                                     ^^^^^^^^^^^^^ `keys::AddressParseError` cannot be formatted with the default formatter
   |
   = help: the trait `std::fmt::Display` is not implemented for `keys::AddressParseError`
   = note: in format strings you may be able to use `{:?}` (or {:#?} for pretty-print) instead
   = note: required by `config::_IMPL_DESERIALIZE_FOR_GenesisConfig::_serde::de::Error::custom`

Since keys::AddressParseError is an error, we should also implement the necessary traits.

error[E0554]: #![feature] may not be used on the stable release channel --> collections/src/lib.rs:2:1 | 2 | #![feature(box_into_raw_non_null)] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0554]: #![feature] may not be used on the stable release channel --> collections/src/lib.rs:3:1 | 3 | #![feature(specialization)] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0554]: #![feature] may not be used on the stable release channel --> collections/src/lib.rs:4:1 | 4 | #![feature(map_get_key_value)] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Compiling indexmap v1.0.2 error: aborting due to 3 previous errors

For more information about this error, try rustc --explain E0554. error: Could not compile nimiq-collections. warning: build failed, waiting for other jobs to finish... error: build failed

The example config file includes many "available settings" that are not actually available, such as the dumb mode, light and nano consensus etc.

These should be removed from the example config file for release.

Currently, only producers of micro blocks during the latest two epochs are known. The slot data should be included in ChainInfo to persist the producer info of historic blocks.

Currently, we close our WebSocket connections by:

1. Having a select between a one-shot channel and the connection processing future.
2. Sending a close event via the one-shot channel and thus terminating the connection processing future.
3. Sending the WebSocket close frame after having terminated the connection processing future.

https://github.com/nimiq/core-rs/blob/52906bda31a837966964d416b8c9c0436b874248/network/src/connection/network_connection.rs#L131-L159

With the most recent version of tungstenite and tokio-tungstenite there might be a nicer way to do it:

• https://github.com/snapview/tokio-tungstenite/pull/53
• https://github.com/snapview/tungstenite-rs/pull/56

I only skimmed over the two pull requests, but it seems that it is now possible to send the close frame via the Sink and shut down the Stream part when receiving the other party's close frame.

In primitives/src/coin.rs there is a comment in the Deserialize implementation which says:

Check that the value does not exceed Javascript's Number.MAX_SAFE_INTEGER.

Shouldn't we check that then on serialization too? Serialization is currently auto-derived and thus just serializes the u64 even if it's greater than Javascripts MAX_SAFE_INTEGER.

Also the checked_add should check this.

Pull request checklist

• (n/a) All tests pass. Demo project builds and runs.
• [x] I have resolved any merge conflicts.

What's in this pull request?

Adds that one needs 2GB of RAM to build. I tried to build this with 1GB of RAM, and it didn't work until I went up to 2GB.

New issue checklist

• [ ] I have read all of the README
• [ ] I have searched existing issues and this is not a duplicate.

General information

• Library version(s): 0.2
• Browser version(s): N/A
• Devices/Simulators/Machine affected: Linux aarch64
• Reproducible in the testnet? (Yes/No): N/A
• Related issues:

Bug report

Expected behavior

Build doesn't fail

Actual behavior

Build fails

Steps to reproduce

cargo +nightly build --release

Crash log? Screenshots? Videos? Sample project?

error: failed to run custom build command for `libargon2-sys v0.2.0 (/home/ubuntu/nimiq/libargon2-sys)`

Caused by:
  process didn't exit successfully: `/home/ubuntu/nimiq/target/release/build/libargon2-sys-66ab85cb0894a5ef/build-script-build` (exit code: 1)
--- stdout
TARGET = Some("aarch64-unknown-linux-gnu")
HOST = Some("aarch64-unknown-linux-gnu")
CC_aarch64-unknown-linux-gnu = None
CC_aarch64_unknown_linux_gnu = None
HOST_CC = None
CC = None
CFLAGS_aarch64-unknown-linux-gnu = None
CFLAGS_aarch64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
CARGO_CFG_TARGET_FEATURE = Some("fp,neon")
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "native" "-Wall" "-Wextra" "-DARGON2_NO_THREADS" "-o" "/home/ubuntu/nimiq/target/release/build/libargon2-sys-1a40ba79533982af/out/native/argon2.o" "-c" "native/argon2.c"
exit code: 0
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "native" "-Wall" "-Wextra" "-DARGON2_NO_THREADS" "-o" "/home/ubuntu/nimiq/target/release/build/libargon2-sys-1a40ba79533982af/out/native/core.o" "-c" "native/core.c"
exit code: 0
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "native" "-Wall" "-Wextra" "-DARGON2_NO_THREADS" "-o" "/home/ubuntu/nimiq/target/release/build/libargon2-sys-1a40ba79533982af/out/native/blake2/blake2b.o" "-c" "native/blake2/blake2b.c"
exit code: 0
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "native" "-Wall" "-Wextra" "-DARGON2_NO_THREADS" "-o" "/home/ubuntu/nimiq/target/release/build/libargon2-sys-1a40ba79533982af/out/native/opt.o" "-c" "native/opt.c"
cargo:warning=In file included from native/opt.c:26:0:
cargo:warning=native/blake2/blamka-round-opt.h:23:10: fatal error: emmintrin.h: No such file or directory
cargo:warning= #include <emmintrin.h>
cargo:warning=          ^~~~~~~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

--- stderr


error occurred: Command "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "native" "-Wall" "-Wextra" "-DARGON2_NO_THREADS" "-o" "/home/ubuntu/nimiq/target/release/build/libargon2-sys-1a40ba79533982af/out/native/opt.o" "-c" "native/opt.c" with args "cc" did not execute successfully (status code exit code: 1).

This issue is a meta issue with the information of all the features that should/could be backported from Albatross after the stable release:

• [ ] Client API support
• [ ] Build genesis block data from config file
• [ ] Move paths of dependencies to root crate

Currently, we only support loading certificates (and private key) from a PKCS#12 file, but most certificate providers (including Let's Encrypt) issue their certificates in separate non-encrypted PEM formatted files, so it makes sense for us to support loading certificates this way too.

This is the checklist of pending items/bugs/tasks that need to be addressed in order to release a 1.0 version:

• [x] Looking into networking layer improvements in albatross and backporting anything that makes sense
• [x] Make sure TLS termination works properly with our current mainnet infrastructure setup
• [x] Backport the updated GitLab CI config from albatross
• [x] Fix bug where incomplete AccountsProof are sent
• [x] Code cleanup (warnings during compilation, etc.)
• [ ] Support https for the metrics server
• [ ] Make sure the name of metrics in Rust match those in JavaScript