There is an implementation of a Boolean Bitfield here:

https://github.com/sigp/lighthouse/tree/master/boolean-bitfield

It (kinda) does the job for now, but it really needs some work done. If you spend some time looking at it I think you'll soon find out what I mean. As an example;

• There is a possibility of overflows: we return the number of bits as a usize, however there can theoretically be usize number of bytes meaning we can have 8 * usize bits.
• It keeps track of the number of true bits as you flip bits on and off. I don't think this is ideal as most cases where we want to know the number of true bits, we'll be receiving some serialized bytes from somewhere else (e.g., p2p nodes) and will need to calculate it manually.

On top of these two points, there's likely many chances for optimization.

Required Functionality

Get

get(n: usize) -> Result<bool, Error>

Get value at index n.

Error if bit out-of-bounds (OOB) of underlying bytes.

Set

set(n: usize, val: bool) -> Result<(bool, Error>

Set bit at index n. Returns the previous value if successful.

Error if bit is OOB of underlying bytes.

Highest Set Bit

highest_set_bit() -> Option<usize>

Returns the index of the highest set bit. Some(n) if a bit set set, None otherwise.

Note: this is useful because we need to reject messages if an unnecessary bit is set (e.g. if there are 10 voters and the 11th bit is set

Number of Underlying Bytes

num_bytes() -> usize

Returns the length of the underlying bytes.

_Note: useful to reject bitfields that are larger than required (e.g., if there are eight voters and two bytes -- only one byte is necessary). _

Number of Set Bits

num_set_bits() -> usize

Returns the total number of set bits (i.e., how many peeps voted).

Note: I'm not 100% sure we'll use this but I suspect we will.

Description

Given is a custom beacon-chain testnet with nodes in physical distinct network locations [A, B]. The nodes have both an ENR supplied in a testnet directory specified as boot-enr.yaml and (due to inability to sufficiently network through ENR) a multi-address command line flag.

The ENR file looks like this: gist/d6eea3ea3356e41bde81864143284ce9#file-4-boot_enr-yaml

The multi-address look like this:

--libp2p-addresses /ip4/51.158.190.99/tcp/9000,/ip4/87.180.203.227/tcp/9000

Version

~/.opt/lighthouse master*
❯ git log -n1
commit f6a6de2c5d9e6fe83b6ded24bad93615f98e2163 (HEAD -> master, origin/master, origin/HEAD)
Author: Sacha Saint-Leger <[email protected]>
Date:   Mon Mar 23 09:21:53 2020 +0100

    Become a Validator guides: update (#928)

~/.opt/lighthouse master*
❯ rustc --version
rustc 1.42.0 (b8cedc004 2020-03-09)

Present Behaviour

In case there is a network fragmentation between A and B, the nodes do not attempt to reconnect to each other. Both nodes know about each other both from the ENR and the multi-address format.

Furthermore, if both A and B are validators, there happens a chain split with two different head slots.

It's possible to reconnect the nodes by restarting the beacon chain nodes manually, however, the chains of A and B are now unable to reorganize to the best head and the peers ban each other.

Mar 23 10:53:27.534 ERRO Disconnecting and banning peer          timeout: 30s, peer_id: PeerId("16Uiu2HAkxE6kBjfoGtSfhSJAE8oib6h3gM972pAj9brmthTHuuP2"), service: network
Mar 23 10:53:27.612 ERRO Disconnecting and banning peer          timeout: 30s, peer_id: PeerId("16Uiu2HAmPz5xFwZfY4CYN6fxf3Yz6LQaDfzCUrA5qwCoTHoCSiNR"), service: network
Mar 23 10:53:38.000 WARN Low peer count                          peer_count: 1, service: slot_notifier

In this case, deleting either A's or B's beacon chain and temporarily stopping the validators is the only way to recover from this issue.

Expected Behaviour

The beacon chain node should aggressively try to maintain connections. It should keep trying to resolve ENR and multi-addresses even after a disconnect.

I don't know how it's designed, but I imagine:

1. having higher timeouts to prevent disconnects in case of short network fragmentation
2. having repeated connection attempts even if previous attempts failed in case of longer or more severe network fragmentation

I have not enough understanding of consensus to have a suggestion for how to handle the reorganization, but having a more stable network would certainly help here.

Steps to resolve

Manual restart including reset of the beacon chain data directory.

Add pools supporting sync committees:

• naive sync aggregation pool
• observed sync contributions pool
• observed sync contributors pool
• observed sync aggregators pool

Add SSZ types and tests related to sync committee signatures.

Sigma Prime core contributors (@paulhauner , @AgeManning , @spble , @kirk-baird, @michaelsproul and others) have recently been discussing licensing for Lighthouse. We would like to change the licensing on Lighthouse to a more permissive, less restrictive license: Apache 2.0 (presently GPL 2.0).

We would like to request our contributors to provide their approval for this change. If you approve this re-licensing, please comment in this issue with the sentence "I agree with re-licensing my work on this project to Apache 2.0" or similar.

If you disagree with this change and would like to discuss further, please feel free to do so here or reach out to us on Gitter.

Thank you all for helping make Lighthouse an even more open project!

Description

Implement the function described here: https://github.com/ethereum/eth2.0-specs/issues/54

Present Behaviour

Function doesn't exist.

Expected Behaviour

Function should exist.

Steps to resolve

AFAIK, the function is still experimental so be on the lookout for bugs and optimisations.

At this stage, I think we should implement it as a separate crate.

Issue Addressed

NA

Primary Change

When investigating memory usage, I noticed that retrieving a block from an early slot (e.g., slot 900) would cause a sharp increase in the memory footprint (from 400mb to 800mb+) which seemed to be ever-lasting.

After some investigation, I found that the reverse iteration from the head back to that slot was the likely culprit. To counter this, I've switched the BeaconChain::block_root_at_slot to use the forwards iterator, instead of the reverse one.

I also noticed that the networking stack is using BeaconChain::root_at_slot to check if a peer is relevant (check_peer_relevance). Perhaps the steep, seemingly-random-but-consistent increases in memory usage are caused by the use of this function.

Using the forwards iterator with the HTTP API alleviated the sharp increases in memory usage. It also made the response much faster (before it felt like to took 1-2s, now it feels instant).

Additional Changes

In the process I also noticed that we have two functions for getting block roots:

• BeaconChain::block_root_at_slot: returns None for a skip slot.
• BeaconChain::root_at_slot: returns the previous root for a skip slot.

I unified these two functions into block_root_at_slot and added the WhenSlotSkipped enum. Now, the caller must be explicit about the skip-slot behaviour when requesting a root.

Additionally, I replaced vec![] with Vec::with_capacity in store::chunked_vector::range_query. I stumbled across this whilst debugging and made this modification to see what effect it would have (not much). It seems like a decent change to keep around, but I'm not concerned either way.

Also, BeaconChain::get_ancestor_block_root is unused, so I got rid of it :wastebasket:.

Additional Info

I haven't also done the same for state roots here. Whilst it's possible and a good idea, it's more work since the fwds iterators are presently block-roots-specific.

Whilst there's a few places a reverse iteration of state roots could be triggered (e.g., attestation production, HTTP API), they're no where near as common as the check_peer_relevance call. As such, I think we should get this PR merged first, then come back for the state root iters. I made an issue here https://github.com/sigp/lighthouse/issues/2377.

Description

I receive this error in my logs, over and over:

{
  "msg": "Failed to update eth1 cache",
  "level": "ERRO",
  "ts": "2020-05-27T09:02:20.147080658-05:00",
  "service": "eth1_rpc",
  "error": "Failed to update eth1 cache: FailedToInsertDeposit(NonConsecutive { log_index: 52, expected: 51 })",
  "retry_millis": "7000"
}

I run lighthouse with the following command: /home/lighthouse/.cargo/bin/lighthouse --logfile /data/lighthouse/logs/beacon.log beacon_node --eth1 --http --ws --datadir /data/lighthouse --http-address 0.0.0.0 --ws-address 0.0.0.0

I run a Nethermind eth1 client, archive sync to Goerli, version 1.8.40

I will say that this is as far as I got, I'm not running a validator yet. I wanted the beacon node to be up and running and in sync before I moved on to the next steps...

Version

Lighthouse 0.1.2

Present Behaviour

It's not clear to me what this log means, I searched for it but I couldn't find it.

Expected Behaviour

Unknown

Issue Addressed

#2282

Proposed Changes

Reduce the outbound requests made to eth1 endpoints by caching the results from eth_chainId and net_version. Further reduce the overall request count by increasing auto_update_interval_millis from 7_000 (7 seconds) to 60_000 (1 minute). This will result in a reduction from ~2000 requests per hour to 360 requests per hour (during normal operation). A reduction of 82%.

Additional Info

If an endpoint fails, its state is dropped from the cache and the eth_chainId and net_version calls will be made for that endpoint again during the regular update cycle (once per minute) until it is back online.

Description

The sync manager shut down and I temporarily lost finality on Schlesi Testnet.

thread 'tokio-runtime-worker-7' panicked at 'elapsed=7481664; when=7481664', /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/wheel/mod.rs:244:5

Version

Using Lighthouse from this branch #1052 @ 310cc4fb504f6b3d6f4b0e9a5f53715a0c848138

Using Schlesi config: https://github.com/goerli/schlesi

Present Behavior

Connected a validator client to a beacon node locally, lost sync manager on the beacon node:

Apr 27 13:18:00.206 INFO Block from local validator              block_slot: 390, block_root: 0x7e2d…96d0, service: http
Apr 27 13:18:06.000 INFO Synced                                  slot: 390, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:18:16.020 INFO Attestation from local validator        slot: 391, index: 0, source: 11, target: 11, service: http
Apr 27 13:18:18.001 INFO Synced                                  slot: 391, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:18:20.041 INFO Attestation from local validator        slot: 391, index: 0, source: 11, target: 11, service: http
thread 'tokio-runtime-worker-7' panicked at 'elapsed=7481664; when=7481664', /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/wheel/mod.rs:244:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Apr 27 13:18:21.453 INFO Sync Manager shutdown                   service: sync, service: network
Apr 27 13:18:30.001 INFO Synced                                  slot: 392, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:18:36.146 ERRO No valid eth1_data votes, `votes_to_consider` empty, outcome: casting `state.eth1_data` as eth1 vote, genesis_time: 1587981600, earliest_block_timestamp: 1587974284, lowest_block_number: 2596126, service: beacon
Apr 27 13:18:36.177 INFO Block from local validator              block_slot: 393, block_root: 0x7d1e…64e9, service: http
Apr 27 13:18:42.001 INFO Synced                                  slot: 393, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:18:54.001 INFO Synced                                  slot: 394, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:19:00.170 ERRO No valid eth1_data votes, `votes_to_consider` empty, outcome: casting `state.eth1_data` as eth1 vote, genesis_time: 1587981600, earliest_block_timestamp: 1587974284, lowest_block_number: 2596126, service: beacon
Apr 27 13:19:00.244 INFO Block from local validator              block_slot: 395, block_root: 0xfcac…31d4, service: http
Apr 27 13:19:06.001 INFO Synced                                  slot: 395, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier
Apr 27 13:19:18.001 INFO Synced                                  slot: 396, epoch: 12, finalized_epoch: 10, finalized_root: 0xd017…bed4, peers: 6, service: slot_notifier

Validator lost ability to submit blocks:

Apr 27 13:18:04.002 INFO Successfully subscribed validators      failed_validators: 2, validators: 2, service: attestation
Apr 27 13:18:06.001 INFO Some validators active                  slot: 390, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:18:16.002 INFO Successfully subscribed validators      failed_validators: 2, validators: 2, service: attestation
Apr 27 13:18:16.020 INFO Successfully published attestations     slot: 391, committee_index: 0, head_block: 0x0f3fbc341d18706b67578838e9bdda5314d0a7eb87b9e48d26825dd9fd84ca0d, count: 1, service: attestation
Apr 27 13:18:18.001 INFO Some validators active                  slot: 391, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:18:20.042 INFO Successfully published aggregate attestations, slot: 391, committee_index: 0, head_block: 0x0f3f…ca0d, signatures: 1, service: attestation
Apr 27 13:18:28.002 CRIT Error during attestation production     error: Failed to subscribe validators: ReqwestError(Error(Status(500), "http://localhost:5052/validator/subscribe")), service: attestation
Apr 27 13:18:30.001 INFO Some validators active                  slot: 392, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:18:36.177 CRIT Error whilst producing block            message: Error from beacon node when publishing block: ReqwestError(Error(Status(500), "http://localhost:5052/validator/block")), service: block
Apr 27 13:18:40.002 CRIT Error during attestation production     error: Failed to subscribe validators: ReqwestError(Error(Status(500), "http://localhost:5052/validator/subscribe")), service: attestation
Apr 27 13:18:42.001 INFO Some validators active                  slot: 393, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:18:52.001 CRIT Error during attestation production     error: Failed to subscribe validators: ReqwestError(Error(Status(500), "http://localhost:5052/validator/subscribe")), service: attestation
Apr 27 13:18:54.001 INFO Some validators active                  slot: 394, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:19:00.244 CRIT Error whilst producing block            message: Error from beacon node when publishing block: ReqwestError(Error(Status(500), "http://localhost:5052/validator/block")), service: block
Apr 27 13:19:04.002 CRIT Error during attestation production     error: Failed to subscribe validators: ReqwestError(Error(Status(500), "http://localhost:5052/validator/subscribe")), service: attestation
Apr 27 13:19:06.001 INFO Some validators active                  slot: 395, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier
Apr 27 13:19:16.002 CRIT Error during attestation production     error: Failed to subscribe validators: ReqwestError(Error(Status(500), "http://localhost:5052/validator/subscribe")), service: attestation
Apr 27 13:19:18.001 INFO Some validators active                  slot: 396, epoch: 12, total_validators: 3, active_validators: 2, proposers: 2, service: notifier

Expected Behavior

Should not panic :)

Steps to resolve

A restart was able to resolve this.

Issue Addressed

#22

Proposed Changes

Simplifies the implementation of boolean-bitfield crate by using the bit-vec crate which handles most of the functionality for us.

Additional Info

Opening this to start the integration process; marked WIP until this changes.

Some notes/questions:

Usage: will we ever be starting with an empty bitfield and expecting to use the API to insert additional bits? There are handy push/pop fns for bit-vec but they aren't exposed right now. We should also note that for now, BooleanBitfield::new() returns an empty bitset and we can't do anything with it. Bringing this up because I expect this is not what we ultimately want.

API question: I implemented the API given on #22 although I have included a len method which returns the number of bits this bitfield is tracking. The API on the issue requests the number of backing bytes. Given that bit-vec tracks the bit count I figure we can just use this. If we really want a byte count then I can do the |x| (x + 7) // 8 trick.

ssz and canonical encoding: if you can point me to some test vectors for ssz {en,de}coding of bitfields then I'll add tests here. in particular, it looks like bit-vec uses big endian encoding so we will need to make sure that matches what is coming off the wire and/or adjust accordingly.

Description

It would be great to have a guide that directs users on how to install Rust and run the test suite. For example:

1. Obtain and install cargo (linux, mac and windows)
2. Use cargo to install stable
3. Use stable to run the test suite (e.g., cargo test --all)

I recommend giving high-level steps that an experienced developer could follow, then linking to more detailed articles that provide "hand holding".

Issue Addressed

Following up from https://github.com/sigp/lighthouse/pull/3223#issuecomment-1158718102, it has been observed that the validator client uses vastly more memory in some compilation configurations than others. Compiling with Cross and then putting the binary into an Ubuntu 22.04 image seems to use 3x more memory than compiling with Cargo directly on Debian bullseye.

Proposed Changes

Enable malloc metrics for the validator client. This will hopefully allow us to see the difference between the two compilation configs and compare heap fragmentation. This PR doesn't enable malloc tuning for the VC because it was found to perform significantly worse. The --disable-malloc-tuning flag is repurposed to just disable the metrics.

Description

We presently don't have a timeout for publishing contribution and proofs:

https://github.com/sigp/lighthouse/blob/564d7da656803f5e06e53a303972580be54500bf/common/eth2/src/lib.rs#L898-L913

It would be nice if we could have a shortened timeout like for attestations:

https://github.com/sigp/lighthouse/blob/564d7da656803f5e06e53a303972580be54500bf/common/eth2/src/lib.rs#L748-L749

This would mean that we can try some other BNs before the slot expires and the message becomes invalid. This is particularly impactful on Gnosis which has a short slot time.

Issue Addressed

Deprecates the step parameter in the blocks by range request

Proposed Changes

• Modifies the BlocksByRangeRequest type to remove the step parameter and everywhere we took it into account before
• Adds a new type to still handle coding and decoding of requests that use the parameter

Additional Info

I went with a deprecation over the type itself so that requests received outside lighthouse_network don't even need to deal with this parameter. After the deprecation period just removing the Old blocks by range request should be straightforward

Issue Addressed

Fixes #1864 and a bunch of other closed but unresolved issues.

Proposed Changes

Allows the deposit caching to recover from NonConsecutive deposit errors by resetting the last processed block to the last valid deposit's block number. Still not sure of the underlying cause of this error, but this should recover the cache so we don't need --eth1-purge-cache anymore :tada:

A huge thanks to @one-three-three-seven for reproducing the error and providing the data that helped testing out the fix :raised_hands:

Still needs a few more tests.

Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/3270

Proposed Changes

Optimize the calculation of historic beacon committees in the HTTP API.

This is achieved by allowing committee caches to be constructed for historic epochs, and constructing these committee caches on the fly in the API. This is much faster than reconstructing the state at the requested epoch, which usually takes upwards of 20s, and sometimes minutes with SPRP=8192. The depth of the randao_mixes array allows us to look back 64K epochs/0.8 years from a single state, which is pretty awesome!

We always use the state_id provided by the caller, but will return a nice 400 error if the epoch requested is out of range for the state requested, e.g.

# Prater
curl "http://localhost:5052/eth/v1/beacon/states/3170304/committees?epoch=33538"

{"code":400,"message":"BAD_REQUEST: epoch out of bounds, try state at slot 1081344","stacktraces":[]}

Queries will be fastest when aligned to slot % SPRP == 0, so the hint suggests a slot that is 0 mod 8192.

Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/2944

Proposed Changes

Remove snapshots from the cache during sync rather than cloning them. This reduces unnecessary cloning and memory fragmentation during sync.

Additional Info

This PR relies on the fact that the block_delay cache is not populated for blocks from sync. Relying on block delay may have the side effect that a change in block_delay calculation could lead to: a) more clones, if block delays are added for syncing blocks or b) less clones, if blocks near the head are erroneously provided without a block_delay. Case (a) would be a regression to the current status quo, and (b) is low-risk given we know that the snapshot cache is current susceptible to misses (hence tree-states).