Subspace Monorepo

Overview

Subspace Network Monorepo

Rust TypeScript

This is a mono repository for Subspace Network implementation, primarily containing Subspace node/client using Substrate framework and farmer app implementations.

Repository structure

The structure of this repository is the following:

  • crates contains Subspace-specific Rust crates used to build node and farmer, most are following Substrate naming conventions
    • subspace-node is an implementation of the node for Subspace protocol
    • subspace-farmer is a CLI farmer app
  • substrate contains modified copies of Substrate's crates that we use for testing

How to run

This is a monorepo with multiple binaries and the workflow is typical for Rust projects:

  • cargo run --release --bin subspace-node -- --dev --tmp to run a node
  • cargo run --release --bin subspace-farmer -- farm to start farming

NOTE: You need to have nightly version of Rust toolchain with wasm32-unknown-unknown target available or else you'll get a compilation error.

You can find readme files in corresponding crates for requirements, multi-node setup and other details.

Issues
  • Remove some duplicated types in RPC

    Remove some duplicated types in RPC

    I make it a draft PR because I'm unsure about some leftover on the duplicated types that unavoidably pull in some types from the substrate, e.g., public_key in Solution, slot in NewSlotInfo. Should be reviewed per commit.

    Close #103

    opened by liuchengxu 6
  • Unify the styling of dependencies in Cargo.toml

    Unify the styling of dependencies in Cargo.toml

    Basically, my personal taste for the dependencies styling, the main idea is : non-substrate deps, sc-*, sp-*, frame-*/pallet-*, subspace deps. Feel free to suggest another one, but a unified style is always good for collaboration.

    Putting the dep items in batches also brings more convenience for the batch editing operation in a text editor.

    No code changes, only styling stuff. I believe it works if it compiles.

    Related #44.

    opened by liuchengxu 5
  • Error: Header Rejected, too far in the future

    Error: Header Rejected, too far in the future

    Error Code

    💔 Verification failed for block 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d received from peer: 12D3KooWPEynXbQeuGwai76xheGav44cCs3n4cpMpCP94g7my1wT, "Header 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d rejected: too far in the future"
    

    Description

    The user contacted me regarding their node crashing due to a power outage, when they reconnected there were no peers connected and their chain was behind the longest known chain.

    As time progressed, in some cases the node would catch up; connect to 4 peers and work for about 1 min then fall back behind and drop all the peers again.

    Expected Behavior

    The node should catch up to the chain, and keep caught up. furthermore, the node should not lose the peer connection.

    Actual Behavior

    The node would try to sync with other nodes and catch up for a moment, then fall far behind again.

    Attempted Troubleshooting

    We rebooted the machine, attempted to reconnect the node, and the issue persisted. From here we purged the chain and attempted sync from Genysis. initially, it was syncing very effectively, but once it got to the longest chain the issue arose again.

    General farmer usage

    Steps to Reproduce

    *This may be hard to replicate.

    1. Run node & farmer as per normal workflow.
    2. Unexpectedly power off the machine.
    3. Boot machine & attempt to restart node & farmer
    4. see if the issue arises.

    Your Environment

    • Version used: Spartan Testnet
    • Operating System and version: Windows 10
    • Desktop, Laptop, or Server: Lenovo Legion
    bug 
    opened by ImmaZoni 5
  • Reduce unwraps in spartan-farmer CLI

    Reduce unwraps in spartan-farmer CLI

    This PR is mostly about refactoring the farmer Command with some unwraps fixed. This is the first step of ultimately eliminating the unwrap or replacing them with expect(), I'll try sending some small PRs when reading the code.

    opened by liuchengxu 4
  • Do not submit unsigned root block extrinsic on nodes that do not produce blocks themselves

    Do not submit unsigned root block extrinsic on nodes that do not produce blocks themselves

    This should allow non-validator nodes to validate blocks properly as well as serve as public RPC node

    opened by nazar-pc 3
  • Subspace upgrade (step 3)

    Subspace upgrade (step 3)

    This PR introduces an unsigned extrinsic containing root block. It is used to persist in the history the state of the archiving process and to set mapping between segment index and Merkle Root in the runtime storage that can later be queried.

    This PR also introduces runtime API for submitting root blocks and it is used in block import pipeline, but no validation is currently happening.

    There are some minor cleanups I did when I was adding new things using existing code as an example.

    The validation and tests are the things for the upcoming PRs.

    opened by nazar-pc 3
  • First version of a tool for downloading blocks from Substrate-based chain

    First version of a tool for downloading blocks from Substrate-based chain

    Trivial implementation for now, can process about 150-200 blocks/s and creates a file for every block, which is extremely inefficient for early blocks in Kusama history that are under 200 bytes each. But it works.

    It is best to sync from local archiving node that syncs from the network, public RPC endpoints have rate limits that slow sync down to 5-7 blocks/s and I don't know what the daily limits are, you might be banned fairly quickly.

    opened by nazar-pc 3
  • Implement K-DHT for `spartan-farmer`

    Implement K-DHT for `spartan-farmer`

    This PR creates a submodule dht in the commands module, that acts as a wrapper around rust-libp2p.

    The dht module is logically divided into two sections:

    1. Client API - these are the methods that we will be using inside spartan-farmer to interact with the DHT.
    2. EventLoop - this is the part of code that handles the actual network events from the Kademlia protocol.

    The EventLoop object is spawned as a background task, so that it works in an async manner.

    Design patterns used here were picked up from the following PR in rust-libp2p:

    1. PR 2186

    TODO

    • [x] Add support for bootstrap.
    • [x] Add Result to method return values.
    • [x] Add ClientConfig to be able to create bootstrap nodes.
    • [x] Use oneshot channels to get Ok/Err from ClientEvents to NetworkEvents to ClientEvents.
    • [x] Add tests.
    opened by whereistejas 3
  • Farmer lib phase3b

    Farmer lib phase3b

    Summary: abstraction of farmer process.

    Now, farmer is an interruptible process, and callable from public API.

    I suggest reviewing all changes at once instead of commit by commit (early commits were not that meaningful, since I was learning the concepts).

    Important note:

    • I'm aware of this abstracted farmer is not being used directly in the main. It is still in the farm.rs. It will move out of it, but currently farm.rs contains also plotting, and the initialization of the arguments required for both farming and plotting, and when this function is removed from there, it will be plot.rs instead of farm.rs.
    • I could have done these initializations in the new abstracted farming process, but then, the same would have to be done for the plotting -> inefficiency and code duplication

    My plan:

    • I'm show-casing the abstracted struct can be called anywhere with this PR.
    • And after the plot abstraction PR (the upcoming one), I'm planning to have another PR, which introduces an initializer function, providing the common arguments for both plotting and farming tasks (in order to reduce duplication). With that PR, farm.rs will be changed significantly (maybe erased), and we will have an initializer, along with abstracted plotting and farming tasks, that can be callable from anywhere, and present in the Public API.
    opened by ozgunozerk 2
  • Farmer bin to lib transition - 1st step

    Farmer bin to lib transition - 1st step

    This PR can be summarized around 2 main changes:

    1. Making the farmer crate also a library (it still has the main.rs)
    2. Phase One of preparing the public API for the farmer-lib

    Since this is a small PR, it would make more sense the review all changes at once

    Making the farmer crate also a library

    For this, the whole crate is re-organized. Mostly copy-paste and moving the files/code pieces around. Also, a common_mod.rs is created to prevent code duplication between main.rs and lib.rs.

    Phase One of farmer-lib Public API -> Identity

    Identity is abstracted into a struct. Now, all the KeyPair related operations are being done using the new Identity struct. Also, this struct is exposed in the lib.rs as public.

    Not completed: In farm.rs, I am yet to change the name of farm_caller. Trying to find a better name/approach at the moment. I'm open to suggestions.

    opened by ozgunozerk 2
  • Remove pre-genesis objects

    Remove pre-genesis objects

    I need this for some of the upcoming work such that I don't have to mess with pre-genesis objects anymore.

    We can't use genesis_build because it only deals with state and we need to fill the block itself.

    I discovered that there exists DigestItem::Other(Vec<u8>), which has comment Some other thing. Unsupported and experimental., but hey, this is exactly what we need! So solution is a custom block struct that when creating genesis block will add that otherwise useless digest item so we can plot some pieces with genesis block alone.

    Reviewing commit by commit will make the most sense.

    opened by nazar-pc 0
  • Farmer Library - Integration tests for `farming`

    Farmer Library - Integration tests for `farming`

    This PR is adding 2 new integration tests for the process farming. The first one is for simulating the happy path, in which, there is a slot, and the farmer tries to find the correct solution. In the second one, we are issuing multiple slots, and also, salt is changing throughout these slots. Farmer tries to find solutions for all of them whilst recommitting in the background for the new salt.

    In the next PR, there will be integration tests for plotting.

    Some bad things that I've done (not necessarily bad maybe, but I would like to avoid these, if possible):

    • RPC trait methods are now &mut self instead of &self. Update: not anymore :)
      • Why?
      • Because, for the MockRPC, we are using channels to issue new challenges and send the expected results. All the receive method variants required the mut on the self.
    • MockRPC is not cloneable, because of the channels in its struct. (this one worries me the most, since we are cloning the client in plotting)
    • Client object is not being sent as an address in the farming.rs, it is now passed by value. Update: also fixed this back

    Note: the changes listed above does not change the behaviour of the original protocol, other tests are passing, and also tested manually with starting a node and a farmer. So, no worries there :)

    opened by ozgunozerk 0
  • Investigate block production stall for the duration of plot recommitment

    Investigate block production stall for the duration of plot recommitment

    For ~116G plot recommitment took several minutes:

    [2021-11-18T22:09:28Z INFO  subspace_farmer::farming] Salt will update to 0c00000000000000 soon, recommitting in background
    [2021-11-18T22:22:14Z INFO  subspace_farmer::farming] Finished recommitment in background for 0c00000000000000 in 766.2087 seconds
    

    And coincidentally block production stalled for about the same time:

    2021-11-18 22:09:27 🙌 Starting consensus session on top of parent 0x3a9921a6129501004c55bfb2289c8f144755a7a6f12e7d8f7c91c33b98209adb    
    2021-11-18 22:09:27 🎁 Prepared block for proposing at 54456 [hash: 0xed3a65676e7103a74c77f9c27546c59af2d494ee6034172cdbe8f8c12a730068; parent_hash: 0x3a99…9adb; extrinsics (3): [0xaa8d…d010, 0x8b00…bc64, 0xdc0e…f635]]    
    2021-11-18 22:09:27 🔖 Pre-sealed block for proposal at 54456. Hash now 0x1cedc221b843158bea21e5ee3ef68898b17b19e4a3f7f67845160bed1ab1dd4f, previously 0xed3a65676e7103a74c77f9c27546c59af2d494ee6034172cdbe8f8c12a730068.    
    2021-11-18 22:09:27 ✨ Imported #54456 (0x1ced…dd4f)    
    2021-11-18 22:09:28 💤 Idle (1 peers), best: #54456 (0x1ced…dd4f), finalized #0 (0xce2f…a5eb), ⬇ 685.7kiB/s ⬆ 687.5kiB/s    
    ...
    2021-11-18 22:22:23 💤 Idle (1 peers), best: #54456 (0x1ced…dd4f), finalized #0 (0xce2f…a5eb), ⬇ 24 B/s ⬆ 24 B/s    
    2021-11-18 22:22:27 🙌 Starting consensus session on top of parent 0x1cedc221b843158bea21e5ee3ef68898b17b19e4a3f7f67845160bed1ab1dd4f    
    2021-11-18 22:22:27 🎁 Prepared block for proposing at 54457 [hash: 0x3d3f28901b730f430733500720ccb1ab725d4357f6e5777d1c65e1689fbe9f6c; parent_hash: 0x1ced…dd4f; extrinsics (3): [0x8674…64b0, 0xe6f9…b16d, 0x770c…351a]]    
    2021-11-18 22:22:27 🔖 Pre-sealed block for proposal at 54457. Hash now 0xdb8b025129fae6b31744d05f0635f17636bcf99ac42e4f05b9d96f6d6138838e, previously 0x3d3f28901b730f430733500720ccb1ab725d4357f6e5777d1c65e1689fbe9f6c.    
    2021-11-18 22:22:27 ✨ Imported #54457 (0xdb8b…838e)    
    

    This is a problem and shouldn't happen. We can reasonably expect that recommitments will affect I/O, but it should not stall block production completely.

    bug 
    opened by nazar-pc 0
  • Feed ownership

    Feed ownership

    pallet-feeds right now doens't remember who created feed and allows anyone to push data to any feed.

    We need to:

    • [ ] track who created feed
    • [ ] limit puts to feed owner
    • [ ] support getting feeds associated with owner
    • [ ] support transfers of feed ownership
    opened by nazar-pc 0
  • Refactor `claim_slot` in authorship

    Refactor `claim_slot` in authorship

    Part of #80

    This PR mainly refactors claim_slot into 3 steps:

    1. Prepare the neccessary data for the next slot.
    2. Notify the subscribers(framers) about the new slot.
    3. Await for a slot solution.

    Should be reviewed by each commit.

    opened by liuchengxu 1
  • Revamp pallet-feeds

    Revamp pallet-feeds

    Part of #80

    The intention of this PR is to unify the terminology about data and object, I believe we prefer object as I see it appeared more often in some places. There are some other improvements like renamings.

    The relayer might need to be updated accordingly.

    Still have some TODOs, but can be reviewed and I'd like to get some feedback.

    TODO:

    • [ ] Weight
    • [ ] Fix tests
    opened by liuchengxu 0
  • Unify the concept of commitment and tag for a piece

    Unify the concept of commitment and tag for a piece

    Ref https://github.com/subspace/subspace/pull/85#discussion_r737073029

    refactor 
    opened by liuchengxu 0
  • Refine the rust documentation

    Refine the rust documentation

    I suggest we go through all the existing crates to see if there is some room to polish or add more descriptive docs, potentially with some small renamings/refactorings.

    • [ ] pallet-feeds
    • [x] pallet-offences-subspace(#98)
    • [x] pallet-subspace(#118)
    • [ ] sc-consensus-subspace
      • #99
    • [x] sc-consensus-subspace-rpc(#115)
    • [x] sp-consensus-subspace(#97)
    • [ ] subspace-archiving
    • [x] subspace-core-primitives(#85)
    • [ ] subspace-farmer
    • [x] subspace-node(#89)
    • [x] subspace-runtime(#88)
    • [x] subspace-solving(#89)
    documentation 
    opened by liuchengxu 0
  • Upgrade test runtime environment to support transactions

    Upgrade test runtime environment to support transactions

    Current test environment provided by forks of substrate-test-runtime and substrate-test-runtime-client doesn't support transactions and makes it harder to maintain quality of the software. We need to fix that and and unlock corresponding tests.

    enhancement help wanted 
    opened by nazar-pc 0
  • Remove BABE and Grandpa from dev dependencies

    Remove BABE and Grandpa from dev dependencies

    Those are used for testing, but, ideally, we'd use our own consensus instead, which would reduce number of dependencies and maybe even CI times.

    enhancement help wanted 
    opened by nazar-pc 5
Owner
subspace
A free and fair blockchain for all
subspace
Joystream Monorepo

Joystream This is the main code repository for all Joystream software. In this mono-repo you will find all the software required to run a Joystream ne

Joystream 115 Nov 20, 2021
Joystream Monorepo

Joystream This is the main code repository for all Joystream software. In this mono-repo you will find all the software required to run a Joystream ne

Joystream 115 Nov 23, 2021