Routing - specialised storage DHT

Related tags

Database sn_routing
Overview

PLEASE NOTE THAT THIS REPOSITORY HAS NOW BEEN ARCHIVED

All routing development is now via the safe_network repository

sn_routing

sn_routing - a specialised storage DHT

Crate LoC
LoC
Documentation MaidSafe website Safe Dev Forum Safe Network Forum

Overview

A secured DHT, based on a kademlia-like implementation, but with some very stark differences. This is a recursive as opposed to iterative network, enabling easier NAT traversal and providing more efficient use of routers and larger networks. This also allows very fast reconfiguration of network changes, aleviating the requirement for a refresh algorithm. A recursive solution based on a network protocol layer that is 'connection oriented' also allows a close group to be aligned with security protocols.

This library makes use of Public-key cryptography to allow a mechanism to ensure nodes are well recognised and cryptographically secured. This pattern allows the creation of a DHT based PKI and this in turn allows a decentralised network to make use of groups as fixed in relation to any address. This is particularly useful in a continually fluid network as described here, creating a server-less and autonomous network.

This is a very under researched area. For a general introduction to some of the ideas behind the design related to XOR Space, watching The Safe Network from First Principles series is recommended. The slides for XOR Distance Metric and Basic Routing lecture are also available here. The last video from the series on how the same ideas were applied to decentralised BitTorrent trackers is available here. A proper formalisation of the Routing algorithm is in progress.

Logging

Messages are logged via the standard log crate, and where enabled, printed via env_logger. By default this prints messages of level "warn" and higher ("error"), but not lower levels ("info", "debug", "trace"). The level can be set explicitly (any of the above or "off"), e.g.:

export RUST_LOG=sn_routing=info

Optionally, the following sub-targets can be controlled independently:

  • stats — messages about connections and sn_routing table size
  • crust — messages from the mock Crust layer (not real Crust)

Example:

export RUST_LOG=sn_routing=info,stats=off

License

Licensed under the General Public License (GPL), version 3 (LICENSE http://www.gnu.org/licenses/gpl-3.0.en.html).

Linking exception

sn_routing is licensed under GPLv3 with linking exception. This means you can link to and use the library from any program, proprietary or open source; paid or gratis. However, if you modify sn_routing, you must distribute the source to your modified version under the terms of the GPLv3.

See the LICENSE file for more details.

Contributing

Want to contribute? Great 🎉

There are many ways to give back to the project, whether it be writing new code, fixing bugs, or just reporting errors. All forms of contributions are encouraged!

For instructions on how to contribute, see our Guide to contributing.

Comments
  • Elder promotion and demotion

    Elder promotion and demotion

    Taking over from https://github.com/maidsafe/routing/pull/1856 Implement part of https://github.com/maidsafe/routing/issues/1835

    This implements the logic for promoting and demoting Elders as needed and updates the splitting logic correspondingly.

    opened by jeanphilippeD 27
  • Renames

    Renames

    Rename

    • vote -> proposal
    • consensus -> agreement
    • accumulate -> aggregate
    • Stage -> Dispatcher
    • Approved -> Core
    • handle_XXX_event -> handle_XXX_agreement
    opened by madadam 26
  • Implements RFC-0023.

    Implements RFC-0023.

    Implements RFC-0023.

    Allow all ImmutableData type managers to obtain a copy of a chunk from any ImmutableData type manager by applying a deterministic naming scheme to the 3 ImmutableData types.

    Refer to https://github.com/maidsafe/rfcs/blob/master/proposed/0023-immutable-data-type-naming/0023-immutable-data-type-naming.md which includes the code used in this PR.


    This change is Reviewable   Multiple assignees: @Fraser999 Fraser999, @afck afck, @dirvine dirvine, @maqi maqi

    opened by ghost 19
  • Removed unused code from src/types.rs

    Removed unused code from src/types.rs

    I assumed that any code not used internally was to be removed. I also assumed the commented tests were to be removed. I hope these were acceptable assumptions to make.

    Everything remaining is currently used in the main build or the tests.

    Review on Reviewable

    opened by hintofbasil 16
  • Display estimation of total number of nodes

    Display estimation of total number of nodes

    In RT print routine, compute average of section length multiplied by 2 ^ section prefix length. This gives an estimation of the total number of nodes in the network. Display also the potential deviation given by max and min values.

    This implements this idea proposed in this post in safe forum.

    opened by Thierry61 15
  • Infinite looping CRUST::NewConnection event

    Infinite looping CRUST::NewConnection event

    debug output using routing master (at v0.2.8) and crust (v0.2.1)

    Handle find group response Sending ConnectRequest to peer ae12ce..b47a4d New CONNECTION; Tcp(V4(127.0.0.1:53078)) CRUST::NewConnection on Tcp(V4(127.0.0.1:53078)) Register unknown connection; Tcp(V4(127.0.0.1:53078)) Send IAM; Tcp(V4(127.0.0.1:53078)) Sending IAM; Tcp(V4(127.0.0.1:53078)) New Message; Tcp(V4(127.0.0.1:53078)) New NON_SIGNED ROUTING MESSAGE; Tcp(V4(127.0.0.1:53078)) Handle I Am Handle I Am Relocated; Tcp(V4(127.0.0.1:53078)) RT (size : 1) added connected node ae12ce..b47a4d on Tcp(V4(127.0.0.1:53078)) Handle CHURN new node ae12ce..b47a4d RT: 31465a..e5f0d2 RT: ae12ce..b47a4d New CONNECTION; Tcp(V4(127.0.0.1:53083)) CRUST::NewConnection on Tcp(V4(127.0.0.1:53083)) Register unknown connection; Tcp(V4(127.0.0.1:53083)) Send IAM; Tcp(V4(127.0.0.1:53083)) Sending IAM; Tcp(V4(127.0.0.1:53083)) New CONNECTION; Tcp(V4(127.0.0.1:53088)) CRUST::NewConnection on Tcp(V4(127.0.0.1:53088)) Register unknown connection; Tcp(V4(127.0.0.1:53088)) Send IAM; Tcp(V4(127.0.0.1:53088)) Sending IAM; Tcp(V4(127.0.0.1:53088))

    paused 
    opened by benjaminbollen 15
  • Message bouncing (fix for issues with lagging message recipient)

    Message bouncing (fix for issues with lagging message recipient)

    This PR replaces the message backlog mechanism with message bouncing:

    • Consider two nodes, A and B.
    • A sends a message to B (it can be a message originating from A or its section or a message that is just relayed through A)
    • B receives it but can't handle it because it isn't in the correct state
    • B wraps the message in Variant::Bounce and includes the version of its own section according to its knowledge. If B is not yet member of any section, it sets it to None.
    • B sends the Bounce back to A
    • A compares the version in the Bounce with the version of the section B is a member of, according to A
    • If the bounce version is lower than the known version, that means that B is lagging behind. A then schedules the message to be resent after a short delay (currently 1 second)
    • When B receives the message again, chances are it has already caught up and is able to handle the message now. If not, it sends the Bounce again.
    • If the bounce version is the same or higher, or if B is no longer a member of any section known to A, then A discards the message

    This PR also modifies the incoming message filter to support the above flow: messages are now inserted into the filter only if they are handled. If they are only relayed and/or bounced, they are not filtered. This modification was necessary, otherwise the resent messages would get filtered out.

    Finally, this PR includes minor refactoring / code cleanup.

    Closes #1941

    opened by madadam 13
  • Crash when running Put and Get of simple_key_value_store

    Crash when running Put and Get of simple_key_value_store

    On Windows, I ran the simple_key_value_store example. There was no network to connect to, and I retried with my machine's network cards disabled to be sure.

    On starting the example with no args, no indication is given whether or not it has connected to a network.

    If I subsequently tried to Put or Get, the example crashed with output:

    thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value'
    
    critical 
    opened by Fraser999 13
  • Dkg

    Dkg

    Rename Agreement -> Dkg as it is Dkg we do the agreements via messages and here we should only be updating section chain and SAP with different members (handling churn)

    BREAKING CHANGE

    opened by dirvine 12
  • Nodes appear to be running as Clients.

    Nodes appear to be running as Clients.

    Running multiple instances of key_value_store as a node we're seeing all connecting to the first but not to each other. If we run a client it also connects only to the first, while storing a value is only stored the first.

    bug critical 
    opened by ghost 12
  • Join limit

    Join limit

    Limit the number of prospective nodes in the bootstrapping process to one per proxy node.

    We also disabled the direction check for now, since we found that it makes the ci_test less reliable. This is to be expected since the notion of a "close node" is currently ill-defined, but we should consider re-enabling it once that issue is fixed.

    Review on Reviewable

    opened by afck 11
Releases(v0.77.10)
Owner
MaidSafe
MaidSafe
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem

Google 31.5k Jan 1, 2023
ForestDB - A Fast Key-Value Storage Engine Based on Hierarchical B+-Tree Trie

ForestDB is a key-value storage engine developed by Couchbase Caching and Storage Team, and its main index structure is built from Hierarchic

null 1.2k Dec 26, 2022
A Key-Value data storage system. - dorea db

Dorea DB ?? Dorea is a key-value data storage system. It is based on the Bitcask storage model Documentation | Crates.io | API Doucment įŽ€äŊ“中文 | English

ZhuoEr Liu 112 Dec 2, 2022
Forage is for Storage

Forage Forage is for Storage Remote storage: Open storage channels to a remote storage provider over Tor Lightweight: Platform-optimized using Blake3-

FuzzrNet 11 Dec 26, 2022
Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

null 5.7k Jan 6, 2023
Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres.

SDB - SignatureDB Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres

Fremantle Industries 5 Apr 26, 2022
SQLite compiled to WASM with pluggable data storage

wasm-sqlite SQLite compiled to WASM with pluggable data storage. Useful to save SQLite in e.g. Cloudflare Durable Objects (example: https://github.com

Markus Ast 36 Dec 7, 2022
Appendable and iterable key/list storage, backed by S3, written in rust

klstore Appendable and iterable key/list storage, backed by S3. General Overview Per key, a single writer appends to underlying storage, enabling many

Eric Thill 3 Sep 29, 2022
OBKV Table Client is Rust Library that can be used to access table data from OceanBase storage layer.

OBKV Table Client is Rust Library that can be used to access table data from OceanBase storage layer. Its access method is different from JDBC, it skips the SQL parsing layer, so it has significant performance advantage.

OceanBase 4 Nov 14, 2022
A mini kv database demo that using simplified bitcask storage model with rust implementation

A mini kv database demo that using simplified bitcask storage model with rust implementation.

Wancheng Long 17 Nov 28, 2022
Storage metadata service.

Storage API This projects hosts the storage metadata API, a backend component which hosts metadata for storage snapshots, the actual data is housed in

Fractal Networks 4 Aug 28, 2022
A high-performance storage engine for modern hardware and platforms.

PhotonDB A high-performance storage engine for modern hardware and platforms. PhotonDB is designed from scratch to leverage the power of modern multi-

PhotonDB 466 Jun 22, 2023
Implementation of the Kademlia DHT protocol in Rust

kademlia-dht Simple implementation of the Kademlia DHT protocol in Rust with state dumping features for educational purposes (not production-ready). T

Leonardo Folgoni 18 Sep 24, 2022
Simple, robust, BitTorrent's Mainline DHT implementation

Mainline Simple, robust, BitTorrent's Mainline DHT implementation. This library is focused on being the best and simplest Rust client for Mainline, es

Nuh 4 Nov 21, 2023
Network Block Storage server, written in Rust. Supports pluggable and chainable underlying storage

nbd-rs Disclaimer DO NEVER USE THIS FOR PRODUCTION Do not use this for any data that you cannot afford to lose any moment. Expect data loss, corruptio

Rainlab Inc 10 Sep 30, 2022
Plugin for macro-, mini-quad (quads) to save data in simple local storage using Web Storage API in WASM and local file on a native platforms.

quad-storage This is the crate to save data in persistent local storage in miniquad/macroquad environment. In WASM the data persists even if tab or br

ilya sheprut 9 Jan 4, 2023
UnlimCloud provides unlimited cloud storage for your files, utilizing Telegram as the storage solution

UnlimCloud provides unlimited cloud storage for your files, utilizing Telegram as the storage solution. Simply log in using your Telegram ID, and you are good to go.

inulute 5 Nov 27, 2023
A blazingly fast static web server with routing, templating, and security in a single binary you can set up with zero code. :zap::crab:

binserve ⚡ ?? A blazingly fast static web server with routing, templating, and security in a single binary you can set up with zero code. ?? UPDATE: N

Mufeed VH 722 Dec 27, 2022
Parameterized routing for generic resources in Rust

Usher Usher provides an easy way to construct parameterized routing trees in Rust. The nodes of these trees is naturally generic, allowing Usher to le

Isaac Whitfield 34 Oct 22, 2022
Dynamic routing system for Rocket

rocket_router Dynamic routing system for Rocket Caveats Each file should export only one route, named the same as the file itself. After adding a new

David Arsene 1 Oct 27, 2021