Description

Hey 👋 This is a WebRTC transport implemented in accordance w/ the spec. It's based on the webrtc-rs library.

Resolves: #1066.

Links to any relevant issues

• https://github.com/libp2p/rust-libp2p/issues/1066
• https://github.com/libp2p/specs/issues/220
• https://github.com/paritytech/smoldot/issues/1712

Change checklist

• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

Updated PR description:

Description

Add QUIC as transport to rust-libp2p, using the quinn-proto QUIC implementation. Initially a continuation of #2159, later switched to use the code from #1334 (see 4a317df41adccb0a1dd65ad3df049683fa2ca4ea).

Links to any relevant issues

• #2883

Open TODOs

• [x] Fix docs
• [x] Review / document how backpressure is enforced
• [x] Add more tests

Change checklist

• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

Commit message body

Add experimental QUIC implementation based on quinn-proto

Co-authored-by: Demi Marie Obenour [email protected] Co-authored-by: Pierre Krieger [email protected] Co-authored-by: Thomas Eizinger [email protected] Co-authored-by: Max Inden [email protected] Co-authored-by: Marco Munizaga [email protected] Co-authored-by: Elena Frank [email protected]

Description

Complying with Semver makes it easier for users to consume rust-libp2p.

Links to any relevant issues

https://github.com/libp2p/rust-libp2p/issues/2635

Change checklist

• [x] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

The extension paper S-Kademlia includes a proposal for lookups over disjoint paths. Within vanilla Kademlia, queries keep track of the closest nodes in a single bucket. Any adversary along the path can thus influence all future paths, in case they can come up with the next-closest (not overall closest) hops. S-Kademlia tries to solve the attack above by querying over disjoint paths using multiple buckets.

To adjust the libp2p Kademlia implementation accordingly this change-set introduces an additional peers iterator: DisjointClosestPeersIter. This new iterator wraps around a set of ClosestPeersIter giving each of them a share of the allowed parallelism, amount of striven for results and set of initial nodes to contact. DisjointClosestPeersIter enforces that each of the ClosestPeersIter explore disjoint paths by having each peer instantly fail that was queried by a different iterator.

Status

This is a follow up (/alternative implementation) to https://github.com/libp2p/rust-libp2p/pull/1412, more particularly to the feedback from @romanb in https://github.com/libp2p/rust-libp2p/pull/1412#pullrequestreview-355844433.

Ready for review.

~~Like https://github.com/libp2p/rust-libp2p/pull/1412 this is in an early stage, ready for high-level feedback. Reviewing commit by commit should be easiest.~~

Open questions:

• ~~Different from the suggestion in https://github.com/libp2p/rust-libp2p/pull/1412#pullrequestreview-355844433 this pull request introduces the disjoint path logic into a separate iterator. The reason is that introducing it into query.rs would mess up the abstraction of closest and fixed iterator. Should DisjointClosestPeersIter and ClosestPeersIter be merged? Should this logic live in query.rs like suggested in the first place?~~

• ~~As of today this implementation depends on the amount of allowed parallelism to be equal or greater to the amount of disjoint paths. Is that something that we want to assume? If not this needs some more in-depth changes.~~

• ~~If one ClosestPeersIter returns PeersIterState::Waiting and one returns PeersIterState::WaitingAtCapacity DisjointClosestPeersIter returns the former. I am not sure this upholds correctness.~~

Description

Adding a libp2p transport built on top of the newly released Arti. Currently only dialing is supported, since Onion Services are not supported by Arti yet and it will likely take some time until it's supported, since it's not funded yet.

The implementation should be considered as a first step towards a stable Tor based transport.

Links to any relevant issues

This implementation follows the discussion in #708 and should close this issue.

Open Questions

~~I added libp2p-onion to the default features. However this might be a bad idea, since arti has a lot of dependencies and therefore increases the build time of libp2p greatly.~~ (There are no default features anymore)

Since ~~there is no documentation yet and~~ it's very untested (because of the transports nature automated testing is hard) ~~libp2p-onion maybe shouldn't be integrated in libp2p at all now~~.

Change checklist

• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] A changelog entry has been made in the appropriate crates

I am using the latest quinn git master. While I could backport to the latest release, I would prefer to get the existing code working first.

Current approach:

• Use quinn-proto (the bare state machine)
• Each connection has a pair of HashMaps to store wakers, one for readers and one for writers. When I/O on a stream becomes possible, the corresponding waker is awoken.
• The mutable state is protected with mutexes, so libp2p-quic is thread-safe.
• The state machine is only advanced on a background task.
• Channels are used for incoming connections (to an endpoint) and incoming streams (to a connection). They are also used to send packets generated by a connection to the background task for transmission.
• Strict mutex aquisition ordering is upheld to prevent deadlocks. Specifically, if both the mutex protecting the endpoint and the mutex protecting a connection need to be taken, the mutex protecting the endpoint must be taken first.

What I would like feedback on:

• Should I be using channels in places where I am currently using mutexes?
• To avoid potential memory exhaustion denial of service attacks, I am using a bounded channel for outgoing packets. If it fills up, packets are dropped. Since QUIC is a reliable protocol, I expect that these will be retransmitted, which is much better than having to buffer an unlimited amount of data.
• How should I handle the case where a connection arrives, but we are not ready for it? We can’t buffer an unlimited number of them, so at some point we will need to impose backpressure. The current plan is to rely on quinn-proto limiting the number of connections that it returns but have not yet been accepted.

Edit

The code is now essentially complete and is ready for review.

From the multistream-select 1.0 simultaneous open protocol extension specification:

In order to support direct connections through NATs with hole punching, we need to account for simultaneous open. In such cases, there is no single initiator and responder, but instead both peers act as initiators. This breaks protocol negotiation in multistream-select, which assumes a single initator.

This draft proposes a simple extension to the multistream protocol negotiation in order to select a single initator when both peers are acting as such.

See https://github.com/libp2p/specs/pull/196/ for details.

This commit implements the above specification, available via Version::V1SimultaneousOpen.

Given that the extension is backward compatible, there is no gain for rust-libp2p to support it without supporting the larger effort of TCP hole punching.

To test this against the Golang implementation:

#[test]
fn sim_open_server() {
    futures::executor::block_on(async {
        let listener = TcpListener::bind("127.0.0.1:5000").await.unwrap();
        let listener_addr = listener.local_addr().unwrap();
        println!("{:?}", listener_addr);

        let connec = listener.accept().await.unwrap().0;
        let protos = vec![b"/b", b"/a"];
        let (proto, io) = dialer_select_proto_serial(connec, protos, Version::V1SimOpen).await.unwrap();
        assert_eq!(proto, b"/a");
        io.complete().await.unwrap();
    });
}

#[test]
fn sim_open_client() {
    futures::executor::block_on(async {
        let connec = TcpStream::connect("127.0.0.1:5000").await.unwrap();
        let protos = vec![b"/b", b"/a"];
        let (proto, io) = dialer_select_proto_serial(connec, protos.into_iter(),Version::V1SimOpen)
            .await.unwrap();
        assert_eq!(proto, b"/a");
        io.complete().await.unwrap();
    });
}

package main

import (
        // "fmt"
        "net"
        multistream "github.com/multiformats/go-multistream"
)

func sim_open_client() {
        conn, err := net.Dial("tcp", "127.0.0.1:5000")
        if err != nil {
                panic(err)
        }

        proto, server, err := multistream.SelectWithSimopenOrFail([]string{"/a"}, conn)
        if err != nil {
                panic(err)
        }

        if proto != "/a" {
                panic(proto)
        }

        if server {
                fmt.Println("Server")
        } else {
                fmt.Println("Client")
        }
}

func sim_open_server() {
        ln, err := net.Listen("tcp", "127.0.0.1:5000")
        if err != nil {
                panic(err)
        }
        conn, err := ln.Accept()
        if err != nil {
                panic(err)
        }

        mux := multistream.NewMultistreamMuxer()
	mux.AddHandler("/a", nil)

        proto, _, err := mux.Negotiate(conn)
        if err != nil {
                panic(err)
        }

        if proto != "/a" {
                panic(proto)
        }
}

I'm looking around for an IPFS implementation I can use in a cross-platform desktop/mobile application. I've been playing with https://github.com/Agorise/c-ipfs, but it seems to be incomplete and unstable from my testing.

I don't necessary need a full IPFS implementation, but would ideally like to be able to interact with nodes powered by go-ipfs. My use case is "offline first" application user data storage, and I'd like to replace my current dependency on Couchbase server infastructure with IPFS.

In order to use this library I would need to create a C++ wrapper for it, but it seems that things are moving and changing fast here so I wonder if its better to wait until the API has settled?

I'm trying to demonstrate floodsub interoperability between Go, JS and Rust. In all cases, the topic name is libp2p-demo-chat. Go and JS use that topic string verbatim, but rust-libp2p hashes it to RDEpsjSPrAZF9JCK5REt3tao.

Thoughts? I'd really like to demo the 3 languages inter-operating at some conferences this fall. Guessing you did this for privacy reasons @tomaka?

Example output below. This one is a Go peer connecting to my bootstrapper:

<NOTICE> Got connection from /ip4/127.0.0.1/tcp/46023
handleIncomingRPC: subopt.GetTopicid() = 'libp2p-demo-chat'

Now a JS peer connecting:

<NOTICE> Got connection from /ip4/127.0.0.1/tcp/56098
handleIncomingRPC: subopt.GetTopicid() = 'libp2p-demo-chat'

Now a rust peer connecting:

<NOTICE> Got connection from /ip4/127.0.0.1/tcp/56076
handleIncomingRPC: subopt.GetTopicid() = 'RDEpsjSPrAZF9JCK5REt3tao'

Still a draft as I am opening this to discuss the details and direction of this.

Description

The end goal is to allow the following

• when calling get_providers give the caller results back in real time, not collect all provider records
• when calling get_providers allow the total number of records to be of a fixed limit, such that not necessarily 20 peers need to be contacted (as is the case in the current impl)

Implementation

There are currently two different commits, the first one implements the basic limit functionality, but this gets replaced with a more general solution in 2868e7bb14c9b267b2c50d811895f10cf3945b46, which streams the results and allows the caller to .finish() the query when they have enough.

Things Left To Do

• [x] decide which queries should be transitioned to the new progress based api
• [x] potentially remove Quorum based on the new functionality, as this now could be done by the caller if using the progression based api
• [x] write changelog

Description

Previously, ConnectionHandler employed the prototype pattern via the IntoConnectionHandler abstraction. This allowed the Swarm to construct an instance of IntoConnectionHandler before the connection was established. This abstraction is however unnecessary. We can model all existing usecases by delaying the call to NetworkBehavour::new_handler until the connection is established. Not only does this delete a lot of code, it also makes several APIs simpler:

• NetworkBehaviours can track state for future connections within themselves (indexed by PeerId) and pass it into the ConnectionHandler upon construction. This removes the need for passing along a handler in NetworkBehaviourAction::Dial.
• Removing Handler from NetworkBehaviourAction::Dial also avoids the need to pass the handler back into the behaviour in inject_dial_failure.
• By making NetworkBehaviour::new_handler fallible, NetworkBehaviours can implement almost arbitrary connection management policies by denying the construction of a ConnectionHandler for a newly established connection.

Resolves #2824.

Links to any relevant issues

Open Questions

Change checklist

• [ ] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

Summary

We have some code based on the dcutr example that starts a first event loop and wait on all listen interfaces for one second. Our code can act both as a client (listener + dialer) and a server (listener only).

We are experimenting a race condition while waiting on all listen interfaces: both NewListenAddr and incoming ConnectionEstablished events can be received if for instance an other peer dial the peer at the same time. One possible fix is to explicitly handle the ConnectionEstablished event but we could also miss other events (SwarmEvent::Behaviour events, ...).

Expected behaviour

When starting get all listen interfaces through NewListenAddr events, and then receive other events.

Actual behaviour

Listen interfaces events are mixed with other kind of events.

Possible Solution

Provide a swarm API that can filter in events. Filtered out events are kept in the swarm for later usage.

Version

• libp2p version (version number, commit, or branch): 0.50.0

Would you like to work on this bug ?

Maybe, depending on the proposed solution and its complexity.

Description

addresses #3299

Notes

Links to any relevant issues

Open Questions

We are now getting the observed_addresses at different places, i.e. previously we had PollParameters on poll with what I assume would be the list of updated external_addresses that would be then used to calculate the list of observed_addresses. Now we get the list of observed_addresses from the ExternalAddrs which is updated on NewExternalAddr, but will it make a difference that it might be updated when an Event has already been added to queued_events?

Change checklist

• [x] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

Description

Changes regarding the #2217

Notes

Links to any relevant issues

Open Questions

Change checklist

• [ ] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] A changelog entry has been made in the appropriate crates

Description

With this commit libp2p-dcutr no longer discards the whole remote payload in case an addr is unparsable, but instead logs the failure and skips the unparsable multiaddr.

Notes

Links to any relevant issues

See https://github.com/libp2p/rust-libp2p/issues/3244 for details.

Open Questions

Change checklist

• [x] I have performed a self-review of my own code
• [ ] I have made corresponding changes to the documentation
• [ ] I have added tests that prove my fix is effective or that my feature works
• [x] A changelog entry has been made in the appropriate crates

In libp2p-dcutr we introduced ActionBuilder as events from the ConnectionHandler could not be enriched with the local node's external addresses from the on_* event handlers. Instead, one had to wait for the NetworkBehaviour::poll method to access the addresses.

https://github.com/libp2p/rust-libp2p/blob/68d0f882bd7fc0b4a83a11d0d034ae9f085f3e62/protocols/dcutr/src/behaviour_impl.rs#L349-L363

Now that https://github.com/libp2p/rust-libp2p/pull/3153 is merged, event handlers have access to the external addresses right away, thus the ActionBuilder abstraction is no longer needed. Let's remove it.

To receive a remote's identify information one opens a stream and waits for the remote to send the identify payload in that stream.

When connecting to a go-libp2p node over a QUIC connection, identify exchanges fail.

https://github.com/libp2p/rust-libp2p/blob/68d0f882bd7fc0b4a83a11d0d034ae9f085f3e62/protocols/identify/src/protocol.rs#L192-L196

Changing the above to flush instead of close resolves the issue. Note that flushing instead of no action at all is needed in case multistream-select V1Lazy is used where the multistream-select message is only send on the stream once userdata is send or it is flushed.

Occurs with the rust-libp2p punchr client when connecting to go-libp2p IPFS nodes.

Opening with little details for now to have a central place to track. @elenaf9 in case you need more details, let me know.