Quick Peer-To-Peer UDP file transfer

Overview

qft

QFT is a small application for Quick (and really reliable) Peer-To-Peer UDP file transfer.

If a friend sent you here...

...look at the "Releases" section on the sidebar. You should see a link titled vX.Y.Z. Click on that, and then choose the right file for your OS: qft for Linux, qft-mac for Mac, and qft.exe for Windows (currently unavailable, sorry.). Download this file, make it executable in case of Linux or Mac, and then follow your friend's instructions on how to receive the file they wanted to send you.

Usage:

  • Find a public QFT helper (for example tudbut.de:4277)
  • On the sender PC, enter qft sender <helper> <shared-phrase> <filename>.
  • On the receiver PC, enter qft receiver <helper> <shared-phrase> <filename>.
  • Both PCs should start transferring after a short while. If they don't, try again.

OR

  • On both PCs, enter qft gui.
  • Select mode
  • Select file to send and file to save to
  • Update the shared phrases to match
  • Click start

Arguments:

qft helper   <bind-port>
qft sender   <helper-address>:<helper-port> <phrase> <filename> [bitrate] [skip]
qft receiver <helper-address>:<helper-port> <phrase> <filename> [bitrate] [skip]

What helpers do

Helpers are NOT relays for data, they are only used to ESTABLISH the connection.

Helpers are there to help with holepunching.

  • P1 connects* to helper
  • P1 sends the phrase to the helper
  • P1 waits for a response
  • Some time passes
  • P2 connects* to the same helper
  • P2 sends the phrase to the helper
  • P2 gets P1's public IP and port
  • P1 gets P2's public IP and port
  • P1 and P2 disconnect* from the helper
  • P1 and P2 start a loop (slightly simplified):
    • fire a packet at eachother multiple times
    • try to receive as many packets from the other one
    • if none are received, loop again
    • if one is received, exit the loop
  • Connection between P1 and P2 is established.

*UDP is a connection-less protocol, there are no handshakes. The word "connection" is used here as an indicator that data will be exchanged between the "connected" parties. The word "disconnect" is used here as an indicator that no more data will be exchanged between the "previously connected" parties.

Debunking some myths about P2P networking

  • "True P2P is only possible without a NAT" - Both my experiments and wikipedia would like to have a word about hole punching. The only issue I have found are some german mobile data providers, but sending large files over mobile data is rarely something you'd want to do - and if so, use something like croc or the magic wormhole, which aren't purely true P2P.
  • "Croc is P2P as well, why does this exist?" - Croc is not Peer-to-Peer. Croc uses a relay server to exchange data between the two clients (unless one of the client has a port-forward set up, which is almost never the case). That is Client-to-Server-to-Client, which is not really Peer-to-Peer. Peer-to-Peer means two clients sending their data directly to eachother, without a server. "Peers make a portion of their resources, such as processing power, disk storage or network bandwidth, directly available to other network participants, without the need for central coordination by servers or stable hosts." - Wikipedia

Tips 'n Tricks

  • You can add a number to the end of both of your commands (after the filename) to boost transfer speeds (higher = faster), but a too large number might cause unreliability due to local network conditions or VPNs. The maximum possible is 65532 (65535 - 3).
  • You can run a helper yourself, as the "helper" mode argument suggests. This helper should simply be run on a server which is reachable from all over the web (a cheap VPS will definitely do).
  • Helpers don't have to be run on a public server, they work in LAN too, but that way, only computers in the same LAN will be able to use them.
  • You can allow streaming (for example when you want to transmit from /dev/stdin) by setting the QFT_STREAM environmental variable.
  • To use qfts and qftr aliases on linux or mac, run (replacing (shell) with your shell name, usually bash or zsh):
echo 'alias qftr="qft receiver tudbut.de:4277"' >> ~/.(shell)rc
echo 'alias qfts="qft sender tudbut.de:4277"' >> ~/.(shell)rc
source ~/.(shell)rc

Cool stuff

  • Files are transferred over UDP, but qft has additional reliability measures in place to avoid broken files.
  • Unreliable internet connection? No problem! QFT will simply pause transmission until the connection is back! Doesn't work? Check out the "Resume a fully stopped transfer" section!
  • Did you know you can hibernate or suspend your computer while it's transferring and it'll continue where it left of, even when only one side is suspended? (Unless your router blocked the port, read the "Resume a fully stopped transfer" section in that case)
  • QFT can withstand heavy weather network conditions: 1000ms ping is just as fast as 10ms ping, packet loss/reorder rates of over 10% are tolerated (but can slow speeds down, especially when ping is high).
  • It's written in 100% pure Rust.

Resume a fully stopped transfer

You most likely won't need this unless the transfer completely died due to a VERY long pause or a computer restart, but if you do:

Stop qft on both ends and start it again with the [skip] parameter in place (if you didn't specify a bitrate before, the default is 256). It will skip those bytes and continue where you specified.

Troubleshooting

It says Connecting... but doesn't connect

One of your ends didn't correctly connect to the helper. Stop the transfer on both ends and try again.

Croc

Many people have mentioned how this is like croc. It isn't, because croc uses a relay that all your data is sent through. This is a bottleneck and also means that the relay admins are responsible for the content that is sent. The relay also buffers a lot of data, meaning its RAM might fill up if the sender's connection is much faster than the receiver's. Croc being tagged "peer-to-peer" is misleading at best because it rarely uses the P2P capabilities (it requires a port-forward to do P2P, which is rarely done). Read the previous section about P2P myths if you think Croc is always peer-to-peer.

Relevant XKCD

Relevant XKCD Image

FAQ

What is a helper?

As explained above, it is used to establish the connection between the two partners.

Why is a helper needed?

Your router uses a thing called Network Address Translation. It is required because otherwise, there would be way too many IP addresses in use and the internet would cease to work during busy times. This NAT is also a problem however, because it is a layer between your PC and the open internet. When there is a new incoming connection, the NAT won't know which PC in your LAN to forward the connection to, so the connection is simply rejected. Any Peer-to-Peer software therefore needs a helper server (also called "STUN" server) which both peers will ask for the other's IP address and port. Both peers can then send a bunch of outgoing connections to eachother. If everything goes well, both peers have sent an outgoing with the right timing, causing both NATs to think they are outgoing connections, when actually, they are a sort of combination of incoming and outgoing ones.

TL;DR: P2P networking is impossible without a helper server, because of Routers. Port-forwarding would be required otherwise, which can be hard to set up.

How to make a public/private helper?

Read the 2nd bullet point in the Tips 'n Tricks section.

You might also like...
P2P File Transfer

P2P File Transfer Overview This is a small project to help solve a problem that should've been solved by now: file transfer. For in-person file transf

Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.

Phantun A lightweight and fast UDP to TCP obfuscator. Table of Contents Phantun Latest release Overview Usage 1. Enable Kernel IP forwarding 2. Add re

A small holepunching implementation written in Rust (UDP)

rust-udp-holepunch A small holepunching implementation written in Rust (UDP) Prerequisites Your rendezvous server must lay in a network which doesn't

Fast User-Space TCP/UDP Stack

Catnip Catnip is a TCP/IP stack that focuses on being an embeddable, low-latency solution for user-space networking. Building and Running 1. Clone Thi

Test the interception/filter of UDP 53 of your local networks or hotspots.

udp53_lookup Test the interception/filter of UDP 53 of your local networks or hotspots. Inspired by BennyThink/UDP53-Filter-Type . What's the purpose?

Tachyon is a performant and highly parallel reliable udp library that uses a nack based model

Tachyon Tachyon is a performant and highly parallel reliable udp library that uses a nack based model. Strongly reliable Reliable fragmentation Ordere

UDP proxy with Proxy Protocol and mmproxy support
UDP proxy with Proxy Protocol and mmproxy support

udppp UDP proxy with Proxy Protocol and mmproxy support. Features Async Support Proxy Protocol V2 SOCKET preserve client IP addresses in L7 proxies(mm

Stream API for tokio-udp.

UDPflow Stream API for tokio-udp. TCP-like UDP stream use tokio::net::UdpSocket; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use udpflow::{UdpListen

Aggressively reliable delivery layer. Above UDP. Nothing else.
Aggressively reliable delivery layer. Above UDP. Nothing else.

Aggressively reliable delivery layer. Above UDP. Nothing else.

Comments
  • Can we use a standerd STUN server as the qft helper?

    Can we use a standerd STUN server as the qft helper?

    Qft is a good job! I am tired of transferring files with scp, especially when the target directory is deep.

    Btw, is it possible to use a standerd stun server as the qft helper?

    opened by Creling 3
  • QFT Helper Info?

    QFT Helper Info?

    Find a public QFT helper (for example tudbut.de:4277)

    Requesting more information on what a "helper" is?

    Why is a helper required/needed?

    Why can't we just use our public WAN IP address?

    How do you find a pulbic helper?

    Can you create your own private helper? How?

    opened by snakyjake1 2
  • shared-phrase collision

    shared-phrase collision

    Since the shared-phrase is user chosen it is possible that more than 1 send and receive party use the same phrase at the same time on the same helper server.

    Right now this could cause either

    1. mixup of receiver B receiving data from sender A and vice versa

    2. both senders to "connect" to each other and sending data at each other while the receivers get no data at all

    The solution would be to let the helper server generate a phrase that is not in use at the moment and send it back to the sender. The sender then can forward the phrase to the receiver and everything else then works like usual.

    This reminds me a lot of croc.

    usability-issue potential-bug 
    opened by fgardt 2
  • What is the required setting of the firewall if both the sender and the receiver are behind NAT?

    What is the required setting of the firewall if both the sender and the receiver are behind NAT?

    Excellent work. I have been looking for the file transfer application from home to the office without the expensive relay server. Although there are techniques like UDT, QUIC, STUN or something else, the qft seems as the only takeaway and nearly workable one. I have recompiled the binaries for linux and raspberry-pi here . The linux version is compiled in static way and is hoped to solved the missing glibc6 problem.

    There are still two problems I met:

    The first is that the qft is workable when at least one of the sender or receiver have global Internet IP. If both the machines are behind the NAT router, the process would stop after sending the first package like this:

    # the receiver
     ./qft receiver tudbut.de:4277 KEYS hi.txt.gz 
    Holepunching IP:PORT (partner) and :38116 (you).
    Connecting...
    Holepunch and connection successful. # [Freeze here, do not receive anything]
    
    # the sender
    ./qft sender   tudbut.de:4277 KEYS hi.txt.gz
    Holepunching IP:PORT (partner) and :53757 (you).
    Connecting...
    Holepunch and connection successful.
    Length: 20003073
    Sent 983040 bytes; Speed: 512 kb/s
    10s passed since last packet ==> Contact broke. Trying to resend packet...
    10s passed since last packet ==> Contact broke. Trying to resend packet... # [Repeat this line]
    
    

    In my opinions, the UPnP setting or something else is needed for any one of the mchine, but I'm not sure what exactly is.

    The other one is that when the packages size is larger than 256, it's more likely that the process will fail or packages will be dropped. I think this should be solved with paralelling transfering of ~1MB blocks with small package size.

    opened by mimi3421 2
Releases(v0.5.5)
Owner
Daniel H.
Daniel H.
Simple Peer-to-Peer Exchange

Near Cetificate Devoloper - Demo Simple Peer-to-Peer Exchange On NEAR How it works? See how p2p exchange work here. Exploring The Code The contract co

null 3 Dec 20, 2021
IDP2P is a peer-to-peer identity protocol which enables a controller to create, manage and share its own proofs as well as did documents

IDP2P Experimental, inspired by ipfs, did:peer and keri Background See also (related topics): Decentralized Identifiers (DIDs) Verifiable Credentials

null 5 Oct 31, 2022
Peer-to-peer communications library for Rust based on QUIC protocol

qp2p Crate Documentation MaidSafe website SAFE Dev Forum SAFE Network Forum Overview This library provides an API to simplify common tasks when creati

MaidSafe 337 Dec 14, 2022
Easy-to-use wrapper for WebRTC DataChannels peer-to-peer connections written in Rust and compiling to WASM.

Easy-to-use wrapper for WebRTC DataChannels peer-to-peer connections written in Rust and compiling to WASM.

null 58 Dec 11, 2022
Peer-to-peer overlay routing

Rust_Pinecone This is a port of the peer-to-peer overlay routing mechanism Pinecone and aims to be interoperable with it, although it isn't yet becaus

null 3 Aug 2, 2022
Mateversum is a peer-to-peer WebXR metaverse project.

Mateversum ?? Mateversum (pronounced: MAH-tay-ver-sum) is a peer-to-peer WebXR metaverse project. The idea is that you'd be able to connect to a netwo

Ashley 23 Dec 21, 2022
Core library for Lightning Network peer-to-peer nostr platform

Mostro server This document explains how Mostro works. Overview Due to the growing need to be able to operate with Bitcoin without giving up personal

Mostro 16 Jan 4, 2023
Simple in-network file transfer with barely any overhead.

fftp fftp is the "Fast File Transport Protocol". It transfers files quickly between computers on a network with low overhead. Motivation FTP uses two

leo 4 May 12, 2022
Transfer file in LAN is so easy

txrx Transfer file in LAN is so easy. Purpose Finding machine IP before executing netcat Multiplatform transfer is collapsing Usage Just run $ txrx fi

Konge 4 Sep 5, 2022
Yet Another File Transfer Protocol.

yaftp Yet Another File Transfer Protocol. Build & Run $> cargo build --release Features C2C Lightweight Per something per session High performence Res

b23r0 20 Sep 22, 2022