Multi-stream HTTP downloader using range requests

Related tags

HTTP Client chooch
Overview

chooch - An Amazing Project

Downloads files faster than wget/curl (in theory) using multiple connections. Chooch recycles the slowest connection and re-connects with a new and unique source port.

In environments where LACP or ECMP (hash-based load balancing across multiple physical interfaces) is implemented, a naive multi-stream downloader may not utilize all the bond interfaces if they re-use the same static connections. Chooch aims to solve this by recycling slow connections and ensuring new connections are made with a new unique source port. Doing this adds entropy to the bond's hashing algorithm which may improve total throughput if the connections are evenly distributed across the it's physical interfaces.

In the future, Chooch may be expanded to support multiple source IPs, or could include a utility to pre-determine the best source port set to use for a given machine/network configuration. However, this optimization would require the download server to be reachable using a static address and port, as the source IP is a common facet used in bond hashing algorithms.

Features

  • Uses multiple connections in parallel to download quicker
  • Terminates the slowest connection and re-connects with a new client port.
  • Pre-allocates the file on-disk

Usage

USAGE:
    chooch [FLAGS] [OPTIONS] <url> <output>

FLAGS:
    -f, --force-overwrite    Overwrites existing output file if it already exists
    -h, --help               Prints help information
    -s, --skip-prealloc      Skips the pre-allocation of the target file
    -V, --version            Prints version information

OPTIONS:
    -i, --bind-ip <bind-ip>          Sets the IP address used to make outgoing connections.
    -c, --chunk-size <chunk-size>     [default: 32MB]
    -w, --workers <worker-count>      [default: 6]

ARGS:
    <url>       The URL you wish to download
    <output>    The output destination for this download

Todo

  • TLS Support
  • Handle requests without content-length for unlimited streaming
  • More options for HTTP requests (headers, method, etc)

PRs welcome :)

You might also like...
HTTPie: human-friendly CLI HTTP client for the API era
HTTPie: human-friendly CLI HTTP client for the API era

HTTPie: human-friendly CLI HTTP client for the API era HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI in

null 25.4k Dec 30, 2022
ratpack: a simpleton's HTTP framework (for rust-lang)

ratpack: a simpleton's HTTP framework (for rust-lang) ratpack is idealized in the simplicity of the sinatra (ruby) framework in its goal, and attempts

ZeroTier, Inc. 5 Jun 29, 2022
A backend providing a HTTP REST like interface for uploading files written in rust.

UploadServer A backend providing a HTTP REST like interface for uploading files written in rust. API Documentation License This project is licensed un

null 4 Nov 20, 2022
rh: user-friendly command-line HTTP client

Rust HTTP Cli The command name in your terminal is rh. rh: user-friendly command-line HTTP client rh is a user-friendly, lightweight and performant co

null 8 Nov 30, 2022
An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC.

🚀 H123 An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC. Warning This is an experimental project and not inte

Naoki Ikeguchi 7 Dec 15, 2022
decode a byte stream of varint length-encoded messages into a stream of chunks

length-prefixed-stream decode a byte stream of varint length-encoded messages into a stream of chunks This crate is similar to and compatible with the

James Halliday 4 Feb 26, 2022
An fast, offline reverse geocoder (1,000 HTTP requests per second) in Rust.

Rust Reverse Geocoder A fast reverse geocoder in Rust. Inspired by Python reverse-geocoder. Links Crate 2.0.0 Docs 1.0.1 Docs Description rrgeo takes

Grant Miner 91 Dec 29, 2022
Library + CLI-Tool to measure the TTFB (time to first byte) of HTTP requests. Additionally, this crate measures the times of DNS lookup, TCP connect and TLS handshake.

TTFB: CLI + Lib to Measure the TTFB of HTTP/1.1 Requests Similar to the network tab in Google Chrome or Mozilla Firefox, this crate helps you find the

Philipp Schuster 24 Dec 1, 2022
xh is a friendly and fast tool for sending HTTP requests. It reimplements as much as possible of HTTPie's excellent design, with a focus on improved performance.
xh is a friendly and fast tool for sending HTTP requests. It reimplements as much as possible of HTTPie's excellent design, with a focus on improved performance.

xh is a friendly and fast tool for sending HTTP requests. It reimplements as much as possible of HTTPie's excellent design, with a focus on improved performance

Mohamed Daahir 3.4k Jan 6, 2023
Signing HTTP requests without heavy SDKs

reqsign   Signing API requests without effort. Most API is simple. But they could be complicated when they are hidden from complex abstraction. reqsig

Xuanwo 34 Dec 16, 2022
An HTTP proxy for assets (mainly images) to route requests through an always-encrypted connection.

camo-rs camo-rs is a frontend-compatible Rust-re-implementation of the now archived NodeJS-based atmos/camo - an HTTP proxy for assets (mainly images)

Dennis Schubert 7 Dec 8, 2022
📡Proxy HTTP/1.1 requests over a sensitive point-to-point link
📡Proxy HTTP/1.1 requests over a sensitive point-to-point link

ptproxy Motivation What's this? Why do I need this? What's a sensitive network link? What's wrong with a VPN? What's wrong with HTTP[S]? What's wrong

Alba Mendez 5 Jul 26, 2023
A range proof SNARK using Spartan

R1CS Gadgets This repository contains R1CS gadgest I have built using (a slightly modified version of) Spartan. Currently, the two gadgets included ar

Imran Khaliq 4 Sep 21, 2022
A more modern http framework benchmarker supporting HTTP/1 and HTTP/2 benchmarks.

rewrk A more modern http framework benchmark utility.

Harrison Burt 273 Dec 27, 2022
A simple web server(and library) to display server stats over HTTP and Websockets/SSE or stream it to other systems.

x-server-stats A simple web server(and library) to display server stats over HTTP and Websockets/SSE or stream it to other systems. x-server(in x-serv

Pratyaksh 11 Oct 17, 2022
An iterator following a space-filling pattern over a given range
An iterator following a space-filling pattern over a given range

rlp-iter rlp-iter (Resolving Lattice Point Iterator) is an iterator that returns a space-filling permutation of integers in a given range. Specificall

Nathan Essex 1 May 27, 2022
ZKP fork for rust-secp256k1, adds wrappers for range proofs, pedersen commitments, etc

rust-secp256k1 rust-secp256k1 is a wrapper around libsecp256k1, a C library by Peter Wuille for producing ECDSA signatures using the SECG curve secp25

null 53 Dec 19, 2022
Bulletproofs and Bulletproofs+ Rust implementation for Aggregated Range Proofs over multiple elliptic curves

Bulletproofs This library implements Bulletproofs+ and Bulletproofs aggregated range proofs with multi-exponent verification. The library supports mul

[ZenGo X] 62 Dec 13, 2022
Extent - replacement for std::ops::{Range,RangeInclusive}

extent This is an alternative to std::ops::{Range,RangeInclusive}, avoiding the quirks of those types (non-Copy, inability to produce empty inclusive

Graydon Hoare 33 Sep 27, 2022
Comments
  • License?

    License?

    What are the licensing requirements for this code? Am I allowed to use it in a closed source commercial project if I keep the origin (creator info, github source, etc.)?

    opened by mhvelplund 1
Owner
Check out @pylonbot
null
GitHub
A more modern http framework benchmarker supporting HTTP/1 and HTTP/2 benchmarks.

rewrk A more modern http framework benchmark utility.

Harrison Burt 273 Dec 27, 2022
Typed, correct GraphQL requests and responses in Rust

graphql_client A typed GraphQL client library for Rust. Features Precise types for query variables and responses. Supports GraphQL fragments, objects,

GraphQL Rust 914 Dec 27, 2022
An HTTP library for Rust

hyper A fast and correct HTTP implementation for Rust. HTTP/1 and HTTP/2 Asynchronous design Leading in performance Tested and correct Extensive produ

null 11k Jan 8, 2023
An easy and powerful Rust HTTP Client

reqwest An ergonomic, batteries-included HTTP Client for Rust. Plain bodies, JSON, urlencoded, multipart Customizable redirect policy HTTP Proxies HTT

Sean McArthur 6.8k Dec 31, 2022
FeignHttp is a declarative HTTP client. Based on rust macros.

FeignHttp is a declarative HTTP client. Based on rust macros. Features Easy to use Asynchronous request Configurable timeout settings Suppor

null 46 Nov 30, 2022
Pretend is a macros-based declarative Rust HTTP client

pretend is a modular, Feign-inspired HTTP, client based on macros. It's goal is to decouple the definition of a REST API from it's implementation.

null 24 Aug 3, 2022
🐱‍👤 Drop-in HTTP replacement module for Garry's Mod

??‍?? gmsv_reqwest This module is a drop-in replacement for Garry's Mod's HTTP function, inspired by gmsv_chttp created by timschumi. The module uses

William 38 Dec 12, 2022
Pyre - A fast python HTTP server inspired by japronto written in rust.

Pyre - A fast python HTTP server inspired by japronto written in rust.

null 135 Nov 26, 2022
Minimal Rust HTTP client for both native and WASM

ehttp: a minimal Rust HTTP client for both native and WASM If you want to do HTTP requests and are targetting both native and web (WASM), then this is

Emil Ernerfeldt 105 Dec 25, 2022
Fast and friendly HTTP server framework for async Rust

Tide Serve the web API Docs | Contributing | Chat Tide is a minimal and pragmatic Rust web application framework built for rapid development. It comes

http-rs 4.1k Jan 2, 2023