Layer 4 load balancer with dynamic configuration loading

Overview

Convey

Layer 4 load balancer with dynamic configuration loading featuring proxy, passthrough and direct server return modes

Features

  • Stats page (at /stats) with basic connection/bytes counters and backend server pool statuses
  • Dynamic configuration re-loading of backend servers and associated weights. Configuration is loaded via a .toml file (see sample.toml for a full example).
  • Tcp-based health checking of backend servers at a configured interval. If a server fails its health check it will be automatically removed from selection and added back once its health checks are successful.

Proxy Features

  • Event-driven TCP load balancer built on tokio.
  • Weighted round-robin load balancing. For uniform round robin simply leave out the weights or set them to be equal.
  • TCP connection termination

Passthrough and Direct Server Return (DSR) Features

  • Packet forwarding (no TCP termination)
  • Minimal internal connection tracking
  • NAT

Usage

Convey 0.3.5

Usage:
  convey
  convey --config=<config_file>
  convey (-p | --passthrough) --config=<config_file>
  convey (-d | --dsr) --config=<config_file>
  convey (-p | --passthrough)
  convey (-d | --dsr)
  convey (-h | --help)
  convey (-v | --version)

Options:
  -h, --help               Show this screen.s
  -p, --passthrough        Run load balancer in passthrough mode (instead of default proxy mode)
  -d, --dsr                Run load balancer in direct server mode (instead of default proxy mode)
  --config=<config_file>   Config file location [default config.toml].
  -v, --version            Show version.

Passthrough mode

For passthrough mode we need a couple iptables rules on the convey load balancer to handle ingress packets from the client and responses from the backend load balanced servers. Since TCP is not terminating we need to ensure the OS does not send a RST in response to any packets destined for a port that does not have a process bound to it. We need to do the same for any packets came back through from a backend server. Convey internally assigns ephemeral ports 32768-61000 to map connections to clients.

passthrough

For passthrough mode on the convey load balancer

sudo iptables -t raw -A PREROUTING -p tcp --dport <LOAD_BALANCER_PORT> -j DROP
sudo iptables -t raw -A PREROUTING -p tcp --sport <BACKEND_SERVER_PORT> --dport 33768:61000 -j DROP

To run

sudo ./target/release/convey --passthrough --config=sample-passthrough.toml

DSR Mode

For dsr mode we need the same iptables rule for ingress packets. Responses from the backend load balanced servers will be going directly to the clients. The "listening" port on the convey load balancer must match the backend load balanced servers listening ports in this mode.

dsr

For dsr mode on the convey load balancer

sudo iptables -t raw -A PREROUTING -p tcp --dport <LOAD_BALANCER_PORT> -j DROP

In dsr mode the backend servers "participate" in that their response packets must be sent directly to the client. Convey does not do any encapsulation so, for example, a gre tunnel is not an option. Instead, Traffic Control can be used as an egress nat.

For dsr mode on backend servers

sudo tc qdisc add dev <LOCAL_INTERFACE> root handle 10: htb

sudo tc filter add dev <LOCAL_INTERFACE> parent 10: protocol ip prio 1 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff match ip dst <LOAD_BALANCER_IP> action ok

sudo tc filter add dev <LOCAL_INTERFACE> parent 10: protocol ip prio 10 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff action nat egress <LOAD_BALANCER_IP>

To run

sudo ./target/release/convey --dsr --config=sample-passthrough.toml

Proxy

No special setup neccessary

proxy

To run

sudo ./target/release/convey --config=sample-proxy.toml

Tests

The easiest way to run tests is to run them as superuser. This is because some of the tests spin up test servers as well as a convey load balancer instance.

sudo ~/.cargo/bin/cargo test

AF_XDP

The feature/xdp branch is a WIP using AF_XDP to loadbalance in passthrough and DSR modes. For now it will be maintained in a separate branch since it requires kernel versions 5.4 or greater.

Build

cargo build --release

Comments
  • Cleanup/CI

    Cleanup/CI

    • [x] Applied cargo fmt
    • [x] Applied cargo clippy
    • [x] Added CircleCI. You will need to activate it on your profile.

    A few remaining Clippy suggestions.

    1. Don't name structs with the name of the module:
    warning: item name ends with its containing module's name. You can always call `BaseConfig` as `config::Base`
      --> src/config.rs:19:1
       |
    19 | / pub struct BaseConfig {
    20 | |     pub frontends: HashMap<String, FrontendConfig>,
    21 | |     pub backends: HashMap<String, BackendPool>,
    22 | |     pub stats: Stats,
    23 | |     pub passthrough: Option<Passthrough>,
    24 | | }
       | |_^
       |
       = note: `-W clippy::module-name-repetitions` implied by `-W clippy::pedantic`
       = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#module_name_repetitions
    

    and

    warning: item name starts with its containing module's name
      --> src/stats.rs:24:1
       |
    24 | / pub struct StatsApi {
    25 | |     stats: Arc<RwLock<Stats>>,
    26 | | }
       | |_^
       |
       = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#module_name_repetitions
    
    1. If your enum is called ReadError all fields inside are errors, so it is kind of redundant to have Error in the enum fields.
    warning: All variants have the same postfix: `Error`
      --> src/config.rs:57:1
       |
    57 | / pub enum ReadError {
    58 | |     IOError(IOError),
    59 | |     ParseError(Vec<toml::de::Error>),
    60 | |     DecodeError(toml::de::Error),
    61 | | }
       | |_^
       |
       = note: `-W clippy::pub-enum-variant-names` implied by `-W clippy::pedantic`
       = help: remove the postfixes and use full paths to the variants instead of glob imports
       = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#pub_enum_variant_names
    
    1. it seems that tokio::prelude is unused here:
    warning: unused import: `tokio::prelude`
      --> src/proxy/mod.rs:11:5
       |
    11 | use tokio::prelude::*;
       |     ^^^^^^^^^^^^^^
       |
       = note: `#[warn(unused_imports)]` on by default
    
    opened by naomijub 8
  • Update deps

    Update deps

    What do you think? I used cargo-outdated to track outdated dependencies. All tests and build seem to pass.

    I would squash this to clean all repeated commits as our sources got different from my experimentations on my on source.

    opened by naomijub 2
  • 1 INSECURE dependency and a few outdated dependencies

    1 INSECURE dependency and a few outdated dependencies

    Check it out at https://deps.rs/crate/convey/0.2.0

    Dependencies

    Crate | Required | Latest | Status -- | -- | -- | -- crossbeam-channel | ^0.3.4 | 0.5.0 | out of date failure | ^0.1.2 | 0.1.8 | insecure

    Dev-ependencies

    Crate | Required | Latest | Status -- | -- | -- | -- assert_fs | ^0.10.0 | 1.0.0 | out of date proptest | ^0.8.7 | 0.10.1 | out of date rand | ^0.6.1 | 0.7.3 | out of date structopt | ^0.2 | 0.3.21 | out of date

    opened by naomijub 0
  • Don't calculate ip checksum

    Don't calculate ip checksum

    Since we're now sending out with IP_HDRINCL set on raw socket the kernel will perform the checksum calculation on the ip header anyway. So there's no need to do it twice

    opened by bparli 0
  • Passthrough performance improvements

    Passthrough performance improvements

    • configure workers to listen and send on independent sockets
    • setup listening L2 socket to fanout for load balancing
    • send on L3 socket
    • remove channels for passing packets
    • remove arp managing
    opened by bparli 0
  • DSR Mode

    DSR Mode

    Does DSR work across servers in different datacentres?

    I have got the standard proxy working but cant get the config working for DSR and cant see where I'm going wrong from the instructions. Is there a log anywhere I could check?

    The interface is listening with no errors, but I cant complete a TCP connection with the server behind it.

    I set up the balancers as per the instructions below, is there anythign I could be missing?

    Thank You for any help

    ` sudo tc qdisc add dev enp0s8 root handle 10: htb

    sudo tc filter add dev enp0s8 parent 10: protocol ip prio 1 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff match ip dst <LOAD_BALANCER_IP> action ok

    sudo tc filter add dev enp0s8 parent 10: protocol ip prio 10 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff action nat egress 192.168.1.117 <LOAD_BALANCER_IP> `

    opened by sowie2016 1
Releases(v0.3.5)
Owner
Ben Parli
Ben Parli
The true next-gen L7 minecraft proxy and load balancer. Built in Rust.

Lure The true next-gen L7 minecraft proxy and load balancer. Built in Rust, Tokio and Valence. Why? Rust is a powerful programming language and a grea

Sammwy 67 Apr 16, 2023
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.

Phantun A lightweight and fast UDP to TCP obfuscator. Table of Contents Phantun Latest release Overview Usage 1. Enable Kernel IP forwarding 2. Add re

Datong Sun 782 Dec 30, 2022
wireguard tool to manage / generate configuration. Maintain one yaml configuration file to quickly build wireguard network.

wgx wireguard tool to manage / generate configuration. Maintain one yaml configuration file to quickly build wireguard network. Usage wgx --h USAGE:

yinheli 6 Nov 3, 2022
Bevy asset loader that transparently supports loading over http(s)

Bevy Web Asset This is a tiny crate that that wraps the standard bevy asset loader, and adds the ability to load assets from http and https urls. Supp

Johan Klokkhammer Helsing 28 Jan 2, 2023
Web3-proxy: a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers.

web3-proxy Web3-proxy is a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers. Signed transactions (eth_sendRawTrans

null 55 Jan 8, 2023
Simple and fast layer 4 proxy in Rust

Fourth 这一波在第四层。 English Fourth是一个Rust实现的Layer 4代理,用于监听指定端口TCP流量,并根据规则转发到指定目标。 功能 监听指定端口代理到本地或远端指定端口 监听指定端口,通过TLS ClientHello消息中的SNI进行分流 安装方法 为了确保获得您架构

Rui Li 17 Nov 8, 2022
Shotover: L7 data-layer proxy

Shotover: L7 data-layer proxy

null 64 Dec 16, 2022
An End-to-End Privacy Computing Protocol on Layer 2

Eigen Network Eigen Network is an end-to-end privacy computation network for a better digital economy based on hybrid privacy computation protocols an

Eigen Lab 24 Oct 13, 2022
Automatically updates your Cloudflare DNS records for specific zones. Especially useful if you have dynamic IP address

Cloudflare DNS updater What does it do? Cloudflare DNS updater updates specified dns records for specified zones effortlessly and automatically. It wa

Niko Huuskonen 8 Aug 30, 2022
Final Project for "Computer Networking Security": A Layer-3 VPN implementation over TLS

Final Project for "Computer Networking Security": A Layer-3 VPN implementation over TLS

Siger Yang 2 Jun 7, 2022
Aggressively reliable delivery layer. Above UDP. Nothing else.

Aggressively reliable delivery layer. Above UDP. Nothing else.

IchHabeKeineNamen 2 Jun 5, 2022
A simple configuration-based module for inter-network RPC in Holochain hApps.

DNA Auth Resolver A simple configuration-based module for inter-network RPC in Holochain hApps. About Usage In the origin zome In the destination DNA

Shadman Baig 0 Feb 4, 2022
A demo app covering building an eBPF load-balancer in Rust

eBPF Rust UDP LoadBalancer Demo This is an example of creating a UDP load-balancer in Rust as an eXpress Data Path (XDP) type eBPF program using the a

Shane Utt 17 Jan 5, 2023
The High Performance Proxy/Load Balancer

Silverwind-The Next Generation High-Performance Proxy English 简体中文 The Silverwind is a high-performance reverse proxy/load balancer. And it could be a

null 112 Apr 7, 2023
The true next-gen L7 minecraft proxy and load balancer. Built in Rust.

Lure The true next-gen L7 minecraft proxy and load balancer. Built in Rust, Tokio and Valence. Why? Rust is a powerful programming language and a grea

Sammwy 67 Apr 16, 2023
Load and resolve Cargo configuration.

cargo-config2 Load and resolve Cargo configuration. This library is intended to accurately emulate the actual behavior of Cargo configuration, for exa

Taiki Endo 6 Jan 10, 2023
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.

Phantun A lightweight and fast UDP to TCP obfuscator. Table of Contents Phantun Latest release Overview Usage 1. Enable Kernel IP forwarding 2. Add re

Datong Sun 782 Dec 30, 2022
GaiaX dynamic template engine is a lightweight cross-platform solution for pure native dynamic card.

GaiaX dynamic template engine is a lightweight cross-platform solution for pure native dynamic card, developed by Alibaba YouKu technology team GaiaX

Alibaba 774 Dec 22, 2022
A dynamic output configuration tool that automatically detects and configures connected outputs based on a set of profiles.

shikane A dynamic output configuration tool that automatically detects and configures connected outputs based on a set of profiles. Each profile speci

Hendrik Wolff 15 May 4, 2023
A rust layered configuration loader with zero-boilerplate configuration management.

salak A layered configuration loader with zero-boilerplate configuration management. About Features Placeholder Key Convension Cargo Features Default

Daniel YU 28 Sep 20, 2022