Rust implementation of TCP + UDP Proxy Protocol (aka. MMProxy)

Saikō Technology

Last update: Dec 29, 2022

Related tags

Network programming mmproxy-rs

Overview

mmproxy-rs

A Rust implementation of MMProxy! 🚀

Rationale

Many previous implementations only support PROXY Protocol for either TCP or UDP, whereas this version supports both TCP and UDP.

Another reason to choose mmproxy-rs may be if you want to avoid interference from Garbage Collection pauses, which is what originally triggered the re-write from the amazing go-mmproxy.

Features

TCP - Accepts PROXY Protocol enabled requests from Nginx, HAProxy
UDP - Accepts PROXY Protocol enabled requests from udppp, Cloudflare Spectrum
No Garbage Collection pauses

Requirements

Install Rust with rustup if you haven't already.

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ cargo --version

Installation

From git:

cargo install --git https://github.com/saiko-tech/mmproxy-rs

From crates.io

cargo install mmproxy

Usage

Usage: mmproxy [-h] [options]

Options:
  -h, --help              Prints the help string.
  -4, --ipv4 <addr>       Address to which IPv4 traffic will be forwarded to.
                          (default: "127.0.0.1:443")
  -6, --ipv6 <addr>       Address to which IPv6 traffic will be forwarded to.
                          (default: "[::1]:443")

  -a, --allowed-subnets <path>
                          Path to a file that contains allowed subnets of the
                          proxy servers.

  -c, --close-after <n>   Number of seconds after which UDP socket will be
                          cleaned up. (default: 60)

  -l, --listen-addr <string>
                          Address the proxy listens on. (default:
                          "0.0.0.0:8443")

  --listeners <n>         Number of listener sockets that will be opened for the
                          listen address. (Linux 3.9+) (default: 1)
  -p, --protocol <p>      Protocol that will be proxied: tcp, udp. (default:
                          tcp)
  -m, --mark <n>          The mark that will be set on outbound packets.
                          (default: 0)

Example

You'll need root permissions or CAP_NET_ADMIN capability set on the mmproxy binary with setcap(8).

address=X.X.X.X # get this via "ip addr" command - don't use 0.0.0.0!
bind_port=8080
upstream_port=8081
sudo ip rule add from 127.0.0.1/8 iif lo table 123
sudo ip route add local 0.0.0.0/0 dev lo table 123
sudo mmproxy -m 123 -l $address:$bind_port -4 127.0.0.1:$upstream_port -p udp

Benchmarking

Tests were run on a Linux 6.0.12-arch1-1 box with an AMD Ryzen 5 5600H @ 3.3GHz (12 logical cores).

TCP mode

Setup

bpf-echo server simulated the upstream service that the proxy sent traffic to. The traffic was generated using tcpkali.

The following command was used to generate load:

tcpkali -c 50 -T 10s -e1 'PROXY TCP4 127.0.0.1 127.0.0.1 \{connection.uid} 25578\r\n' -m 'PING\r\n' 127.0.0.1:1122

which specifies 50 concurrent connections, a runtime of 10 seconds, sending a PROXYv1 header for each connection, and using the message PING\r\n over TCP.

Results

	↓ Mbps	↑ Mbps	↓ pkt/s	↑ pkt/s
no-proxy	34662.036	53945.378	3173626.3	4630027.6
go-mmproxy	27527.743	44128.818	2520408.4	3787491.3
mmproxy-rs	27228.169	50173.384	2492924.1	4306284.7

UDP Mode

Setup

iperf client -> udppp -> mmproxy-rs/go-mmproxy -> iperf server

$ udppp -m 1 -l 25578 -r 25577 -h "127.0.0.1" -b "127.0.0.1" -p          // udppp
# mmproxy -l "127.0.0.1:25577" -4 "127.0.0.1:1122" -p udp -c 1           // mmproxy-rs
# mmproxy -l "127.0.0.1:25577" -4 "127.0.0.1:1122" -p udp -close-after 1 // go-mmproxy
$ iperf -sup 1122                                                        // iperf server
$ iperf -c 127.0.0.1 -p 25578 -Rub 10G                                   // iperf client

Results

	transfer	bandwidth
no-proxy	6.31 GBytes	5.42 Gbits/sec
go-mmproxy	3.13 GBytes	2.69 Gbits/sec
mmproxy-rs	3.70 GBytes	3.18 Gbits/sec

The iperf test was run in reverse mode, with the server sending data to the client. The results suggest that mmproxy-rs has higher throughput from upstream to downstream compared to go-mmproxy.

Acknowledgements and References

Comments

TCP listener doesn't scale well
Benchmarks done with tcpkali:

$ tcpkali -c 50 -T 10s -e1 'PROXY TCP4 127.0.0.1 127.0.0.1 \{connection.uid} 1122\r\n' -m 'PING\r\n' 127.0.0.1:25577 mmproxy-rs: Aggregate bandwidth: 14582.664↓, 14615.142↑ Mbps go-mmproxy: Aggregate bandwidth: 27613.029↓, 27637.372↑ Mbps

Some initial profiling with strace showed that the Go runtime is making heavy use of the splice syscall while the Rust implementation has user-space buffers that the kernel copies data into, and out of (tokio::io::copy_bidirectional).

splice() is a Linux-specific system call that moves data between a file descriptor and a pipe without a round trip to user space.

enhancement
opened by brkp 15
ERROR : while handling a connection: Transport endpoint is not connected

Hello there

i have try it with tcp connection - http

i have test with send-proxy and send-proxy-v2

and got this error on console :

[2022-12-30T06:05:13Z INFO mmproxy::listener::tcp] [new conn] [origin: 127.0.0.1:20115 [src: 149.200.186.14:13662] [2022-12-30T06:05:13Z ERROR mmproxy::listener::tcp] while handling a connection: Transport endpoint is not connected (os error 107)

[2022-12-30T06:05:13Z INFO mmproxy::listener::tcp] [new conn] [origin: 127.0.0.1:20117 [src: 149.200.186.14:13628] [2022-12-30T06:05:13Z ERROR mmproxy::listener::tcp] while handling a connection: Transport endpoint is not connected (os error 107)

[2022-12-30T06:05:13Z INFO mmproxy::listener::tcp] [new conn] [origin: 127.0.0.1:20121 [src: 149.200.186.14:13569] [2022-12-30T06:05:13Z ERROR mmproxy::listener::tcp] while handling a connection: Transport endpoint is not connected (os error 107)

Any advice please
question

opened by rezq2009 10
Why does iproute2 not want to send packets to the docker network?

vds 1: https://ibb.co/cXFjbq4 vds 2: https://ibb.co/tBWsvkK I configure iproute2 with this script https://pastebin.com/nxgJAMbF , I run the script on vds 2, on vds 1 I drive iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 25565 -j DNAT --to 10.228.228.2:25565 iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 10.228.228.2:80 after I try to go to the site, the site enters but the server running in docker does not enter, what could have gone wrong? docker: https://pastebin.com/xwaJpGfs
invalid

opened by lootoos 6
installation instructions/crates.io badge

Added installation instructions and a crates.io badge/link to the README. Also, @markus-wa can you add me as a collaborator on crates.io (name: brkp) so that I can also push changes to there without bothering you first?

Adresses #1

opened by brkp 3
Wait for performance improvements to the no-proxy level?

no-proxy | 34662.036 | 53945.378 | 3173626.3 | 4630027.6 ( Mbps)

go-mmproxy | 27527.743 | 44128.818 | 2520408.4 | 3787491.3 ( Mbps)

mmproxy-rs | 27228.169 | 50173.384 | 2492924.1 | 4306284.7 ( Mbps)

mmproxy-rs:will it be possible to reach the no-proxy level?

Is it really possible for you to pull this off?
wontfix

opened by lootoos 3

Improve Error Messages

This PR introduces context messages for each error message, providing additional information to help understand the cause of the error. Here's an example before/after:

Before:

[2023-01-02T13:15:03Z INFO  mmproxy::listener::tcp] listening on: 127.0.0.1:25577
[2023-01-02T13:15:06Z INFO  mmproxy::listener::tcp] [new conn] [origin: 127.0.0.1:53566 [src: 127.0.0.1:0]
[2023-01-02T13:15:06Z ERROR mmproxy::listener::tcp] while handling a connection: Operation not permitted (os error 1)

After:

[2023-01-02T13:12:59Z INFO  mmproxy::listener::tcp] listening on: 127.0.0.1:25577
[2023-01-02T13:13:16Z INFO  mmproxy::listener::tcp] [new conn] [origin: 127.0.0.1:46780] [src: 127.0.0.1:0]
[2023-01-02T13:13:16Z ERROR mmproxy::listener::tcp] failed to set ip transparent on the upstream socket: Operation not permitted (os error 1)

opened by brkp 1

How to configure Docker's NAT?

How do I configure this to log in to the server? If the rules are removed via iptables -t nat -F - then the server enters but the Ip of the players is "172.18.0.1"
invalid

opened by lootoos 1
Bump tokio from 1.23.0 to 1.23.1
Bumps tokio from 1.23.0 to 1.23.1.

Release notes

Sourced from tokio's releases.

Tokio v1.23.1

This release forward ports changes from 1.18.4.

Fixed

net: fix Windows named pipe server builder to maintain option when toggling pipe mode (#5336).

#5336: tokio-rs/tokio#5336

Commits

1a997ff chore: prepare Tokio v1.23.1 release

a8fe333 Merge branch 'tokio-1.20.x' into tokio-1.23.x

ba81945 chore: prepare Tokio 1.20.3 release

763bdc9 ci: run WASI tasks using latest Rust

9f98535 Merge remote-tracking branch 'origin/tokio-1.18.x' into fix-named-pipes-1.20

9241c3e chore: prepare Tokio v1.18.4 release

699573d net: fix named pipes server configuration builder

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
how to perform a performance test of mmproxy-rs?

iperf server install on vds? iperf client on home pc? next, run iperf clinet - by specifying the address of the iperf server? Are there any ways to test TCP mmproxy-rs? What are the commands for testing TCP mmproxy-rs?
question

opened by lootoos 6