mitmproxy has its own asyncio.StreamReader/StreamWriter UDP analog wrapper implementation based on Python's asyncio module.

To integrate the UDP functionality from mitmproxy_wireguard into mitmproxy, a similar wrapper interface (UdpStream?) around the existing UDP functionality will be required.

This PR is an attempt at fixing https://github.com/mitmproxy/mitmproxy/issues/5694:

• First, we now try to consume more than one packet (if available) before iterating over the smoltcp sockets. Iterating over the smoltcp sockets is fast already, but batching makes things even faster.
• Somewhat unrelatedly, I discovered during debugging that we create a second socket for resent SYN packets. We now make sure this doesn't happen anymore.
• Remove a very expensive logging call (see below).
• Finally, we simply increase the channel sizes. The previous settings have been a bit too conservative.

These changes should mitigate the reported problem to the extent that I can reproduce it anymore, but unfortunately is not a complete fix. The bigger pressing problem seems to be that pyo3-log is slow and blocking - the following block takes about 100ms when mitmweb is running under load:

for _ in 0..100 {
    log::debug!("slow");
}

It seems like mitmdump isn't affected, and numbers get much worse once I add more mitmweb WebSocket connections. So there's clearly some work that needs to be done on the Python side as well.

I'm running mitmdump inside Termux. While trying to update to the latest version, the mitmproxy-wireguard wheel failed to build locally:

~ $ pip install -U --upgrade-strategy eager mitmproxy
Requirement already satisfied: mitmproxy in /data/data/com.termux/files/usr/lib/python3.10/site-packages (8.1.1)
Collecting mitmproxy
  Using cached mitmproxy-9.0.0-py3-none-any.whl (1.6 MB)
Requirement already satisfied: wsproto<1.3,>=1.0 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (1.1.0)
Collecting wsproto<1.3,>=1.0
  Using cached wsproto-1.2.0-py3-none-any.whl (24 kB)
Requirement already satisfied: urwid<2.2,>=2.1.1 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (2.1.2)
Requirement already satisfied: pyperclip<1.9,>=1.6.0 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (1.8.2)
Collecting pyOpenSSL<22.2,>=22.1
  Using cached pyOpenSSL-22.1.0-py3-none-any.whl (57 kB)
Requirement already satisfied: msgpack<1.1.0,>=1.0.0 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (1.0.4)
Requirement already satisfied: asgiref<3.6,>=3.2.10 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (3.5.2)
Requirement already satisfied: flask<2.3,>=1.1.1 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (2.1.2)
Collecting flask<2.3,>=1.1.1
  Using cached Flask-2.2.2-py3-none-any.whl (101 kB)
Requirement already satisfied: hyperframe<7,>=6.0 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (6.0.1)
Requirement already satisfied: protobuf<5,>=3.14 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (4.21.2)
Collecting protobuf<5,>=3.14
  Using cached protobuf-4.21.9-py3-none-any.whl (291 kB)
Requirement already satisfied: pyparsing<3.1,>=2.4.2 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (3.0.9)
Requirement already satisfied: ruamel.yaml<0.18,>=0.16 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (0.17.21)
Requirement already satisfied: sortedcontainers<2.5,>=2.3 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (2.4.0)
Requirement already satisfied: passlib<1.8,>=1.6.5 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (1.7.4)
Requirement already satisfied: zstandard<0.19,>=0.11 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (0.18.0)
Collecting cryptography<38.1,>=38.0
  Using cached cryptography-38.0.1.tar.gz (599 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: h11<0.15,>=0.11 in /data/data/com.termux/files/usr/lib/python3.10/site-packages (from mitmproxy) (0.12.0)
Collecting h11<0.15,>=0.11
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 852.4 kB/s eta 0:00:00
Collecting mitmproxy-wireguard<0.2,>=0.1.6
  Using cached mitmproxy_wireguard-0.1.14.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      💥 maturin failed
        Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
        Caused by: `cargo metadata` exited with an error: error: failed to load manifest for workspace member `/data/data/com.termux/files/usr/tmp/pip-install-fyxt3ct_/mitmproxy-wireguard_05d31e8eb7024923b70ca8f0f700dfe5/test-client`
      
      Caused by:
        failed to read `/data/data/com.termux/files/usr/tmp/pip-install-fyxt3ct_/mitmproxy-wireguard_05d31e8eb7024923b70ca8f0f700dfe5/test-client/Cargo.toml`
      
      Caused by:
        No such file or directory (os error 2)
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/data/data/com.termux/files/usr/tmp/pip-modern-metadata-_vzn0zuw', '--interpreter', '/data/data/com.termux/files/usr/bin/python3']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /data/data/com.termux/files/usr/tmp/pip-modern-metadata-_vzn0zuw --interpreter /data/data/com.termux/files/usr/bin/python3`
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip

My best guess would be that this happens because https://github.com/decathorpe/mitmproxy_wireguard/blob/bbd32cd9b431dcec3e9486b88208f15d55cd1325/Cargo.toml#L65 is included in the PyPi source archive, but the test client code isn't.

Software versions:

~ $ rustc -Vv
rustc 1.64.0
binary: rustc
commit-hash: unknown
commit-date: unknown
host: aarch64-linux-android
release: 1.64.0
LLVM version: 15.0.1
~ $ python --version
Python 3.10.8
~ $ maturin --version
maturin 0.13.7

See https://github.com/mitmproxy/mitmproxy/issues/5592#issuecomment-1292141069 😃

Edit: With rustfmt complaining again I've also added an autofix.ci job. You may need to install the GitHub App for that. 🙃

This PR simplifies our CI setup and hardens it against supply chain attacks:

• Most importantly, all third-party actions are removed. We shouldn't use those as they introduce unnecessary supply chain risk unless pinned and audited.
• We now pin maturin and install twine via apt.
• To simplify Linux builds, we now use zig for manylinux-compatible compilation.

I've also taken the liberty to switch rustfmt to stable without any custom settings, which makes the CI setup easier and looks just as good to me. Feel free to revert this change and judge me for it, I don't have particular opinions other than a slight preference for the most simple approach.

This PR adds an implementation of UdpStream, a fake stream interface for UDP packets. It provides the same interface as TcpStream, except that most methods don't do anything (since UDP is not connection-based), except:

• UdpStream.read(n): returns up to n bytes from the payload of the received UDP packet, and zero bytes once the initial payload has been read
• UdpStream.write(data): sends a datagram as response to the initially received datagram (or drops it if the network device's send queue is full)

Adapted the test echo server for the new interface, and it seems to work as expected (i.e. it responds "HELLO" if you send a datagram containing the text "hello" with nc -u).

Fixes #7

The source socket of WireGuard UDP packets (i.e. the socket address of the WireGuard client outside the WireGuard tunnel) can now be accessed with TcpStream.get_extra_info("original_src").

There's currently no way to do the same for UDP packets, since the callback for received UDP packets only accepts (payload, src_addr, dst_addr) as arguments.

The original packet destination address is required for implementing a "transparent mode" based on mitmproxy_wireguard. Its sockets are handled internally and not by the operating system, so it needs to expose this information in TcpStream instances.

I've been working on .pyi/PyO3 support for pdoc (https://github.com/mitmproxy/pdoc/issues/239), ended up with some initial docstrings here as a byproduct. :)

See individual commits below.

Some thoughts on the next steps:

• One thing that's slightly annoying about having multiple smoltcp interfaces is that we need to route

   server.tcp_send(connection_id=42, data=b"...")
  

  to the correct smoltcp interface. Maybe it's easier to just use one interface here and map from IPs to peers on the WireGuard side.

• I haven't quite figured out what's the best way to get the raw TCP events out without intertwining everything. I'll keep going on that.

If you want to coordinate on chat, send me an email with what works best for you (ideally Slack, Discord). Alternatively we can stay to mail. :)

This PR fixes the CI failures over at mitmproxy/mitmproxy: https://github.com/mitmproxy/mitmproxy/actions/runs/3374019278/jobs/5599200642

The problem here is as follows:

1. Write and Drain get batched into a single iteration.
2. We first poll the interface (nothing to do yet) before calling sock.send
3. We wait infinitely because we have no reason for another loop.

Previously, the following Drain would always trigger another loop. This PR changes it so that we send before polling. I need to go over this tomorrow again with a fresh mind to check if that's enough. I could imagine some weird read/flow control interactions as well which may require us to poll first, then handle sockets, and then poll again.

Leaving the host empty defaults to binding to both 0.0.0.0 and ::1. However, rust's UDPSocket doesn't bind to both, it binds to the first successful.

Happens here: https://github.com/decathorpe/mitmproxy_wireguard/blob/main/src/server.rs#L125..L135 Documentation here: https://doc.rust-lang.org/std/net/struct.UdpSocket.html#method.bind

Best fix is probably to just support a single listen address, since multiple listen sockets opens another can of worms.

It looks like the unmaintained actions-rs actions will be affected by the deprecation of old NodeJS 12 based actions: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/

The easiest solution is probably to run cargo commands directly, even if doing that means we'll lose commit and PR annotations :(

Looks like we're also using old versions of the actions/checkout action, but updating from actions/checkout@v2 actions/checkout@v3 should solve the problem there.

The same problem also seems to apply to actions/upload-artifact, actions/setup-python, and actions/download-artifact actions ... not sure if we can just bump to new versions, as for some actions, the way to pass them arguments has changed.