"Last Mile" streaming server and client

Overview

TSLM - Terminal Stream Last Mile


This is an asynchronous WebSocket server written in Rust using tokio-tungstenite. It allows a WebSocket client to perform the following actions:

  • Create a channel.
  • Notify the channel with relevant data.
  • Subscribe to a channel.
  • Receive messages from the subscribed channels.

The purpose of this server is to act as a websocket gateway.

As a gateway publishers should be able to selectively push messages from the internal/private/intranet applications to the gateway for it to fan out messages to subscribers connected from public networks without compromising security.

In the probable event of getting hacked the gateway server should not be able to connect inward into the intranet limiting the area vulnerable to attacks.


IMPORTANT NOTE:

This server is a weekend project, and while functional, it might not have the full range of features you might expect from a more extensive, dedicated development effort.


Acknowledgements

  • The current message schemas are experimental and terrible JSON. The reason is that this initial implementation is serializing enums directly as it is quite practical for experimentation but terrible as API definitions. Message schemas will change drastically in the next versions.

How to run

From source

This is the source code so it is possible to just build and run locally running:

cargo run

And get a local server running just to play around and debug.

Note: The code expects your local to have openssl installed.

As docker container

The repo also provides example dockerfiles at ./docker that can be used to build a docker image that you can push to your registry and run.

Build the dependencies

There are 3 dockerfiles at ./docker:

  • 'build-amazonlinux-openssl-rust.dockerfile' that builds a 'builder' image.
  • 'exec-amazonlinux-openssl.dockerfile' that build the execution environment image.
  • 'last-mile.dockerfile' uses these 2 previously images as builder and execution environments.

So first go into ./docker and build the 2 images:

# from ./docker
./docker_build.sh
./docker_exec.sh

Build the image

Now you can run ./docker_build.sh that will use the 'last-mile.dockerfile' to build the final image.

Run the image

'./docker-run.sh' is an example, just mounts ./config to a path where the container expects to find the config and runs the server.

Conclusion

The docker builder image takes care of installing rust, openssl and other dependencies the project might require to build.

Then the docker exec image continues by setting up the environment again but only with the minimal required binary dependencies and the required executables to run the server so the image should be as small as possible.

These things combined make the build process faster as it caches and reduces the final image size.

The examples are cross compiling to x86_64 which is the architecture I am using now at amazon, just change the target architecture to the one you are running in the ./docker/last-mile.dockerfile instructions.


TODO

  • Server configuration.
  • Multiple network interface/ip/port binding.
  • Client fallback mechanism by protocol.
  • Channel backlog.
  • Identified messages for replies.
  • Auth with 3rd party sso using oauth2.
  • Using white listed public keys and hashes to validate publishers.
  • Channel configuration.
  • Docker to make it easier for others to build and run locally and give the server a try.
You might also like...
Acts as an IRC server and a nostr client. Connect with your IRC client using your nostr private key as the password.

nostr-irc Acts as an IRC server and a nostr client. Connect with your IRC client using your nostr private key as the password. Experimental code, use

QUIC proxy that allows to use QUIC to connect to an SSH server without needing to patch the client or the server.

quicssh-rs πŸ˜„ quicssh-rs is a QUIC proxy that allows to use QUIC to connect to an SSH server without needing to patch the client or the server. quicss

Materialize simplifies application development with streaming data. Incrementally-updated materialized views - in PostgreSQL and in real time. Materialize is powered by Timely Dataflow.
Materialize simplifies application development with streaming data. Incrementally-updated materialized views - in PostgreSQL and in real time. Materialize is powered by Timely Dataflow.

Materialize is a streaming database for real-time applications. Get started Check out our getting started guide. About Materialize lets you ask questi

Easy Hadoop Streaming and MapReduce interfaces in Rust

Efflux Efflux is a set of Rust interfaces for MapReduce and Hadoop Streaming. It enables Rust developers to run batch jobs on Hadoop infrastructure wh

New generation decentralized data warehouse and streaming data pipeline
New generation decentralized data warehouse and streaming data pipeline

World's first decentralized real-time data warehouse, on your laptop Docs | Demo | Tutorials | Examples | FAQ | Chat Get Started Watch this introducto

An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ
An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ

An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ

tectonicdb is a fast, highly compressed standalone database and streaming protocol for order book ticks.

tectonicdb crate docs.rs crate.io tectonicdb tdb-core tdb-server-core tdb-cli tectonicdb is a fast, highly compressed standalone database and streamin

A web-based streaming service with improved privacy, performance, and simplicity.

Majiix Majiix is a web-based open-source streaming service. The backend that handles the core streaming logic is written in Rust and makes use of cutt

Polaris is a music streaming application, designed to let you enjoy your music collection from any computer or mobile device.
Polaris is a music streaming application, designed to let you enjoy your music collection from any computer or mobile device.

Polaris is a music streaming application, designed to let you enjoy your music collection from any computer or mobile device. Polaris works by streami

A simple CLI for UPnP media file streaming

Slingr A simple CLI for streaming media files over a local network to UPnP media renderers. Designed to work with cheap HDMI/DLNA/UPnP/Miracast Dongle

Streaming Network Overlay Connection Arbitration Tunnel

SNOCAT Streaming Network Overlay Connection Arbitration Tunnel snocat is a framework for forwarding streams across authenticated, encrypted QUIC tunne

A highly efficient daemon for streaming data from Kafka into Delta Lake

kafka-delta-ingest The kafka-delta-ingest project aims to build a highly efficient daemon for streaming data through Apache Kafka into Delta Lake. Thi

Fluvio is a high-performance distributed streaming platform that's written in Rust

Fluvio is a high-performance distributed streaming platform that's written in Rust, built to make it easy to develop real-time applications.

High-level RTSP multimedia streaming library, in Rust

High-level RTSP multimedia streaming library, in Rust. Good support for ONVIF RTSP/1.0 IP surveillance cameras, as needed by Moonfire NVR. Works around brokenness in cheap closed-source cameras.

GStreamer HTTP Live Streaming Plugin

A highly configurable GStreamer HLS sink plugin. Based on the hlssink2 element. The flexhlssink is written in Rust and has various options to configure the HLS output playlist generation.

A highly efficient daemon for streaming data from Kafka into Delta Lake

A highly efficient daemon for streaming data from Kafka into Delta Lake

Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust.

led_matrix_zmq Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust. This repository includes: Rust client and server l

Rust-advent - Learning Rust by solving advent of code challenges (Streaming live on Twitch every Monday)
Rust-advent - Learning Rust by solving advent of code challenges (Streaming live on Twitch every Monday)

Rust advent πŸ¦€ 🐚 Learning Rust by implementing solutions for Advent of Code problems. πŸŽ₯ HEY, we are live-streaming our attempts to solve the exercis

Streaming data over unix sockets, in Rust

Unix-socket based client/server In order to dig into Sōzu channels, I had to dig into the workings of unix sockets. What this repo contains a small so

Comments
  • Consider adopting AsyncAPI spec

    Consider adopting AsyncAPI spec

    AsyncAPI spec is an specification used to describe your asynchronous APIs. You can then generate documentation, code, and validate your message payloads among other things.

    I suggest this project to have its own asyncapi.yaml file and, additionally, to analyze how AsyncAPI tools ecosystem can help.

    Feel free to reach me. Happy to help on this task!

    enhancement 
    opened by smoya 2
SockJS server for rust language

SockJS server SockJS server for Actix framework. API Documentation Cargo package: sockjs SockJS is built with Actix web Minimum supported Rust version

Actix 63 Oct 7, 2022
RuTTY - Rust TTY Server

RuTTY - Rust TTY Server Demo RuTTY (aka Ruthie) is a CLI-powered websocket server written in Rust that allows you to expose your commands via browser.

Gershon Papi 4 Jun 4, 2023
Command-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions

websocat Netcat, curl and socat for WebSockets. Examples Connect to public echo server $ websocat ws://echo.websocket.org 123 123 ABC ABC Serve and c

Vitaly Shukela 5k Jan 4, 2023
websocket client

#websocket client async fn test_websocket()->anyhow::Result<()> { wasm_logger::init(wasm_logger::Config::default()); let (tx, rx) = futures_c

null 1 Feb 11, 2022
A webserver and websocket pair to stop your viewers from spamming !np and "what's the song?" all the time.

spotify-np ?? spotify-np is a Rust-based local webserver inspired by l3lackShark's gosumemory application, but the catch is that it's for Spotify! ??

Noire 2 Aug 27, 2022
ChatApp made using the standard library net module and tui-rs.

chatui Simple chat application. You'll need both chatui_server and chatui_client to run this application. Installation With cargo cargo install chatui

Gauravsingh Sisodia 6 Dec 15, 2021
Clearly a repo about websockets and their comparison...

Tags Go vs Typescript: Video 1 of the series is tag go-vs-ts-video-1 Current Project Video 2 and 3 are going to be likely Go vs Rust vs Typescript, bu

ThePrimeagen 344 Dec 31, 2022
A MITM Proxy Written in Rust πŸ¦€! Toolkit for HTTP/1, HTTP/2, and WebSockets with SSL/TLS Capabilities. Learning Project.

Man In The Middle Proxy Description Rust-based Man in the Middle proxy, an early-stage project aimed at providing visibility into network traffic. Cur

null 158 Mar 9, 2023
Rust client for Timeplus Proton, a fast and lightweight streaming SQL engine

Rust Client for Timeplus Proton Rust client for Timeplus Proton. Proton is a streaming SQL engine, a fast and lightweight alternative to Apache Flink,

Timeplus 4 Feb 27, 2024
Streaming STOMP client for Rust

tokio-stomp An async STOMP client (and maybe eventually, server) for Rust, using the Tokio stack. It aims to be fast and fully-featured with a simple

null 7 Jun 15, 2022