chain nats.io servers with transformation & processing pipelines

Overview

NATS proxy service

Simple tool to forward specific topics from one nats.io cluster to the same server or another. Provides support to process messages with deno Javascript or TypeScript code.

Example

Imagine that we use nats.io to relay events for every confirmed or canceled order in our shopping platform:

./naps --source nats://aws:4222 --destination nats://aks:4222 --topics "orders.>"

Processing Example

If the --script flag is present, naps will spawn a Deno runtime with all v8 capabilities plus promises and all event loop goodies, allowing you to, for example, only keep the confirmed ones and relay them to the myapp.orders.confirmed. You only have to code a recv function with the following signature:

interface RecvResult {
    topic: string,
    msg: string
};

function recv(topic: string, data: Uint8Array): boolean | RecvResult {
    //... your code here...
}
  • If the function returns true, the message will be simply forwarded to the same topic. Do note that the message will end up twice in the topic
  • If the function returns false, this message will be discarded
  • Finally, when RecvResult is returned, that data will be sent over the nats wire.

Example command:

./naps --source nats://aws:4222 --destination nats://aks:4222 --topics "myapp.v1.orders" --script "
    import { Buffer } from 'http://deno.land/x/node_buffer/index.ts';
    
    interface Order {
        status: 'confirmed' | 'canceled',
        user: string,
        amount: number,
        item: any
    };
    
    function processOrder(data: Buffer): RecvResult {
        const orderRaw = data.toString();
        const order = JSON.parse(orderRaw) as Order;
        
        // Skip orders that are not confirmed
        if (order.status !== 'confirmed') {
            return false;
        }
        return {
            topic: 'myapp.v1.orders.confirmed',
            msg: orderRaw
        };
    }

    function recv(topic, uint8array) {
        switch (topic) {
            case "myapp.v1.orders":
                return processOrder(Buffer.from(uint8array))
            default:
                // nothing to do...
        }
    }
"

Thanks

  • Thanks to the rust community for such a good documentation and wide range of libraries which have made this journey far easier.
  • Thanks to the denoland community for pointing me into the right direction. Specially Andreu Botella, denoland contributor who patiently answered all my questions and guided me to a decent solution. Many thanks, man!

TODOs

  • Support for NATS TLS connections
  • JetStream
  • Feature Sagas by allowing to return multiple RecvResult when employing deno runtime
You might also like...
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.

Phantun A lightweight and fast UDP to TCP obfuscator. Table of Contents Phantun Latest release Overview Usage 1. Enable Kernel IP forwarding 2. Add re

Synchronized shadow state of Solana programs available for off-chain processing.

Solana Shadow The Solana Shadow crate adds shadows to solana on-chain accounts for off-chain processing. This create synchronises all accounts and the

Fast Hilbert space-filling curve transformation using a LUT
Fast Hilbert space-filling curve transformation using a LUT

Fast Hilbert Fast Hilbert 2D curve computation using an efficient Lookup Table (LUT). Convert from discrete 2D space to 1D hilbert space and reverse V

3d transformation gizmo built on top of the egui library.
3d transformation gizmo built on top of the egui library.

egui-gizmo 3d transformation gizmo built on top of the egui library. Try it out in a web demo Usage let gizmo = Gizmo::new("My gizmo") .view_matri

ObfusEval is the benchmarking tool to evaluate the reliability of the code obfuscating transformation.

ObfusEval ObfusEval is the benchmarking tool to evaluate the reliability of the code obfuscating transformation. The following two metrics related the

Expression transformation language. An esolang.

Etl Etl (pronounced like "nettle" without the n) is an esoteric programming language for transforming expressions. There are three data types in Etl:

Peakrs Dataframe is a library and framework facilitates the extraction, transformation, and loading (ETL) of data.

Peakrs Dataframe Peakrs Dataframe is a library and framework facilitates the extraction, transformation, and loading (ETL) of data. Its first applicat

Following "ZK HACK III - Building On-chain Apps Off-chain Using RISC Zero"

RISC Zero Rust Starter Template Welcome to the RISC Zero Rust Starter Template! This template is intended to give you a starting point for building a

A system to programmatically run data pipelines

Factotum A dag running tool designed for efficiently running complex jobs with non-trivial dependency trees. The zen of Factotum A Turing-complete job

Execution of and interaction with external processes and pipelines

subprocess The subprocess library provides facilities for execution of and interaction with external processes and pipelines, inspired by Python's sub

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

rust-bert Rust native Transformer-based models implementation. Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing

Connect GStreamer pipelines to Jitsi Meet conferences

gst-meet: Integrate Jitsi Meet conferences with GStreamer pipelines Note: gst-meet is in an alpha state and is under active development. The command-l

Provides a way to use enums to describe and execute ordered data pipelines. 🦀🐾

enum_pipline Provides a way to use enums to describe and execute ordered data pipelines. 🦀 🐾 I needed a succinct way to describe 2d pixel map operat

Framework for large distributed pipelines
Framework for large distributed pipelines

Rain Rain is an open-source distributed computational framework for processing of large-scale task-based pipelines. Rain aims to lower the entry barri

Using bevy and custom render pipelines in order to render many objects in a forest using chunks for performance.

bevy_efficient_forest_example Using bevy and custom render pipelines in order to render many objects in a forest using chunks for performance. Grass i

A fast and robust MLOps tool for managing data and pipelines

xvc A Fast and Robust MLOps Swiss-Army Knife in Rust ⌛ When to use xvc? Machine Learning Engineers: When you manage large quantities of unstructured d

OpenAPI-based test coverage analysis tool that helps teams improve integration test coverage in CI/CD pipelines
OpenAPI-based test coverage analysis tool that helps teams improve integration test coverage in CI/CD pipelines

Ready-to-use OpenAPI test coverage analysis tool that helps teams improve integration CoveAPI is an advanced test coverage analysis tool based on the

Write CI/CD pipelines using TypeScript

Katoa Katoa is a community fork of Cidada, a tool created by Fig which was sunset in late 2023 following acquisition by AWS. This fork and the underly

An API for managing your servers

Intecture APIs Intecture is an API for managing your servers. Visit intecture.io. API docs can be found here: intecture.io/api/intecture_api/. Intectu

Releases(v0.2.1)
Owner
Marquitos
🇪🇸 你喜欢我的裤子吗 🇨🇳
Marquitos
Transforms UDP stream into (fake) TCP streams that can go through Layer 3 & Layer 4 (NAPT) firewalls/NATs.

Phantun A lightweight and fast UDP to TCP obfuscator. Table of Contents Phantun Latest release Overview Usage 1. Enable Kernel IP forwarding 2. Add re

Datong Sun 782 Dec 30, 2022
🤖 brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers

brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers (plain text data). That is, although it can be accessed from a browser, brwrs will not correctly interpret the browser's GET request.

daCoUSB 3 Jul 30, 2021
Safe Rust crate for creating socket servers and clients with ease.

bitsock Safe Rust crate for creating socket servers and clients with ease. Description This crate can be used for Client <--> Server applications of e

Lorenzo Torres 3 Nov 25, 2021
A library to quickly create OAuth2.1 compliant servers from scratch.

oauth21-server A library to easily create an OAuth 2.1 compliant authorization server. The motivation to develop this library comes from the fact that

Revanth Pothukuchi 3 Mar 14, 2022
Web3-proxy: a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers.

web3-proxy Web3-proxy is a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers. Signed transactions (eth_sendRawTrans

null 55 Jan 8, 2023
An HTTP server wrapper for omnisette. Supports both V1 (Provision) and V3 of anisette servers.

omnisette-server An HTTP server wrapper for omnisette. Supports both V1 (Provision) and V3 of anisette servers. Setup First, download the Apple Music

SideStore Team 5 Mar 29, 2023
A Markov chain based Discord chat bot.

A Markov chain based Discord chat bot. Building It is recommended to use cargo.

Dominik Miedziński 1 Dec 26, 2021
Off-chain services for Gnosis Protocol v2

Cow Protocol Services This repository contains backend code for Cow Protocol Services written in Rust. Order Book The orderbook crate provides the htt

CoW Protocol 42 Jan 3, 2023
nats-spy is a terminal tool to help you to monitor NATS messages.

nats-spy nats-spy is a terminal tool to help you to monitor NATS messages. Install Homebrew (macOS) brew install alihanyalcin/nats-spy/nats-spy Usage

Alihan Doğuş Yalçın 23 Oct 23, 2022
Rust client for NATS, the cloud native messaging system.

A Rust client for the NATS messaging system. Status Motivation Rust may be the most interesting new language the NATS ecosystem has seen. We believe t

NATS - The Cloud Native Messaging System 651 Jan 3, 2023