Retina is a network analysis framework that supports 100+ Gbps traffic analysis on a single server with no specialized hardware.

Overview

Retina

build-status doc-status

Retina is a network analysis framework that enables operators and researchers to ask complex questions about high-speed (>100gbE) network links. Retina allows users to easily subscribe to subsets of parsed application-layer sessions, reassembled network flows, or raw packets in real-time and to run arbitrary analysis code in a standard Rust-based software environment. Retina optimizes for:

  • Expressiveness: Retina supports arbitrarily complex processing of individual packets, reassembled connections, or parsed application-layer sessions using a simple filter and callback interface.

  • Performance: Retina is capable of real-time traffic analysis in high volume (100G+) environments, such as ISPs or academic institutions.

  • Deployability: Retina is readily deployable on a single multi-core server with commodity 100gbE NICs (e.g., Mellanox ConnectX-5 or Intel E810).

  • Security: Retina leverages compile-time memory safety guarantees offered by Rust to safely and efficiently process network traffic.

Documentation

A detailed description of Retina's architecture and its performance can be found in our SIGCOMM'22 paper: Retina: Analyzing 100 GbE Traffic on Commodity Hardware.

Documentation for using and developing against Retina can be found here. It includes a comprehensive description of supported filter syntax and subscribable types.

Getting Started

Install Rust and DPDK. Detailed instructions can be found in INSTALL.

Add $DPDK_PATH/lib/x86_64-linux-gnu to your LD_LIBRARY_PATH, where DPDK_PATH points to the DPDK installation directory.

Fork or clone the main git repository:

git clone [email protected]:stanford-esrg/retina.git

Write your first Retina application (see examples):

use retina_core::config::default_config;
use retina_core::subscription::TlsHandshake;
use retina_core::Runtime;
use retina_filtergen::filter;

#[filter("tls.sni ~ '^.*\\.com$'")]
fn main() {
    let cfg = default_config();
    let callback = |tls: TlsHandshake| {
        println!("{:?}", tls);
    };
    let mut runtime = Runtime::new(cfg, filter, callback).unwrap();
    runtime.run();
}

Build all examples in release mode:

cargo build --release

Run basic in release mode:

sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH RUST_LOG=error ./target/release/basic

Development

Build a single application in debug mode:

cargo build --bin my_app

Run in debug mode:

sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH RUST_LOG=debug ./target/debug/my_app

Filter expansion (requires Rust nightly):

cargo expand --manifest-path=examples/my_app/Cargo.toml

Contributing

Contributions welcome! Please run cargo fmt and cargo clippy before making a pull request.

Reproducibility

A Docker image is available to run Retina without the hassle of installing DPDK and other dependencies. It is, however, not suitable for performance testing as it uses the DPDK PCAP driver and is limited to a single core. The GitHub repository also includes a tutorial and a video to start learning about Retina.

A CloudLab image is available to reproduce a few of the experiments shown in the paper on the CloudLab public testbed. The repository also includes the scripts and information to reproduce these experiments on your own testbed.

Acknowledgements

Retina was developed with support from the National Science Foundation under award CNS-2124424, and through gifts from Google, Inc., Cisco Systems, Inc., and Comcast Corporation.

Comments
  • Program fails with Online config - Intel x710 with DPDK

    Program fails with Online config - Intel x710 with DPDK

    admin@AN450:~/retina$ sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH RUST_LOG=error ./target/release/client_randoms -c configs/online.toml Initializing Retina runtime... Ethdev port_id=0 invalid rss_hf: 0x3afbc, valid value: 0x7ef8 thread 'main' panicked at 'Failed to initialize port.: Failed to configure Port 0', /home/admin/retina/core/src/runtime/online.rs:69:14 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace [2022-10-12T18:28:17Z ERROR retina_core::port] Failed to close Port 0.

    opened by hassaan-ali-arista 7
  • cargo build --release Stuck

    cargo build --release Stuck

    Running this command takes forever:

    admin@AN450:~/retina/examples$ cargo build --release --bin basic
        Updating git repository `https://github.com/thegwan/pcap`
        Updating crates.io index
           Fetch [                         ]   0.72%, 1.56KiB/s   
    
    opened by hassaan-ali-arista 2
  • 3-way Handshake Scenaio

    3-way Handshake Scenaio

    Hi,

    I was trying to filter out the TCP packets involved in a 3-way handshake scenario. Are there any pre-built filters that I can utilize to capture the SYN, SYN-ACK and ACK packets?

    opened by hassaan-ali-arista 1
  • VDEV: Set default MTU to 65535

    VDEV: Set default MTU to 65535

    In general VDEVs are not able to change the MTU, and do not support scattering. So if GRO is enabled packets will be truncated and malformed.

    @thegwan is there a way to support truncated packets? It would be interesting as traces are often truncated too.

    opened by tbarbette 1
  • Allow to pass supplementary arguments and relax device constraints

    Allow to pass supplementary arguments and relax device constraints

    Add the dpdk_supl_args argument to pass arguments such as ["--no-huge","--no-pci"] allowing to try Retina in restricted environment, i.e. with Docker using PCAP virtual device (https://doc.dpdk.org/guides/nics/pcap_ring.html).

    Also, if RSS is not supported do not fail but warn the user.

    Similarly, if some statistics are not available, do not fail but warn the user.

    Verify that the MTU is in the device constraint. If setting it is not supported, do not fail but warn user.

    Verify the VLAN_STRIP offload is possible. If not, just don't activate it.

    opened by tbarbette 1
  • Install instr

    Install instr

    Added notes to INSTALL.md:

    • Can run Retina offline out of the box on a VM
    • Link to bug for meson build failure with DPDK 21.08
    • Basic changes needed for ARM-based VMs
    opened by thearossman 0
  • Spin example

    Spin example

    Reduces visibility of the DPDK module to crate-private. Adds an example to approximate callback complexity by busy looping for a given number of cycles.

    opened by thegwan 0
  • If mlx5 is not available the build fails

    If mlx5 is not available the build fails

    Compilation fails without mlx5 installed.

    I had to remove the default = ["mlx5"] line from core/cargo.toml. It seems --no-default-feature does not work because core is a dependency.

    opened by tbarbette 1
Owner
Stanford Security Research
Stanford Security Research
Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic

Nethoscope Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic. Check video on how it

Vesa Vilhonen 86 Dec 5, 2022
An experimental fork of a16z's Helios Ethereum client which can run its network traffic over the Nym mixnet

Helios (Nym mixnet fork) Helios is a fully trustless, efficient, and portable Ethereum light client written in Rust. This fork of Helios includes nasc

Nym 4 Mar 3, 2023
A multiplexed p2p network framework that supports custom protocols

Tentacle Overview This is a minimal implementation for a multiplexed p2p network based on yamux that supports mounting custom protocols. Architecture

漂流 188 Dec 19, 2022
Twinsies is a specialized reference-counting pointer where the item is jointly owned in 2 places

twinsies Twinsies is a special shared pointer, similar to an Arc, where two specific objects (called [Joint]) share joint ownership of the underlying

Nathan West 17 Feb 1, 2023
A specialized blockchain for testing use cases with the FRAME NFTs Pallet.

Substrate NFTs Node The Substrate NFTs node is a specialized blockchain for testing use cases with the FRAME NFTs Pallet. ?? The purpose of this node

Sacha Lansky 4 May 25, 2023
Prefix tree (ordered map and set) data structure using 100% safe Rust

PFX: A 100% safe, blob-oriented prefix tree This crate provides a prefix tree map and set data structure, implemented purely in safe Rust. The API is

Árpád Goretity  4 Apr 3, 2024
Cross-chain bridge message delivery network. We are hiring, [email protected]

Introduction Implementation of a https://darwinia.network node in Rust based on the Substrate framework. This repository contains runtimes for the Dar

Darwinia Network 225 Nov 8, 2022
ARYA Network is a polkadot/substrate based chain for Non-fungible Token platform on which we can own sell and buy the NFT's on polkadot network.

ARYA Network ARYA Network is a polkadot/substrate based chain for Non-fungible Token platform on which we can own sell and buy the NFT's on polkadot n

Pankaj Chaudhary 6 Dec 20, 2022
dWallet Network, a composable modular signature network is the home of dWallets

Welcome to dWallet Network dWallet Network, a composable modular signature network is the home of dWallets. A dWallet is a noncollusive and massively

dWallet Labs 8 Feb 26, 2024
A network application framework for Rust

This crate is deprecated! This crate is deprecated without an immediate replacement. Discussion about a successor can be found in tokio-rs/tokio#118.

Tokio 700 Dec 12, 2022
ROT Cipher implementation in rust, supports rot1 to rot25.

rotcipher-rs ROT Cipher implementation in rust, supports rot1 to rot25. Supported ROT ciphers ROT1 up to ROT25, it is also possible to make custom ROT

null 0 Sep 7, 2022
Quick poc of the rsync wire protocol in Rust. Supports delta transfer.

Rsync wire protocol in Rust This is a quick poc of the rsync wire protocol in Rust. It supports delta transfer. The code is really a mess right now, a

LightQuantum 4 Feb 18, 2023
Glommio Messaging Framework (GMF) is a high-performance RPC system designed to work with the Glommio framework.

Glommio Messaging Framework (GMF) The GMF library is a powerful and innovative framework developed for facilitating Remote Procedure Calls (RPCs) in R

Mohsen Zainalpour 29 Jun 13, 2023
Manager for single node Rancher clusters

Bovine Manage single node Rancher clusters with a single binary, bovine. % bovine run Pulling [rancher/rancher:latest], this may take awhile... Ranche

Nick Gerace 51 Feb 17, 2022
Yi Token by Crate Protocol: the primitive for auto-compounding single token staking pools.

yi Yi Token by Crate Protocol: the primitive for auto-compounding single token staking pools. About Yi is a Solana primitive for building single-sided

Crate Protocol 12 Apr 7, 2022
Most useful information about your system in a single command.

mymy Access the most common information about your system using a single command. Mymy is a command line tool that provides the most helpful informati

Théo Crevon 5 Apr 4, 2023
Single and multi-threaded custom ingestion crate for Stellar Futurenet, written in Rust.

rs-ingest Ingestion library written in rust for Futurenet rs-ingest Ingestion library written in rust for Futurenet Features Running offline Single-th

Xycloo Labs 3 Aug 14, 2023
The powerful analysis platform to explore and visualize data from blockchain.

Mars: The powerful analysis platform to explore and visualize data from Web3 Features Blazing Fast Create from scratch with Rust. Pipeline Processor E

DeepETH 66 Dec 17, 2022
Heimdall is an advanced Ethereum smart contract toolkit for forensic and heuristic analysis.

Heimdall is an advanced EVM toolkit which aims to make dealing with smart contracts on EVM based chains easier. Installation & Usage Heimdall's update

Jonathan Becker 489 Jan 2, 2023