Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux.

Overview

crates.io API documentation license

Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux.

When is this Rust crate useful?

  • Performance or latency is crucial, and
  • you run Linux.

A typical use case could be audio/video streaming.

Don't need maximum performance and minimum latency, and want a higher level protocol with serialization and lots of bells and whistles built-in? Try D-Bus.

Also, a unix socket is easier to set up and is not that much slower (see benchmark below).

As for Linux, this crate uses memfd sealing to ensure safety between untrusted processes, and ringbuffer signaling is done using eventfd for best performance. These two features are Linux only.

Getting started

You probably want to start in the sharedring module, which sets up a ringbuffer between untrusted processes (it's a wait-free/lock-free, bounded, SPSC queue). Another useful function is mem::write_once for a scenario where you write data once and make it available for reading afterwards. The mem and ringbuf modules contain building blocks that might be useful in other use cases.

The downside of using memfd based shared memory is that you need to set it up by transferring file descriptors, using some other way of communication. Using D-Bus would be the standard way of doing that - it's also possible using unix sockets.

There is also a client/server example in the examples directory that can help you get started. Enjoy!

Benchmark

Sharedring vs unix sockets vs D-Bus Notice the log scale: for a 64K packet, sharedring is a three times faster than unix sockets, and 60 times faster than D-Bus. (D-Bus is a higher level protocol, so that comparison is to some degree comparing apples and oranges.)

License

The code is Apache 2.0 / MIT dual licensed. Any code submitted in Pull Requests, discussions or issues is assumed to have this license, unless explicitly stated otherwise.

You might also like...
Utility that takes logs from anywhere and sends them to Telegram.

logram Utility that takes logs from anywhere and sends them to Telegram. Supports log collection from files, journald and docker containers. More abou

A safe and ergonomic alternative to rm

rip (Rm ImProved) rip is a command-line deletion tool focused on safety, ergonomics, and performance. It favors a simple interface, and does not imple

A simple, fast and user-friendly alternative to 'find'

fd [中文] [한국어] fd is a program to find entries in your filesytem. It is a simple, fast and user-friendly alternative to find. While it does not aim to

CLI utility to move (or rename) your files to a new location and redirect all of its symbolic links, to the new path

Move Links CLI utility to move (or rename) your files to a new location and redirect all of its symbolic links, to the new path (or name). Usage execu

A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.
A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.

[[TOC]] Comtrya This is better, yes? Warning This tool is ridiculously early in its development. I'm building out features as I migrate my dotfiles ov

A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.
A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.

Comtrya This is better, yes? Warning This tool is ridiculously early in its development. I'm building out features as I migrate my dotfiles over. Thos

Parallel finance a decentralized lending protocol built on top of the Polkadot ecosystem. Our unique approach will allow users to earn
Parallel finance a decentralized lending protocol built on top of the Polkadot ecosystem. Our unique approach will allow users to earn "double interests" from staking and lending their tokens simultaneously.

Parallel Finance A new Cumulus-based Substrate node, ready for hacking 🚀 Getting Started Follow these steps to get started with the Cumulus Template

idf-env tool helps set up and manage ESP-IDF installations

idf-env Tool for maintaining ESP-IDF environment. Quick start Install serial drivers for ESP boards on Windows. Execute following command in PowerShel

It is a backup tool that creates backups and stores them on an object storage
It is a backup tool that creates backups and stores them on an object storage

Hold My Backup It is a backup tool that creates backups and stores them on an object storage. By default it uses minio but you can use AWS: S3 as well

Comments
  • Is backing buffer dynamically resizable?

    Is backing buffer dynamically resizable?

    Hey there! First of all, great work!

    Right now I am playing around with shmem_ipc as a potential backend for my tiny RPC library. However, I am encountering a problem where after writing > CAPACITY via send_raw data on the other side becomes garbled and incomplete, failing my validation. It seems to me like buffer just gets completely saturated and stops the flow of data.

    Is there a way around it? Maybe I can somehow reset the pointer of this buffer to zero before I send each new message or something like that?

    Thanks!

    opened by skyne98 5
  • HugeTLB and mlock support?

    HugeTLB and mlock support?

    Hi, I'm trying to add hugetlb and mlock support for the shared memory region to further increase the performance at https://github.com/dovahcrow/shmem-ipc. I can create a PR if you want to make these changes into the main repo.

    ~Note that the implementation is not finished: I am stuck at using set_len to increase the size of the memfd (https://github.com/dovahcrow/shmem-ipc/blob/master/src/mem.rs#L81): it always returns EINVAL, and I haven't found a solution yet.~ By reading the Linux source code, it turned out hugetlb does not need set_len!

    Any suggestion is welcomed!

    opened by dovahcrow 3
  • Possible unexpected blocking behavior?

    Possible unexpected blocking behavior?

    The eventfds are only read in the block* method on the sender and receiver. It seems possible that writing to these eventfd will block if they are never read, once the counter in the eventfd reaches maximum

    A write(2) call adds the 8-byte integer value supplied in its buffer to the counter. The maximum value that may be stored in the counter is the largest unsigned 64-bit value minus 1 (i.e., 0xfffffffffffffffe). If the addition would cause the counter's value to exceed the maximum, then the write(2) either blocks until a read(2) is performed on the file descriptor, or fails with the error EAGAIN if the file descriptor has been made nonblocking.

    opened by dovahcrow 3
  • impl Sync for sharedring::{Receiver, Sender} ?

    impl Sync for sharedring::{Receiver, Sender} ?

    It seems to me that it is safe to implement Sync for Receiver and Sender because the recv and send methods require &mut which implies at most one reader and writer as https://docs.rs/shmem-ipc/latest/shmem_ipc/sharedring/struct.Sender.html#safety required.

    opened by dovahcrow 2
Owner
null
Basic system information fetcher, with a focus on performance.

Table of Contents: About Changelog Dependencies Benchmarks Features Installation Platform Support About Macchina Macchina lets you view basic system i

null 677 Dec 28, 2022
Everyday-use client-side map-aware Arch Linux mirror ranking tool

Rate Arch Mirrors This is a tool, which fetches mirrors, skips outdated/syncing Arch Linux mirrors, then uses info about submarine cables and internet

Nikita Almakov 196 Jan 2, 2023
Transform Linux Audit logs for SIEM usage

Linux Audit – Usable, Robust, Easy Logging TLDR: Instead of audit events that look like this… type=EXECVE msg=audit(1626611363.720:348501): argc=3 a0=

null 465 Jan 1, 2023
MILD - Minimal Install Linux Desktop

MILD - Minimal Install Linux Desktop MILD is a simple and straightforward text-mode installer that aims to install a "D.E."(Desktop Environment) with

Pedro Rosendo 3 Jul 23, 2022
A simple and fast download accelerator, written in Rust

zou A simple and fast download accelerator, written in Rust Zou is a Snatch fork by @k0pernicus. Snatch is a fast and interruptable download accelerat

Antonin Carette 173 Dec 4, 2022
A system handler to get information and interact with processes written in Rust

A system handler to get information and interact with processes written in Rust

Guillaume Gomez 1.1k Jan 3, 2023
OCI Runtime, Image and Distribution Spec in Rust

oci-spec-rs Open Container Initiative (OCI) Specifications for Rust This library provides a convenient way to interact with the specifications defined

Containers 119 Dec 29, 2022
desktop automation, screen capture, input listen/simulation, asar compress, color picker for nodejs and electron based on rust

desktop automation, screen capture, input listen/simulation, asar compress, color picker for nodejs and electron based on rust. 基于 Rust / WASM 提供截图、取色、键鼠事件监听模拟、压缩解压、图像处理、获取已安装应用等跨平台功能的现代异步 Nodejs 模块,占用空间小, 安装便捷, 使用简单, 高性能, 资源占用极小, 可取代 iohook 和 robotjs

null 48 Dec 15, 2022
GTK application for browsing and installing fonts from Google's font archive

Font Finder This project is a from-scratch implementation of TypeCatcher in Rust. It is a GTK3 application for browsing through and installing fonts f

Michael Murphy 252 Dec 26, 2022
A project for automatically generating and maintaining Debian repositories from a TOML spec.

Debian Repository Builder A simple utility for constructing and maintaining Debian repositories. Configuration of a repo is based on the directory hie

Pop!_OS 52 Feb 7, 2022