Network Block Storage server, written in Rust. Supports pluggable and chainable underlying storage

Overview

nbd-rs

Disclaimer

DO NEVER USE THIS FOR PRODUCTION

Do not use this for any data that you cannot afford to lose any moment. Expect data loss, corruption/bit rot, and every other possible storage disaster.

If you use alpha level software for your data, you might end up like we did previously => https://github.com/sheepdog/sheepdog/issues/425

Roadmap

  • Successfully serve dummy empty file (filled with zeroes), enough to satisfy qemu-img info
  • 0.0.1 Serve a raw image read-only from a file
    • consider mmap
  • 0.0.2 Read/Write access
    • Trim support (remove entire chunk object if trimmed)
  • 0.0.3 Storage Backend Abstraction
  • 0.0.4 Sharded File Implementation. Shard image file into 4M chunks (my-image.{n}.chunk)
  • 0.0.5 Object Storage Abstraction
  • 0.0.6 S3 Object Storage
  • 0.0.7 Cache Backend Implementation
  • Retry Backend Implementation
  • Trim Support
  • Transmission Phase Refactor. Move to async/queue
  • Refactor NBDSession. Leverage Interior Mutability.
  • Config/Argument syntax refactor.
  • Memory-based Object Storage
  • Test Suite
    • Unit Tests
    • Integration Tests
  • Known Issues
    • Fix panic at sudden disconnection
    • Fix panic at invalid export name

Stretch goals

  • Speed & BW optimizations, lazy disk-write, page-cache, etc.
  • Multi-volume support
  • Multi-connection support
  • Research Disconnect/Reconnection behavior
  • S3 backing support for shards
  • Stateless Multi-server support (S3 & Redis backend)
    • Redis: state cache, etc. S3: storage
  • Encryption

Build

cargo build

The executable binary is located at ./target/debug/nbd-rs.

Run

Arguments:

[-e | --export EXPORT_NAME; DRIVER (raw, sharded); (cache)? CONN_STR]...
  Examples:
    -e my_export raw cache:file:/test/
    -e my_export sharded cache:file:/test/
    -e my_export raw cache:s3:http://username:password@${S3_HOST}/path
    -e my_export raw s3:http://username:password@${S3_HOST}/path
    -e my_export sharded s3:http://username:password@${S3_HOST}/path

Examples of export argument:

# Single Export (-e | --export), Raw, File, Log Level: DEBUG
RUST_LOG=debug ./target/debug/nbd-rs --export my_raw_export raw file:/export/path/

# Single Export, Sharded, S3, Log Level: TRACE
RUST_LOG=trace ./target/debug/nbd-rs -e my_raw_export sharded s3:/export/path/

# Single Export with Cache, Sharded, S3
./target/debug/nbd-rs -e my_sharded_export sharded cache:s3:http://username:password@${S3_HOST}/path

# Multiple Exports
./target/debug/nbd-rs -e my_raw_export raw file:/export/path/ -e my_sharded_export sharded s3:http://username:password@${S3_HOST}/path

NBD-rs will panic if no export has been specified.

Container

docker build -t dkr.local/nbd-rs:dev .
docker run -it --rm -p 10809:10809 dkr.local/nbd-rs:dev --export ${EXPORT_NAME} raw file:/test/

See Run section for more information on arguments.

Test

qemu-img info nbd:localhost:10809;exportname=${EXPORT_NAME}

Write local image to NBD:

# Connect to drive by reading
nbd-client -N ${IMAGE_NAME} localhost /dev/nbd0

# Write
dd if=${LOCAL_IMAGE} of=/dev/nbd0 bs=1M status=progress oflag=sync,direct

# Disconnect
nbd-client -d /dev/nbd0

Boot alpine with qemu:

qemu-system-x86_64   -enable-kvm   -machine q35,accel=kvm   -m 2048  -drive file=nbd:127.0.0.1:10809:exportname=${EXPORT_NAME},format=raw   -display gtk   -serial mon:stdio

COPYING

GPL-3.0

Copyright 2021, Rainlab Inc. Tokyo

You might also like...
Jex Compiler Server - Server that runs Jex code

Server that compiles and runs Jex code.

QUIC proxy that allows to use QUIC to connect to an SSH server without needing to patch the client or the server.

quicssh-rs 😄 quicssh-rs is a QUIC proxy that allows to use QUIC to connect to an SSH server without needing to patch the client or the server. quicss

RDE1 (Rusty Data Exfiltrator) is client and server tool allowing auditor to extract files from DNS and HTTPS protocols written in Rust. 🦀
RDE1 (Rusty Data Exfiltrator) is client and server tool allowing auditor to extract files from DNS and HTTPS protocols written in Rust. 🦀

Information: RDE1 is an old personal project (end 2022) that I didn't continue development on. It's part of a list of projects that helped me to learn

An online version of the popular game four in a row, written in Rust on the server side and Flutter + Dart on the client.
An online version of the popular game four in a row, written in Rust on the server side and Flutter + Dart on the client.

Four in a Row - Server An online version of the popular game four in a row, written in Rust on the server side and Flutter + Dart on the client. Downl

The Safe Network Core. API message definitions, routing and nodes, client core api.

safe_network The Safe Network Core. API message definitions, routing and nodes, client core api. License This Safe Network repository is licensed unde

Astar Network is an interoperable blockchain based the Substrate framework and the hub for dApps within the Polkadot Ecosystem
Astar Network is an interoperable blockchain based the Substrate framework and the hub for dApps within the Polkadot Ecosystem

Astar Network is an interoperable blockchain based the Substrate framework and the hub for dApps within the Polkadot Ecosystem. With Astar Network and

Fast and easy-to-use event-driven network library.
Fast and easy-to-use event-driven network library.

message-io is a fast and easy-to-use event-driven network library. The library handles the OS socket internally and offers a simple event message API

A decentralized, censorship-resistant, and incentive-compatible packet-routing overlay network

About Earendil is a decentralized, censorship-resistant packet-routing overlay network designed for performance and censorship resistance. It enables

Filen.io is a cloud storage provider with an open-source desktop client.

Library to call Filen.io API from Rust Filen.io is a cloud storage provider with an open-source desktop client. My goal is to write a library which ca

Releases(v0.1.0)
Owner
Rainlab Inc
Engineering Company based in Tokyo. Mostly nerds.
Rainlab Inc
An HTTP server wrapper for omnisette. Supports both V1 (Provision) and V3 of anisette servers.

omnisette-server An HTTP server wrapper for omnisette. Supports both V1 (Provision) and V3 of anisette servers. Setup First, download the Apple Music

SideStore Team 5 Mar 29, 2023
Docker containers on a synthetic network. Run applications in a context that lets you manipulate their network conditions.

Synthetic Network Docker containers on a synthetic network. Run applications in a context that lets you manipulate their network conditions. Dependenc

Daily 58 Dec 15, 2022
DNS Server written in Rust for fun, see https://dev.to/xfbs/writing-a-dns-server-in-rust-1gpn

DNS Fun Ever wondered how you can write a DNS server in Rust? No? Well, too bad, I'm telling you anyways. But don't worry, this is going to be a fun o

Patrick Elsen 26 Jan 13, 2023
A fast, stable, efficient, and lightweight intranet penetration, port forwarding tool supports multiple connections, cascading proxy, and transmission encryption

A fast, stable, efficient, and lightweight intranet penetration, port forwarding tool supports multiple connections, cascading proxy, and transmission encryption

editso 1.3k Dec 30, 2022
A simple web server(and library) to display server stats over HTTP and Websockets/SSE or stream it to other systems.

x-server-stats A simple web server(and library) to display server stats over HTTP and Websockets/SSE or stream it to other systems. x-server(in x-serv

Pratyaksh 11 Oct 17, 2022
A multi-targets ping tool and library, which supports 10,000 packets/second, accurate latency

mping-rs a multi-targets ping tool, which supports 10,000 packets/second, accurate latency. 一个高频ping工具,支持多个目标。 正常的ping一般用来做探测工具,mping还可以用来做压测工具。 Go版本:

smallnest 25 Oct 29, 2023
Bevy asset loader that transparently supports loading over http(s)

Bevy Web Asset This is a tiny crate that that wraps the standard bevy asset loader, and adds the ability to load assets from http and https urls. Supp

Johan Klokkhammer Helsing 28 Jan 2, 2023
🔍 Fully-featured metrics collection agent for First Tech Challenge competitions. Supports Prometheus.

Scout Scout is a fully-featured free and open source metrics collector for FTC competitions. The project is licensed under the GNU LGPLv3 license. Fea

hivemind 3 Oct 24, 2023
Dav-server-rs - Rust WebDAV server library. A fork of the webdav-handler crate.

dav-server-rs A fork of the webdav-handler-rs project. Generic async HTTP/Webdav handler Webdav (RFC4918) is defined as HTTP (GET/HEAD/PUT/DELETE) plu

messense 30 Dec 29, 2022
axum-server is a hyper server implementation designed to be used with axum framework.

axum-server axum-server is a hyper server implementation designed to be used with axum framework. Features Conveniently bind to any number of addresse

null 79 Jan 4, 2023