s3d is an S3 daemon for the Edge written in Rust

Related tags

Miscellaneous s3d
Overview


s3d is an S3 daemon for the Edge written in Rust

The goal of s3d is to provide a daemon for edge platforms (e.g. factory servers 🏭 planes πŸ›© ships 🚒 cars 🏎 laptops πŸ’» mobile devices πŸ“± wearables ⌚ ...) that connect to a central object storage (aka Hub).

Since edge applications operate in subideal environments (high latency network, disconnections, hardware failures, eavesdropping, ...), s3d aims to make the applications run smoothly while it handles the flows of data, security, networking, local capacity, metadata caching, write queueing, etc.

Worth mentioning that the choice of the Rust language is a natural fit for the edge systems, as it is a modern language with a focus on functionality, safety and performance. s3d builds with the rust toolchain into a single binary that loads a yaml config file, which makes it easy to set up and configure in standalone linux or containerized environments like Kubernetes.

Info

This project is still πŸ›Έ πŸ›Έ πŸ›Έ Experimental πŸš€ πŸš€ πŸš€

This means it's a great time to affect its direction!

If you find it interesting or want to contribute, please feel free to communicate using these options:

Docs

Quick start

To start using s3d you need to install it, run it, and configure your data workflows.

The following steps illustrate a basic flow of s3d usage:

# Install the s3d binary using Rust toolchain
cargo install s3d

# Sets up the configuration, hub connection, and local storage
s3d init

# Run the daemon to serve S3 clients can connect to read and write objects
# Start and stop will spawn/kill the daemon in the background
s3d run
s3d start
s3d stop

# Fetch reads metadata from the hub and stores locally
# This includes the list of objects and their metadata, but excludes objects contents
s3d fetch [bucket/prefix]

# Pull is like fetch but includes objects contents.
s3d pull [bucket/prefix]

# Simple access to objects from the CLI
s3d get bucket/key > file
s3d put bucket/key < file
s3d ls [bucket/prefix]

# Show local changes not pushed to the hub
s3d diff [bucket/prefix]

# Show bucket status and local store stats (objects, sizes, etc).
s3d status [bucket/prefix]

# Push bucket changes (merge by last modified time).
s3d push [bucket/prefix]

# Remove objects from local store based on age
s3d prune [bucket/prefix]
You might also like...
A gui api explorer written in Rust.

Zzz - GUI Api platform Pronounced "Zees"; as in "catching some Z's". A pun on RESTful APIs. example URL: https://jsonplaceholder.typicode.com/todos/ T

Cross-platform GUI written in Rust using ADB to debloat non-rooted android devices
Cross-platform GUI written in Rust using ADB to debloat non-rooted android devices

Cross-platform GUI written in Rust using ADB to debloat non-rooted android devices. Improve your privacy, the security and battery life of your device.

A variation of the solana helloworld program example with a client written in Rust instead of Typescript

Simple Solana Smart Contract Example This repository demonstrates how to create and invoke a program on the Solana blockchain. In Solana the word prog

A powerful minecraft bedrock software written in Rust with a powerful Typescript plugin API.
A powerful minecraft bedrock software written in Rust with a powerful Typescript plugin API.

Netrex A powerful minecraft bedrock software written in RustLang. Why Netrex? It's written in Rust. Unique and straight to the point. Typescript Plugi

Unofficial Bitwarden compatible server written in Rust, formerly known as bitwarden_rs

Alternative implementation of the Bitwarden server API written in Rust and compatible with upstream Bitwarden clients*, perfect for self-hosted deploy

oreboot is a fork of coreboot, with C removed, written in Rust.

oreboot is a downstream fork of coreboot, i.e. oreboot is coreboot without 'c'.

TAT agent is an agent written in Rust, which run in CVM or Lighthouse instances.

TAT agent is an agent written in Rust, which run in CVM, Lighthouse or CPM 2.0 instances. Its role is to run commands remotely without ssh login, invoked from TencentCloud Console/API. Commands include but not limited to: Shell, PowerShell, Python. TAT stands for TencentCloud Automation Tools. See more info at https://cloud.tencent.com/product/tat.

Brainfuck interpreter written in rust

Brainfuck Interpreter Written in Rust Simple Interpreter that runs bare Brainfuck and extends it with debug command # which prints content of first te

DWARF packaging utility, written in Rust, supporting GNU extension and DWARF 5 package formats.

thorin thorin is an DWARF packaging utility for creating DWARF packages (*.dwp files) out of input DWARF objects (*.dwo files; or *.o files with .dwo

Comments
  • Wish: expose serve() & co for testing

    Wish: expose serve() & co for testing

    After seeing your comment in https://github.com/datenlord/s3-server/issues/108#issuecomment-1060436328 and given that I'm getting cargo audit errors by using datenlord/s3-server on my CI, which pulls Rusoto (deprecated), I've considered switching to your crate to substitute a test that is using s3-server from @datenlord.

    All I'd need is to have the serve() method you have in server.rs and perhaps some accessory ones to just get/put some example file(s) on my small testsuite, but currently the crate only seems to be available as a standalone binary instead of lib:

    % cargo build
        Updating crates.io index
    warning: s3d-test-example v0.1.0 (/Users/rvalls/dev/umccr/s3d-test-example) ignoring invalid dependency `s3d` which is missing a lib target
       Compiling s3d-test-example v0.1.0 (/Users/rvalls/dev/umccr/s3d-test-example)
        Finished dev [unoptimized + debuginfo] target(s) in 1.36s
    
    opened by brainstorm 0
  • Automate release flow with attached prebuilt binary

    Automate release flow with attached prebuilt binary

    For example - https://github.com/redhat-et/microshift/blob/main/.github/workflows/nightly.yaml

    Some related options to consider:

    • https://github.com/marketplace/actions/release-drafter
    • https://github.com/marketplace/actions/release-changelog-builder
    • https://trstringer.com/github-actions-create-release-upload-artifacts/
    opened by guymguym 1
  • Build Failed When Running `make`

    Build Failed When Running `make`

    The build failed when running make on my local machine, the error seems to have originated from cargo, but adding --stacktrace in the arguments failed to produce a more useful output.

    > Task :rust-runtime:fixManifests FAILED
    
    FAILURE: Build failed with an exception.
    
    * What went wrong:
    Execution failed for task ':rust-runtime:fixManifests'.
    > A problem occurred starting process 'command 'cargo''
    

    Cargo version: cargo 1.59.0 (49d8809dc 2022-02-10)

    opened by RobotSail 2
Releases(v0.0.1)
Wallpaper daemon for Wayland

wpaperd wpaperd is a minimal wallpaper daemon for Wayland. It allows the user to choose a different image for each output (aka for each monitor) just

Danilo Spinella 53 Dec 28, 2022
A highly modular Bitcoin Lightning library written in Rust. Its Rust-Lightning, not Rusty's Lightning!

Rust-Lightning is a Bitcoin Lightning library written in Rust. The main crate, lightning, does not handle networking, persistence, or any other I/O. Thus, it is runtime-agnostic, but users must implement basic networking logic, chain interactions, and disk storage. More information is available in the About section.

Lightning Dev Kit 850 Jan 3, 2023
A language server for lua written in rust

lua-analyzer lua-analyzer is a lsp server for lua. This is mostly for me to learn the lsp protocol and language analysis so suggestions are helpful. T

null 61 Dec 11, 2022
Opensource diagnostic software for Daimler vehicles, inspired by Xentry and DAS, written in Rust

OPENSTAR An opensource diagnostic application for Daimler vehicles inspired by DAS and Xentry. Some of the work here is based on OpenVehicleDiag If yo

Ashcon Mohseninia 21 Nov 20, 2022
Benchmarking web frameworks written in rust with rewrk tool.

Web Framework Benchmarks Benchmarking web frameworks written in rust with rewrk tool.

null 103 Dec 8, 2022
Yet another ROS2 client library written in Rust

RclRust Target CI Status Document Foxy (Ubuntu 20.04) Introduction This is yet another ROS2 client library written in Rust. I have implemented it inde

rclrust 42 Dec 1, 2022
A little bit fast and modern Ruby version manager written in Rust

A little bit fast and modern Ruby version manager written in Rust Features Pure Rust implementation not using ruby-build Cross-platform support (macOS

Takayuki Maeda 510 Jan 5, 2023
Music bot written in Rust

Akasuki What is Akasuki? Akasuki is a simple discord music bot written in rust. Highlights Select your music using discord's new select menu feature,

Forbidden A 0 Dec 19, 2021
A Discord bot for sending GeoGuessr challenge links that uses the GeoGuessr API written in rust.

GeoGuessr-bot-rs This is a simple implementation of a discord bot that send GeoGuessr-challenge links on demand. Features: Slash-commands Lightning-fa

Luukas PΓΆrtfors 6 Nov 18, 2022
Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust.

led_matrix_zmq Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust. This repository includes: Rust client and server l

Dan 2 Sep 6, 2022