Rustus - TUS protocol implementation in Rust.

Overview

Rustus

Tus protocol implementation written in Rust.

Features

This implementation has several features to make usage as simple as possible.

  • Rustus is robust, since it uses asynchronous Rust;
  • It can store information about files in databases;
  • You can specify directory structure to organize your uploads;
  • Highly configurable;

Supported info storages

  • FileSystem
  • PostgresSQL
  • Mysql
  • SQLite
  • Redis

Supported data storages

  • FileSystem

Installation

Since I haven't configured build automation yet, you can build it from source using cargo.

cargo install --path .

Or you can build a docker image.

docker build --tag="rustus:latest" --cache-from=s3rius/tuser:latest -f deploy/Dockerfile .

Docker image and binaries will be available soon.

Architecture

Files and info about them are separated from each other. In order to modify original file rustus searches for information about the file in information storage.

However, automatic migration between different information storages is not supported.

Configuration

You can configure rustus via command line or environment variables. All options are listed in rustus --help.

Roadmap

  • Data storage interface;
  • Info storage interface;
  • Core TUS protocol;
  • Extensions interface;
  • Creation extension;
  • Creation-defer-length extension;
  • Creation-with-upload extension;
  • Termination extension;
  • Route to get uploaded files;
  • Database support for info storage;
  • Redis support for info storage;
  • Notification interface;
  • Notifications via http hooks;
  • Notifications via RabbitMQ;
  • S3 as data storage store support;
  • Executable files notifications;
  • Rustus helm chart;
  • Cloud native rustus operator.
Comments
  • CORS Support

    CORS Support

    In working up a POC with rustus and uppy, uppy is unable to upload directly to rustus because of a CORS error. scanning through the code, it appears that rustus doesn't have native cors handling, is that correct?

    opened by Kaelten 27
  • Healthcheck URL 404s

    Healthcheck URL 404s

    When working towards deployments I went and checked the /health route and at least for me locally 0.5.6 is return a 404

    Am I misreading the code on how I should query it?

    ❯ curl localhost:1081/health -v
    *   Trying 127.0.0.1:1081...
    * Connected to localhost (127.0.0.1) port 1081 (#0)
    > GET /health HTTP/1.1
    > Host: localhost:1081
    > User-Agent: curl/7.79.1
    > Accept: */*
    >
    * Mark bundle as not supporting multiuse
    < HTTP/1.1 404 Not Found
    < content-length: 0
    < tus-resumable: 1.0.0
    < tus-version: 1.0.0
    < access-control-expose-headers: tus-extension, upload-concat, location, tus-version, tus-checksum-algorithm, tus-resumable, upload-offset, content-length, upload-length, tus-max-size, content-type, upload-metadata, upload-defer-length
    < vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
    < date: Fri, 22 Jul 2022 23:15:46 GMT
    
    opened by Kaelten 5
  • Unexpected hook behaviors.

    Unexpected hook behaviors.

    I've been testing hooks for the last little while and there's a few things that are not behaving as I expected.

    I'm using the webhook settings with all hooks enabled and in both of the below log segments I'm logging the additional info with this line.

    f'    {payload.upload.id}: hook {hook_name}; method {payload.request.method}; is_final: {payload.upload.is_final}; is_partial: {payload.upload.is_partial};'
    

    When doing a simple upload without concat I see the following logs:

        8f40fed6-4431-4ee3-8506-2e08871af41d: hook pre-create; method POST; is_final: False; is_partial: False;
    INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
        8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-create; method POST; is_final: False; is_partial: False;
    INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
        8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-receive; method PATCH; is_final: False; is_partial: False;
    INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
        8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-finish; method PATCH; is_final: False; is_partial: False;
    INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
    

    When doing an upload using concatenation of two parts I see this as the logs:

        dad05558-2c08-4250-aaea-8238736c7ebb: hook pre-create; method POST; is_final: False; is_partial: True;
        46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook pre-create; method POST; is_final: False; is_partial: True;
    INFO:     127.0.0.1:62958 - "POST /rustus/callback HTTP/1.1" 200 OK
    INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
        dad05558-2c08-4250-aaea-8238736c7ebb: hook post-create; method POST; is_final: False; is_partial: True;
    INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
        46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook post-create; method POST; is_final: False; is_partial: True;
    INFO:     127.0.0.1:62960 - "POST /rustus/callback HTTP/1.1" 200 OK
        dad05558-2c08-4250-aaea-8238736c7ebb: hook post-finish; method PATCH; is_final: False; is_partial: True;
        46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook post-finish; method PATCH; is_final: False; is_partial: True;
    INFO:     127.0.0.1:62960 - "POST /rustus/callback HTTP/1.1" 200 OK
    INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
        cf318548-2c74-4a4f-bf02-c528f248e9d5: hook pre-create; method POST; is_final: True; is_partial: False;
    INFO:     127.0.0.1:62962 - "POST /rustus/callback HTTP/1.1" 200 OK
        cf318548-2c74-4a4f-bf02-c528f248e9d5: hook post-create; method POST; is_final: True; is_partial: False;
    INFO:     127.0.0.1:62962 - "POST /rustus/callback HTTP/1.1" 200 OK
    

    A few things that surprised me:

    • None of the hooks for the simple upload are labeled as is_final
    • There is only a pre-create and post-create event for the concatenation call.
    • Only the concat related events are set as is_final
    • There's no post-receive events for partial uploads

    This leads me to the following thoughts, and also wondering if these are correct assumptions:

    1. Fragmented uploads (concat enabled) are identifiable by looking at the is_partial flag.
    2. To understand upload progress I'd have to look at the offset/size information provided through post_receive hooks.
    3. Since concat enabled uploads don't get post_receive hooks I can't tell what's going on with them as they upload.
    4. To understand when a upload is finished I have to look for post-create hooks that have is_final: True as well as post-finish hooks.

    Is all of the above correct? Are any of the above pointing to small bugs or logic errors in how the hooks are triggered?

    opened by Kaelten 5
  • Concatenation extension regression in 0.5.3

    Concatenation extension regression in 0.5.3

    When uploading with the concatenation extension against 0.5.2 things work as expected. However in 0.5.3 the combined file stays 0 bytes and the parts are not removed. I've enabled debug logging, and there's nothing in the logs to indicate a failure or issue.

    rustus  | [2022-06-23][00:40:05+00:00][INFO] "PATCH /60a7d100-2b2b-447b-8c48-c080f0b4e6da/ HTTP/1.1" "-" "204" "172.19.0.1" "302.296001"
    rustus  | [2022-06-23][00:40:05+00:00][INFO] "PATCH /1009fb2b-263e-4cea-9019-d8a879155417/ HTTP/1.1" "-" "204" "172.19.0.1" "307.389292"
    rustus  | [2022-06-23][00:40:05+00:00][INFO] "POST / HTTP/1.1" "-" "201" "172.19.0.1" "5.008000"
    
    opened by Kaelten 4
  • Post-finish webhook fails to fire with a single-chunk upload and `creation-with-upload` enabled.

    Post-finish webhook fails to fire with a single-chunk upload and `creation-with-upload` enabled.

    When enabling creation-with-upload for an upload that is small enough to finish as part of the upload creation step, the post-finish webhook does not fire.

    I successfully see the post-finish webhook when the file requires multiple chunks to upload or when I disable creation-with-upload.

    opened by Kaelten 3
  • Architecture improvements

    Architecture improvements

    At this point project is released, but we have a poor architecture decisions that were made to develop the application fast.

    At first we must separate InfoStorage and Storage from each other. Because today FileStorage makes way more things than just storing data on a file system.

    The main idea is to rebuild Storage trait that would use FileInfo structure instead of a file_id string and remove get_file_info method. Also we have to save InfoStorage as a part of the actix state in order to use it in controllers. It gives us more flexibility and makes it easy to develop new storage types.

    opened by s3rius 3
  • Extra path separator when using `RUSTUS_URL=/`

    Extra path separator when using `RUSTUS_URL=/`

    When I use '/' for the path instead of /files things work, but the path that is generated for the files contain an extra /. Here's a log snippet that shows the issue.

    rustus  | [2022-06-22][01:03:13+00:00][INFO] "POST / HTTP/1.1" "-" "201" "172.19.0.1" "66.578333"
    rustus  | [2022-06-22][01:03:14+00:00][INFO] "PATCH //10ac0e0e-d67d-4e2c-880b-1346793008b5 HTTP/1.1" "-" "204" "172.19.0.1" "263.382625"
    

    In these situations I've found that using a url join library often is the cleanest way to eliminate duplications and ensure urls are built properly. Failing that doing a regex replace so that /+ is replaced with / works fairly well as well.

    opened by Kaelten 2
  • Add pre-terminate hook

    Add pre-terminate hook

    It's would be super nice if rustus will run pre-terminate hook an it would be blocking. Because now you can delete files freely without verifying authorization.

    opened by s3rius 2
  • Add prometheus metrics

    Add prometheus metrics

    It would be really nice to add prometheus metrics to rustus. In this issue we discuss which metrics we want to see on /metrics page.

    They can be later used in frontend page.

    opened by s3rius 2
  • LRU cache for files

    LRU cache for files

    Maybe it'll be a great idea to have some kind of LRU cache to hold recently created/written files. That way we'll skip file opening in each request if we touched specified files recently

    opened by Kolaer 2
  • Add pre-allocate feature to file storage

    Add pre-allocate feature to file storage

    Added simple pre-allocate functionality, should be enough for simpler cases but needs testing for corner cases.

    Also added a couple of tests just for the fun of it.


    Closes #23

    opened by Kolaer 2
  • Request: Option to disable health check access logging

    Request: Option to disable health check access logging

    When I deploy services, I prefer to filter out the logging of the health check endpoint since these are so often spammy and not representative of actual performance or issues. Having a toggle in Rustus to enable/disable logging for that one endpoint would be great.

    opened by Kaelten 1
  • Webhook Documentation Suggestions

    Webhook Documentation Suggestions

    While understanding hooks I ran across a few thoughts.

    1. The documentation provided for the schema doesn't include the variations of possible values/types depending on when the hook is triggered.
    2. The fastapi example could benefit from providing the pydantic models needed to parse the api.
    3. It strikes me as odd that only URI is uppercase, guessing this is an encoding artifact based on how rust's serialization works
    4. I was a little surprised that the hook name wasn't available in the payload, but not a huge deal either way.

    To help address the first two thoughts, here is the schema I've reversed engineered from watching the hooks over several types of uploads. I'm not sure I've captured all the variations yet.

    import datetime
    import uuid
    
    from pydantic import BaseModel, conint, Field
    
    class UploadInfo(BaseModel):
        id: uuid.UUID
        offset: conint(ge=0)
        length: conint(ge=0) | None
        path: str | None
        created_at: datetime.datetime
        deferred_size: bool
        is_partial: bool
        is_final: bool
        parts: list[uuid.UUID] | None
        storage: str
        metadata: dict
    
    
    class RequestInfo(BaseModel):
        uri: str = Field(alias='URI')
        method: str
        remote_addr: str
        headers: dict
    
    
    class RustusPayload(BaseModel):
        upload: UploadInfo
        request: RequestInfo
    
    

    I'd also suggest providing example payloads for each event type that covers the variations or at least document when a field can be in different states. i.e. parts is None unless it's a concat finalization in which it gives the ids of the other parts.

    opened by Kaelten 7
  • Bump lapin from 1.10.0 to 2.1.1

    Bump lapin from 1.10.0 to 2.1.1

    Bumps lapin from 1.10.0 to 2.1.1.

    Changelog

    Sourced from lapin's changelog.

    2.1.1 (2022-04-08)

    Bug Fixes

    Fix potential race condition on connection error

    2.1.0 (2022-03-23)

    Bug Fixes

    Fix potential race condition on connection opening

    2.0.3 (2022-02-25)

    Misc

    • Update amq-protocol

    2.0.2 (2022-02-23)

    Bug Fixes

    • Fix some potential hang with rustls

    2.0.1 (2022-02-08)

    Misc

    • Update README

    2.0.0 (2022-02-02)

    Misc

    • Make most of the API based on async fns
    • Update to amq-protocol 7.0
    • Reworked connector
    • Dropped third-party reactors/executors integration
    • New third-party reactors integration through reactor-trait
    • New third-party executors integration through executor-trait
    • Misc internal cleanups
    • Default reactor to async-io
    • Switch from log to tracing
    • Make most internals use async rust
    • Switch to edition 2021
    • Default to rustls for TLS handling
    • basic_publish no longer takes ownership of the data
    • Dropped Clone impl where it didn't make sense
    • Acker is now public
    Commits
    • 203dff0 v2.1.1
    • 38d0fc9 avoid race condition on connection error
    • 1317dbb v2.1.0
    • a51b5a3 connection: properly update ConnectionStep before sending frames
    • 1fc8e47 log more information on connection state error
    • 1393f27 examples: reformat tokio imports
    • 258edb1 properly notify connection waiter when there was an error while connection
    • 4911142 Add an example using Tokio runtime (#362)
    • 68359bb v2.0.3
    • 635004f update amq-protocol and regen code
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • Bump tokio-amqp from 1.1.0 to 2.0.0

    Bump tokio-amqp from 1.1.0 to 2.0.0

    Bumps tokio-amqp from 1.1.0 to 2.0.0.

    Changelog

    Sourced from tokio-amqp's changelog.

    2.0.0 (2022-02-02)

    Misc

    • Make most of the API based on async fns
    • Update to amq-protocol 7.0
    • Reworked connector
    • Dropped third-party reactors/executors integration
    • New third-party reactors integration through reactor-trait
    • New third-party executors integration through executor-trait
    • Misc internal cleanups
    • Default reactor to async-io
    • Switch from log to tracing
    • Make most internals use async rust
    • Switch to edition 2021
    • Default to rustls for TLS handling
    • basic_publish no longer takes ownership of the data
    • Dropped Clone impl where it didn't make sense
    • Acker is now public

    1.10.0 (2022-01-30)

    Bug Fixes

    • Fix potential panic on shutdown

    1.9.0 (2021-11-23)

    Bug Fixes

    • Fix handling of RabbitMQ replies when they arrive unordered

    1.8.1 (2021-11-12)

    Misc

    • Logging updates

    1.8.0 (2021-08-22)

    Misc

    • Update to nom 7

    1.7.1 (2021-05-06)

    Bug Fixes

    • Fix handling of unresponsive servers

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
Releases(0.5.14)
Owner
Pavel Kirilin
Web developer, Linux fan, Tech enthusiast.
Pavel Kirilin
RakNet Protocol implementation by Rust.

rust-raknet RakNet Protocol implementation by Rust. Raknet is a reliable udp transport protocol that is often used for communication between game clie

b23r0 152 Nov 21, 2022
🥧 Savoury implementation of the QUIC transport protocol and HTTP/3

quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF. It provides a low level API for processing QUIC packet

Cloudflare 6.9k Nov 27, 2022
Easy protocol definitions in Rust

protocol Documentation Easy protocol definitions in Rust. This crate adds a custom derive that can be added to types, allowing structured data to be s

Dylan McKay 150 Nov 5, 2022
A Constrained Application Protocol(CoAP) library implemented in Rust.

coap-rs A fast and stable Constrained Application Protocol(CoAP) library implemented in Rust. Features: CoAP core protocol RFC 7252 CoAP Observe optio

Covertness 168 Nov 14, 2022
A Rust library for parsing the SOME/IP network protocol (without payload interpretation).

someip_parse A Rust library for parsing the SOME/IP network protocol (without payload interpretation). Usage Add the following to your Cargo.toml: [de

Julian Schmid 18 Oct 31, 2022
Peer-to-peer communications library for Rust based on QUIC protocol

qp2p Crate Documentation MaidSafe website SAFE Dev Forum SAFE Network Forum Overview This library provides an API to simplify common tasks when creati

MaidSafe 328 Nov 25, 2022
A simple tool in Rust to split urls in their protocol, host, port, path and query parts.

rexturl A simple tool to split urls in their protocol, host, port, path and query parts. Install cargo install rexturl or clone the source code and r

Volker Schwaberow 3 Oct 22, 2022
🤖 brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers

brwrs is a new protocol running over TCP/IP that is intended to be a suitable candidate for terminal-only servers (plain text data). That is, although it can be accessed from a browser, brwrs will not correctly interpret the browser's GET request.

daCoUSB 3 Jul 30, 2021
A multi-protocol network relay

A multi-protocol network relay

zephyr 42 Oct 23, 2022
The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL.

Graph Node The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL. Graph Node is an open s

Mindy.wang 2 Jun 18, 2022
🤖 Autonomous Twitter bot that posts analytics of the APYs available on the Solend Protocol.

Solend APY Twitter Bot Solana Ignition Hackathon 2021 View Demo · Report Bug · Request Feature Table of Contents About The Project Motivation Challeng

Manuel Gil 4 Sep 23, 2022
Yet Another File Transfer Protocol.

yaftp Yet Another File Transfer Protocol. Build & Run $> cargo build --release Features C2C Lightweight Per something per session High performence Res

b23r0 20 Sep 22, 2022
An End-to-End Privacy Computing Protocol on Layer 2

Eigen Network Eigen Network is an end-to-end privacy computation network for a better digital economy based on hybrid privacy computation protocols an

Eigen Lab 24 Oct 13, 2022
Shade Protocol is an array of connected privacy-preserving dApps built on Secret Network

Shade Protocol Core Contracts Contract Reference Description mint doc Handles asset burning and silk minting oracle doc Handles asset price queries tr

Secure Secrets 58 Nov 15, 2022
IDP2P is a peer-to-peer identity protocol which enables a controller to create, manage and share its own proofs as well as did documents

IDP2P Experimental, inspired by ipfs, did:peer and keri Background See also (related topics): Decentralized Identifiers (DIDs) Verifiable Credentials

null 5 Oct 31, 2022
UDP proxy with Proxy Protocol and mmproxy support

udppp UDP proxy with Proxy Protocol and mmproxy support. Features Async Support Proxy Protocol V2 SOCKET preserve client IP addresses in L7 proxies(mm

b23r0 9 Nov 22, 2022
Off-chain services for Gnosis Protocol v2

Cow Protocol Services This repository contains backend code for Cow Protocol Services written in Rust. Order Book The orderbook crate provides the htt

CoW Protocol 40 Nov 28, 2022
Bioyino is a distributed statsd-protocol server with carbon backend.

Bioyino The StatsD server written in Rust Description Bioyino is a distributed statsd-protocol server with carbon backend. Features all basic metric t

avito.tech 209 Nov 8, 2022
Futures-based QUIC implementation in Rust

Pure-rust QUIC protocol implementation Quinn is a pure-rust, future-based implementation of the QUIC transport protocol undergoing standardization by

null 2.4k Nov 27, 2022