In working up a POC with rustus and uppy, uppy is unable to upload directly to rustus because of a CORS error. scanning through the code, it appears that rustus doesn't have native cors handling, is that correct?

When working towards deployments I went and checked the /health route and at least for me locally 0.5.6 is return a 404

Am I misreading the code on how I should query it?

❯ curl localhost:1081/health -v
*   Trying 127.0.0.1:1081...
* Connected to localhost (127.0.0.1) port 1081 (#0)
> GET /health HTTP/1.1
> Host: localhost:1081
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 0
< tus-resumable: 1.0.0
< tus-version: 1.0.0
< access-control-expose-headers: tus-extension, upload-concat, location, tus-version, tus-checksum-algorithm, tus-resumable, upload-offset, content-length, upload-length, tus-max-size, content-type, upload-metadata, upload-defer-length
< vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
< date: Fri, 22 Jul 2022 23:15:46 GMT

I've been testing hooks for the last little while and there's a few things that are not behaving as I expected.

I'm using the webhook settings with all hooks enabled and in both of the below log segments I'm logging the additional info with this line.

f'    {payload.upload.id}: hook {hook_name}; method {payload.request.method}; is_final: {payload.upload.is_final}; is_partial: {payload.upload.is_partial};'

When doing a simple upload without concat I see the following logs:

    8f40fed6-4431-4ee3-8506-2e08871af41d: hook pre-create; method POST; is_final: False; is_partial: False;
INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
    8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-create; method POST; is_final: False; is_partial: False;
INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
    8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-receive; method PATCH; is_final: False; is_partial: False;
INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK
    8f40fed6-4431-4ee3-8506-2e08871af41d: hook post-finish; method PATCH; is_final: False; is_partial: False;
INFO:     127.0.0.1:63439 - "POST /rustus/callback HTTP/1.1" 200 OK

When doing an upload using concatenation of two parts I see this as the logs:

    dad05558-2c08-4250-aaea-8238736c7ebb: hook pre-create; method POST; is_final: False; is_partial: True;
    46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook pre-create; method POST; is_final: False; is_partial: True;
INFO:     127.0.0.1:62958 - "POST /rustus/callback HTTP/1.1" 200 OK
INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
    dad05558-2c08-4250-aaea-8238736c7ebb: hook post-create; method POST; is_final: False; is_partial: True;
INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
    46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook post-create; method POST; is_final: False; is_partial: True;
INFO:     127.0.0.1:62960 - "POST /rustus/callback HTTP/1.1" 200 OK
    dad05558-2c08-4250-aaea-8238736c7ebb: hook post-finish; method PATCH; is_final: False; is_partial: True;
    46c25c7d-6bfe-4486-a07e-9210cbd6f3d2: hook post-finish; method PATCH; is_final: False; is_partial: True;
INFO:     127.0.0.1:62960 - "POST /rustus/callback HTTP/1.1" 200 OK
INFO:     127.0.0.1:62959 - "POST /rustus/callback HTTP/1.1" 200 OK
    cf318548-2c74-4a4f-bf02-c528f248e9d5: hook pre-create; method POST; is_final: True; is_partial: False;
INFO:     127.0.0.1:62962 - "POST /rustus/callback HTTP/1.1" 200 OK
    cf318548-2c74-4a4f-bf02-c528f248e9d5: hook post-create; method POST; is_final: True; is_partial: False;
INFO:     127.0.0.1:62962 - "POST /rustus/callback HTTP/1.1" 200 OK

A few things that surprised me:

• None of the hooks for the simple upload are labeled as is_final
• There is only a pre-create and post-create event for the concatenation call.
• Only the concat related events are set as is_final
• There's no post-receive events for partial uploads

This leads me to the following thoughts, and also wondering if these are correct assumptions:

1. Fragmented uploads (concat enabled) are identifiable by looking at the is_partial flag.
2. To understand upload progress I'd have to look at the offset/size information provided through post_receive hooks.
3. Since concat enabled uploads don't get post_receive hooks I can't tell what's going on with them as they upload.
4. To understand when a upload is finished I have to look for post-create hooks that have is_final: True as well as post-finish hooks.

Is all of the above correct? Are any of the above pointing to small bugs or logic errors in how the hooks are triggered?

When uploading with the concatenation extension against 0.5.2 things work as expected. However in 0.5.3 the combined file stays 0 bytes and the parts are not removed. I've enabled debug logging, and there's nothing in the logs to indicate a failure or issue.

rustus  | [2022-06-23][00:40:05+00:00][INFO] "PATCH /60a7d100-2b2b-447b-8c48-c080f0b4e6da/ HTTP/1.1" "-" "204" "172.19.0.1" "302.296001"
rustus  | [2022-06-23][00:40:05+00:00][INFO] "PATCH /1009fb2b-263e-4cea-9019-d8a879155417/ HTTP/1.1" "-" "204" "172.19.0.1" "307.389292"
rustus  | [2022-06-23][00:40:05+00:00][INFO] "POST / HTTP/1.1" "-" "201" "172.19.0.1" "5.008000"

When enabling creation-with-upload for an upload that is small enough to finish as part of the upload creation step, the post-finish webhook does not fire.

I successfully see the post-finish webhook when the file requires multiple chunks to upload or when I disable creation-with-upload.

At this point project is released, but we have a poor architecture decisions that were made to develop the application fast.

At first we must separate InfoStorage and Storage from each other. Because today FileStorage makes way more things than just storing data on a file system.

The main idea is to rebuild Storage trait that would use FileInfo structure instead of a file_id string and remove get_file_info method. Also we have to save InfoStorage as a part of the actix state in order to use it in controllers. It gives us more flexibility and makes it easy to develop new storage types.

When I deploy services, I prefer to filter out the logging of the health check endpoint since these are so often spammy and not representative of actual performance or issues. Having a toggle in Rustus to enable/disable logging for that one endpoint would be great.

When I use '/' for the path instead of /files things work, but the path that is generated for the files contain an extra /. Here's a log snippet that shows the issue.

rustus  | [2022-06-22][01:03:13+00:00][INFO] "POST / HTTP/1.1" "-" "201" "172.19.0.1" "66.578333"
rustus  | [2022-06-22][01:03:14+00:00][INFO] "PATCH //10ac0e0e-d67d-4e2c-880b-1346793008b5 HTTP/1.1" "-" "204" "172.19.0.1" "263.382625"

In these situations I've found that using a url join library often is the cleanest way to eliminate duplications and ensure urls are built properly. Failing that doing a regex replace so that /+ is replaced with / works fairly well as well.

It's would be super nice if rustus will run pre-terminate hook an it would be blocking. Because now you can delete files freely without verifying authorization.

It would be really nice to add prometheus metrics to rustus. In this issue we discuss which metrics we want to see on /metrics page.

They can be later used in frontend page.

Maybe it'll be a great idea to have some kind of LRU cache to hold recently created/written files. That way we'll skip file opening in each request if we touched specified files recently

While understanding hooks I ran across a few thoughts.

1. The documentation provided for the schema doesn't include the variations of possible values/types depending on when the hook is triggered.
2. The fastapi example could benefit from providing the pydantic models needed to parse the api.
3. It strikes me as odd that only URI is uppercase, guessing this is an encoding artifact based on how rust's serialization works
4. I was a little surprised that the hook name wasn't available in the payload, but not a huge deal either way.

To help address the first two thoughts, here is the schema I've reversed engineered from watching the hooks over several types of uploads. I'm not sure I've captured all the variations yet.

import datetime
import uuid

from pydantic import BaseModel, conint, Field

class UploadInfo(BaseModel):
    id: uuid.UUID
    offset: conint(ge=0)
    length: conint(ge=0) | None
    path: str | None
    created_at: datetime.datetime
    deferred_size: bool
    is_partial: bool
    is_final: bool
    parts: list[uuid.UUID] | None
    storage: str
    metadata: dict


class RequestInfo(BaseModel):
    uri: str = Field(alias='URI')
    method: str
    remote_addr: str
    headers: dict


class RustusPayload(BaseModel):
    upload: UploadInfo
    request: RequestInfo

I'd also suggest providing example payloads for each event type that covers the variations or at least document when a field can be in different states. i.e. parts is None unless it's a concat finalization in which it gives the ids of the other parts.

Bumps lapin from 1.10.0 to 2.1.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps tokio-amqp from 1.1.0 to 2.0.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Create S3 Data storage to be able to store files on S3.

The main idea is to create many file chunks 5MB each and concatenate them after upload is finished.