Unlike GPL, the AGPL triggers when software is accessed over a network. This would stop Spacedrive and its libraries from being forked, modified and served as a SaaS platform without the modifications being made public. This does not prevent Spacedrive and its libraries from being used inside commercial projects, but it does extend the boundary of the original GPL license.

For all of Spacedrive's code to be relicensed we must get consent from all contributors. If you are a contributor, please comment on this PR indicating that you accept the license change. If you do not accept this change or fail to accept it by 27 June 8:00PM PST then your changes will be removed.

• [x] @jamiepine
• [x] @Brendonovich
• [x] @maxichrome
• [x] @oscartbeaumont
• [x] @benja
• [x] @xPolar
• [x] @LayZeeDK
• [x] @KodingDev
• [x] @sr2echa
• [x] @CreatingBytes
• [x] @FahimFBA
• [x] @mmattbtw
• [x] @Harry-Hopkinson
• [x] @XSPGMike
• [x] @he1d1
• [x] @IllusionMan1212
• [x] @ecwyne
• [x] @krumware
• [x] @stevharve
• [x] @mark-strudwick
• [x] @chroxify
• [x] @TheUltDev
• [ ] @̶i̶l̶k̶k̶a̶ (Edit: Only commit fixed a broken link in the readme, not a code change. No response, not waiting.)
• [x] @CodePurble
• [x] @abhnva
• [x] @cnrad
• [x] @MarquesCoding
• [x] @alsonick
• [x] @CodeWithBryan
• [x] @̶f̶i̶v̶e̶s̶t̶o̶n̶e̶s̶ (Edit: Added a single apostrophe to the readme. No response, not waiting.)
• [ ] @̶y̶k̶a̶b̶u̶s̶a̶l̶a̶h̶ (Edit: The only commit by this user was already overwritten. No response, not waiting.)
• [x] @kor-pixel
• [x] @̶r̶y̶a̶n̶h̶a̶t̶i̶c̶u̶s̶ (Edit: Can not be found on the contributor list. No response, not waiting.)

No external contributions should be accepted into this repository until this pull request has been merged, which will not happen until all contributors have agreed to the license change.

I accept this license change.

This is the PR for ENG-132 (Support for Multiple Libraries). I would like to get this merged in before ENG-42 (P2P Communication) which will hopefully be ready later this week.

This PR is a draft as it requires UI changes so it can be merged into main without breaking the functionality of the app. The UI pages required are documented in ENG-132.

• Using tokio on all filesystem operations
• Some minor tweaks to be more consistent on paths between &str, AsRef and PathBuf
• Using logging instead of println
• Updating uuid crate

For example in the modal for adding a new device (trigged from "Add device" on Overview page), when pressing the enter key inside the text field the modal does not get submitted. _{Oscar Beaumont on Linear}

This is a full rewrite of setup-system.ps1 that sets up a user's machine for Spacedrive development.

It currently:

1. Checks for Rust and Cargo
2. Installs pnpm (if not installed)
3. Installs the latest version of Node.js using pnpm
4. Installs LLVM (compiler for ffmpeg-rust)
5. Downloads ffmpeg and set as an environment variable

The CONTRIBUTING.md has also been updated to include info on running the script.

This PR helps close issue #77 by creating a working Windows setup script.

After a metric ton of trial-and-error, by way of this PR, the Spacedrive app shows macOS' native frosted-glass blur effect behind the sidebar. Took some Objective-C binding magic, but to see it working as expected is so great!

Closes ENG-81

This changes keyboard-shortcut functionality to properly use the Tauri (Rust-side) shortcuts engine. This allows, for instance, macOS' app-specific shortcut remapping — which is particularly important for power users — among other benefits (such as macOS' highlighting of shortcut position in the Menu Bar when pressed, which is absent in the current main build).

Describe the bug

The sidebar for Spacedrive is transparent and not blurred like it's supposed to be (specifically on macOS).

Reproduction

1. git pull && pnpm i && pnpm prep && pnpm desktop dev
2. Observe sidebar

Expected behavior

I'd assume that the sidebar background would just be a solid color because I don't think blur can be native on Linux (or Windows).

Platform and versions

➜  spacedrive git:(main) pnpm --version && cargo --version && rustc --version
7.5.2
cargo 1.61.0 (a028ae4 2022-04-29)
rustc 1.61.0 (fe5b13d68 2022-05-18)

➜  spacedrive git:(main) neofetch
             /////////////                matt@pop-os 
         /////////////////////            ----------- 
      ///////*767////////////////            MS-7B98 1.0 
    //////7676767676*//////////////          Pop!_OS 22.04 LTS 
   /////76767//7676767//////////////         6 hours, 34 mins 
  /////767676///*76767///////////////        kitty 
 ///////767676///76767.///7676*///////       Catppuccin-Macchiato-Dark-BL 
/////////767676//76767///767676////////      12.32GiB / 31.29GiB (39%) 
//////////76767676767////76767/////////      Lil B - The Frozen Tape - Patrick Star 
///////////76767676//////7676//////////
////////////,7676,///////767///////////                           
/////////////*7676///////76////////////                           
///////////////7676////////////////////
 ///////////////7676///767////////////
  //////////////////////'////////////
   //////.7676767676767676767,//////
    /////767676767676767676767/////
      ///////////////////////////
         /////////////////////
             /////////////

### Stack trace

_No response_

### Additional context

_No response_

This PR is massive, please criticize it to shreds - it's important that we get this right.

The Key Manager now uses DashMap in order to provide concurrency and better performance over HashMap.

There are a few known bugs/issues:

• The dropdown UI is pretty bad, and doesn't fit in too well with the rest of Spacedrive (and the TS formatting is bad too, I don't know how we format that though)
• The key numbering is volatile, and can/will be confusing
• We have no way to set key names within the UI
• We have no way to view the plaintext key within the UI
• Object/container stats are not attached (this isn't major, as they would all be 0 and therefore wouldn't show anyway)
• The UI offers no differentiation between the default key and normal keys
• There is a bug where you are unable to remove a key instantly while mounting another key
• We have no way to set the master password, and possibly no way to verify that it's correct until the user attempts to run an action. I will be adding a master password validation feature, which will essentially insert a ghost key into the library, which is only ever used to ensure the entered master password is correct. If we're going to give the user a master password during library onboarding, we'll need a way to make sure it's correct and doesn't deviate from what was provided.

Go ahead and close this if needs be, there is still quite a lot to do - I just didn't want this branch to be sat around forever.

Here is a preview of the current key manager (key mounting on dev builds isn't this fast, but they will be in prod):

https://user-images.githubusercontent.com/77554505/199038410-0d21c5c5-4ec6-48e3-aad8-2e139978c22f.mp4

I updated the DIstro identifier as the one in place was having issues with openSUSE. Also had to update FFMPEG to ^4.4.0 due to openSUSE not having on main PPA. Added in node check to setup as a requirement needed for all builds except mobile. I am working towards automating the checks next so instead of failing we would be able to install rust/pnpm/node through script.

Closes #(issue)

…r users without cargo or pnpm installed

set -e prevents the if statements from running when which exits with non-zero status are never informed about installing cargo or pnpm, removing it fixes that.

Updates Tauri dependencies for Debian in setup-system.sh

This PR completes the Linux - Debian task of issue #77

Fixes #77 Linux - Debian task

[WIP] Improved macOS file system watcher.

The notify crate we are currently using doesn't give us low-level information about the event and it also emits the events as the OS exposed them (which is generally wrong). I am working on a crate that will expose the events correctly to Spacedrive and internally is written in low-level API's so it can be efficient and make proper use of the information available to determine the proper type of the event.

Describe the bug

PS C:\Users\29422\Desktop\spacedrive> pnpm desktop dev

> @sd/[email protected] desktop C:\Users\29422\Desktop\spacedrive
> pnpm --filter @sd/desktop -- "dev"


> @sd/[email protected] dev C:\Users\29422\Desktop\spacedrive\apps\desktop
> tauri dev

     Running BeforeDevCommand (`pnpm exec vite --clearScreen=false --mode development`)

  VITE v3.2.5  ready in 1873 ms

  ➜  Local:   http://localhost:8001/
  ➜  Network: use --host to expose
warning: C:\Users\29422\Desktop\spacedrive\apps\mobile\rust\Cargo.toml: unused manifest key: target.cfg(not(target_os = "ios")).features
        Info Watching C:\Users\29422\Desktop\spacedrive\core for changes...
        Info Watching C:\Users\29422\Desktop\spacedrive\crates/* for changes...
        Info Watching C:\Users\29422\Desktop\spacedrive\crates/sync/example/api for changes...
        Info Watching C:\Users\29422\Desktop\spacedrive\apps/desktop/src-tauri for changes...
        Info Watching C:\Users\29422\Desktop\spacedrive\apps/mobile/rust for changes...
        Info Watching C:\Users\29422\Desktop\spacedrive\apps/server for changes...
    Finished dev [unoptimized + debuginfo] target(s) in 3.57s
C:\Users\29422\Desktop\spacedrive\apps\desktop:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @sd/[email protected] dev: `tauri dev`
Exit status 3221225781
 ELIFECYCLE  Command failed with exit code 1.

Reproduction

run pnpm desktop dev

Expected behavior

No response

Platform and versions

PS C:\Users\29422\Desktop\spacedrive> pnpm --version && cargo --version && rustc --version
7.20.0
cargo 1.65.0 (4bc8f24d3 2022-10-20)
rustc 1.65.0 (897e37553 2022-11-02)

OS: win 10 22h2

Stack trace

No response

Additional context

No response

This PR is dependent on #468 and is not ready for merge, just created this to document changes.

Changes:

• [x] We now pass around &mut WorkerContext to avoid need for Mutex. I was able to move some stuff around so we only ever need one instance of WorkerContext which satisfies Rust's borrow checker rule of having a single mutable reference. This means no need for lock().await to update the job report.
• [x] Remove WorkerEvent enum along with the associated channel and Tokio task. This channel was extra complexity along with another point of failure and was just not needed. For example here we send a message than use done_tx to await the response. There is no reason we shouldn't just do the DB call and other logic on the current thread given it's already awaiting that operation.
• [x] CurrentDirFileIdentifierJob, FileEncryptorJob and FileDecryptorJob were not put into the JobManager's resume functionality. This means they would never be resumed.
  • [x] The longer-term fix for this is to implement the JobRestored trait and a constant called const JOB_RESTORER: Lazy<HashMap<&'static str, Box<dyn JobRestorer>>>. When you ingest a job if it doesn't exist in the JOB_RESTORER constant it will refuse to run. This will help us catch the mistake before it causes issues.
• [x] Instead of resuming jobs on core startup we now resume them when each library is loaded within the library manager. Resuming them at startup is gonna cause problems when we have encrypted libraries that can't be loaded at startup (because they have a password). The same thing was done for the LocationManager as it's gonna have the same problems. For the LocationManager it would mean when creating a library it would not enabled the FS listener without restarting the app.
• [x] Removed external job queuing (LibraryCtx.queue_job(...), JobManager.ingest_queue(...)). These methods facilitate writing code that assumes we only have a single worker thread. This is currently the case but I don't think it will always be the case so writing code around it is just gonna cause us errors. Eg. We do .queue_job(...).queue_job(...).spawn_job(...) in one point which assumes the spawned job will run first then the two queued jobs. With multiple worker threads, they could all just start at the same time.
  • [ ] As of writing I haven't come up with a solution but just using .spawn(...).spawn(...).spawn(...) and reordering the jobs achieves the equivalent functionality of the previous code without needing the queue method which encourages the single-threaded assumptions. Gonna work on a solution.
• [x] Removed DynJob and replace it with a simple system that uses a tokio task for both queued and spawned jobs. Upon queuing a job it will spawn onto a tokio::task and await a channel that triggers the job to start. This system is so much easier to understand the code doesn't do a heap of message passing. In terms of performance, if a web server can handle 10k reqs/seconds and spawns a new tokio task for each thread, we should be fine spawning a few extra tasks for the added simplicity. This also mostly removes the concept of a Worker.
• [x] Simplify spawn_job call. From library.spawn_job(Job::new(FileEncryptorJobInit { ... }, FileEncryptorJob {})).await to library.spawn_job(FileEncryptorJobInit { ... }).await
• [ ] Refactor the StatefulJob trait
  • [ ] Replace the empty individual job struct's with the old StatefulJob::Data struct.
  • [ ] Merge StatefulJob::Init into StatefulJob::Data as a statemachine style thing. This should also allow removing the JobInitData trait which would be nice. Or is this a stupid idea???
  • [ ] Improved type safety. Rust has a powerful type system that can be used to prevent the system from getting invalid states. The job system uses Option::take which means the system could get into invalid states if a mistake is made. Why not just prevent us from ever being able to make a mistake.
• [ ] Crash safety. The job state is now persisted to the database during execution at a set interval of steps. This means if the core were to crash or shutdown incorrectly we don't have to restart the job, we can just resume from the last checkpoint. (Partially implemented)
• [ ] Allow pausing individual jobs. The existing system had a global shutdown channel that would stop all of the jobs. The new system has a channel per job such that lets you ask a job to shutdown or to pause and trigger the next job.
  • [ ] Backend
  • [ ] Frontend
• [ ] Reintroduce using Hash on StatelessJob::Init so that two identical jobs can't be run at the same time. This was a regression from my changes.
• [ ] Make the job's time elapsed on the frontend work again.
  • The old system would push updates to the frontend every second pointlessly using up rspc IPC bandwidth. Instead the frontend will just keep updating the time elapsed from a static time_started field on the job. When it receives the update that the job has been paused or stopped it can stop ticking up the time elapsed and use the value in that payload. This is similar to what Vercel does and it means if the stop or pause events never reach the frontend it will keep counting forever but this is a fine tradeoff, imo.
• [ ] Pushing certain data directly to frontend's RQ cache to prevent the core from feeling lagging while under heavy load.
  • When indexing a large location a lot of invalidate_query events are sent over the rspc IPC bus. This causes it to become saturated and makes the app lag. The way an invalidate_query event works is as follows: "Frontend invalidate key x" -> "Backend what is the new data for key x" -> "Frontend here is the new data for key x". This can be improved.
  • [ ] Batch invalidations. Ask the frontend to invalidate a bunch at once so it can fetch all the new values at once with it's built-in query batching.
  • [ ] Pushing new data. Eg. "Frontend I have the following new data for key x.". This means 2 less IPC calls for the same functionality. I suspect this is probs gonna be less typesafe or more error-prone than the normal invalidate_query call so for now, it will only be used for stuff that would benefit from it.
• [ ] Resume safety - When a job is resumed it should check that the resources it depends on still exist in the DB or it should automatically remove itself instead of erroring out. Idk if the current system handles this or not I wanna test and make sure it's good.
• [ ] Add the prune preview media job. Currently, all preview media is created but never deleted. Delete it if it's not being used by any library.

We are probably gonna want to statically bundle ffmpeg on macOS and this PR is to test ways of doing that against our CI workflow to see if they will work.

Please feel free to close this PR if anyone isn't comfortable with balloon hashing being implemented.

The parameter levels of this PR aren't final, I feel as though they're a little low but I will need to test on multiple machines to find the optimal balance. This PR not only adds balloon hashing, but it cleans up a massive amount of the crypto code (primarily header serialization/deserialization).

The size of this PR did get a little out of hand, but it also has #484 and #487 squash merged into it (this is so I could continue working on the latest version of the codebase).

The UI work did not get completed, so there are currently 6 items in the Select boxes. This is not final, and I plan to open an issue specifically for fixing the UI crypto code (I'll likely need some help for this one).

I'll keep this as a draft until the prior two PRs get merged, as the squash merge will need to be excluded with a git cherry-pick.