Rustic - a backup tool that provides fast, encrypted, deduplicated backups

Related tags

Data processing rustic
Overview

rustic - a restic-compatible backup tool written in pure Rust

crate Docs Apache2/MIT licensed Crates.io Downloads

Rustic is a backup tool that provides fast, encrypted, deduplicated backups. It can read the restic repo format desribed in the design document and writes a compatible repo format which can also be read by restic.

Note that rustic currently is in an alpha release and misses functionalities and tests. It is not yet considered to be ready for use in a production environment.

Open points:

  • Add more backends, backup-sources and restore-destinations
  • Add missing commands
  • Allow for parallel repo access
  • Parallelize the code, maybe use async code
  • Improve error handling
  • Add tests and benchmarks
  • Add CI

Open issues:

  • restore does not yet restore metadata

Comparison with restics:

Improvements:

  • Huge decrease in memory requirement
  • Can use .gitignore files
  • Snapshots save total size and node count, this is also shown in the rustic snapshots command
  • cat tree command accepts a snapshot and path to cat the tree blob

Differences:

  • backup uses glob patterns to include/exclude instead of exclude files
  • file/dir permissions have different format in go and rust (but the important information is identical)

Current limitations:

  • No repo initialization implemented; create a repo with restic and use it with rustic
  • Missing commands, e.g.: init, key, mount, copy, dump, find, forget, prune
  • Backup location, backup source and restore destinations only on local file system
  • Runs only on linux
  • No parallel repo access implemented (no locking; duplicate blobs are not supported)
  • Not speed optimized (and no cache implemented)
  • ... and many more

License

Licensed under either of:

at your option.

Contribution

Contributions in form of issues or PRs are very welcome.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments
  • config command invalidates config file (Error: crypto error)

    config command invalidates config file (Error: crypto error)

    I just created a new local repo and after a few backups and prunes, i'm getting this error on any command

    password is correct
Error: crypto error

    I'm not sure how to get more information as check and other commands don't work

    bug critical 
    opened by nickchomey 15
  • Prune error

    Prune error "Error: used pack 00b1373b does not exist!"

    hi, when i prune my repo after a successful backup, I get the following error:

    Error: used pack 00b1373b does not exist!

    What does this mean? Do I need to fix this? If yes, how do I fix this?

    Thanks!"

    opened by thht 13
  • Delete local files that are not in the snapshot

    Delete local files that are not in the snapshot

    Is it possible for rustic to delete local files that are not in teh snapshot?

    The use case is that you want to restore a folder to a specific point in time, removing any changes that might have been made after that point in time. It seems to me that this is a crucial feature in order for any backup tool, and is the default behaviour in most tools that I am familiar with but it is missing in restic (see this issue https://github.com/restic/restic/issues/2348)

    enhancement 
    opened by nickchomey 11
  • Customize progress bars update

    Customize progress bars update

    Option to show progress info with given frequency as a line of text without control characters would be super useful. See restic's RESTIC_PROGRESS_FPS for the behavior I have in mind.

    Option to force progress bars off altogether would work as an alternative when rustic is to be executed unattended under logged screen sessions. As of now session logfiles which take ~20 KiB each with restic use hundreds of megabytes each with rustic.

    CLI 
    opened by borkd 9
  • Compression levels?

    Compression levels?

    What are the compression levels? There isn't any guidance in the help display

    --set-compression <LEVEL>
            set compression level, 0 equals no compression

    And which type of compression is used? zstd?

    documentation 
    opened by nickchomey 9
  • Behavior when cache is missing

    Behavior when cache is missing

    Hi, I might have a quite peculiar use case... I need to run rustic on an ephemeral machine. This means, that there is not going to be a persistent cache. The backup source is also on a remote, mounted by rclone.

    My question is: Does rustic in this case need to reread all the files? I.e. will this lead to a redownload of the whole source everytime I start the backup even if not much has changed?

    From my first trials, it at least seems that rustic treats all files as "new", although it realizes that it these files (or their data more specifically) are already in the repo:

    Files:       41 new, 0 changed, 0 unchanged
--
Dirs:        7 new, 0 changed, 0 unchanged
Added to the repo: 248 B (raw: 364 B)
processed 41 files, 180.2 MiB
    opened by thht 7
  • rustic `check` needs a lot system cpu

    rustic `check` needs a lot system cpu

    rustic check run on a 64TB repo gets killed with OOM.

     - checking trees...
[00:29:17] █████████████████████████████████████░░░       4479/4751                                                                      
Killed
    performance 
    opened by aawsome 7
  • rustic with rclone 1.61.0 doesn't work

    rustic with rclone 1.61.0 doesn't work

    error is

    Error: url must start with http://! url: [http://127.0.0.1:40417/]

    using rclone 1.59 works fine its serve output is

    2022/12/21 02:12:21 NOTICE: S3 bucket octane-site-staging: Serving restic REST API on http://127.0.0.1:46095/

    where as 1.61 is

    2022/12/21 02:14:01 NOTICE: S3 bucket octane-site-staging: Serving restic REST API on [http://127.0.0.1:38023/]
    opened by cameronbraid 6
  • rclone not recognized

    rclone not recognized

    RClone is not recignized for me because the version is not prefixed with a v:

    Error: output of 'rclone version' doesn't start with 'rclone v'

    $ rclone version
rclone 1.60.1
- os/version: nixos 23.05 (Stoat) (64 bit)
- os/kernel: 6.1.0 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: dynamic
- go/tags: cmount
    duplicate 
    opened by NobbZ 5
  • feature request: `--no-cache` equivalent

    feature request: `--no-cache` equivalent

    It would be super helpful if rustic would gain a --no-cache equivalent for its commands. Caching index and snapshots is not a big deal, but data is the source of problems which prevents me from troubleshooting #48 further.

    Why? Caching makes sense when one works with a slow or expensive backend. When reads are cheap, as it might be the case of local filesystem access, implementing caching that actually speeds things up even further at scale is a difficult ask.

    Local storage is performant enough in terms of performance and capacity to forego caching completely for the repo referenced in #48. A --no-cache mode would reduce CPU overhead and, more importantly, significantly reduce flash wear on every check or prune.

    opened by borkd 5
  • can init repo using rclone, but cannot backup to it or view snapshots

    can init repo using rclone, but cannot backup to it or view snapshots 

    ❯ RUSTIC_PASSWORD=a RUSTIC_REPOSITORY=rclone:live:octane-sites rustic init
key 8d7a5781 successfully added.
repository f23b2bbd successfully created.
❯ RUSTIC_PASSWORD=a RUSTIC_REPOSITORY=rclone:live:octane-sites rustic snapshots
[WARN] Error error sending request for url (http://127.0.0.1:33911/keys/8d7a5781fec2de5baeed68791e51060a3a16c3a00e0eab136470df0317d05e3a): operation timed out at 736.624523ms, retrying
^C
❯ rclone ls live:octane-sites
      155 config
      362 keys/8d7a5781fec2de5baeed68791e51060a3a16c3a00e0eab136470df0317d05e3a
❯ RUSTIC_PASSWORD=a RUSTIC_REPOSITORY=rclone:live:octane-sites rustic backup /tmp/a
[WARN] Error error sending request for url (http://127.0.0.1:33931/keys/8d7a5781fec2de5baeed68791e51060a3a16c3a00e0eab136470df0317d05e3a): operation timed out at 350.420717ms, retrying

    ❯ rustic --version
rustic v0.4.0
    opened by cameronbraid 4
  • build(deps): bump comfy-table from 6.1.3 to 6.1.4

    build(deps): bump comfy-table from 6.1.3 to 6.1.4

    Bumps comfy-table from 6.1.3 to 6.1.4.

    Release notes

    Sourced from comfy-table's releases.

    v6.1.4

    [6.1.4] - 2022-12-31

    Added

    • New preset ASCII_FULL_CONDENSED #97
    Changelog

    Sourced from comfy-table's changelog.

    [6.1.4] - 2022-12-31

    Added

    • New preset ASCII_FULL_CONDENSED #97
    Commits
    • 7966fbd chore: Release comfy-table version 6.1.4
    • bae38bc chore: Bump rstest
    • eb4ad25 docs: Add v6.1.4 Changelog
    • b4204fb Merge pull request #97 from alexander-beedie/ascii-condensed-preset
    • b735bf8 fix 'iter_kv_map' clippy lint
    • 307dc1d new preset "ASCII_FULL_CONDENSED"
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies rust 
    opened by dependabot[bot] 0
  • restore of single file: Allow to rename file

    restore of single file: Allow to rename file

    You can do rustic restore <SNAP>:/path/to/file /my_path/to/dir/ and file will be restored as /my_path/to/dir/file.

    If you run rustic restore <SNAP>:/path/to/file /my_path/to/newfile, you'll get /my_path/to/newfile/file which is a bit non-intuitive. rustic should be able to identify when to put file into a dir and when the filename is specified:

    • if the restore path ends with /, always treat it as a dir
    • else check if the path exists and if it is a dir; then treat it as dir. Otherwise restore to the given filename.
    enhancement 
    opened by aawsome 0
  • build(deps): bump clap_complete from 3.2.5 to 4.0.7

    build(deps): bump clap_complete from 3.2.5 to 4.0.7

    Bumps clap_complete from 3.2.5 to 4.0.7.

    Release notes

    Sourced from clap_complete's releases.

    v4.0.7

    [4.0.7] - 2022-09-30

    Features

    • (derive) Populate implicit ArgGroup (#3165)

    Fixes

    • (derive) Support #[group(skip)] on Parser derive
    • (derive) Tell users about implicit arg groups when running into group name conflicts
    • (error) Don't report unrelated groups in conflict or requires errors

    v4.0.6

    [4.0.6] - 2022-09-30

    Features

    v4.0.5

    [4.0.5] - 2022-09-30

    v4.0.4

    [4.0.4] - 2022-09-29

    Fixes

    • (error) Specialize the self-conflict error to look like clap v3

    v4.0.3

    [4.0.3] - 2022-09-29

    Fixes

    • (error) Quote literals consistently
    • (error) Stylize escape (--) suggestions
    • (error) Format help flag as a literal

    v4.0.2

    [4.0.2] - 2022-09-28

    Fixes

    • (parser) SetFalse should conflict with itself like SetTrue and Set
    • (parser) Allow one-off overrides

    v4.0.1

    [4.0.1] - 2022-09-28

    ... (truncated)

    Changelog

    Sourced from clap_complete's changelog.

    [4.0.7] - 2022-09-30

    Features

    • (derive) Populate implicit ArgGroup (#3165)

    Fixes

    • (derive) Support #[group(skip)] on Parser derive
    • (derive) Tell users about implicit arg groups when running into group name conflicts
    • (error) Don't report unrelated groups in conflict or requires errors

    [4.0.6] - 2022-09-30

    Features

    [4.0.5] - 2022-09-30

    [4.0.4] - 2022-09-29

    Fixes

    • (error) Specialize the self-conflict error to look like clap v3

    [4.0.3] - 2022-09-29

    Fixes

    • (error) Quote literals consistently
    • (error) Stylize escape (--) suggestions
    • (error) Format help flag as a literal

    [4.0.2] - 2022-09-28

    Fixes

    • (parser) SetFalse should conflict with itself like SetTrue and Set
    • (parser) Allow one-off overrides

    [4.0.1] - 2022-09-28

    Fixes

    • (derive) Ensure #[clap(...)] attribute still works

    [4.0.0] - 2022-09-28

    Highlights

    ... (truncated)

    Commits
    • 0eccd55 chore: Release
    • 1e37c25 docs: Update changelog
    • dcd5fec Merge pull request #4572 from epage/group
    • dde22e7 style: Update for latest clippy
    • dd8435d perf(parser): Reduce duplicate lookups
    • 4a34b9d perf(parser): Reduce lookups for conflicts
    • 9242e1d Merge pull request #4566 from hitenkoku/fix/4565
    • b72ad58 fix: Added cfg in help_template
    • c6058de fix(help_template): Added cfg
    • a7a8241 docs(command): Added deprecated suffix in help string
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies rust 
    opened by dependabot[bot] 0
  • build(deps): bump indicatif from 0.17.0 to 0.17.2

    build(deps): bump indicatif from 0.17.0 to 0.17.2

    Bumps indicatif from 0.17.0 to 0.17.2.

    Release notes

    Sourced from indicatif's releases.

    0.17.2

    A small maintenance release which makes indicatif more portable and fixes some minor regressions.

    • Use portable-atomic to fix build on some 32-bit platforms (#484, thanks to @​messense)
    • Implement multi-line progress message support (#443, thanks to @​happenslol)
    • Reset estimator of progress rate on backwards movement (#483, thanks to @​rlee287)
    • Fix percent initial value when there is no length (#491, thanks to @​devmatteini)
    • Bumped the MSRV to 1.56 (#482)

    On behalf of @​djc and @​chris-laplante, thanks to all contributors!

    0.17.1

    2.5 months after the large 0.17 release, we (finally) have a release that addresses most of the regressions found in 0.17. There is ongoing work on changes in the estimation algorithm, tracked in #394, which has regressed for some users.

    Note that we made some technically semver-breaking change of adding a missing Sync bound to the ProgressTracker bounds (#471). We're assuming that most users don't (yet) have custom ProgressTracker impls, and that users who do have probably built one that is Sync anyway.

    Fixed regressions

    • Fixed unicode-width feature spelling (#456)
    • Only tick if the ticker is disabled (#458)
    • Rework MultiProgress zombie line handling (#460)
    • Fix incorrect link in documentation (#469, thanks to @​Jedsek)
    • Take a reference for ProgressBar::style() (#476, thanks to @​andrewchambers)

    Other changes

    Thanks from @​djc and @​chris-laplante to all contributors!

    Commits
    • 25afbed Bump version number to 0.17.2
    • 8e220fd Fix clippy lints
    • 5b8b905 Fix percent initial value when there is no length
    • 2c85ff8 Add an armv5te test job to CI
    • 44ec391 Use portable-atomic to fix build on some 32-bit platforms
    • 14b5ef2 Update test to ensure reset occurs after rewind
    • 997567d Reset estimator of progress rate on backwards movement
    • 517398b Bump MSRV to 1.56
    • 222df5b Add additional tests for multi-progress multiline rendering
    • be579da Improve multiline support in format_style
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies rust 
    opened by dependabot[bot] 1
  • hot/cold: Repair hot repo

    hot/cold: Repair hot repo

    When using a hot/cold repo scenario, the hot repo should be in sync with the cold repo for files it contains. The check command does check the consistency of the hot repo. However, if there are consistency errors, there is no way to repair the hot repo except by manually removing/copying repo files. There should be a repair hot-repo command which is doing this automatically.

    enhancement 
    opened by aawsome 0
Releases(v0.4.1)
Owner
null
GitHub
Provides multiple-dtype columner storage, known as DataFrame in pandas/R

brassfibre Provides multiple-dtype columner storage, known as DataFrame in pandas/R. Series Single-dtype 1-dimentional vector with label (index). Crea

Sinhrks 21 Nov 28, 2022
This library provides a data view for reading and writing data in a byte array.

Docs This library provides a data view for reading and writing data in a byte array. This library requires feature(generic_const_exprs) to be enabled.

null 2 Nov 2, 2022
A fast, powerful, flexible and easy to use open source data analysis and manipulation tool written in Rust

fisher-rs fisher-rs is a Rust library that brings powerful data manipulation and analysis capabilities to Rust developers, inspired by the popular pan

Syed Vilayat Ali Rizvi 5 Aug 31, 2023
A fast, powerful, flexible and easy to use open source data analysis and manipulation tool written in Rust

fisher-rs fisher-rs is a Rust library that brings powerful data manipulation and analysis capabilities to Rust developers, inspired by the popular pan

null 5 Sep 6, 2023
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

Polars Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow Blazingly fast DataFrames in Rust, Python & Node.js Polars is

null 11.8k Jan 8, 2023
A tool to stream the chats of Twitch channels as a CSV.

twitch2csv A tool to stream the chats of Twitch channels as a CSV. Installation You can use cargo to install this tool: cargo install -f twitch2csv Us

Clément Renault 2 Nov 20, 2021
PostQuet: Stream PostgreSQL tables/queries to Parquet files seamlessly with this high-performance, Rust-based command-line tool.

STATUS: IN DEVELOPMENT PostQuet: Streaming PostgreSQL to Parquet Exporter PostQuet is a powerful and efficient command-line tool written in Rust that

Per Arneng 4 Apr 11, 2023
A dataframe manipulation tool inspired by dplyr and powered by polars.

dply is a command line tool for viewing, querying, and writing csv and parquet files, inspired by dplyr and powered by polars. Usage overview A dply p

null 14 May 29, 2023
rustic_core - library for fast, encrypted, deduplicated backups that powers rustic-rs

Library for fast, encrypted, and deduplicated backups About This library is powering rustic-rs. A backup tool that provides fast, encrypted, deduplica

rustic 9 Sep 29, 2023
It is a backup tool that creates backups and stores them on an object storage

Hold My Backup It is a backup tool that creates backups and stores them on an object storage. By default it uses minio but you can use AWS: S3 as well

Taylan Dogan 13 Feb 17, 2022
Trustworthy encrypted command line authenticator app compatible with multiple backups.

cotp - command line totp authenticator I believe that security is of paramount importance, especially in this digital world. I created cotp because I

Reply 71 Dec 30, 2022
Fast, deduplicated content and database seeding for WordPress

Sprout Fast, deduplicated content and database seeding for WordPress. Documentation | Install | Releases Store your uploads and database in a secure,

Tom Lawton 14 Feb 22, 2024
A rustic tcp + serialization abstraction.

Wire An abstraction over TCP and Serialization "put a struct in one side and it comes out the other end" Wire is a library that makes writing applicat

Ty Overby 33 May 12, 2021
Rustic bindings for sqlite3

Rust-Sqlite3 Rustic bindings for sqlite3. OBSOLETE in favor of jgallagher/rusqlite Copyright (c) 2014-2017 Dan Connolly and contributors Share and enj

Dan Connolly 57 Sep 26, 2022
Lupus is a utility to administer backups with future integration with rsync

Lupus is a utility to administer backups with future integration with rsync. Many other features are either included or planned such as chat bridges using rcon and or parsing the pipe output from programs/games.

null 3 Sep 19, 2022
Incremental, multi-version remote backup tool for block devices.

bsync Incremental, multi-version remote backup tool for block devices. The on-disk backup format is a SQLite database and I've been dogfooding this on

Heyang Zhou 7 Aug 21, 2022
A command-line tool and Docker image to automatically backup Git repositories from GitHub or anywhere

A command-line tool and Docker image to automatically backup Git repositories from GitHub or anywhere

Jake Wharton 256 Dec 27, 2022
A simple tool for extracting files from iOS backup archive.

iBackupExtractor A simple tool for extracting files from iOS backup archive. iOS backup files are not stored with their original directory layouts. Re

Cyandev 132 Oct 10, 2023
Bijou is a tiny yet fast encrypted file system.

Bijou ✨??✨ Bijou (['bi:ʒu], French for "jewel") is a tiny yet fast encrypted filesystem, built upon RocksDB. Bijou provides a FUSE interface, as well

Mivik 5 Sep 27, 2023
Valheim Docker powered by Odin. The Valheim dedicated gameserver manager which is designed with resiliency in mind by providing automatic updates, world backup support, and a user friendly cli interface.

Valheim Docker If you are looking for a guide on how to get started click here Mod Support! It is supported to launch the server with BepInEx but!!!!!

Michael 657 Dec 30, 2022