Zero-details, privacy-focused in-app file system.

Overview

ZboxFS Logo ZboxFS

GitHub action Crates.io Crates.io GitHub last commit license GitHub stars

ZboxFS is a zero-details, privacy-focused in-app file system. Its goal is to help application store files securely, privately and reliably. By encapsulating files and directories into an encrypted repository, it provides a virtual file system and exclusive access to authorised application.

Unlike other system-level file systems, such as ext4, XFS and Btrfs, which provide shared access to multiple processes, ZboxFS is a file system that runs in the same memory space as the application. It provides access to only one process at a time.

By abstracting IO access, ZboxFS supports a variety of underlying storage layers, including memory, OS file system, RDBMS and key-value object store.

Disclaimer

ZboxFS is under active development, we are not responsible for any data loss or leak caused by using it. Always back up your files and use at your own risk!

Features

  • Everything is encrypted 🔒 , including metadata and directory structure, no knowledge can be leaked to underlying storage
  • State-of-the-art cryptography: AES-256-GCM (hardware), XChaCha20-Poly1305, Argon2 password hashing and etc., powered by libsodium
  • Support varieties of underlying storages, including memory, OS file system, RDBMS, Key-value object store and more
  • Files and directories are packed into same-sized blocks to eliminate metadata leakage
  • Content-based data chunk deduplication and file-based deduplication
  • Data compression using LZ4 in fast mode, optional
  • Data integrity is guaranteed by authenticated encryption primitives (AEAD crypto)
  • File contents versioning
  • Copy-on-write (COW 🐮 ) semantics
  • ACID transactional operations
  • Built with Rust ♥️

Comparison

Many OS-level file systems support encryption, such as EncFS, APFS and ZFS. Some disk encryption tools also provide virtual file system, such as TrueCrypt, LUKS and VeraCrypt.

This diagram shows the difference between ZboxFS and them.

Comparison

Below is the feature comparison list.

ZboxFS OS-level File Systems Disk Encryption Tools
Encrypts file contents ✔️ partial ✔️
Encrypts file metadata ✔️ partial ✔️
Encrypts directory ✔️ partial ✔️
Data integrity ✔️ partial ✖️
Shared access for processes ✖️ ✔️ ✔️
Deduplication ✔️ ✖️ ✖️
Compression ✔️ partial ✖️
Content versioning ✔️ ✖️ ✖️
COW semantics ✔️ partial ✖️
ACID Transaction ✔️ ✖️ ✖️
Varieties of storages ✔️ ✖️ ✖️
API access ✔️ through VFS through VFS
Symbolic links ✖️ ✔️ depends on inner FS
Users and permissions ✖️ ✔️ ✔️
FUSE support ✖️ ✔️ ✔️
Linux and macOS support ✔️ ✔️ ✔️
Windows support ✔️ partial ✔️

Supported Storage

ZboxFS supports a variety of underlying storages. Memory storage is enabled by default. All the other storages can be enabled individually by specifying its corresponding Cargo feature when building ZboxFS.

Storage URI identifier Cargo Feature
Memory "mem://" N/A
OS file system "file://" storage-file
SQLite "sqlite://" storage-sqlite
Redis "redis://" storage-redis
Zbox Cloud Storage "zbox://" storage-zbox-native

* Visit zbox.io to learn more about Zbox Cloud Storage.

Specs

Algorithm and data structure Value
Authenticated encryption AES-256-GCM or XChaCha20-Poly1305
Password hashing Argon2
Key derivation BLAKE2B
Content dedup Rabin rolling hash
File dedup Merkle tree
Index structure Log-structured merge-tree
Compression LZ4 in fast mode

Limits

Limit Value
Data block size 8 KiB
Maximum encryption frame size 128 KiB
Super block size 8 KiB
Maximum filename length No limit
Allowable characters in directory entries Any UTF-8 character except /
Maximum pathname length No limit
Maximum file size 16 EiB
Maximum repo size 16 EiB
Max number of files No limit

Metadata

Metadata Value
Stores file owner No
POSIX file permissions No
Creation timestamps Yes
Last access / read timestamps No
Last change timestamps Yes
Access control lists No
Security Integrated with crypto
Extended attributes No

Capabilities

Capability Value
Hard links No
Symbolic links No
Case-sensitive Yes
Case-preserving Yes
File Change Log By content versioning
Filesystem-level encryption Yes
Data deduplication Yes
Data checksums Integrated with crypto
Offline grow No
Online grow Auto
Offline shrink No
Online shrink Auto

Allocation and layout policies

Feature Value
Address allocation scheme Append-only, linear address space
Sparse files No
Transparent compression Yes
Extents No
Copy on write Yes

Storage fragmentation

Fragmentation Value
Memory storage No
File storage fragment unit size < 32 MiB
RDBMS storage No
Key-value storage No
Zbox cloud storage fragment unit size < 128 KiB

How to use

For reference documentation, please visit documentation.

Requirements

Supported Platforms

  • 64-bit Debian-based Linux, such as Ubuntu
  • 64-bit macOS
  • 64-bit Windows
  • 64-bit Android, API level >= 21

32-bit and other OS are NOT supported yet.

Usage

Add the following dependency to your Cargo.toml:

[dependencies]
zbox = "0.9.1"

If you don't want to install libsodium by yourself, simply specify libsodium-bundled feature in dependency, which will automatically download, verify and build libsodium.

[dependencies]
zbox = { version = "0.9.1", features = ["libsodium-bundled"] }

Example

extern crate zbox;

use std::io::{Read, Write, Seek, SeekFrom};
use zbox::{init_env, RepoOpener, OpenOptions};

fn main() {
    // initialise zbox environment, called first
    init_env();

    // create and open a repository in current OS directory
    let mut repo = RepoOpener::new()
        .create(true)
        .open("file://./my_repo", "your password")
        .unwrap();

    // create and open a file in repository for writing
    let mut file = OpenOptions::new()
        .create(true)
        .open(&mut repo, "/my_file.txt")
        .unwrap();

    // use std::io::Write trait to write data into it
    file.write_all(b"Hello, World!").unwrap();

    // finish writing to make a permanent content version
    file.finish().unwrap();

    // read file content using std::io::Read trait
    let mut content = String::new();
    file.seek(SeekFrom::Start(0)).unwrap();
    file.read_to_string(&mut content).unwrap();
    assert_eq!(content, "Hello, World!");
}

Build with Docker

ZboxFS comes with Docker support, which made building ZboxFS easier. Check each repo for more details.

Static linking with libsodium

By default, ZboxFS uses dynamic linking when it is linked with libsodium. If you want to change this behavior and use static linking, you can enable below two environment variables.

On Linux/macOS,

export SODIUM_LIB_DIR=/path/to/your/libsodium/lib
export SODIUM_STATIC=true

On Windows,

set SODIUM_LIB_DIR=C:\path\to\your\libsodium\lib
set SODIUM_STATIC=true

And then re-build the code.

cargo build

Performance

The performance test is run on a Macbook Pro 2017 laptop with spec as below.

Spec Value
Processor Name: Intel Core i7
Processor Speed: 3.5 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Memory: 16 GB
OS Version: macOS High Sierra 10.13.6

Test result:

Read Write TPS
Baseline (memcpy): 3658.23 MB/s 3658.23 MB/s N/A
Baseline (file): 1307.97 MB/s 2206.30 MB/s N/A
Memory storage (no compress): 605.01 MB/s 186.20 MB/s 1783 tx/s
Memory storage (compress): 505.04 MB/s 161.11 MB/s 1180 tx/s
File storage (no compress): 445.28 MB/s 177.39 MB/s 313 tx/s
File storage (compress): 415.85 MB/s 158.22 MB/s 325 tx/s

To run the performance test on your own computer, please follow the instructions in CONTRIBUTING.md.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms of conditions.

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Community

License

ZboxFS is licensed under the Apache 2.0 License - see the LICENSE file for details.

Comments
  • "zero knowledge" does not seem to be the right term

    The project's description says

    Zbox is a zero-knowledge, privacy-focused embeddable file system.
    

    This seems to be false: there is nothing "zero knowledge" about this (or any) filesystem. Zero knowledge is a property of a proof; informally, it means that the proof reveals nothing except its correctness [1].

    I think the term you're looking for is something closer to "semantic security" [2], meaning that the filesystem does not leak anything about the contents except their length.

    I am aware that several other projects currently misuse this terminology in the same way you are. It's harmful to your users (in that it misinforms them), to the community (in that it confuses useful pieces of terminology), and to your project (in that it looks unschooled) to follow their example.

    [1] https://en.wikipedia.org/wiki/Zero-knowledge_proof [2] https://en.wikipedia.org/wiki/Semantic_security

    opened by kwantam 16
  • WebAssembly support

    WebAssembly support

    Hey there! As far as I can see, ZboxFS has its ways of using the native FS and other services, therefore making me wonder if it can be successfully compiled to WASM and use as an in-memory secure file system. Are there any plans for it?

    Thanks!

    enhancement 
    opened by skyne98 12
  • SIGABRT on opening Repo

    SIGABRT on opening Repo

    I'm getting a SIGABRT on opening a repo. The error happens both with libsodium-bundled and with the system libsodium.

    Host: Arch Linux Zbox: 0.8.3

    fn main() {
        let mut repo = zbox::RepoOpener::new()
            .open("mem://", "password")
            // .open("file://file.zbox", "password")
            .unwrap();
    }
    
    Program received signal SIGABRT, Aborted.
    0x00007ffff7dd1755 in raise () from /usr/lib/libc.so.6
    (gdb) bt
    #0  0x00007ffff7dd1755 in raise () from /usr/lib/libc.so.6
    #1  0x00007ffff7dbc851 in abort () from /usr/lib/libc.so.6
    #2  0x0000555555587245 in sodium_misuse () at sodium/core.c:199
    #3  0x0000555555715f7d in _sodium_malloc (size=<optimized out>)
        at sodium/utils.c:578
    #4  sodium_malloc (size=<optimized out>) at sodium/utils.c:610
    #5  0x00005555555ec5fa in zbox::base::crypto::SafeBox<T>::new_empty ()
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/base/crypto.rs:163
    #6  0x000055555592178a in zbox::volume::storage::file::file_armor::FileArmor<T>::new (base=0x555555a74b60)
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/file/file_armor.rs:183
    #7  0x0000555555595cc9 in zbox::volume::storage::file::file::FileStorage::new (
        base=0x55555599d007)
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/file/file.rs:40
    #8  0x00005555555feaff in zbox::volume::storage::storage::parse_uri (uri=...)
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/storage.rs:45
    #9  0x00005555555fec48 in zbox::volume::storage::storage::Storage::new (uri=...)
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/storage.rs:144
    #10 0x00005555555b328f in zbox::volume::volume::Volume::new (uri=...)
        at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/volume.rs:42
    #11 0x00005555555fc91d in zbox::fs::fs::Fs::open (uri=..., pwd=...,
    --Type <RET> for more, q to quit, c to continue without paging--
         at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/fs/fs.rs:162
    #12 0x000055555559087f in zbox::repo::Repo::open (uri=..., pwd=..., read_only=false) at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/repo.rs:682
    #13 0x0000555555590266 in zbox::repo::RepoOpener::open (self=0x7fffffffe5b8, uri=..., pwd=...) at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/repo.rs:252
    #14 0x0000555555587520 in zbox_repro::main () at src/main.rs:2
    
    wontfix 
    opened by theduke 7
  • Deeper explanation in website or docs

    Deeper explanation in website or docs

    The current explanation on the website is shallow and does not go deep into the internal workings of the library. For instance "Multiple storages, including memory, OS file system, RDBMS, key-value object store" does not explain how to interact with other storage systems like memory or RDBMS.

    Another example, "State-of-the-art cryptography: AES-256-GCM (hardware), XChaCha20-Poly1305, Argon2 and etc." does not explain which is cryptography is used by default or how it's all interconnected.

    It would be great if you explained more, or gave a document detailing the internal architecture of zbox.

    Also the chat on gitter seems abandoned

    enhancement 
    opened by charleschege 7
  • Issue with file reader after using `set_len`

    Issue with file reader after using `set_len`

    Hi @burmecia,

    another issue I stumbled upon (this is the last test case that fails against the pyfilesystem test suite): when a reader already exists for a file, extending the file with set_len and then reading the file (for instance with read_to_end) will not take into account the newly added bytes.

    Here's a test case, that's shows that file.read_to_end will read the same file content twice although the call to set_len should extend the file:

    // cargo-deps: zbox="*"
    extern crate zbox;
    
    use ::std::io::Read;
    use ::std::io::Seek;
    use ::std::io::SeekFrom;
    
    fn main() {
        ::zbox::init_env();
    
        let mut repo = zbox::RepoOpener::new()
            .create(true)
            .open("mem://", "your password")
            .unwrap();
    
        let mut file = zbox::OpenOptions::new()
            .create(true)
            .write(true)
            .read(true)
            .open(&mut repo, "/my_file.txt")
            .unwrap();
        file.write_once(b"My file").unwrap();
        file.seek(SeekFrom::Start(0)).unwrap();
    
        let mut res = Vec::new();
        file.read_to_end(&mut res);
        println!("{:?}", res);
    
        file.set_len(15).unwrap();
        file.seek(SeekFrom::Start(0)).unwrap();
    
        let mut res2 = Vec::new();
        file.read_to_end(&mut res2);
        println!("{:?}", res2);
    }
    
    opened by althonos 7
  • Zbox characteristics as a filesystem

    Zbox characteristics as a filesystem

    Most of README is about encryption and security, also some about backend support.

    But I don't see typical filesystem metrics like maximum file size, maximum filesystem size, maximum number of files, overhead, fragmentation resilience and so on.

    Shall README also mention those parameters?

    For example, is zbox suitable for storage of very large number of small files? How slow is to delete those files afterwards?

    opened by vi 6
  • Absolute file URLs on Windows

    Absolute file URLs on Windows

    I'm able to use the file storage on Windows 10 using urls like file://../data/foo, but file://D:/data/foo fails on the open call with the following error message:

    thread 'main' panicked at 'opening archive failed: Io(Os { code: 123, kind: Other, message: "The filename, directory name, or volume label syntax is incorrect." })', src\libcore\result.rs:1189:5
    

    I'm generating the file URLs using Url::from_file_path, so the syntax should be ok. I can also paste this URL into Chrome and opens the folder view.

    opened by anlumo 5
  • ZBox claims misleading and inaccurate. Please consider rewording the lack of leakage.

    ZBox claims misleading and inaccurate. Please consider rewording the lack of leakage.

    SpiderOak made the same mistake with its misuse of the term "zero knowledge" and since migrated to "no knowledge", which hasn't fixed the semantic issue but it has avoided the overlap of terms - by coining another inaccurate term, which you've adopted.

    An extract from https://zbox.io/about/ , emphasis mine.

    Only encryption is not enough to protect privacy. In addition to strong encryption, Zbox also stores all data in a number of same-sized blocks, which leaves no details to any underlying storages, including Zbox cloud storage. By using this method, only [the] application itself can hold the key and have access to their data.

    "Same-sized blocks" may reduce some leakage but this does not imply that nothing remains leaked.

    1. The server knows which users interact with the service and when.
    2. The server knows which blocks the user reads.
    3. The server knows which blocks the user writes.

    To prevent (1) the users must identify to the service using a credential that demonstrates their authority without revealing their identity.

    To prevent (2) a Private Information Retrieval (PIR) or Oblivious Transfer (OT) protocol must be used; to prevent the server from learning which blocks the client asks for and to prevent the server from learning which value it responds with. PIR may reveal other blocks, potentially owned by other users. Even if encrypted, this is probably not desired. Limiting (2) to OT.

    To prevent (3) the dual to PIR whose name I've forgotten may be used to enable a client to upload a block without revealing to the server the block they've pushed into the set.

    In so far that I've read, there have been no mentions for PIR or OT or any form of anonymous credentials in ZBox' documentation. When a user reads your claims, they may be led into a false sense of security by correctly interpreting what you've written and not as intended.

    Please consider rewording these claims or invest into the cryptographic protocols that make them true. Obviously there are costs in the latter and the choices of which to use depends on how many "cloud" replicas there exist and who controls them. I.e. If there are several replicas owned by distinct parties then you can use some of the cheaper PIR protocols that rely on a ratio of the servers not colluding. If you've only one server, you'll need CPIR (computationally bound) which builds on Diffie-Hellman, or a computationally bound OT.


    BTW: The /about/ page says "ZBos" instead of "ZBox" in the 5th paragraph.

    opened by WildCryptoFox 4
  • Unable to open Sqlite storage

    Unable to open Sqlite storage

    Hi!

    I can create a sqlite backend file, open some files in it, read them and close. But i am not able to reopen the same sqlite file again. Then zbox crashes with the given error below. Somehow in a frame below the RepoOpener call.

    To reproduce simply execute the attached rust program two times.

    It seems that I am able to use the file backend, but not the sqlite backend. If I replace sqlite://./myRepoSqlite with file://./myRepo it works. :thinking:.

    use std::io::prelude::*;
    use std::io::{Seek, SeekFrom};
    use zbox::{init_env, OpenOptions, RepoOpener};
    
    fn main() {
        // initialise zbox environment, called first
        init_env();
    
        // create and open a repository
        let mut repo = RepoOpener::new()
            .create(true)
            .open("sqlite://./myRepoSqlite", "your password")
            .unwrap();
    
        // List files
        let things = repo.read_dir("/").unwrap();
        println!("{:?}", things);
    
        {
            // create and open a file for writing
            let mut file = OpenOptions::new()
                .create(true)
                .open(&mut repo, "/my_file.txt")
                .unwrap();
            
            file.seek(SeekFrom::End(0)).unwrap();
    
            // use std::io::Write trait to write data into it
            file.write_all(b"Hello, world!\n").unwrap();
    
            // finish writting to make a permanent content version
            file.finish().unwrap();
    
            // read file content using std::io::Read trait
            let mut content = String::new();
            file.seek(SeekFrom::Start(0)).unwrap();
            file.read_to_string(&mut content).unwrap();
    
            println!("{}", content);
        }
    }
    
    

    Dependencies:

    [dependencies]
    zbox = {version="0.8.2", features=["storage-file", "storage-sqlite"]}
    

    Error:

    RUST_BACKTRACE=1 cargo run
        Finished dev [unoptimized + debuginfo] target(s) in 0.05s
         Running `target/debug/crypting`
    thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2695:10
    stack backtrace:
       0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
                 at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
       1: std::sys_common::backtrace::_print
                 at src/libstd/sys_common/backtrace.rs:71
       2: std::panicking::default_hook::{{closure}}
                 at src/libstd/sys_common/backtrace.rs:59
                 at src/libstd/panicking.rs:197
       3: std::panicking::default_hook
                 at src/libstd/panicking.rs:211
       4: std::panicking::rust_panic_with_hook
                 at src/libstd/panicking.rs:474
       5: std::panicking::continue_panic_fmt
                 at src/libstd/panicking.rs:381
       6: rust_begin_unwind
                 at src/libstd/panicking.rs:308
       7: core::panicking::panic_fmt
                 at src/libcore/panicking.rs:85
       8: core::panicking::panic_bounds_check
                 at src/libcore/panicking.rs:61
       9: <usize as core::slice::SliceIndex<[T]>>::index
                 at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2695
      10: core::slice::<impl core::ops::index::Index<I> for [T]>::index
                 at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2552
      11: <alloc::vec::Vec<T> as core::ops::index::Index<I>>::index
                 at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/vec.rs:1687
      12: <zbox::volume::storage::sqlite::sqlite::SqliteStorage as zbox::volume::storage::Storable>::get_super_block
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/storage/sqlite/sqlite.rs:329
      13: zbox::volume::storage::storage::Storage::get_super_block
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/storage/storage.rs:196
      14: zbox::volume::super_block::SuperBlk::load_arm
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/super_block.rs:131
      15: zbox::volume::super_block::SuperBlk::load
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/super_block.rs:159
      16: zbox::volume::volume::Volume::open
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/volume.rs:95
      17: zbox::fs::fs::Fs::open
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/fs/fs.rs:167
      18: zbox::repo::Repo::open
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/repo.rs:682
      19: zbox::repo::RepoOpener::open
                 at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/repo.rs:247
      20: crypting::main
                 at src/main.rs:10
      21: std::rt::lang_start::{{closure}}
                 at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libstd/rt.rs:64
      22: std::panicking::try::do_call
                 at src/libstd/rt.rs:49
                 at src/libstd/panicking.rs:293
      23: __rust_maybe_catch_panic
                 at src/libpanic_unwind/lib.rs:85
      24: std::rt::lang_start_internal
                 at src/libstd/panicking.rs:272
                 at src/libstd/panic.rs:394
                 at src/libstd/rt.rs:48
      25: std::rt::lang_start
                 at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libstd/rt.rs:64
      26: main
      27: __libc_start_main
      28: _start
    

    Environment information:

    $ rustc -V && cargo version && uname -a && lsb_release -d
    rustc 1.36.0 (a53f9df32 2019-07-03)
    cargo 1.36.0 (c4fcfb725 2019-05-15)
    Linux STEFANO-PC 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux
    Description:	Arch Linux
    

    Zipped Sqlite file for inspection: myRepoSqlite.zip

    opened by senden9 4
  • Pre-bundled libsodium

    Pre-bundled libsodium

    Using zbox could be a little easier if libsodium was downloaded and built during cargo build. Although simple, libsodium is an extra configuration step, and It introduces complexity for local development and automated CI builds.

    The rust_sodium project does this in a custom build script.

    Zbox could offer an optional pre-bundled version with a feature flag e.g. cargo build --features libsodium-bundled which would download and build a recent stable build of libsodium.

    enhancement 
    opened by xmclark 4
  • Why is Zbox so slow?

    Why is Zbox so slow?

    The benchmark results in README show that Zbox's read/write throughput is one order of magnitude lower than the baseline. And my impression is that AES-GCM can easily achieve a throughput over 1GB/s. So, encryption/decryption should not be the root of the problem. Then, what is the performance bottleneck? Is it the Merkle trees (if Zbox uses them internally)? Is improving performance on the top of your TODO list?

    I believe Zbox is a very good idea. And it could be even more useful if the performance overhead is reduced to the minimal.

    There are no design docs about the internals of Zbox. And I haven't got the time to read the source code. So, I hope you can shed some light on the performance issue. Thanks.

    opened by tatetian 4
  • Fixed windows msvc download_and_install_libsodium in build.rs

    Fixed windows msvc download_and_install_libsodium in build.rs

    I was recently working on a project using zboz, from windows with msvc.

    I noticed that whenever I add the "libsodium-bundled" feature, cargo throws a File Not Found error, more precisely it cannot find libsodium.lib in the temporary ZIP archive. I checked and indeed the path is wrong. Currently I understand it to be:

    • libsodium/x64/Release/v142/static/libsodium.lib
    • libsodium/Win32/Release/v142/static/libsodium.lib

    Also after changing the paths, cargo throws another error but this time from reqwest: "reqwest::get" is asynchronous. I added the "blocking" feature in Cargo.toml only in the windows target and changed the code to "reqwest::blocking::get"

    opened by raccoman 1
  • storage-sqlite RepoOpener not creating path

    storage-sqlite RepoOpener not creating path

    let repo_dir = format!("sqlite://{}/vaults/default.sqlite", "./my-app-data-dir/");
    
    let mut repo = RepoOpener::new()
        .create(true)
        .open(&repo_dir, &master_password)
        .unwrap();
    
    println!("Done! {}", repo_dir);
    

    If path does not exist it will fail

    opened by raccoman 0
  • Question: reason not support symbolic link?

    Question: reason not support symbolic link?

    This project established a foundation to keep data in safe way. But in some situation we need symbolic link, can you please explain reason not support symbolic link? Especially if there have any technical stopper?

    opened by cetsupport 0
  • Failed open repo in file mode

    Failed open repo in file mode

    Errors:

    thread 'main' panicked at 'failed to open repo: InvalidUri', src/main.rs:14:10
    
    extern crate zbox;
    
    use std::io::{Read, Write, Seek, SeekFrom};
    use zbox::{init_env, RepoOpener, OpenOptions};
    
    fn main() {
        // initialise zbox environment, called first
        init_env();
    
        // create and open a repository in current OS directory
        let mut repo = RepoOpener::new()
            .create(true)
            .open("file://./my_repo", "your password")
            .expect("failed to open repo");
    
        // create and open a file in repository for writing
        let mut file = OpenOptions::new()
            .create(true)
            .open(&mut repo, "/my_file.txt")
            .unwrap();
    
        // use std::io::Write trait to write data into it
        file.write_all(b"Hello, World!").unwrap();
    
        // finish writing to make a permanent content version
        file.finish().unwrap();
    
        // read file content using std::io::Read trait
        let mut content = String::new();
        file.seek(SeekFrom::Start(0)).unwrap();
        file.read_to_string(&mut content).unwrap();
        assert_eq!(content, "Hello, World!");
    }
    
    [package]
    name = "zbox-try"
    version = "0.1.0"
    edition = "2021"
    
    # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
    
    [dependencies]
    zbox = "0.9.2"
    

    Versions:

    uname -a
    Darwin Eren-MacBook-Air.local 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 arm64
    -
    cargo -V
    cargo 1.60.0-nightly (25fcb135d 2022-02-01)
    -
    rustc --version
    rustc 1.60.0-nightly (f624427f8 2022-02-06)
    
    opened by xerenahmed 0
  • Snapshots?

    Snapshots?

    Does zboxfs support snapshots?

    Our application involves user media which can very easily become corrupt (e.g device removed during write), and the ability to roll back to a previous snapshot would be hugely beneficial.

    enhancement 
    opened by TomMettam 2
  • FR: `zbox::Repo::ongoing_transaction`

    FR: `zbox::Repo::ongoing_transaction`

    ZboxFS seems to think in terms of transactions. There are transaction-related error cases in zbox::Error, there are limitations based on transactions (e.g. inability to write multiple files in parallel), but there is few things about transaction available in API.

    If transactions are repository-wide (not file-scoped), I suggest to add something like fn ongoing_transaction(&self) -> Option<TransactionInfo>; in zbox::Repo, which should tell if a transaction is ongoing. Additional information like number of uncommitted bytes, affected file or datetime may be provided in TransactionInfo.

    wontfix 
    opened by vi 5
Releases(0.9.2)
  • 0.9.2(Sep 8, 2021)

  • 0.9.1(Jan 29, 2020)

  • 0.9.0(Jan 23, 2020)

    This is a binary incompatible minor release. In this release,

    • Added RepoOpener.dedup_file option, #72
    • Fixed crash issue when file storage space is full, #67
    • Fixed concurrency issue by adding exclusive tx, #44
    • Upgrade rust dependency to 1.38.0
    • Improved faulty fuzz test code
    • Other minor changes, upgrades and document improvement
    Source code(tar.gz)
    Source code(zip)
  • 0.8.8(Dec 26, 2019)

    This is a minor release. In this release,

    • Added Repo.copy_dir_all method, #53
    • VersionReader can retrieve version information now, #56
    • Repo can be destroyed using Repo.destroy method unconditionally, #57, #61
    • Fixed weak reference issue for File object so it cannot be read after repo is closed, #59
    • RepoInfo.uri method now doesn't mask secretive information in URI, #62
    • Replaced TravisCI with GitHub Action
    • Other minor changes and document improvement
    Source code(tar.gz)
    Source code(zip)
  • 0.8.7(Oct 12, 2019)

    This is a minor release. In this release,

    • Android binding is now released, check more details at https://github.com/zboxfs/zbox-android
    • Added force option to RepoOpener, #51
    • Memory storage is now persistent during the process's lifetime, #50
    • Empty location in URI is now invalid, #49
    • LZ4 source code is now embedded in source tree
    • Other minor changes and document improvement
    Source code(tar.gz)
    Source code(zip)
  • 0.8.6(Sep 9, 2019)

    This is a minor patch release. In this release,

    • Refactored wasm binding building pipeline to use wasm_pack
    • Upgraded lz4 to 1.9.2
    • Changed fnode sub_nodes cache eviction to commit phase
    • Moved content cache eviction to commit phase
    • Varies dependencies upgrades and document fixes
    Source code(tar.gz)
    Source code(zip)
  • 0.8.5(Sep 2, 2019)

    In this release,

    • Used correct hash key for sector id calculation in zbox storage, fix #46
    • Added missing shrank segment data buffer, fix #47
    • Changed order when adding new file version, fix #48
    • Fixed building issue on docs.rs
    • Upgraded some dependencies
    Source code(tar.gz)
    Source code(zip)
  • 0.8.4(Aug 24, 2019)

    This is a minor release to improve code quality and bug fix. In this release,

    • Fixed eid to cstr lifetime issue, #41
    • Replaced openssl with rust-tls, #43
    • Changed default version_limit to 1
    • Changed default dedup_chunk option to false
    • Other minor code and documentation improvements
    Source code(tar.gz)
    Source code(zip)
  • 0.8.3(Jul 10, 2019)

    This is a minor release to improve code quality and bug fix. In this release,

    • Fixed sqlite storage reopen issue #41
    • Moved out wasm binding to separate git repo
    • Removed wasm-logger dependency
    • Used Rust's new trait object syntax dyn
    Source code(tar.gz)
    Source code(zip)
  • 0.8.2(Jun 7, 2019)

    This release is mainly to add log level support for JavaScript binding. Now in js side initEnv can set different log level. For example,

    await zbox.initEnv({ logLevel: 'debug' });
    

    or turn off log output

    await zbox.initEnv({ logLevel: 'off' });
    
    Source code(tar.gz)
    Source code(zip)
  • 0.8.1(Jun 5, 2019)

    This is a bug fix release.

    • moved out c/c++ binding to separate repo: https://github.com/zboxfs/zbox-c
    • moved out Android Java binding to separate repo: https://github.com/zboxfs/zbox-android
    • fixed wrong c types definitions in lz4 binding
    • added missing zbox_version() for wasm binding
    Source code(tar.gz)
    Source code(zip)
  • 0.8.0(Jun 4, 2019)

    This release is a binary incompatible release.

    1. added super block repair functionality, the dual super blocks are identical now
    2. removed bytes crate dependency
    3. improved build.rs for windows
    4. move out docker image build files to separate git repos
    5. added zbox_version() function
    Source code(tar.gz)
    Source code(zip)
  • 0.7.1(Apr 24, 2019)

  • 0.7.0(Apr 23, 2019)

    This release is a API and binary incompatible release. Some major refactoring have been done in storage and transaction control.

    • Added WebAssembly support
    • Added zbox storage support
    • Introduced LSMT (Log Structured Merge Tree) as index manager for file and zbox storage
    • In storage interface, WAL IO is separated from data IO
    • Added Node.js binding
    • Removed external LZ4 dependency
    • Many other minor bug fixes and performance improvements
    Source code(tar.gz)
    Source code(zip)
  • 0.6.1(Sep 8, 2018)

    In this release,

    • Several bugs found in fuzz test have been fixed
    • Some missing FFI parts are completed
    • Two new docker images are created for building Zbox
    Source code(tar.gz)
    Source code(zip)
  • 0.6.0(Aug 31, 2018)

    This release is a binary incompatible release. There are some major changes have been done for volume, storage and transaction control.

    • Memory and file storage module have been rewritten, storage interface has been redesigned to be able to support more types of storage in the future
    • Volume module has been refactored, block and address cache are implemented in this module now, transaction operations are completed removed from volume and storage
    • Trans module has been rewritten, introduced Armor trait, Wal and WalQueue object
    • Fuzz test has been greatly simplified, now only one file can do all the fuzz test
    • New faulty storage is developed to support random IO error test, which is integrated with fuzz test
    • Compress is now optional and is disabled by default
    • Add a new dedup_chunk option to Repo and File, which can turn on/off the data chunk deduplication
    • Performance has been greatly improved due to the new volume and storage architecture design
    • Many other minor bug fixes
    Source code(tar.gz)
    Source code(zip)
  • 0.5.1(Apr 25, 2018)

    In this release, some major changes are done for behaviors of read/write and rename on File. In general, their behaviors are more like normal file operations.

    • Read/write will now affect the file position as normal Unix file operations will do
    • Read will always perform on the latest file version
    • Rename now behaves like rename(2)
    Source code(tar.gz)
    Source code(zip)
  • 0.5.0(Mar 25, 2018)

    In this release FFI interface is added, and provided C/C++ binding as the initial stage. In the future, more language binding will be added based on this FFI. Also, fixed some minor bugs.

    Source code(tar.gz)
    Source code(zip)
  • 0.4.0(Feb 9, 2018)

    This is a repository incompatible release, to upgrade to version all repository files need to be re-imported.

    In this release,

    • Add version_limit option to RepoOpener, which is the default settings for maximum number of file versions
    • Some repo and fs code refactoring
    Source code(tar.gz)
    Source code(zip)
  • 0.3.0(Feb 9, 2018)

    This is a repository incompatible release, to upgrade to version all repository files need to be re-imported.

    In this release,

    • Virtual IO layer is introduced to FileStorage to support random IO error test
    • Performance test and fuzz test are moved to integration test directory
    • FileStorage entity map number reduced to 4 in order to improve performance
    Source code(tar.gz)
    Source code(zip)
  • 0.2.1(Jan 20, 2018)

  • 0.2.0(Jan 20, 2018)

    Note: This release is NOT repo binary compatible with v0.1.4.

    In this release:

    • Change content split to be on chunk boundary
    • Segment can shrink now, this can dramatically reduce disk space waste
    • Add merkle tree as content hashing algorithm, improve 60% file storage write performance
    • Add performance test tool
    Source code(tar.gz)
    Source code(zip)
  • 0.1.4(Dec 20, 2017)

    • Add Windows support
    • Improve and add data persistence for fuzz tools
    • Fix several critical bugs in fnode and file storage
    • Some other code improvements
    Source code(tar.gz)
    Source code(zip)
  • 0.1.3(Dec 7, 2017)

    This release is mainly for changing misleading term zero-knowledge. There also some other fixes and improvements.

    • Fix issue in the sample code in README
    • Add create_new() function to RepoOpener.
    Source code(tar.gz)
    Source code(zip)
  • 0.1.2(Dec 5, 2017)

Owner
Zbox
Zero-details, privacy-focused cloud storage for applications.
Zbox
Open your operating system's file browser from Garry's Mod.

Open your operating system's file browser from Garry's Mod.

Earu 5 Apr 13, 2022
Temporary file library for rust

tempfile A secure, cross-platform, temporary file library for Rust. In addition to creating temporary files, this library also allows users to securel

Steven Allen 782 Dec 27, 2022
the file filesystem: mount semi-structured data (like JSON) as a Unix filesystem

ffs: the file filesystem ffs, the file filessytem, let's you mount semi-structured data as a fileystem---a tree structure you already know how to work

Michael Greenberg 176 Dec 31, 2022
⚡ Garry's Mod module that boosts performance by moving -condebug file I/O to a separate thread

This is a Garry's Mod server module that moves -condebug file I/O out of the main thread, which should significantly improve performance for noisy servers.

William 32 Dec 28, 2022
fftp is the "Fast File Transport Protocol". It transfers files quickly between computers on a network with low overhead.

fftp fftp is the "Fast File Transport Protocol". It transfers files quickly between computers on a network with low overhead. Motivation FTP uses two

leo 4 May 12, 2022
rswatch 🔎 is simple, efficient and reliable file watcher.

rswatch File watcher rswatch is a simple, reliable and efficient file watcher, it can watch a single file or a directory and run arbitrary commands wh

Eugene 3 Sep 23, 2022
Rustronomy-fits - a Rustronomy tool for FITS file I/O

Rustronomy-fits - a Rustronomy tool for FITS file I/O Rustronomy-fits provides I/O tools for reading, writing and parsing FITS files. It is currently

Raúl 3 Dec 8, 2022
Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.

Spacedrive A file explorer from the future. spacedrive.com » Download for macOS · Windows · Linux · iOS · watchOS · Android ~ Links will be added once

Spacedrive 16.2k Jan 7, 2023
Dufs is a distinctive utility file server that supports static serving, uploading, searching, accessing control, webdav...

Dufs (Old Name: Duf) Dufs is a distinctive utility file server that supports static serving, uploading, searching, accessing control, webdav... Featur

null 2.1k Dec 31, 2022
MASQ Network 121 Dec 20, 2022
An open source, programmed in rust, privacy focused tool for reading programming resources (like stackoverflow) fast, efficient and asynchronous from the terminal.

Falion An open source, programmed in rust, privacy focused tool for reading programming resources (like StackOverFlow) fast, efficient and asynchronou

Obscurely 17 Dec 20, 2022
Open-source tool to enforce privacy & security best-practices on Windows and macOS, because privacy is sexy 🍑🍆

privacy-sexy Open-source tool to enforce privacy & security best-practices on Windows and MacOs, because privacy is sexy ?? ?? privacy-sexy is a data-

Subconscious Compute 3 Oct 20, 2022
Enhance Rust errors with file and line details using the `#[wherr]` macro for clearer debugging.

wherr Crate Discuss wherr on Hacker News Enhance Rust's ? operator by appending file and line number details to errors, simplifying the debugging proc

Joel Jakobsson 49 Sep 6, 2023
Show details about outdated packages in your NixOS system.

nix-olde is a tool to show details about outdated packages in your NixOS system using https://repology.org/ database. It can use both default <nixpkgs

Sergei Trofimovich 14 Jan 24, 2023
Safeguard your financial privacy with zero-knowledge proofs.

Spinner The Spinner project (https://spinner.cash) takes a privacy first approach to protect users crypto assets. It is a layer-2 protocol built on th

Spinner 21 Dec 28, 2022
Detects whether a terminal supports color, and gives details about that support

Detects whether a terminal supports color, and gives details about that support. It takes into account the NO_COLOR environment variable. This crate i

Kat Marchán 30 Dec 29, 2022
The purpose of this example is to provide details as to how one would go about using GraphQL with the Rust Language.

Zero to GraphQL Using Rust The purpose of this example is to provide details as to how one would go about using GraphQL with the Rust Language. Thus,

Conrad Taylor 1 Jan 7, 2022
A Rust framework to develop and use plugins within your project, without worrying about the low-level details.

VPlugin: A plugin framework for Rust. Website | Issues | Documentation VPlugin is a Rust framework to develop and use plugins on applications and libr

VPlugin 11 Dec 31, 2022
CLI command for retrieving weather details.

rusty-forecast rusty-forecast is a Linux CLI command for retrieving weather details. INSTALLATION git clone https://github.com/husseinhareb/rusty-fo

Hussein Hareb 10 May 27, 2024
Tight Model format is a lossy 3D model format focused on reducing file size as much as posible without decreasing visual quality of the viewed model or read speeds.

What is Tight Model Format The main goal of the tmf project is to provide a way to save 3D game assets compressed in such a way, that there are no not

null 59 Mar 6, 2023