The project's description says

Zbox is a zero-knowledge, privacy-focused embeddable file system.

This seems to be false: there is nothing "zero knowledge" about this (or any) filesystem. Zero knowledge is a property of a proof; informally, it means that the proof reveals nothing except its correctness [1].

I think the term you're looking for is something closer to "semantic security" [2], meaning that the filesystem does not leak anything about the contents except their length.

I am aware that several other projects currently misuse this terminology in the same way you are. It's harmful to your users (in that it misinforms them), to the community (in that it confuses useful pieces of terminology), and to your project (in that it looks unschooled) to follow their example.

[1] https://en.wikipedia.org/wiki/Zero-knowledge_proof [2] https://en.wikipedia.org/wiki/Semantic_security

Hey there! As far as I can see, ZboxFS has its ways of using the native FS and other services, therefore making me wonder if it can be successfully compiled to WASM and use as an in-memory secure file system. Are there any plans for it?

Thanks!

I'm getting a SIGABRT on opening a repo. The error happens both with libsodium-bundled and with the system libsodium.

Host: Arch Linux Zbox: 0.8.3

fn main() {
    let mut repo = zbox::RepoOpener::new()
        .open("mem://", "password")
        // .open("file://file.zbox", "password")
        .unwrap();
}

Program received signal SIGABRT, Aborted.
0x00007ffff7dd1755 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7dd1755 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff7dbc851 in abort () from /usr/lib/libc.so.6
#2  0x0000555555587245 in sodium_misuse () at sodium/core.c:199
#3  0x0000555555715f7d in _sodium_malloc (size=<optimized out>)
    at sodium/utils.c:578
#4  sodium_malloc (size=<optimized out>) at sodium/utils.c:610
#5  0x00005555555ec5fa in zbox::base::crypto::SafeBox<T>::new_empty ()
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/base/crypto.rs:163
#6  0x000055555592178a in zbox::volume::storage::file::file_armor::FileArmor<T>::new (base=0x555555a74b60)
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/file/file_armor.rs:183
#7  0x0000555555595cc9 in zbox::volume::storage::file::file::FileStorage::new (
    base=0x55555599d007)
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/file/file.rs:40
#8  0x00005555555feaff in zbox::volume::storage::storage::parse_uri (uri=...)
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/storage.rs:45
#9  0x00005555555fec48 in zbox::volume::storage::storage::Storage::new (uri=...)
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/storage/storage.rs:144
#10 0x00005555555b328f in zbox::volume::volume::Volume::new (uri=...)
    at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/volume/volume.rs:42
#11 0x00005555555fc91d in zbox::fs::fs::Fs::open (uri=..., pwd=...,
--Type <RET> for more, q to quit, c to continue without paging--
     at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/fs/fs.rs:162
#12 0x000055555559087f in zbox::repo::Repo::open (uri=..., pwd=..., read_only=false) at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/repo.rs:682
#13 0x0000555555590266 in zbox::repo::RepoOpener::open (self=0x7fffffffe5b8, uri=..., pwd=...) at /home/theduke/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.3/src/repo.rs:252
#14 0x0000555555587520 in zbox_repro::main () at src/main.rs:2

The current explanation on the website is shallow and does not go deep into the internal workings of the library. For instance "Multiple storages, including memory, OS file system, RDBMS, key-value object store" does not explain how to interact with other storage systems like memory or RDBMS.

Another example, "State-of-the-art cryptography: AES-256-GCM (hardware), XChaCha20-Poly1305, Argon2 and etc." does not explain which is cryptography is used by default or how it's all interconnected.

It would be great if you explained more, or gave a document detailing the internal architecture of zbox.

Also the chat on gitter seems abandoned

Hi @burmecia,

another issue I stumbled upon (this is the last test case that fails against the pyfilesystem test suite): when a reader already exists for a file, extending the file with set_len and then reading the file (for instance with read_to_end) will not take into account the newly added bytes.

Here's a test case, that's shows that file.read_to_end will read the same file content twice although the call to set_len should extend the file:

// cargo-deps: zbox="*"
extern crate zbox;

use ::std::io::Read;
use ::std::io::Seek;
use ::std::io::SeekFrom;

fn main() {
    ::zbox::init_env();

    let mut repo = zbox::RepoOpener::new()
        .create(true)
        .open("mem://", "your password")
        .unwrap();

    let mut file = zbox::OpenOptions::new()
        .create(true)
        .write(true)
        .read(true)
        .open(&mut repo, "/my_file.txt")
        .unwrap();
    file.write_once(b"My file").unwrap();
    file.seek(SeekFrom::Start(0)).unwrap();

    let mut res = Vec::new();
    file.read_to_end(&mut res);
    println!("{:?}", res);

    file.set_len(15).unwrap();
    file.seek(SeekFrom::Start(0)).unwrap();

    let mut res2 = Vec::new();
    file.read_to_end(&mut res2);
    println!("{:?}", res2);
}

Most of README is about encryption and security, also some about backend support.

But I don't see typical filesystem metrics like maximum file size, maximum filesystem size, maximum number of files, overhead, fragmentation resilience and so on.

Shall README also mention those parameters?

For example, is zbox suitable for storage of very large number of small files? How slow is to delete those files afterwards?

I'm able to use the file storage on Windows 10 using urls like file://../data/foo, but file://D:/data/foo fails on the open call with the following error message:

thread 'main' panicked at 'opening archive failed: Io(Os { code: 123, kind: Other, message: "The filename, directory name, or volume label syntax is incorrect." })', src\libcore\result.rs:1189:5

I'm generating the file URLs using Url::from_file_path, so the syntax should be ok. I can also paste this URL into Chrome and opens the folder view.

SpiderOak made the same mistake with its misuse of the term "zero knowledge" and since migrated to "no knowledge", which hasn't fixed the semantic issue but it has avoided the overlap of terms - by coining another inaccurate term, which you've adopted.

An extract from https://zbox.io/about/ , emphasis mine.

Only encryption is not enough to protect privacy. In addition to strong encryption, Zbox also stores all data in a number of same-sized blocks, which leaves no details to any underlying storages, including Zbox cloud storage. By using this method, only [the] application itself can hold the key and have access to their data.

"Same-sized blocks" may reduce some leakage but this does not imply that nothing remains leaked.

1. The server knows which users interact with the service and when.
2. The server knows which blocks the user reads.
3. The server knows which blocks the user writes.

To prevent (1) the users must identify to the service using a credential that demonstrates their authority without revealing their identity.

To prevent (2) a Private Information Retrieval (PIR) or Oblivious Transfer (OT) protocol must be used; to prevent the server from learning which blocks the client asks for and to prevent the server from learning which value it responds with. PIR may reveal other blocks, potentially owned by other users. Even if encrypted, this is probably not desired. Limiting (2) to OT.

To prevent (3) the dual to PIR whose name I've forgotten may be used to enable a client to upload a block without revealing to the server the block they've pushed into the set.

In so far that I've read, there have been no mentions for PIR or OT or any form of anonymous credentials in ZBox' documentation. When a user reads your claims, they may be led into a false sense of security by correctly interpreting what you've written and not as intended.

Please consider rewording these claims or invest into the cryptographic protocols that make them true. Obviously there are costs in the latter and the choices of which to use depends on how many "cloud" replicas there exist and who controls them. I.e. If there are several replicas owned by distinct parties then you can use some of the cheaper PIR protocols that rely on a ratio of the servers not colluding. If you've only one server, you'll need CPIR (computationally bound) which builds on Diffie-Hellman, or a computationally bound OT.

BTW: The /about/ page says "ZBos" instead of "ZBox" in the 5th paragraph.

Hi!

I can create a sqlite backend file, open some files in it, read them and close. But i am not able to reopen the same sqlite file again. Then zbox crashes with the given error below. Somehow in a frame below the RepoOpener call.

To reproduce simply execute the attached rust program two times.

It seems that I am able to use the file backend, but not the sqlite backend. If I replace sqlite://./myRepoSqlite with file://./myRepo it works. :thinking:.

use std::io::prelude::*;
use std::io::{Seek, SeekFrom};
use zbox::{init_env, OpenOptions, RepoOpener};

fn main() {
    // initialise zbox environment, called first
    init_env();

    // create and open a repository
    let mut repo = RepoOpener::new()
        .create(true)
        .open("sqlite://./myRepoSqlite", "your password")
        .unwrap();

    // List files
    let things = repo.read_dir("/").unwrap();
    println!("{:?}", things);

    {
        // create and open a file for writing
        let mut file = OpenOptions::new()
            .create(true)
            .open(&mut repo, "/my_file.txt")
            .unwrap();
        
        file.seek(SeekFrom::End(0)).unwrap();

        // use std::io::Write trait to write data into it
        file.write_all(b"Hello, world!\n").unwrap();

        // finish writting to make a permanent content version
        file.finish().unwrap();

        // read file content using std::io::Read trait
        let mut content = String::new();
        file.seek(SeekFrom::Start(0)).unwrap();
        file.read_to_string(&mut content).unwrap();

        println!("{}", content);
    }
}

Dependencies:

[dependencies]
zbox = {version="0.8.2", features=["storage-file", "storage-sqlite"]}

Error:

RUST_BACKTRACE=1 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/crypting`
thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2695:10
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: rust_begin_unwind
             at src/libstd/panicking.rs:308
   7: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
   8: core::panicking::panic_bounds_check
             at src/libcore/panicking.rs:61
   9: <usize as core::slice::SliceIndex<[T]>>::index
             at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2695
  10: core::slice::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libcore/slice/mod.rs:2552
  11: <alloc::vec::Vec<T> as core::ops::index::Index<I>>::index
             at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/liballoc/vec.rs:1687
  12: <zbox::volume::storage::sqlite::sqlite::SqliteStorage as zbox::volume::storage::Storable>::get_super_block
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/storage/sqlite/sqlite.rs:329
  13: zbox::volume::storage::storage::Storage::get_super_block
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/storage/storage.rs:196
  14: zbox::volume::super_block::SuperBlk::load_arm
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/super_block.rs:131
  15: zbox::volume::super_block::SuperBlk::load
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/super_block.rs:159
  16: zbox::volume::volume::Volume::open
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/volume/volume.rs:95
  17: zbox::fs::fs::Fs::open
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/fs/fs.rs:167
  18: zbox::repo::Repo::open
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/repo.rs:682
  19: zbox::repo::RepoOpener::open
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/zbox-0.8.2/src/repo.rs:247
  20: crypting::main
             at src/main.rs:10
  21: std::rt::lang_start::{{closure}}
             at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libstd/rt.rs:64
  22: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:293
  23: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:85
  24: std::rt::lang_start_internal
             at src/libstd/panicking.rs:272
             at src/libstd/panic.rs:394
             at src/libstd/rt.rs:48
  25: std::rt::lang_start
             at /rustc/a53f9df32fbb0b5f4382caaad8f1a46f36ea887c/src/libstd/rt.rs:64
  26: main
  27: __libc_start_main
  28: _start

Environment information:

$ rustc -V && cargo version && uname -a && lsb_release -d
rustc 1.36.0 (a53f9df32 2019-07-03)
cargo 1.36.0 (c4fcfb725 2019-05-15)
Linux STEFANO-PC 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux
Description:	Arch Linux

Zipped Sqlite file for inspection: myRepoSqlite.zip

Using zbox could be a little easier if libsodium was downloaded and built during cargo build. Although simple, libsodium is an extra configuration step, and It introduces complexity for local development and automated CI builds.

The rust_sodium project does this in a custom build script.

Zbox could offer an optional pre-bundled version with a feature flag e.g. cargo build --features libsodium-bundled which would download and build a recent stable build of libsodium.

The benchmark results in README show that Zbox's read/write throughput is one order of magnitude lower than the baseline. And my impression is that AES-GCM can easily achieve a throughput over 1GB/s. So, encryption/decryption should not be the root of the problem. Then, what is the performance bottleneck? Is it the Merkle trees (if Zbox uses them internally)? Is improving performance on the top of your TODO list?

I believe Zbox is a very good idea. And it could be even more useful if the performance overhead is reduced to the minimal.

There are no design docs about the internals of Zbox. And I haven't got the time to read the source code. So, I hope you can shed some light on the performance issue. Thanks.

I was recently working on a project using zboz, from windows with msvc.

I noticed that whenever I add the "libsodium-bundled" feature, cargo throws a File Not Found error, more precisely it cannot find libsodium.lib in the temporary ZIP archive. I checked and indeed the path is wrong. Currently I understand it to be:

• libsodium/x64/Release/v142/static/libsodium.lib
• libsodium/Win32/Release/v142/static/libsodium.lib

Also after changing the paths, cargo throws another error but this time from reqwest: "reqwest::get" is asynchronous. I added the "blocking" feature in Cargo.toml only in the windows target and changed the code to "reqwest::blocking::get"

let repo_dir = format!("sqlite://{}/vaults/default.sqlite", "./my-app-data-dir/");

let mut repo = RepoOpener::new()
    .create(true)
    .open(&repo_dir, &master_password)
    .unwrap();

println!("Done! {}", repo_dir);

If path does not exist it will fail

This project established a foundation to keep data in safe way. But in some situation we need symbolic link, can you please explain reason not support symbolic link? Especially if there have any technical stopper?

Errors:

thread 'main' panicked at 'failed to open repo: InvalidUri', src/main.rs:14:10

extern crate zbox;

use std::io::{Read, Write, Seek, SeekFrom};
use zbox::{init_env, RepoOpener, OpenOptions};

fn main() {
    // initialise zbox environment, called first
    init_env();

    // create and open a repository in current OS directory
    let mut repo = RepoOpener::new()
        .create(true)
        .open("file://./my_repo", "your password")
        .expect("failed to open repo");

    // create and open a file in repository for writing
    let mut file = OpenOptions::new()
        .create(true)
        .open(&mut repo, "/my_file.txt")
        .unwrap();

    // use std::io::Write trait to write data into it
    file.write_all(b"Hello, World!").unwrap();

    // finish writing to make a permanent content version
    file.finish().unwrap();

    // read file content using std::io::Read trait
    let mut content = String::new();
    file.seek(SeekFrom::Start(0)).unwrap();
    file.read_to_string(&mut content).unwrap();
    assert_eq!(content, "Hello, World!");
}

[package]
name = "zbox-try"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
zbox = "0.9.2"

Versions:

uname -a
Darwin Eren-MacBook-Air.local 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 arm64
-
cargo -V
cargo 1.60.0-nightly (25fcb135d 2022-02-01)
-
rustc --version
rustc 1.60.0-nightly (f624427f8 2022-02-06)

Does zboxfs support snapshots?

Our application involves user media which can very easily become corrupt (e.g device removed during write), and the ability to roll back to a previous snapshot would be hugely beneficial.

ZboxFS seems to think in terms of transactions. There are transaction-related error cases in zbox::Error, there are limitations based on transactions (e.g. inability to write multiple files in parallel), but there is few things about transaction available in API.

If transactions are repository-wide (not file-scoped), I suggest to add something like fn ongoing_transaction(&self) -> Option<TransactionInfo>; in zbox::Repo, which should tell if a transaction is ongoing. Additional information like number of uncommitted bytes, affected file or datetime may be provided in TransactionInfo.