the file filesystem: mount semi-structured data (like JSON) as a Unix filesystem

Overview

ffs: the file filesystem

Main workflow Crates.io

ffs, the file filessytem, let's you mount semi-structured data as a fileystem---a tree structure you already know how to work with!

Working with semi-structured data using command-line tools is hard. Tools like jq help a lot, but learning a new language for simple manipulations is a big ask. By mapping hard-to-parse trees into a filesystem, you can keep using the tools you know.

Example

Run ffs [file] to mount file.blah at the mountpoint file. The final, updated version of the file will be outputted on stdout.

object/name $ echo 1 >object/nose $ mkdir object/pockets $ cd object/pockets/ $ echo keys >pants $ echo pen >shirt $ cd .. $ cd .. $ umount object $ [1]+ Done ffs -o object_edited.json object.json $ cat object_edited.json {"eyes":2,"fingernails":10,"human":true,"name":"Mikey Indiana","nose":1,"pockets":{"pants":"keys","shirt":"pen"}} ">
$ cat object.json 
{ "name": "Michael Greenberg", "eyes": 2, "fingernails": 10, "human": true }
$ ffs -o object_edited.json object.json &
[1] 60182
$ tree object
object
├── eyes
├── fingernails
├── human
└── name

0 directories, 4 files
$ echo Mikey Indiana >object/name
$ echo 1 >object/nose
$ mkdir object/pockets
$ cd object/pockets/
$ echo keys >pants
$ echo pen >shirt
$ cd ..
$ cd ..
$ umount object
$ 
[1]+  Done                    ffs -o object_edited.json object.json
$ cat object_edited.json 
{"eyes":2,"fingernails":10,"human":true,"name":"Mikey Indiana","nose":1,"pockets":{"pants":"keys","shirt":"pen"}}

You can specify an explicit mountpoint by running ffs -m MOUNT file; you can specify an output file with -o OUTPUT. You can edit a file in place by running ffs -i file---when the volume is unmounted, the resulting output will be written back to file.

External dependencies

You need an appropriate FUSE or macFUSE along with pkg-config.

See the GitHub build workflow for examples of external dependency installation.

Comments
  • Use `Filesystem::destroy` instead of `Drop::drop`

    Use `Filesystem::destroy` instead of `Drop::drop`

    Per https://github.com/cberner/fuser/issues/153#issuecomment-864348771, better to rely on the explicit contract than on the the current implementation in fuser calling mem::drop.

    enhancement upstream 
    opened by mgree 3
  • Support other formats.

    Support other formats.

    Broad refactor of how things work.

    Adds support for TOML, along with CLI stuff for setting formats (which are automatically inferred from filenames).

    You can convert between things!

    opened by mgree 2
  • Output broken on Linux

    Output broken on Linux

    • [x] ~~choose target~~ ba64e76
    • [x] ~~actually output~~ ba64e76
    • [x] ~~fsync causes output (see #4)~~ doesn't seem to actually work; see 867cb6b for broken test
    bug 
    opened by mgree 2
  • Invalid mount attempt leaves behind empty directory/member

    Invalid mount attempt leaves behind empty directory/member

    Trying to mount a JSON file within the virtual filesystem fails (which is ok, I guess):

    [email protected] ~/Downloads> ./ffs -i ffs1.json [email protected] ~/Downloads> cat ffs1.json {"option":"test"} [email protected] ~/Downloads> cp ffs1.json ffs1/ [email protected] ~/Downloads> ls ffs1/ ffs1.json option [email protected] ~/Downloads> ./ffs -i ffs1/ffs1.json fusermount: bad mount point ffs1/ffs1: Permission denied thread 'main' panicked at 'called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/main.rs:38:41 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

    This actually creates the subdirectory ffs1/ffs1 but does not mount anything there. Unmounting the filesystem results in an additional entry "ffs1: {}" in the json file. I'd suggest that a failed command should not leave behind any new files/members.

    opened by RaphaelWimmer 1
  • Name munging is never undone

    Name munging is never undone

    The JSON object { ".": 5 } will create a file named dot holding the number 5... but it will dump back to JSON without restoring the name.

    The best solution is probably to track an "unmunged" name, and files that don't undergo a rename call go back exactly how they were.

    This unmunged name is metadata, so the appropriate mechanism for working with it is up in the air (see #2).

    bug 
    opened by mgree 1
  • Revise mount CLI interface

    Revise mount CLI interface

    Rather than taking a mountpoint as an argument, use an optional --mount flag.

    If the input is over STDIN, you have to specify a --mount.

    If the input is from a file foo.EXT, then try to make a directory foo. If that file already exists, give up and tell the user about --mount. If it doesn't, great: do that. When things unmount, double check the directory is empty and then remove it.

    Need to revise all the tests, and then test a few configurations of this feature.

    enhancement 
    opened by mgree 1
  • Invalid unmount on macOS

    Invalid unmount on macOS

    Investigate weird unmounts on macOS, producing:

    WARN fuser::mnt::fuse2: umount failed with Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
    

    See actions log. Linux seems fine.

    bug 
    opened by mgree 1
  • Lazy loading

    Lazy loading

    Lazily populate the FS. The current implementation passes tests but has hefty technical debt.

    • [x] Rename fs.rs into eager.rs. 24d9704
    • [x] Use the new Nodelike parsing/unparsing logic to simplify loading for eager::FS, too. 9268a77
    • [x] Let lazy loading rewrite input into a different output format by calling resolve_node to fixpoint. aba72bc
    • [x] CLI flags for controlling laziness. df22c78

    It's tempting to have eager mode work with the lazy FS by just forcing resolution during startup. There would be some overhead, though. How much? We can do benchmarking on three modes: lazy, lazy with --force-early, and the existing eager. dd152f1 shows that lazy with --force-early is on par with eager. So once we can have lazy outputting in whatever format, we can drop eager entirely.

    opened by mgree 0
  • Fix nested mount bug

    Fix nested mount bug

    #42 reports an issue with nested mounts (which are just disallowed by FUSE and macFUSE), which we were handling poorly.

    Check that mount2 worked correctly and report messages.

    opened by mgree 0
  • Empty file mounting fails

    Empty file mounting fails

    $> touch x.json $> ffs x.son thread 'main' panicked at 'JSON: Error("EOF while parsing a value", line: 1, column: 0)', src/format.rs:187:53 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

    enhancement format 
    opened by nkh 3
  • Auto-detect list directories

    Auto-detect list directories

    An idea from @angelhof: notice directories where every file has numerical name suffixes (e.g., role0, role1, or simpliciter 0, 1, 2, ...) and automatically treat them as lists.

    The best way to do this is to use the auto type for directories, too, and new directories have that type assigned.

    enhancement 
    opened by mgree 0
  • Better error messages

    Better error messages

    On a malformed JSON file:

    : [email protected]:~/talks/20211025plos [master] ; ~/ffs/target/release/ffs -i demo.json 
    thread 'main' panicked at 'JSON: Error("expected value", line: 7, column: 12)', src/format.rs:322:45
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    
    enhancement 
    opened by mgree 0
  • Signal support

    Signal support

    For example i3blocks supports multiple signals which you can trigger using pkill -SIGRTMIN+10 i3blocks, which then causes an action.

    Could you add atleast two signals, one for re-reading INPUT file and another for writing OUTPUT file. This would open ffs for uses in scripts and would allow updating INPUT and/or OUTPUT files without needing to restart ffs.

    enhancement 
    opened by grimnight 3
  • Pack/unpack approach

    Pack/unpack approach

    Just load/unload data directly on the filesystem.

    Progress indicator. https://docs.rs/indicatif/0.16.2/indicatif/

    Multithreading? (Is that ever a performance win?)

    Pack

    BFS?

    Unpack

    Flag to follow symlinks, possibly even out of the root.

    -i flag for prompting on weirdness.

    enhancement 
    opened by mgree 0
Releases(latest)
Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.

Spacedrive A file explorer from the future. spacedrive.com » Download for macOS · Windows · Linux · iOS · watchOS · Android ~ Links will be added once

Spacedrive 15.7k Nov 30, 2022
The reference implementation of the Linux FUSE (Filesystem in Userspace) interface

libfuse About FUSE (Filesystem in Userspace) is an interface for userspace programs to export a filesystem to the Linux kernel. The FUSE project consi

null 4.2k Dec 3, 2022
A Tauri Plugin to watch the filesystem for changes

Tauri Plugin FSWatch This plugin provides a "classical" Tauri Plugin Interface to watch changes on files and directories through notify. Architecture

Tauri 32 Nov 24, 2022
Supertag is a tag-based filesystem, written in Rust, for Linux and MacOS

Supertag is a tag-based filesystem, written in Rust, for Linux and MacOS. It provides a tag-based view of your files by removing the hierarchy constraints typically imposed on files and folders. In other words, it allows you to think about your files not as objects stored in folders, but as objects that can be filtered by folders.

Andrew Moffat 536 Nov 22, 2022
A virtual filesystem layer for WASI

wasi-vfs A virtual filesystem layer for WASI. NOTICE: This project currently supports only WASI applications on the top of wasi-libc This project prov

Yuta Saito 61 Nov 23, 2022
Find files with SQL-like queries

Find files with SQL-like queries

null 3.4k Dec 1, 2022
Temporary file library for rust

tempfile A secure, cross-platform, temporary file library for Rust. In addition to creating temporary files, this library also allows users to securel

Steven Allen 754 Nov 26, 2022
Zero-details, privacy-focused in-app file system.

ZboxFS ZboxFS is a zero-details, privacy-focused in-app file system. Its goal is to help application store files securely, privately and reliably. By

Zbox 1.4k Dec 2, 2022
⚡ Garry's Mod module that boosts performance by moving -condebug file I/O to a separate thread

This is a Garry's Mod server module that moves -condebug file I/O out of the main thread, which should significantly improve performance for noisy servers.

William 30 Nov 21, 2022
fftp is the "Fast File Transport Protocol". It transfers files quickly between computers on a network with low overhead.

fftp fftp is the "Fast File Transport Protocol". It transfers files quickly between computers on a network with low overhead. Motivation FTP uses two

leo 4 May 12, 2022
rswatch 🔎 is simple, efficient and reliable file watcher.

rswatch File watcher rswatch is a simple, reliable and efficient file watcher, it can watch a single file or a directory and run arbitrary commands wh

Eugene 3 Sep 23, 2022
Open your operating system's file browser from Garry's Mod.

Open your operating system's file browser from Garry's Mod.

Earu 5 Apr 13, 2022
Rustronomy-fits - a Rustronomy tool for FITS file I/O

Rustronomy-fits - a Rustronomy tool for FITS file I/O Rustronomy-fits provides I/O tools for reading, writing and parsing FITS files. It is currently

Raúl 2 Oct 29, 2022
Minty is an amazingly fast file deduplication app built in rust with a rust user interface.

minty Project Minty has a new look and feel!!! Minty is an amazingly fast file deduplication app built in rust with a rust user interface. I say super

null 26 Nov 20, 2022
Dufs is a distinctive utility file server that supports static serving, uploading, searching, accessing control, webdav...

Dufs (Old Name: Duf) Dufs is a distinctive utility file server that supports static serving, uploading, searching, accessing control, webdav... Featur

null 1.7k Nov 29, 2022
Simple utility to backup/sync data between devices to the cloud

RSink Simple utility to backup/sync data between devices to the cloud Features ?? Blazingly Fast ⛽ Lightweight ?? Cross-platform, runs everywhere ?? S

Abdulrahman Salah 4 Sep 3, 2022
shavee is a Program to automatically decrypt and mount ZFS datasets using Yubikey HMAC as 2FA or any USB drive with support for PAM to auto mount home directories.

shavee is a simple program to decrypt and mount encrypted ZFS user home directories at login using Yubikey HMAC or a Simple USB drive as 2FA written in rust.

Ashutosh Verma 36 Nov 25, 2022
Valq - macros for querying and extracting value from structured data by JavaScript-like syntax

valq   valq provides a macro for querying and extracting value from structured data in very concise manner, like the JavaScript syntax. Look & Feel: u

Takumi Fujiwara 22 Oct 17, 2022
High level FFI binding around the sys mount & umount2 calls, for Rust

sys-mount High level FFI bindings to the mount and umount2 system calls, for Rust. Examples Mount This is how the mount command could be written with

Pop!_OS 29 Nov 7, 2022
mount nhentai as a folder

Why? why not Seriously, why? a friend misread a mount output filesystem as "nsfw" so i had to make it real How do I use this cargo run <mount point> <

null 14 Jun 14, 2021