The reference implementation of the Linux FUSE (Filesystem in Userspace) interface

Related tags

libfuse
Overview

libfuse

About

FUSE (Filesystem in Userspace) is an interface for userspace programs to export a filesystem to the Linux kernel. The FUSE project consists of two components: the fuse kernel module (maintained in the regular kernel repositories) and the libfuse userspace library (maintained in this repository). libfuse provides the reference implementation for communicating with the FUSE kernel module.

A FUSE file system is typically implemented as a standalone application that links with libfuse. libfuse provides functions to mount the file system, unmount it, read requests from the kernel, and send responses back. libfuse offers two APIs: a "high-level", synchronous API, and a "low-level" asynchronous API. In both cases, incoming requests from the kernel are passed to the main program using callbacks. When using the high-level API, the callbacks may work with file names and paths instead of inodes, and processing of a request finishes when the callback function returns. When using the low-level API, the callbacks must work with inodes and responses must be sent explicitly using a separate set of API functions.

Development Status

libfuse is shipped by all major Linux distributions and has been in production use across a wide range of systems for many years. However, at present libfuse does not have any active, regular contributors. The current maintainer continues to apply pull requests and makes regular releases, but unfortunately has no capacity to do any development beyond addressing high-impact issues. When reporting bugs, please understand that unless you are including a pull request or are reporting a critical issue, you will probably not get a response. If you are using libfuse, please consider contributing to the project.

Supported Platforms

  • Linux (fully)
  • BSD (mostly/best-effort)
  • For OS-X, please use OSXFUSE

Installation

You can download libfuse from https://github.com/libfuse/libfuse/releases. To build and install, you must use Meson and Ninja. After extracting the libfuse tarball, create a (temporary) build directory and run Meson:

$ mkdir build; cd build
$ meson ..

Normally, the default build options will work fine. If you nevertheless want to adjust them, you can do so with the meson configure command:

$ meson configure # list options
$ meson configure -D disable-mtab=true # set an option

To build, test, and install libfuse, you then use Ninja:

$ ninja
$ sudo python3 -m pytest test/
$ sudo ninja install

Running the tests requires the py.test Python module. Instead of running the tests as root, the majority of tests can also be run as a regular user if util/fusermount3 is made setuid root first:

$ sudo chown root:root util/fusermount3
$ sudo chmod 4755 util/fusermount3
$ python3 -m pytest test/

Security implications

The fusermount3 program is installed setuid root. This is done to allow normal users to mount their own filesystem implementations.

To limit the harm that malicious users can do this way, fusermount3 enforces the following limitations:

  • The user can only mount on a mountpoint for which they have write permission

  • The mountpoint must not be a sticky directory which isn't owned by the user (like /tmp usually is)

  • No other user (including root) can access the contents of the mounted filesystem (though this can be relaxed by allowing the use of the allow_other and allow_root mount options in /etc/fuse.conf)

If you intend to use the allow_other mount options, be aware that FUSE has an unresolved security bug: if the default_permissions mount option is not used, the results of the first permission check performed by the file system for a directory entry will be re-used for subsequent accesses as long as the inode of the accessed entry is present in the kernel cache - even if the permissions have since changed, and even if the subsequent access is made by a different user. This is of little concern if the filesystem is accessible only to the mounting user (which has full access to the filesystem anyway), but becomes a security issue when other users are allowed to access the filesystem (since they can exploit this to perform operations on the filesystem that they do not actually have permissions for).

This bug needs to be fixed in the Linux kernel and has been known since 2006 but unfortunately no fix has been applied yet. If you depend on correct permission handling for FUSE file systems, the only workaround is to use default_permissions (which does not currently support ACLs), or to completely disable caching of directory entry attributes.

Building your own filesystem

FUSE comes with several example file systems in the example directory. For example, the passthrough examples mirror the contents of the root directory under the mountpoint. Start from there and adapt the code!

The documentation of the API functions and necessary callbacks is mostly contained in the files include/fuse.h (for the high-level API) and include/fuse_lowlevel.h (for the low-level API). An autogenerated html version of the API is available in the doc/html directory and at http://libfuse.github.io/doxygen.

Getting Help

If you need help, please ask on the [email protected] mailing list (subscribe at https://lists.sourceforge.net/lists/listinfo/fuse-devel).

Please report any bugs on the GitHub issue tracker at https://github.com/libfuse/libfuse/issues.

Issues
  • Allow passing `/dev/fuse` file descriptor from parent process

    Allow passing `/dev/fuse` file descriptor from parent process

    This adds a new "pre-mounted" mode of operation in which the FUSE file system helper is launched after the /dev/fuse file descriptor has been opened opened and the file system been mounted by a suitably privileged process.

    Pre-mounted mode is requested by passing an empty string as mountpoint, which reflects the fact that the FUSE helper doesn't actually perform or trigger any mounting itself.

    The main benefit of pre-mounted mode is that no privileged operations need to be performed by the file system implementation itself directly or indirectly, so the FUSE process can run with zero privilege. Moreover, mechanisms like securebits and no_new_privs can be used to prevent subprocesses from re-acquiring privilege, which further helps reduce risk for the case the FUSE helper gets exploited by a malicious file system.

    Below is an example that illustrates this. Note that I'm using shell for presentation purposes, the assumption is that there's a privileged daemon that handles mounting and spawning the FUSE helper in a suitable sandbox.

    Make binaries and libs executable by anyone

    $ chmod o+rx build/lib/libfuse3.so* example/hello $ export LD_LIBRARY_PATH=$PWD/build/lib

    example/hello can mount successfully with privilege

    $ sudo sh -c "LD_LIBRARY_PATH=build/lib ./example/hello /mnt/tmp" $ sudo cat /mnt/tmp/hello Hello World! $ sudo umount /mnt/tmp

    example/hello fails to mount without privilege

    $ sudo capsh --drop=all --secbits=0x3f -- -c 'LD_LIBRARY_PATH=build/lib ./example/hello -f /mnt/tmp' fusermount3: mount failed: Operation not permitted

    Pre-mounting allows example/hello to work without privilege

    $ sudo sh -c ' exec 0<>/dev/fuse mount -i -o nodev,nosuid,noexec,fd=0,rootmode=40000,user_id=0,group_id=0 -t fuse hello /mnt/tmp capsh --drop=all --secbits=0x3f -- -c "LD_LIBRARY_PATH=build/lib example/hello -f """ ' & [1] 55491 $ sudo cat /mnt/tmp/hello Hello World! $ sudo umount /mnt/tmp [1]+ Done sudo sh -c ' exec 0<>/dev/fuse mount -i -o nodev,nosuid,noexec,fd=0,rootmode=40000,user_id=0,group_id=0 -t fuse hello /mnt/tmp capsh --drop=all --secbits=0x3f -- -c "LD_LIBRARY_PATH=build/lib example/hello -f """ '

    opened by saittam 41
  • Deadlock in get_path2()

    Deadlock in get_path2()

    We have some testcases that prove this comment correct ๐Ÿ˜„

    /* FIXME: locking two paths needs deadlock checking */

    (https://github.com/libfuse/libfuse/blob/master/lib/fuse.c#L1258)

    Considering tackling this since I can reproduce the problem. Did you have any thoughts as to how a fix should be implemented?

    bug 
    opened by bnaylor 32
  • passthrough_ll update

    passthrough_ll update

    This fixes a couple of bugs in the passthrough_ll example filesystem and adds support for lots of missing operations. Options are added to configure a base path and to control caching.

    opened by szmi 20
  • Advise file system developers not to use file handles in nonportable ways

    Advise file system developers not to use file handles in nonportable ways

    Background

    The FUSE protocol allows a server to assign a different filehandle each and every time that a file is opened. Multiple filehandles can be concurrently valid for the same file. Operations like FUSE_READ and FUSE_WRITE are supposed to include the file handle associated with the particular file descriptor that initiated the operation. However, that isn't possible in all cases. The FUSE_WRITE_CACHE bit indicates whether the file handle is valid for a FUSE_WRITE operation, and the FATTR_FH flag indicates whether it's valid for a FUSE_SETATTR operation.

    Problem

    There are other cases, too, when the file handle can be invalid. In fact, the entire concept of a file handle is really a Linuxism. Other operating systems enforce a distinction between the file layer and the vnode layer. That's why OSX, FreeBSD, OpenBSD, and Illumos can almost never guarantee that they're sending the right file handle for any operation. NetBSD takes a more extreme position; it doesn't allow multiple concurrent fuse file handles for a single file (which leads to other problems). But since operations other than FUSE_WRITE and FUSE_SETATTR don't have anything analogous to the FUSE_WRITE_CACHE or FATTR_FH bits, there's no way for those operating systems to indicate that the file handle is guessed.

    Mitigating Circumstances

    The impact of this bug is small simply because few if any fuse servers care about the accuracy of fuse file handles. I surveyed 42 real-world FUSE filesystems* and none of them ever check the writepage bit. Only one of them (catfs) checks FATTR_FH, and it can deal with the absence of that flag. I can't find any evidence that any FUSE filesystem actually wants to assign distinct file handles on each concurrent FUSE_OPEN and requires the FUSE client to keep them straight.

    Solution

    Ideally the FUSE protocol would require servers to assign the same file handle for every concurrent open of a single file. Unfortunately, that's not the current situation and such a change would be backwards-incompatible. The second best solution would be to update the kernel protocol to include some equivalent of the FUSE_WRITE_CACHE bit for every operation that includes a file handle. However, that would require all clients and servers alike to update their kernel API version, so uptake would be extremely slow. The path of least resistance would simply be to document the problem. I recommend the following:

    • Advise FUSE server developers to assign the same file handle for every concurrent open of a single file.
    • Rename the writepage bit to fh_invalid, and add a #define for backwards-compatibility
    • Patch libfuse to unconditionally set fh_invalid for most commands on non-Linux operating systems.

    If the maintainers agree, I'll prepare a PR.

    cc @cemeyer @kendmerry

    *: afuse, chironfs, cryptofs, curlftpfs, encfs, ext2, ext4fuse, funionfs, fusepak, gitfs, gstfs, gunzip, hfsfuse, httpfs, ifuse, ltfs, mhddfs, mp3fs, ntfs, ntfs-compression, pod, rar2fs, s3backer, s3fs, simple-mtpfs, smbnetfs, sqlfs, squashfuse, sshfs, unionfs, wdfs, webdavfs, wikipediafs, zip, MooseFS, CephFS, RedoxFS, gcsf, catfs, qcow2-fuse, shotwellvfs, and one closed-source file system.

    needs-info 
    opened by asomers 20
  • write() after release()

    write() after release()

    This is one of the files being rsynced, roughly in the middle of the overall process. I see following in the debug output:

    unique: 459221, opcode: CREATE (35), nodeid: 4358, insize: 103, pid: 8654
    create flags: 0x80c1 /mega/whatever-dir/whatever-file 0100600 umask=0000
       create[0] flags: 0x80c1 /mega/whatever-dir/whatever-file
    getattr /mega/whatever-dir/whatever-file
       NODEID: 15944
       unique: 459221, success, outsize: 160
    unique: 459222, opcode: FLUSH (25), nodeid: 15944, insize: 64, pid: 8654
    flush[0]
       unique: 459222, success, outsize: 16
    unique: 459223, opcode: RELEASE (18), nodeid: 15944, insize: 64, pid: 0
    unique: 459224, opcode: OPEN (14), nodeid: 15944, insize: 48, pid: 8654
    open flags: 0x8002 /mega/whatever-dir/whatever-file
    release[0] flags: 0x8001
       open[0] flags: 0x8002 /mega/whatever-dir/whatever-file
       unique: 459224, success, outsize: 32
       unique: 459223, success, outsize: 16
    unique: 459225, opcode: SETATTR (4), nodeid: 15944, insize: 128, pid: 8534
    getattr /mega/whatever-dir/whatever-file
       unique: 459225, success, outsize: 120
    unique: 459226, opcode: WRITE (16), nodeid: 15944, insize: 88, pid: 11192
    write[0] 8 bytes to 0 flags: 0x8002
    

    So it seems release is executed after open, and then it tries to write to released file.
    Yes, the program is multithreaded. Is there anything to be done? What are usual workarounds for this?

    bug needs-info 
    opened by Kaned1as 19
  • Pass fuse_file_info in chown, chmod, utimensat, etc

    Pass fuse_file_info in chown, chmod, utimensat, etc

    Hi,

    I would like to get the file handle that was set in open when fchown, fchmod, and futimens are called. This is necessary to distinguish between syscalls, such as, chown and fchown, which make a difference when the underlying implementantion is inode-based and not path-based.

    Is it possible to create new FUSE handlers fchown, fchmod, futimens, that receive the fuse_file_info structure, which contains the file handle created in open?

    Thanks, Jose

    enhancement 
    opened by jabolopes 19
  • Improve/define FreeBSD support

    Improve/define FreeBSD support

    Meson wants to use -ldl but this is not required on FreeBSD, that functionality is built into libc.

    [email protected]:~/libfuse/build % meson ..
    WARNING: You are using 'US-ASCII' which is not a a Unicode-compatible locale.
    WARNING: You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
    WARNING: Please switch to a UTF-8 locale for your platform.
    The Meson build system
    Version: 0.40.1
    Source dir: /home/vagrant/libfuse
    Build dir: /home/vagrant/libfuse/build
    Build type: native build
    Project name: libfuse3
    Native c compiler: cc (clang 3.4.1)
    Build machine cpu family: x86_64
    Build machine cpu: x86_64
    Checking for function "fork": YES
    Checking for function "fstatat": YES
    Checking for function "openat": YES
    Checking for function "readlinkat": YES
    Checking for function "pipe2": YES
    Checking for function "splice": NO
    Checking for function "vmsplice": NO
    Checking for function "posix_fallocate": YES
    Checking for function "fdatasync": NO
    Checking for function "utimensat": YES
    Checking for function "setxattr": NO
    Checking for function "iconv": YES
    Checking whether type "struct stat" has member "st_atim": YES
    Checking whether type "struct stat" has member "st_atimespec": YES
    Configuring config.h using configuration
    Dependency threads found: YES
    
    Meson encountered an error in file lib/meson.build, line 20, column 0:
    C library 'dl' not found
    
    bug 
    opened by bnaylor 17
  • Installing failure on Ubuntu 16.04

    Installing failure on Ubuntu 16.04

    On stock Ubuntu 16.04, installing building/installing with meson does not produce working sshfs for example. Building and installing proceeds fine (after manually installing latest meson). However, sshfs --version says:

    shfs: error while loading shared libraries: libfuse3.so.3: cannot open shared object file: No such file or directory

    This happens because installation has placed libfuse to /usr/local/lib/x86_64-linux-gnu/ and that path is not searched for libraries in stock Ubuntu 16.04 system.

    When I built libfuse/sshfs with autotools, this issue did not happen. But autotools support has been dropped so this is no longer an option. I am not 100% sure if this issue is in libfuse or meson or ninja or Ubuntu. But I sure would like to get it fixed.

    bug needs-info 
    opened by tksuoran 16
  • 'll' operations fail when writeback cache is enabled

    'll' operations fail when writeback cache is enabled

    Using FUSE3.2.6 to create file system, create a file on the mount directory, the file size is 1G. Format the file and mount it to the directory. After a few seconds, see the directory through the "ll" command.

    #df -h 
    fuse_hello1          1.0G  1.0G     0 100% /root/libfuse-fuse-3.2.6/example/fileDir
    
    #ll -h /root/libfuse-fuse-3.2.6/example/fileDir
    total 512K
    -rw-r--r-- 1 root root 1.0G Nov 16 15:46 file
    
    #mkfs.xfs /root/libfuse-fuse-3.2.6/example/fileDir/file
    #mount /root/libfuse-fuse-3.2.6/example/fileDir/file /root/libfuse-fuse-3.2.6/example/mountDir
    
    • When writeback cache is enabled, the error occurs.
    #ll /root/libfuse-fuse-3.2.6/example/mountDir
    ls: cannot access mountDir: Input/output error
    
    • When writeback cache is disabled, successful.
    # ll mountDir/
    total 0
    

    enable writeback_cache๏ผšconn->want |= FUSE_CAP_WRITEBACK_CACHE; Incorrect usage or other reasons?

    needs-info 
    opened by Roay 16
  • Init script is installed in /usr/local/etc, not /etc

    Init script is installed in /usr/local/etc, not /etc

    Hi all, I'm trying to install fuse and have followed the steps advised: meson ninja sudo python3 -m pytest test/ (all passed) but I receive an error during the ninja install:

    sudo ~/divers_apps/ninja/ninja install returns:

    [0/1] Installing files. Installing lib/libfuse3.so.3.0.0 to /usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.0.0 Installing util/fusermount3 to /usr/local/bin/fusermount3 Installing util/mount.fuse3 to /usr/local/sbin/mount.fuse3 Installing fuse.h to /usr/local/include/fuse3 Installing fuse_common.h to /usr/local/include/fuse3 Installing fuse_lowlevel.h to /usr/local/include/fuse3 Installing fuse_opt.h to /usr/local/include/fuse3 Installing cuse_lowlevel.h to /usr/local/include/fuse3 Installing /home/tom/divers_apps/fuse/fuse-3.0.2/doc/fusermount3.1 to /usr/local/share/man/man1 Installing /home/tom/divers_apps/fuse/fuse-3.0.2/doc/mount.fuse.8 to /usr/local/share/man/man8 Installing /home/tom/divers_apps/fuse/build/meson-private/fuse3.pc to /usr/local/lib/x86_64-linux-gnu/pkgconfig Running custom install script '/home/tom/divers_apps/fuse/fuse-3.0.2/util/install_helper.sh etc bin lib/x86_64-linux-gnu' update-rc.d: error: unable to read /etc/init.d/fuse3

    Would you by chance have any idea? Thanks!

    bug 
    opened by TomBugnon 16
  • Fix: fd and memory leak in mount.fuse.c

    Fix: fd and memory leak in mount.fuse.c

    The command isn't freed and the fuse_fd isn't closed if execl failed. Fix it.

    Signed-off-by: Lixiaokeng [email protected]

    opened by lixiaokeng 0
  • fuse3: revert

    fuse3: revert "fuse_daemonize(): chdir to '/' even if not running in the background"

    The nvme-cuse in SPDK 21.01 uses cuse in foreground mode. If fuse_daemonize function calls chdir to '/', then all functions call with relative-path parameter will report error due to the working directory changed. The bugfix in SPDK can't fix the all sence, like: apps call chdir self or multi-thread race condition etc. Prefer revert patch in fuse3 to fix the bug in cuse foreground mode.

    Signed-off-by: Lixiaokeng[email protected] Signed-off-by: Weifeng Su[email protected]

    opened by lixiaokeng 8
  • Add CMake support - plus !

    Add CMake support - plus !

    This is a rather substantial change which adds CMake build support. Almost all of the changes are additional CMakeLists.txt files to support that. It also adds a Jenkins file which can be used in place of Travis. There are minor modifications to a very few source code files to eliminate warnings.

    I supply this in the hopes that someone might find it useful.

    You mileage may vary !

    opened by Smit-tay 2
  • fuse_operations::utimens not called when updating modification time only

    fuse_operations::utimens not called when updating modification time only

    When calling touch -d on a file under a fuse file system, it correctly calls fuse_operations::utimens, but when calling touch -md (i.e. instructing touch to only update modification time), I see the SETATTR call in the fuse log, but fuse_operations::utimens never gets called, giving the file system no chance to handle it.

    Reported here: https://github.com/cryfs/cryfs/issues/379

    opened by smessmer 1
  • auto_unmount doesn't work without allow_other

    auto_unmount doesn't work without allow_other

    In https://github.com/libfuse/libfuse/blob/77d662459a0fcdf358d515477d33795837e859d5/util/fusermount.c#L1219 open is executed as root which does not have access to the mount point if allow_other was not used and the real user id is not 0. Since allow_other usually cannot be specified by unprivileged users, auto_unmount has no effect for unprivileged users.

    @kevin-vigor Should we maybe drop privileges before calling open?

    bug 
    opened by mahkoh 1
  • Consider using name_to_handle_at() instead of open(O_PATH) in low-level examples

    Consider using name_to_handle_at() instead of open(O_PATH) in low-level examples

    Using https://man7.org/linux/man-pages/man2/name_to_handle_at.2.html instead of open with O_PATH and openat would be more lightweight and avoid the need for large numbers of open file descriptors.

    However, open_by_handle_at(2) currently needs CAP_DAC_READ_SEARCH in the initial user namespace, which makes it impossible to use in containers and such.

    enhancement 
    opened by Nikratio 1
  • test_passthrough_hp fails

    test_passthrough_hp fails

    uname -a Linux walid-MacBookPro 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

    sudo python3 -m pytest test/ ==================================================== test session starts ==================================================== platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/walid/build/test, inifile: pytest.ini collected 43 items

    test/test_ctests.py::test_write_cache[False] PASSED [ 2%] test/test_ctests.py::test_write_cache[True] PASSED [ 4%] test/test_ctests.py::test_notify1[True-notify_inval_inode] PASSED [ 6%] test/test_ctests.py::test_notify1[True-invalidate_path] PASSED [ 9%] test/test_ctests.py::test_notify1[True-notify_store_retrieve] PASSED [ 11%] test/test_ctests.py::test_notify1[False-notify_inval_inode] PASSED [ 13%] test/test_ctests.py::test_notify1[False-invalidate_path] PASSED [ 16%] test/test_ctests.py::test_notify1[False-notify_store_retrieve] PASSED [ 18%] test/test_ctests.py::test_notify_file_size[True] PASSED [ 20%] test/test_ctests.py::test_notify_file_size[False] PASSED [ 23%] test/test_examples.py::test_hello[hello-options0-invoke_directly] PASSED [ 25%] test/test_examples.py::test_hello[hello-options0-invoke_mount_fuse] PASSED [ 27%] test/test_examples.py::test_hello[hello-options0-invoke_mount_fuse_drop_privileges] PASSED [ 30%] test/test_examples.py::test_hello[hello-options1-invoke_directly] PASSED [ 32%] test/test_examples.py::test_hello[hello-options1-invoke_mount_fuse] PASSED [ 34%] test/test_examples.py::test_hello[hello-options1-invoke_mount_fuse_drop_privileges] PASSED [ 37%] test/test_examples.py::test_hello[hello_ll-options0-invoke_directly] PASSED [ 39%] test/test_examples.py::test_hello[hello_ll-options0-invoke_mount_fuse] PASSED [ 41%] test/test_examples.py::test_hello[hello_ll-options0-invoke_mount_fuse_drop_privileges] PASSED [ 44%] test/test_examples.py::test_hello[hello_ll-options1-invoke_directly] PASSED [ 46%] test/test_examples.py::test_hello[hello_ll-options1-invoke_mount_fuse] PASSED [ 48%] test/test_examples.py::test_hello[hello_ll-options1-invoke_mount_fuse_drop_privileges] PASSED [ 51%] test/test_examples.py::test_passthrough[False-passthrough-False] PASSED [ 53%] test/test_examples.py::test_passthrough[False-passthrough-True] SKIPPED [ 55%] test/test_examples.py::test_passthrough[False-passthrough_fh-False] PASSED [ 58%] test/test_examples.py::test_passthrough[False-passthrough_fh-True] SKIPPED [ 60%] test/test_examples.py::test_passthrough[False-passthrough_ll-False] PASSED [ 62%] test/test_examples.py::test_passthrough[False-passthrough_ll-True] PASSED [ 65%] test/test_examples.py::test_passthrough[True-passthrough-False] PASSED [ 67%] test/test_examples.py::test_passthrough[True-passthrough-True] SKIPPED [ 69%] test/test_examples.py::test_passthrough[True-passthrough_fh-False] PASSED [ 72%] test/test_examples.py::test_passthrough[True-passthrough_fh-True] SKIPPED [ 74%] test/test_examples.py::test_passthrough[True-passthrough_ll-False] PASSED [ 76%] test/test_examples.py::test_passthrough[True-passthrough_ll-True] PASSED [ 79%] test/test_examples.py::test_passthrough_hp[False] PASSED [ 81%] test/test_examples.py::test_passthrough_hp[True] FAILED [ 83%]

    ========================================================= FAILURES ========================================================== _________________________________________________ test_passthrough_hp[True] _________________________________________________ Traceback (most recent call last): File "/home/walid/build/test/test_examples.py", line 202, in test_passthrough_hp tst_rmdir(src_dir, mnt_dir) File "/home/walid/build/test/test_examples.py", line 417, in tst_rmdir assert name in os.listdir(mnt_dir) AssertionError: assert 'testfile_200' in ['testfile_195', 'testfile_196', 'testfile_198', 'testfile_199', 'testfile_197', 'testfile_194']

    • where ['testfile_195', 'testfile_196', 'testfile_198', 'testfile_199', 'testfile_197', 'testfile_194'] = ('/tmp/tmph9hb3buu/mnt')
    • where = os.listdir ===================================================== warnings summary ====================================================== /home/walid/.local/lib/python3.6/site-packages/_pytest/mark/structures.py:327 /home/walid/.local/lib/python3.6/site-packages/_pytest/mark/structures.py:327 /home/walid/.local/lib/python3.6/site-packages/_pytest/mark/structures.py:327: PytestUnknownMarkWarning: Unknown pytest.mark.uses_fuse - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html PytestUnknownMarkWarning,

    -- Docs: https://docs.pytest.org/en/latest/warnings.html ================================================== short test summary info ================================================== SKIPPED [4] /home/walid/build/test/test_examples.py:126: example does not support writeback caching FAILED test/test_examples.py::test_passthrough_hp[True] - AssertionError: assert 'testfile_200' in ['testfile_195', 'testf... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! =================================== 1 failed, 31 passed, 4 skipped, 2 warnings in 29.03s ====================================

    bug 
    opened by jedai47 3
  • Add support for stream-like files

    Add support for stream-like files

    Starting from 2008 it became possible to implement FUSE filesystems with pipe/socket-like files. For this a filesystem server should set file_info->nonseekable and the kernel will correspondingly use FMODE_NONSEEKABLE for opened file handle. See commit cafdcb253e4c "If open sets fi->nonseekable, libfuse will tell the kernel that the file is not seekable" which added nonseekable support to libfuse.

    Unfortunately in 2014 a kernel regression was introduced(*): all file IO switched to be done under lock (file->f_pos_lock) with the idea that file position changes should be atomic. However for stream-like files that were using nonseekable open it created possibility of deadlock in between read and write:

    - file read could be waiting for data from filesystem server
    - filesystem server is generating data in reply to client write to that file
    - write to that file is waiting for f_pos_lock taken by read
    -> deadlock
    

    Unfortunately now the regression is not possible to fix on just kernel side because a) it is already 5 years after 2014, and b) there are existing FUSE filesystems that use nonseekable open and position for corresponding file(+). In other words because of backward compatibility today we cannot change the kernel to avoid taking f_pos_lock and not to pass file position for filesystem server for nonseekable FUSE file handles.

    The solution to the deadlock problem, that is already merged to Linux kernel, is to introduce new FUSE open flag (FOPEN_STREAM) that is explicitly marking file handle as not having position at all, and thus for such file handle the kernel will avoid taking f_pos_lock.

    git.kernel.org/linus/10dce8af3422
    git.kernel.org/linus/bbd84f33652f
    

    When implementing stream-like files filesystem servers should be using both FOPEN_NONSEEKABLE and FOPEN_STREAM:

    - on kernels released before 2014, since in-kernel FUSE client
      ignores unknown FOPEN flags, this will continue to work ok as before.
    - on kernels with FOPEN_STREAM support, this will avoid
      read/write deadlock.
    - on kernels released after 2014 but without FOPEN_STREAM
      support, the deadlock will be there but using FOPEN_STREAM
      does not make the situation worse.
    
    - FOPEN_STREAM was merged into Linux 5.2 (see commits ^^^) and
      was backported to all long-term stable Linux kernel versions
      that are taking their roots after kernel regression point:
    
      Linux 5.1  git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v5.1.4-110-g5bb2a758f2ed
      Linux 4.19 git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v4.19.45-82-gfea685000caf
      Linux 4.14 git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v4.14.124-35-g585724f83b0c
      Linux 4.9  git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v4.9.180-83-gcfd8d2e79524
      Linux 4.4  git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v4.4.180-241-gc9696a8f3e64
      Linux 3.18 https://android.googlesource.com/kernel/common/+/9225b2f9d042
      Linux 3.16 git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=v3.16.69-123-g6d654f9d5d1f
    

    Note: we cannot change libfuse to use use both FOPEN_NONSEEKABLE and FOPEN_STREAM on just file->info_nonseekable=1 for the same backward compatibility reason why we cannot change the kernel to avoid using file position for nonseekable opens (see "+" again). Thus let's introduce new flag file_info->stream and let filesystems that implement stream-like files set both file_info->stream and file_info->nonseekable_open.

    Even though example/poll.c implements only read, I've changed that example to use file_info->stream=1 too, because file implemented there does not use file position at all and because that example is kind of canonical on how to implement "pipe-like" files with libfuse.

    Since this patch is trying to fix regression introduced in 2014, it should be applied to all libfuse branches covering time after that moment:

    - master (current 3.x development)
    - fuse_3_0_bugfix
    - fuse-2_9_bugfix
    - fuse_2_8_bugfix
    

    /cc @Nikratio, @szmi

    (*) git.kernel.org/linus/9c225f2655e3 (+) e.g. gvfs, see

    https://codesearch.debian.net/search?q=-%3Enonseekable+%3D
    https://gitlab.gnome.org/GNOME/gvfs/blob/1.40.0-6-gcbc54396/client/gvfsfusedaemon.c#L1080
    https://gitlab.gnome.org/GNOME/gvfs/blob/1.40.0-6-gcbc54396/client/gvfsfusedaemon.c#L1247-1346
    https://gitlab.gnome.org/GNOME/gvfs/blob/1.40.0-6-gcbc54396/client/gvfsfusedaemon.c#L1399-1481
    
    opened by navytux 11
  • passthrough_ll can skip entries during readdir on non-Linux platforms

    passthrough_ll can skip entries during readdir on non-Linux platforms

    passthrough_ll's FUSE_READDIR implementation saves the last offset of a DIR* object when it returns. Subsequent calls check the provided offset, and seekdir(3) to if it's different than the saved offset. However, that's not a valid use of seekdir. seekdir's argument must be something that was previously returned by telldir(3). The offset provided in the FUSE_READDIR command is a file offset, something that should be used with lseek(2).

    The impact of this bug is that readdir(3) operations can skip entries. Here's the general order of events

    • user program calls readdir.
    • libc calls getdirentries(2) on FreeBSD or getdents64(2) on Linux, using a buffer of size X
    • Kernel calls FUSE_OPENDIR and FUSE_READDIR.
    • passthrough_ll calls readdir on the underlying file system
    • libc calls getdirentries or getdents64 with a buffer of size Y on the underlying file system.
    • passthrough_ll returns to the kernel
    • kernel fills up the user program's buffer but runs out of space before it adds all entries
    • libc saves the actual offset of the last dirent structure.
    • user progam calls readdir again.
    • libc calls getdirentries or getdents64 using the saved offset
    • Kernel calls FUSE_READDIR using the saved offset
    • passthrough_ll sees that the provided offset is different than the saved one, and calls seekdir
    • seekdir does nothing at all, since the offset was not previously returned by telldir.
    • passthrough_ll calls readdir on the underlying filesystem, resuming where it left off previously. The directory entries that didn't fit within the user's first buffer are therefore lost.

    At first glance it seems that there shouldn't be a problem because X == Y. However, the problem arises because the structure used by getdirentries is different than the structure used by the FUSE protocol. FUSE's fuse_dirent is not nul-terminated, but getdirentries's struct dirent and getdents64's struct linux_dirent are. Still, the problem doesn't arise on Linux because struct linux_dirent is smaller than struct fuse_dirent. Here's a chart summarizing the structure sizes, for a name of length n:

    | OS | struct | size| | ------------ | -------------------|------| | FreeBSD | dirent | 24 + n + 1 | | Linux | linux_dirent64 | 20 + n + 1 | | FUSE | fuse_dirent | 24 + n |

    The solution is for passthrough_ll to use getdirentries and lseek directly instead of readdir and seekdir.

    bug 
    opened by asomers 3
  • Semantics of ioctl()'s arg parameter are undefined

    Semantics of ioctl()'s arg parameter are undefined

    https://github.com/libfuse/libfuse/blob/master/include/fuse.h#L662

    The void *arg argument is not mentioned in the function description.

    bug 
    opened by trapexit 4
Releases(fuse-3.10.4)
Zero-details, privacy-focused in-app file system.

ZboxFS ZboxFS is a zero-details, privacy-focused in-app file system. Its goal is to help application store files securely, privately and reliably. By

Zbox 1.2k Jun 8, 2021
Find files with SQL-like queries

Find files with SQL-like queries

null 2.9k Jun 13, 2021
Highly parallelized, blazing fast directory tree analyzer

Parallel Disk Usage (pdu) Highly parallelized, blazing fast directory tree analyzer. Description pdu is a CLI program that renders a graphical chart f

Khแบฃi 75 Jun 9, 2021
Expanding opportunities standard library std::fs and std::io

fs_extra A Rust library that provides additional functionality not present in std::fs. Documentation Migrations to 1.x.x version Key features: Copy fi

Denis Kurilenko 98 Jun 2, 2021
Temporary file library for rust

tempfile A secure, cross-platform, temporary file library for Rust. In addition to creating temporary files, this library also allows users to securel

Steven Allen 517 Jun 13, 2021
CoreOS Disk Image Rehydrator

In CoreOS we ship a lot of disk images, one for each platform. These are almost entirely the same thing, mostly just with an ignition.platform.id stamp in each disk image, wrapped in a hypervisor/platform specific container.

Colin Walters 4 May 18, 2021
Rust implemention of Ascon

Ascon Pure Rust implementation of the lightweight Authenticated Encryption and Associated Data (AEAD) Ascon-128 and Ascon-128a. Security Notes This cr

Sebastian Ramacher 3 May 3, 2021
Extended attribute library for rust.

xattr A small library for setting, getting, and listing extended attributes. Supported Platforms: Linux, MacOS, FreeBSD, and NetBSD. API Documentation

Steven Allen 16 Feb 9, 2021