This adds a new "pre-mounted" mode of operation in which the FUSE file system helper is launched after the /dev/fuse file descriptor has been opened opened and the file system been mounted by a suitably privileged process.

Pre-mounted mode is requested by passing an empty string as mountpoint, which reflects the fact that the FUSE helper doesn't actually perform or trigger any mounting itself.

The main benefit of pre-mounted mode is that no privileged operations need to be performed by the file system implementation itself directly or indirectly, so the FUSE process can run with zero privilege. Moreover, mechanisms like securebits and no_new_privs can be used to prevent subprocesses from re-acquiring privilege, which further helps reduce risk for the case the FUSE helper gets exploited by a malicious file system.

Below is an example that illustrates this. Note that I'm using shell for presentation purposes, the assumption is that there's a privileged daemon that handles mounting and spawning the FUSE helper in a suitable sandbox.

Make binaries and libs executable by anyone

$ chmod o+rx build/lib/libfuse3.so* example/hello $ export LD_LIBRARY_PATH=$PWD/build/lib

example/hello can mount successfully with privilege

$ sudo sh -c "LD_LIBRARY_PATH=build/lib ./example/hello /mnt/tmp" $ sudo cat /mnt/tmp/hello Hello World! $ sudo umount /mnt/tmp

example/hello fails to mount without privilege

$ sudo capsh --drop=all --secbits=0x3f -- -c 'LD_LIBRARY_PATH=build/lib ./example/hello -f /mnt/tmp' fusermount3: mount failed: Operation not permitted

Pre-mounting allows example/hello to work without privilege

$ sudo sh -c ' exec 0<>/dev/fuse mount -i -o nodev,nosuid,noexec,fd=0,rootmode=40000,user_id=0,group_id=0 -t fuse hello /mnt/tmp capsh --drop=all --secbits=0x3f -- -c "LD_LIBRARY_PATH=build/lib example/hello -f """ ' & [1] 55491 $ sudo cat /mnt/tmp/hello Hello World! $ sudo umount /mnt/tmp [1]+ Done sudo sh -c ' exec 0<>/dev/fuse mount -i -o nodev,nosuid,noexec,fd=0,rootmode=40000,user_id=0,group_id=0 -t fuse hello /mnt/tmp capsh --drop=all --secbits=0x3f -- -c "LD_LIBRARY_PATH=build/lib example/hello -f """ '

We have some testcases that prove this comment correct 😄

/* FIXME: locking two paths needs deadlock checking */

(https://github.com/libfuse/libfuse/blob/master/lib/fuse.c#L1258)

Considering tackling this since I can reproduce the problem. Did you have any thoughts as to how a fix should be implemented?

On benchmarking metadata operations with a single threaded bonnie++ and "max_idle_threads" limited to 1, 'top' was showing suspicious 160% cpu usage. Profiling the system with flame graphs showed that an astonishing amount of CPU time was spent in thread creation and destruction.

After verifying the code it turned out that fuse_do_work() was creating a new thread every time all existing idle threads were already busy. And then just a few lines later after processing the current request it noticed that it had created too many threads and destructed the current thread. I.e. there was a thread creation/destruction ping-pong.

Code is changed to only create new threads if the max number of threads is not reached.

Furthermore, thread destruction is disabled, as creation/destruction is expensive in general.

With this change cpu usage of passthrough_hp went from ~160% to ~80% (with different values of max_idle_threads). And bonnie values got approximately faster by 90%. This is a with single threaded bonnie++ bonnie++ -x 4 -q -s0 -d -n 30:1:1:10 -r 0

Without this patch, using the default max_idle_threads=10 and just a single bonnie++ the thread creation/destruction code path is not triggered. Just one libfuse and one application thread is just a corner case - the requirement for the issue was just n-application-threads >= max_idle_threads.

Signed-off-by: Bernd Schubert [email protected]

Background

The FUSE protocol allows a server to assign a different filehandle each and every time that a file is opened. Multiple filehandles can be concurrently valid for the same file. Operations like FUSE_READ and FUSE_WRITE are supposed to include the file handle associated with the particular file descriptor that initiated the operation. However, that isn't possible in all cases. The FUSE_WRITE_CACHE bit indicates whether the file handle is valid for a FUSE_WRITE operation, and the FATTR_FH flag indicates whether it's valid for a FUSE_SETATTR operation.

Problem

There are other cases, too, when the file handle can be invalid. In fact, the entire concept of a file handle is really a Linuxism. Other operating systems enforce a distinction between the file layer and the vnode layer. That's why OSX, FreeBSD, OpenBSD, and Illumos can almost never guarantee that they're sending the right file handle for any operation. NetBSD takes a more extreme position; it doesn't allow multiple concurrent fuse file handles for a single file (which leads to other problems). But since operations other than FUSE_WRITE and FUSE_SETATTR don't have anything analogous to the FUSE_WRITE_CACHE or FATTR_FH bits, there's no way for those operating systems to indicate that the file handle is guessed.

Mitigating Circumstances

The impact of this bug is small simply because few if any fuse servers care about the accuracy of fuse file handles. I surveyed 42 real-world FUSE filesystems* and none of them ever check the writepage bit. Only one of them (catfs) checks FATTR_FH, and it can deal with the absence of that flag. I can't find any evidence that any FUSE filesystem actually wants to assign distinct file handles on each concurrent FUSE_OPEN and requires the FUSE client to keep them straight.

Solution

Ideally the FUSE protocol would require servers to assign the same file handle for every concurrent open of a single file. Unfortunately, that's not the current situation and such a change would be backwards-incompatible. The second best solution would be to update the kernel protocol to include some equivalent of the FUSE_WRITE_CACHE bit for every operation that includes a file handle. However, that would require all clients and servers alike to update their kernel API version, so uptake would be extremely slow. The path of least resistance would simply be to document the problem. I recommend the following:

• Advise FUSE server developers to assign the same file handle for every concurrent open of a single file.
• Rename the writepage bit to fh_invalid, and add a #define for backwards-compatibility
• Patch libfuse to unconditionally set fh_invalid for most commands on non-Linux operating systems.

If the maintainers agree, I'll prepare a PR.

cc @cemeyer @kendmerry

*: afuse, chironfs, cryptofs, curlftpfs, encfs, ext2, ext4fuse, funionfs, fusepak, gitfs, gstfs, gunzip, hfsfuse, httpfs, ifuse, ltfs, mhddfs, mp3fs, ntfs, ntfs-compression, pod, rar2fs, s3backer, s3fs, simple-mtpfs, smbnetfs, sqlfs, squashfuse, sshfs, unionfs, wdfs, webdavfs, wikipediafs, zip, MooseFS, CephFS, RedoxFS, gcsf, catfs, qcow2-fuse, shotwellvfs, and one closed-source file system.

This fixes a couple of bugs in the passthrough_ll example filesystem and adds support for lots of missing operations. Options are added to configure a base path and to control caching.

This is one of the files being rsynced, roughly in the middle of the overall process. I see following in the debug output:

unique: 459221, opcode: CREATE (35), nodeid: 4358, insize: 103, pid: 8654
create flags: 0x80c1 /mega/whatever-dir/whatever-file 0100600 umask=0000
   create[0] flags: 0x80c1 /mega/whatever-dir/whatever-file
getattr /mega/whatever-dir/whatever-file
   NODEID: 15944
   unique: 459221, success, outsize: 160
unique: 459222, opcode: FLUSH (25), nodeid: 15944, insize: 64, pid: 8654
flush[0]
   unique: 459222, success, outsize: 16
unique: 459223, opcode: RELEASE (18), nodeid: 15944, insize: 64, pid: 0
unique: 459224, opcode: OPEN (14), nodeid: 15944, insize: 48, pid: 8654
open flags: 0x8002 /mega/whatever-dir/whatever-file
release[0] flags: 0x8001
   open[0] flags: 0x8002 /mega/whatever-dir/whatever-file
   unique: 459224, success, outsize: 32
   unique: 459223, success, outsize: 16
unique: 459225, opcode: SETATTR (4), nodeid: 15944, insize: 128, pid: 8534
getattr /mega/whatever-dir/whatever-file
   unique: 459225, success, outsize: 120
unique: 459226, opcode: WRITE (16), nodeid: 15944, insize: 88, pid: 11192
write[0] 8 bytes to 0 flags: 0x8002

So it seems release is executed after open, and then it tries to write to released file.
Yes, the program is multithreaded. Is there anything to be done? What are usual workarounds for this?

Hi,

I would like to get the file handle that was set in open when fchown, fchmod, and futimens are called. This is necessary to distinguish between syscalls, such as, chown and fchown, which make a difference when the underlying implementantion is inode-based and not path-based.

Is it possible to create new FUSE handlers fchown, fchmod, futimens, that receive the fuse_file_info structure, which contains the file handle created in open?

Thanks, Jose

Meson wants to use -ldl but this is not required on FreeBSD, that functionality is built into libc.

vagrant@:~/libfuse/build % meson ..
WARNING: You are using 'US-ASCII' which is not a a Unicode-compatible locale.
WARNING: You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
WARNING: Please switch to a UTF-8 locale for your platform.
The Meson build system
Version: 0.40.1
Source dir: /home/vagrant/libfuse
Build dir: /home/vagrant/libfuse/build
Build type: native build
Project name: libfuse3
Native c compiler: cc (clang 3.4.1)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Checking for function "fork": YES
Checking for function "fstatat": YES
Checking for function "openat": YES
Checking for function "readlinkat": YES
Checking for function "pipe2": YES
Checking for function "splice": NO
Checking for function "vmsplice": NO
Checking for function "posix_fallocate": YES
Checking for function "fdatasync": NO
Checking for function "utimensat": YES
Checking for function "setxattr": NO
Checking for function "iconv": YES
Checking whether type "struct stat" has member "st_atim": YES
Checking whether type "struct stat" has member "st_atimespec": YES
Configuring config.h using configuration
Dependency threads found: YES

Meson encountered an error in file lib/meson.build, line 20, column 0:
C library 'dl' not found

Using FUSE3.2.6 to create file system, create a file on the mount directory, the file size is 1G. Format the file and mount it to the directory. After a few seconds, see the directory through the "ll" command.

#df -h 
fuse_hello1          1.0G  1.0G     0 100% /root/libfuse-fuse-3.2.6/example/fileDir

#ll -h /root/libfuse-fuse-3.2.6/example/fileDir
total 512K
-rw-r--r-- 1 root root 1.0G Nov 16 15:46 file

#mkfs.xfs /root/libfuse-fuse-3.2.6/example/fileDir/file
#mount /root/libfuse-fuse-3.2.6/example/fileDir/file /root/libfuse-fuse-3.2.6/example/mountDir

• When writeback cache is enabled, the error occurs.

#ll /root/libfuse-fuse-3.2.6/example/mountDir
ls: cannot access mountDir: Input/output error

• When writeback cache is disabled, successful.

# ll mountDir/
total 0

enable writeback_cache：conn->want |= FUSE_CAP_WRITEBACK_CACHE; Incorrect usage or other reasons?

On stock Ubuntu 16.04, installing building/installing with meson does not produce working sshfs for example. Building and installing proceeds fine (after manually installing latest meson). However, sshfs --version says:

shfs: error while loading shared libraries: libfuse3.so.3: cannot open shared object file: No such file or directory

This happens because installation has placed libfuse to /usr/local/lib/x86_64-linux-gnu/ and that path is not searched for libraries in stock Ubuntu 16.04 system.

When I built libfuse/sshfs with autotools, this issue did not happen. But autotools support has been dropped so this is no longer an option. I am not 100% sure if this issue is in libfuse or meson or ninja or Ubuntu. But I sure would like to get it fixed.

Hi all, I'm trying to install fuse and have followed the steps advised: meson ninja sudo python3 -m pytest test/ (all passed) but I receive an error during the ninja install:

sudo ~/divers_apps/ninja/ninja install returns:

[0/1] Installing files. Installing lib/libfuse3.so.3.0.0 to /usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.0.0 Installing util/fusermount3 to /usr/local/bin/fusermount3 Installing util/mount.fuse3 to /usr/local/sbin/mount.fuse3 Installing fuse.h to /usr/local/include/fuse3 Installing fuse_common.h to /usr/local/include/fuse3 Installing fuse_lowlevel.h to /usr/local/include/fuse3 Installing fuse_opt.h to /usr/local/include/fuse3 Installing cuse_lowlevel.h to /usr/local/include/fuse3 Installing /home/tom/divers_apps/fuse/fuse-3.0.2/doc/fusermount3.1 to /usr/local/share/man/man1 Installing /home/tom/divers_apps/fuse/fuse-3.0.2/doc/mount.fuse.8 to /usr/local/share/man/man8 Installing /home/tom/divers_apps/fuse/build/meson-private/fuse3.pc to /usr/local/lib/x86_64-linux-gnu/pkgconfig Running custom install script '/home/tom/divers_apps/fuse/fuse-3.0.2/util/install_helper.sh etc bin lib/x86_64-linux-gnu' update-rc.d: error: unable to read /etc/init.d/fuse3

Would you by chance have any idea? Thanks!

I encountered the same issue as #211, except in this case it's for option remount. Based on that issue and the fix in #221, maybe this should be solved the same way with a one line addition of remount to the ignored options list? If that's the correct solution then I'm willing to make a PR -- although it's such a trivial change it might be easier for a maintainer to do it instead of dealing with the PR process. I'm open to whatever works best for you.

Here's how I encountered this on NixOS with an open-vm-tools (VMware guest tools) mount:

$ sudo systemctl cat run-vmblock\\x2dfuse.mount
# /etc/systemd/system/run-vmblock\x2dfuse.mount
[Unit]
ConditionVirtualization=vmware
Description=VMware vmblock fuse mount
Documentation=https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/vmblock-fuse/design.txt

[Mount]
Options=subtype=vmware-vmblock,default_permissions,allow_other
Type=fuse
What=/nix/store/1y562cdla1a6jxvs96z14vl7lqps5mgd-test-open-vm-tools-12.1.5/bin/vmware-vmblock-fuse
Where=/run/vmblock-fuse

$ sudo systemctl reload run-vmblock\\x2dfuse.mount; journalctl -u run-vmblock\\x2dfuse.mount --since "-5s"
Job failed. See "journalctl -xe" for details.
Dec 21 13:49:57 dev systemd[1]: Reloading VMware vmblock fuse mount...
Dec 21 13:49:57 dev mount[172663]: fuse: unknown option(s): `-o remount'
Dec 21 13:49:57 dev systemd[1]: run-vmblock\x2dfuse.mount: Mount process exited, code=exited, status=3/NOTIMPLEMENTED
Dec 21 13:49:57 dev systemd[1]: Reload failed for VMware vmblock fuse mount.

And some version info:

$ systemctl --version
systemd 252 (252.1)
+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

$ /nix/store/1y562cdla1a6jxvs96z14vl7lqps5mgd-test-open-vm-tools-12.1.5/bin/vmware-vmblock-fuse --version
FUSE library version 3.11.0
using FUSE kernel interface version 7.31
fusermount3 version: 3.11.0

Follow-up to https://github.com/libfuse/libfuse/pull/710

The IO for FUSE requests and responses can be further customized by allowing to write custom functions for reading/writing the responses. This includes overriding the splice IO. Also extended the hello_ll example (with #ifdefs) to provide an example of utilizing the custom IO (and a test for it). When no custom IO is provided, libfuse will behave the same as it did before any changes in this PR were made.

As it turns out, the changes in https://github.com/libfuse/libfuse/pull/710 are not enough to provide the user necessary control of the IO handling of the request and response data. Different types of file descriptor require different mechanisms of IO interaction. For example, some file descriptor types have boundaries (SOCK_DGRAM, EOF, etc...), while other types of might be unbounded (SOCK_STREAMS, ...). For unbounded communication, one call to read does not necessarily return all the data expected, especially in cases where the request are large. This requires further calls to read in a loop. You also would have to read the header of the FUSE request first, and then read the remaining packet data to ensure that you get all of the data in the request.

The fuse_custom_session_fd() (renamed fuse_session_custom_io() in this PR) function has not been part or a libfuse release yet (only part of the master branch), so my hope is that it's not considered a breaking change to modify that function.

Related to Issue #707

This PR extends the unoffical patch provided by Miklos Szeredi in this conversation by

• hopefully allowing the correct discovery if fuse_lowlevel_notify_expire_entry is available or not
• adding some comments/description

This new function allows dentries to be marked as only expired, instead of always be invalidated as suggested in this conversation.

The (final?) kernel patch provided Miklos Szeredi is here

We tested the first unofficial patch provided by Miklos Szeredi in the email conversation and it worked.

From prctl(2):

   PR_SET_IO_FLUSHER (since Linux 5.6)
          If a user process is involved in the block layer or filesystem I/O  path,  and
          can allocate memory while processing I/O requests it must set arg2 to 1.  This
          will put the process in the IO_FLUSHER state, which allows it  special  treat‐
          ment  to make progress when allocating memory. [..]

          The calling process must have the CAP_SYS_RESOURCE capability.[...]

          Examples  of  IO_FLUSHER  applications are FUSE daemons, SCSI device emulation
          daemons, and daemons that perform error handling like multipath path  recovery
          applications.

As I understand, if this flag is not set and the kernel needs to reclaim memory that is in-use by a FUSE filesystem, this can led to a deadlock.

libfuse currently does not set this flag. Maybe we should attempt to set it? But then, this seems like an unusual thing for a library to do and requires CAP_SYS_RESOURCE (which we can't ensure because it's a property of the binary that links to libfuse).

Maybe we should check if this is set and print a warning if not? Filesystems that know that they don't require this could explicitly disable the warning somehow.

Or just document it somewhere within libfuse?

As described in https://github.com/libfuse/libfuse/issues/695, partial locking of paths can cause a deadlock. Partial locking was added to prevent starvation, but it's unclear what specific cases of starvation were of concern. As far as I was able to determine, since we support reader locks, this means that to starve the queue element, we'd need a constant stream of queued requests that lock the path for write. Write locks are used when the element is being removed, so this stream of requests that starve the rename or lock operations seems unlikely.