---- os::linux::tests::test_empty_extent stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_empty_extent' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- os::linux::tests::test_extent_fetch_many stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_extent_fetch_many' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6

---- os::linux::tests::test_extent_fetch stdout ----
Error: Operation not supported (os error 95)
thread 'os::linux::tests::test_extent_fetch' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', <::std::macros::panic macros>:5:6

When copying, the progress bar is longer than the size of my terminal and it overflows to the next line. This is what I get when I copy a file.

$ cargo run A.mp4 B.mp4
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/xcp A.mp4 B.mp4`
[00:00:00] [--------------------------------------------------------------------[00:00:00] 
[#############>------------------------------------------------------[00:00:00] 
[####################>-----------------------------------------------[00:00:00] 
[###########################>----------------------------------------[00:00:00] 
[########################################>---------------------------[00:00:00] 
[###############################################>--------------------[00:00:00] 
[######################################################>-------------[00:00:00] 
[#############################################################>------[00:00:00] 
[####################################################################[00:00:00] 
[####################################################################[00:00:00] 
[###############################################################################
#] 751.60MB/751.60MB (0s)

~~This seems to be a fairly simple matter of switching over to libc functions that take off64_t instead of off_t, which on 32-bit platforms is i32 instead of i64.~~

This seems to be a fairly simple matter of fallibly converting certain syscall parameters to off_t rather than casting them to i64. The conversion should only actually be fallible on 32-bit platforms, where off_t is i32 instead of i64.

With these changes, cargo test succeeds on my Raspberry Pi 4 with Arch Linux ARM.

(Kudos to the Rust compiler for barfing on the cast mismatches, else the program might have silently eaten large files instead of failing to compile.)

A PR for some features that would be helpful to me. Feel free to close if these are not desired. I'm also happy to make changes.

closes #5

Commit Message:

add option for no-target-director, analagous to cp's no-target-directory. Expected behavior is that when copying a directory to another directory, instead of creating a sub-folder in target, overwrite target.

add option for hidden files. Default is to ignore hidden files.

This is a little hacked in; it seems as though ignore's Walk could handle both .gitignore and hidden files on its own, but I may have missed something.

Hello,

I tried to build xcp 0.9.2 on Termux by cargo install, but it failed to compile.

The command I typed:

$ cargo install --locked xcp

Versions:

rustc 1.64.0
cargo 1.64.0
Termux Variables:
TERMUX_API_VERSION=0.50.1
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=9719
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://mirrors.pku.edu.cn/termux/termux-main/ stable main
Updatable packages:
All packages up to date
termux-tools version:
1.29.3
Android version:
12
Kernel build information:
Linux localhost 4.14.212-Velvet-Cyprus_v2.1 #1 SMP PREEMPT Tue Sep 13 07:06:30 UTC 2022 aarch64 Android
Device manufacturer:
Xiaomi
Device model:
Redmi Note 9S

And the output:

error[E0433]: failed to resolve: could not find `linux` in `os`
  --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:22:14
   |                                                                                    22 | use std::os::linux::fs::MetadataExt;
   |              ^^^^^ could not find `linux` in `os`                                  
error[E0599]: no method named `st_blocks` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:13
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |             ^^^^^^^^^ method not found in `std::fs::Metadata`                         |
    = help: items from traits can only be used if the trait is in scope                 help: the following trait is implemented but not in scope; perhaps add a `use` for it:
    |                                                                                   17  | use std::os::android::fs::MetadataExt;
    |                                                                                   
error[E0599]: no method named `st_size` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:32
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |                                ^^^^^^^ method not found in `std::fs::Metadata`        |
    = help: items from traits can only be used if the trait is in scope                 help: the following trait is implemented but not in scope; perhaps add a `use` for it:
    |                                                                                   17  | use std::os::android::fs::MetadataExt;
    |                                                                                   
error[E0599]: no method named `st_blksize` found for struct `std::fs::Metadata` in the current scope
   --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:182:49
    |                                                                                   182 |     Ok(stat.st_blocks() < stat.st_size() / stat.st_blksize())
    |                                                 ^^^^^^^^^^ method not found in `std::fs::Metadata`
    |                                                                                       = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:      |
17  | use std::os::android::fs::MetadataExt;                                                |
                                                                                        error[E0308]: mismatched types
    --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/xcp-0.9.2/src/os/linux.rs:285:41
     |                                                                                  285  |             libc::ioctl(fd.as_raw_fd(), FS_IOC_FIEMAP, req_ptr)
     |             -----------                 ^^^^^^^^^^^^^ expected `i32`, found `u64`     |             |
     |             arguments to this function are incorrect                                  |
note: function defined here                                                                 --> /data/data/com.termux/files/home/.local/share/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.134/src/unix/linux_like/android/mod.rs:2824:12                         |
2824 |     pub fn ioctl(fd: ::c_int, request: ::c_int, ...) -> ::c_int;                      |            ^^^^^
help: you can convert a `u64` to an `i32` and panic if the converted value doesn't fit       |
285  |             libc::ioctl(fd.as_raw_fd(), FS_IOC_FIEMAP.try_into().unwrap(), req_ptr)
     |                                                      ++++++++++++++++++++                                                                                                Some errors have detailed explanations: E0308, E0433, E0599.                            For more information about an error, try `rustc --explain E0308`.
error: could not compile `xcp` due to 5 previous errors

Copying a file from a btrfs source to an exfat destination fails with the error:

Error: Operation not supported (os error 95)

However, both cp and rsync have no issues copying the file. rsync with -P even displays the progress correctly.

xcp: 0.9.1 rust: 1.63.0-1 kernel: 5.18.18 OS: Fedora 36 Arch: x86_64

add option for no-target-directory, analagous to cp's no-target-directory. Expected behavior is that when copying a directory to another directory, instead of creating a sub-folder in target, overwrite target.

I've packaged and merged xcp earlier today, http://mail-index.netbsd.org/pkgsrc-changes/2022/10/04/msg261531.html Please consider adding this to the README.md Thanks.

Readme mentions, it will thrash the spinning disk drive by reading metadata and copying at the same time. Isn't Receiving multiple read/write request at a time common for HDD on multi program and multi user system. Is there something happening which increases this significantly? How many parallel request will is send for one file, so that HDD thrashing becomes an issue? Is there a way to avoid it through configuration?

There is a known issue with copy_file_range() that will result in files copied from virtual filesystems (e.g. /proc, /sys) to copy the incorrect number of bytes (usually 0 in the case of /proc, but possibly other sizes). LWN has a good summary here:

https://lwn.net/Articles/846403/

Detecting such filesystems/files is hard, as there is no "FS is virtual" flag, so all file sizes are inherently suspect when working this way. This issue is exaserbated in the case of xcp's block-parallel driver (parblock), which needs to know the file size up-front to allocate work on the queues.

This issue needs more investigation, and possibly some fundamental rearchitecting in the case of parallel block copy.

If I start xcp normally, it shows as if the file were copied instantly, while in reality it is copied to cache (/proc/meminfo shows Dirty: as big as the file), with further actual copying happening outside xcp's control (i.e. progress bar).

If I start xcp in a memory-limited cgroup, it copies the file gradually (dirty bytes does not exceed the limit I set for it, I see destination file size gradually increasing), but the progress bar is stuck at 0% until the file finishes growing to its normal size. After that it starts actually copying the file (with normal progress bar). The data is written to vfat USB drive twice: first zero bytes, then actual content.

Shall xcp detect that it is copying the data to a USB drive and activate some O_DIRECT mode so that copy does not touch the cache and is easily abortable by user?

The core of the detection of sparse files is:

https://github.com/tarka/xcp/blob/ea28a4dbec5a9f43210fe72b5c6f827eba9bbaf6/src/os/linux.rs#L182

I may be wrong, but I think the use of st_blksize here is a mistake.

man 2 stat describes st_blksize as 'the "preferred" block size for efficient filesystem I/O.', and st_blocks as 'the number of blocks allocated to the file, in 512-byte units.' I checked with Rust code on a regular (non-sparse) file:

[src/main.rs:7] meta.st_blksize() = 4096
[src/main.rs:8] meta.st_blocks() = 1512
[src/main.rs:9] meta.st_size() = 771765
[src/main.rs:10] meta.st_size() / meta.st_blksize() = 188
[src/main.rs:11] meta.st_size() / 512 = 1507

I think these lines are the corresponding code in GNU cp. They use a constant ST_NBLOCKSIZE (from this header file) rather than anything from the stat struct.

If so, the impact is probably minor - I think st_blksize is only likely to be bigger than 512, so it should never erroneously decide that a dense file is sparse. And it can still detect very sparse files (on my system, where holes are 7/8 of the total size), where it's likely to be most important to optimise copying. But it looks like it will miss other sparse files.

Copying to a virtualbox filesystem leads to xcp printing an error and not copying the files.

[pid 31325] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=20701696, ...}) = 0
[pid 31325] openat(AT_FDCWD, "/media/sf_c/temp/test.out", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
[pid 31325] ftruncate(4, 20701696)      = 0
[pid 31325] flistxattr(3, NULL, 0)      = -1 EOPNOTSUPP (Operation not supported)

after adding --no-perms

[pid 31423] openat(AT_FDCWD, "/media/sf_c/temp/test.out", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
[pid 31423] ftruncate(4, 20701696)      = 0
[pid 31423] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_TYPE|STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_CTIME|STATX_INO|STATX_SIZE|STATX_BLOCKS|STATX_BTIME|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=20701696, ...}) = 0
[pid 31423] copy_file_range(3, NULL, 4, NULL, 20701696, 0) = -1 EINVAL (Invalid argument)

kernel 5.10.11

Merry Christmas, first of all!

How would you feel about a feature to remove copied files? Alternatively (or, even better, in addition) xcp could support running an external command after one or more files have been copied.