fcp is a significantly faster alternative to the classic Unix cp(1) command

Overview

fcp

fcp is a significantly faster alternative to the classic Unix cp(1) command.

fcp aims to handle the most common use-cases for cp with much higher performance.

fcp does not aim to completely replace cp with its myriad options.

Installation

Pre-built binaries

Pre-built binaries for some systems can be found under this repository's releases.

Via cargo

fcp can be installed using cargo by running the following:

cargo install fcp

Usage

Usage information can be found by running fcp --help, and has been reproduced below:

fcp

USAGE:
    fcp SOURCE DESTINATION_FILE
    Copy SOURCE to DESTINATION_FILE, overwriting DESTINATION_FILE if it exists

    fcp SOURCE ... DESTINATION_DIRECTORY
    Copy each SOURCE into DESTINATION_DIRECTORY

Benchmarks

fcp doesn't just claim to be faster than cp, it is faster than cp. As different operating systems display different performance characteristics, the same benchmarks were run on both macOS and Linux.

macOS

The following benchmarks were run on a 2018 MacBook Pro2 (2.9 GHz 6-Core Intel Core i9, 16 GiB RAM, SSD)

Large Files

The following shows the result of a benchmark which copies a directory containing 13 different 512 MB files using cp and fcp, with fcp being approximately 822x faster on average (note the units of the x-axis for each plot)3:

fcp is approximately 822x faster than cp, with fcp's average time to copy being approximately 4.5 milliseconds, while cp's average time to copy is approximately 3.7 seconds

Linux Kernel Source

The following shows the result of a benchmark which copies the source tree of the Linux kernel using cp and fcp, with fcp being approximately 6x faster on average:

fcp is approximately 6x faster than cp, with fcp's average time to copy being approximately 5.1 seconds, while cp's average time to copy is approximately 30 seconds

Linux

The following benchmarks were run on a bare-metal AWS EC2 instance (a1.metal, 16 CPUs, 32 GiB RAM, SSD)

Linux Kernel Source

The following shows the result of a benchmark which copies the source tree of the Linux kernel using cp and fcp, with fcp being approximately 10x faster on average:

fcp is nearly 10x faster than cp, with fcp's average time to copy being approximately 675 milliseconds, while cp's average time to copy is approximately 6.02 seconds

Large Files

The following shows the result of a benchmark which copies a directory containing 13 different 512 MB files using cp and fcp, with fcp being approximately 1.4x faster on average:

fcp is approximately 1.4x faster than cp, with fcp's average time to copy being approximately 8 seconds, while cp`'s average time to copy is approximately 11.3 seconds

[1] See the benchmarks.

[2] While in general you should avoid benchmarking on laptops, fcp is a developer tool and many developers work primarily on laptops. Also unlike with Linux where you can rent by the second, the minimum tenancy for AWS EC2 macOS instances is 24 hours, and these benchmarks took less than an hour.

[3] The massive difference in performance in this case is due to fcp using fclonefileat and fcopyfile under the hood.

Comments
  • Request for musl build?

    Request for musl build?

    I recently discovered this cool program and thought I'd try it out on the one non-laptop SSD system I have. Unfortunately:

    $ fcp
    fcp: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by fcp)
    

    In the past when I've seen things like this with Rust projects, I've asked if the developers could make a musl build instead of (or in addition to) a glibc build. Do you think fcp could have a musl build? Or perhaps it requires glibc?

    (Of course, it might be "I should update my OS", but it's a government system so OS updates are...let's call it deliberate in pace.)

    opened by mathomp4 5
  • windows build fails with `cargo install fcp`

    windows build fails with `cargo install fcp`

    error
    ❯ cargo install fcp
        Updating crates.io index
      Installing fcp v0.2.1
       Compiling autocfg v1.1.0
       Compiling crossbeam-utils v0.8.8
       Compiling cfg-if v1.0.0
       Compiling lazy_static v1.4.0
       Compiling rayon-core v1.9.2
       Compiling scopeguard v1.1.0
       Compiling libc v0.2.123
       Compiling num_cpus v1.13.1
       Compiling bitflags v1.3.2
       Compiling either v1.6.1
       Compiling memoffset v0.6.5
       Compiling crossbeam-epoch v0.9.8
       Compiling rayon v1.5.2
       Compiling crossbeam-channel v0.5.4
       Compiling nix v0.22.3
       Compiling crossbeam-deque v0.8.1
       Compiling fcp v0.2.1
    error[E0433]: failed to resolve: could not find `unix` in `os`
     --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:9:14
      |
    9 | use std::os::unix::fs::{MetadataExt, PermissionsExt};
      |              ^^^^ could not find `unix` in `os`
    
    error[E0433]: failed to resolve: could not find `sys` in `nix`
     --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:6:10
      |
    6 | use nix::sys::stat::Mode;
      |          ^^^ could not find `sys` in `nix`
    
    error[E0432]: unresolved import `nix::unistd`
     --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:7:5
      |
    7 | use nix::unistd;
      |     ^^^^^^^^^^^ no `unistd` in the root
    
    error[E0433]: failed to resolve: could not find `unix` in `os`
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:10:14
       |
    10 | use std::os::unix::fs::{self as unix, DirBuilderExt, FileTypeExt, OpenOptionsExt, PermissionsExt};
       |              ^^^^ could not find `unix` in `os`
    
    error[E0432]: unresolved import `std::os::unix`
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:10:14
       |
    10 | use std::os::unix::fs::{self as unix, DirBuilderExt, FileTypeExt, OpenOptionsExt, PermissionsExt};
       |              ^^^^ could not find `unix` in `os`
    
    error[E0433]: failed to resolve: use of undeclared type `Mode`
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:79:16
       |
    79 |     let mode = Mode::from_bits_truncate(permissions.mode().try_into()?);
       |                ^^^^ use of undeclared type `Mode`
    
    error[E0599]: no method named `mode` found for struct `DirBuilder` in the current scope
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:61:10
       |
    61 |         .mode(mode)
       |          ^^^^ method not found in `DirBuilder`
    
    error[E0599]: no method named `mode` found for struct `OpenOptions` in the current scope
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:69:10
       |
    69 |         .mode(mode)
       |          ^^^^ method not found in `OpenOptions`
    
    error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:79:53
       |
    79 |     let mode = Mode::from_bits_truncate(permissions.mode().try_into()?);
       |                                                     ^^^^ method not found in `Permissions`
    
    error[E0599]: no method named `is_fifo` found for struct `std::fs::FileType` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:102:29
        |
    102 |         } else if file_type.is_fifo() {
        |                             ^^^^^^^ help: there is an associated function with a similar name: `is_file`
    
    error[E0599]: no method named `is_socket` found for struct `std::fs::FileType` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:104:29
        |
    104 |         } else if file_type.is_socket() {
        |                             ^^^^^^^^^ method not found in `std::fs::FileType`
    
    error[E0599]: no method named `is_char_device` found for struct `std::fs::FileType` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:106:29
        |
    106 |         } else if file_type.is_char_device() {
        |                             ^^^^^^^^^^^^^^ method not found in `std::fs::FileType`
    
    error[E0599]: no method named `is_block_device` found for struct `std::fs::FileType` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\filesystem.rs:108:29
        |
    108 |         } else if file_type.is_block_device() {
        |                             ^^^^^^^^^^^^^^^ method not found in `std::fs::FileType`
    
    error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:48:72
       |
    48 |                 let mut dest = fs::create(dest, metadata.permissions().mode())?;
       |                                                                        ^^^^ method not found in `Permissions`
    
    error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
      --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:62:70
       |
    62 |     fs::create_dir(dest, fs::symlink_metadata(source)?.permissions().mode())?;
       |                                                                      ^^^^ method not found in `Permissions`
    
    error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:102:64
        |
    102 |         .map(|ancestor| fs::metadata(ancestor).map(|meta| meta.ino()));
        |                                                                ^^^ method not found in `Metadata`
    
    error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:108:68
        |
    108 |         .map(|source| fs::symlink_metadata(source).map(|meta| meta.ino()))
        |                                                                    ^^^ method not found in `Metadata`
    
    error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:201:46
        |
    201 |         (_, Ok(metadata)) if source_metadata.ino() == metadata.ino() => fatal(format!(
        |                                              ^^^ method not found in `Metadata`
    
    error[E0599]: no method named `ino` found for struct `Metadata` in the current scope
       --> C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\fcp-0.2.1\src\lib.rs:201:64
        |
    201 |         (_, Ok(metadata)) if source_metadata.ino() == metadata.ino() => fatal(format!(
        |                                                                ^^^ method not found in `Metadata`
    
    Some errors have detailed explanations: E0432, E0433, E0599.
    For more information about an error, try `rustc --explain E0432`.
    error: could not compile `fcp` due to 19 previous errors
    warning: build failed, waiting for other jobs to finish...
    error: failed to compile `fcp v0.2.1`, intermediate artifacts can be found at `C:\Users\hitus\AppData\Local\Temp\cargo-installK8TlaQ`
    
    Caused by:
      build failed
    
    wontfix 
    opened by tshrpl 1
  • Benchmarks question: what is copying time?

    Benchmarks question: what is copying time?

    Benchmark plots show some time period in seconds.

    What does that time mean? The time between CLI tool start and when it exits? Or the time when copying actually finishes under the hood (so e.g. removable storage can be ejected)?

    Also were benchmarks performed hot (when involved data was already in RAM) or cold (e.g. like after Mac analogue of /proc/sys/vm/drop_caches, if it exists)?

    Also how benchmarks look with NFS or removable storage?

    opened by vi 1
  • Feature request: Speed limit and IOPS limit.

    Feature request: Speed limit and IOPS limit.

    Sometimes (especially when the 12309 or something gets in the way) unfortunate filesystem copy operation may affect the whole system.

    One of workarounds may be limiting speed.

    Ideally it should support setting or changing the limit after copy operation has already started (like with pv).

    opened by vi 1
  • Feature request: setting maximum thread count.

    Feature request: setting maximum thread count.

    Sometimes I want to economize threads. Also I expect that using only one thread can be beneficial if I know that the copying would involve a rotating storage.

    opened by vi 1
  • Compile error:

    Compile error: "use of unstable library feature 'array_from_ref' "

    Hi, when I typed cargo install fcp the following error was given:

    error[E0658]: use of unstable library feature 'array_from_ref'
       --> /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/fcp-0.2.0/src/lib.rs:190:61
        |
    190 |         (Ok(metadata), _) if metadata.is_dir() => copy_into(array::from_ref(source), dest),
        |                                                             ^^^^^^^^^^^^^^^
        |
        = note: see issue #77101 <https://github.com/rust-lang/rust/issues/77101> for more information
    
    error: aborting due to previous error
    
    For more information about this error, try `rustc --explain E0658`.
    error: could not compile `fcp`
    
    To learn more, run the command again with --verbose.
    warning: build failed, waiting for other jobs to finish...
    error: failed to compile `fcp v0.2.0`, intermediate artifacts can be found at `/tmp/cargo-installoQO1VQ`
    
    Caused by:
      build failed
    

    So what should I do to get it to compile? Also, maybe the answer should be put on the readme page to help others :)

    opened by Gabriel-Alves-Cunha 1
  • Arch Linux AUR package ready

    Arch Linux AUR package ready

    Nothing to report. Just a heads up that I've packaged it for the AUR. Thanks for sharing this interesting 'cp' implementation.

    https://aur.archlinux.org/packages/fcp-git/

    opened by cjsthompson 1
  • Exit with an error when multiple sources have the same file name

    Exit with an error when multiple sources have the same file name

    Previously when presented with this scenario, fcp would attempt to copy both sources to the same destination, with essentially undefined results as multiple threads would concurrently write to the same file. Now we check for duplicated file names before any copying is performed, resolving this issue. Corresponding test-cases have been added.

    opened by Svetlitski 0
  • Further refactor library internals

    Further refactor library internals

    • Rename 'copy_many' to 'copy_into' to better reflect its purpose
    • Remove unnecessary braces from match arms
    • Assorted improvements to make code better follow Rust idioms
    opened by Svetlitski 0
  • Refactoring

    Refactoring

    Assorted refactoring to improve the quality of the codebase.

    There are no functional changes, other than fixing a bug within the testing machinery (namely stack overflow while hydrating fixtures). This is not user-visible, as it affects only testing utilities.

    opened by Svetlitski 0
  • [Feature] Support clonefile(2) on macOS

    [Feature] Support clonefile(2) on macOS

    fcp is an awesome project, and amazingly fast.

    However, it could be even faster if it used the clonefile(2) syscall on macOS. This is instantaneous and creates a copy-on-write file backed by the same filesystem blocks as the original. For copies that do not cross onto different filesystems / devices, this is effectively identical to a manual copy, while being O(1) instead of O(n).

    Rendered man page here: https://www.manpagez.com/man/2/clonefile/ Raw man page here: https://opensource.apple.com/source/xnu/xnu-3789.21.4/bsd/man/man2/clonefile.2.auto.html

    opened by zachriggle 0
  • [Feature] A progress meter would be nice

    [Feature] A progress meter would be nice

    Currently fcp is a "do one thing, do it well" tool, but it lacks a lot of introspection.

    It would be nice if there were a --progress flag that printed out the number of files / bytes copied per second, so that one could gauge how fast a copy is performing.

    opened by zachriggle 2
  • Deadlock when copying to a FAT32 volume

    Deadlock when copying to a FAT32 volume

    My wife asked me to transfer her photos from the backup location on my server to her usb thumb drive. In order than she can mount this drive on Windows I have it formatted as FAT32. I thought I'd give fcp a go to speed up the transfer since it usually takes a very long time with rsync or cp. While the transfer did begin at an amazing speed, it eventually deadlocks waiting for a FUTEX that apparently never gets released. I was able to determine this by stracing the process.

    I attempted this several times to see if it was a fluke and it deadlocked every time after transfering about 8-10G of data

    opened by nrdxp 0
  • $Sign up for GitHub Sponsors $

    $Sign up for GitHub Sponsors $

    fcp has made a lot of daily chores much faster, especially with Thunderbolt 3 (2.6GBps) and Apple Silicon internal gen4 SSD drives.

    Please consider joining https://github.com/sponsors ‼️

    opened by zachriggle 1
  • Support for common flags from GNU ls

    Support for common flags from GNU ls

    fcp is pretty great, and I make a lot of use out of it!

    One feature that would be nice is a few flags...

    [] -n/--no-clobber [] -a/--archive [] -d/--no-dereference [] --preserve={all, mode,ownership,timestamps,context,links,xattr}

    The -a and -n are very to prevent overwriting files, as well as preserving extra metadata (especially on macOS!)

    opened by zachriggle 1
Releases(v0.2.1)
Owner
Kevin Svetlitski
Computer Science at UC Berkeley; Former software performance engineering intern at @facebook; Former full-stack web development intern @onboardiq
Kevin Svetlitski
Pink is a command-line tool inspired by the Unix man command.

Pink is a command-line tool inspired by the Unix man command. It displays custom-formatted text pages in the terminal using a subset of HTML-like tags.

null 3 Nov 2, 2023
Reviving the Research Edition Unix speak command

This repository contains the source code of Unix speak program that appeared in the Third (1973) to Sixth (1975) Research Unix editions, slightly adjusted to run on a modern computer. Details on the code's provenance and the methods employed for reviving it can be found in this blog post.

Diomidis Spinellis 31 Jul 27, 2022
xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement

xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement, but as a companion utility with some more user-friendly feedback and some optimisations that make sense under certain tasks (see below).

Steve Smith 310 Jan 5, 2023
Work to enable a Classic Mac (24-bit 68000) with ~16MB of RAM.

Apple SE FDHD ROM analysis In order to build a Mac clone that doesn't fully emulate the hardware (which is possible because the ROM abstracts hardware

Simon Frankau 6 Oct 6, 2023
Terminal command correction, alternative to thefuck written in Rust

Pay Respects Typed a wrong command? Pay Respects will try to correct your wrong console command by simply pressing F! ?? Blazing fast suggestion: You

iff 11 Aug 9, 2023
A zero-dependency crate for writing repetitive code easier and faster.

akin A zero-dependency crate for writing repetitive code easier and faster. Check Syntax for information about how to use it. Why? Example Syntax NONE

LyonSyonII 36 Dec 29, 2022
Provide CRUD CLI for Moco Activities with Jira Cloud Sync Option for faster time tracking.

Moco CLI Provide CRUD CLI for Moco Activities with Jira Cloud Sync Option for faster time tracking. Available commands Login Jira Must be called befor

Emanuel Vollmer 7 Nov 18, 2022
A Faster(⚡) formatter, linter, bundler, and more for JavaScript, TypeScript, JSON, HTML, Markdown, and CSS Lapce Plugin

Lapce Plugin for Rome Lapce-rome is a Lapce plugin for rome, The Rome is faster ⚡ , A formatter, linter, compiler, bundler, and more for JavaScript, T

xiaoxin 7 Dec 16, 2022
png_defringe_rs is a port of Immorpher's PNG Defringe program written in Rust to achieve easier installation and faster performance.

png_defringe_rs png_defringe_rs is a port of Immorpher's PNG Defringe program written in Rust to achieve easier installation and faster performance. U

null 2 Nov 17, 2022
Technically, this does exactly what sleep does but completes much faster!

hypersleep Technically does everything that sleep does but it is "blazingly fast!" For example, $ time sleep 1 real 0m1.005s user 0m0.001s sys

Nigel 4 Oct 27, 2022
Backend service to build customer facing dashboards 10x faster. Written in Rust.

Frolic is an open source backend service (written in Rust) to build customer facing dashboards 10x faster. You can directly connect your database to t

Frolic 82 Aug 7, 2023
A reasonable web framework. Others are faster, but this is likely to be more economical.

intercity A web framework for people wanting to build microservices good and build other stuff good too. Intercity is a reasonable web framework. Othe

Tim McNamara 3 Nov 2, 2023
Recompile Rust faster. Good for your flow state.

plonk Plonk is a development-time build tool for Rust projects. cargo install cargo-plonk # fn main() { # lib::say_hello(); # } $ cargo build -p exam

Divy Srivastava 16 Dec 12, 2023
Scriptable tool to read and write UEFI variables from EFI shell. View, save, edit and restore hidden UEFI (BIOS) Setup settings faster than with the OEM menu forms.

UEFI Variable Tool (UVT) UEFI Variable Tool (UVT) is a command-line application that runs from the UEFI shell. It can be launched in seconds from any

null 4 Dec 11, 2023
A Rust curses library, supports Unix platforms and Windows

pancurses pancurses is a curses library for Rust that supports both Linux and Windows by abstracting away the backend that it uses (ncurses-rs and pdc

Ilkka Halila 360 Jan 7, 2023
Spawn multiple concurrent unix terminals in Discord

Using this bot can be exceedingly dangerous since you're basically granting people direct access to your shell.

Simon Larsson 11 Jun 1, 2021
A tool for automating terminal applications in Unix.

expectrl A tool for automating terminal applications in Unix. Using the library you can: Spawn process Control process Expect/Verify responces It was

Maxim Zhiburt 132 Dec 14, 2022
A small unix and windows lib to search for executables in PATH folders.

A small unix and windows lib to search for executables in path folders.

Robiot 2 Dec 25, 2021