fastmod is a fast partial replacement for the codemod tool

Last update: Jun 25, 2022

fastmod

fastmod is a fast partial replacement for codemod. Like codemod, it is a tool to assist you with large-scale codebase refactors, and it supports most of codemod's options. fastmod's major philosophical difference from codemod is that it is focused on improving the use case "I want to use interactive mode to make sure my regex is correct, and then I want to apply the regex everywhere". For this use case, it offers much better performance than codemod. Accordingly, fastmod does not support codemod's --start, --end, or --count options, nor does it support anything like codemod's Python API.

Examples

Let's say you're deprecating your use of the tag. From the command line, you might make progress by running:

(.*?)' \ '${2}' ">
fastmod -m -d /home/jrosenstein/www --extensions php,html \
    '(.*?)' \
    '${2}'

For each match of the regex, you'll be shown a colored diff and asked if you want to accept the change (the replacement of the tag with a tag), reject it, or edit the line in question in your $EDITOR of choice.

NOTE: Whereas codemod uses Python regexes, fastmod uses the Rust regex crate, which supports a slightly different regex syntax and does not support look around or backreferences. In particular, use ${1} instead of \1 to get the contents of the first capture group, and use $$ to write a literal $ in the replacement string. See the regex crate's documentation for details.

A consequence of this syntax is that the use of single quotes instead of double quotes around the replacement text is important, because the bash shell itself cares about the $ character in double-quoted strings. If you must double-quote your input text, be careful to escape $ characters properly!

fastmod also offers a usability improvement over codemod: it accepts files or directories to process as extra positional arguments after the regex and substitution. For instance, the example above could have been rewritten as

(.*?)' \ '${2}' \ /home/jrosenstein/www ">
fastmod -m --extensions php,html \
    '(.*?)' \
    '${2}' \
    /home/jrosenstein/www

This makes it possible to use fastmod to process a list of files from somewhere else if needed. Note, however, that fastmod does its own parallel directory traversal internally, so doing find ... | xargs fastmod ... may be much slower than using fastmod by itself.

Requirements

fastmod is primarily supported on macOS and Linux.

fastmod has also been reported to work reasonably well on Windows. The major portability concerns are 1) the use of $EDITOR with a fallback and 2) the console UI, which is descended from codemod's ncurses-based text coloring & screen clearing code. Windows-specific issues and PRs will be considered as long as they aren't too invasive. For example, if something doesn't work on Windows because a Linux/Mac-specific API was used instead of equivalent POSIX or Rust standard library calls, we would be happy to fix that. On the other hand, we would like to avoid taking a direct winapi dependency or substantially increasing the size of our dependency graph for Windows-only enhancements.

Building fastmod

fastmod is written in (stable) Rust and compiles with Rust's cargo build system. To build:

$ git clone https://github.com/facebookincubator/fastmod.git
$ cd fastmod
$ cargo build --release
$ ./target/release/fastmod --help
...

Installing fastmod

The easiest way to install fastmod is simply cargo install fastmod. If you have built fastmod from source following the directions above, you can install your build with cargo install.

How fastmod works

fastmod uses the ignore crate to walk the given directory hierarchy using multiple threads in parallel while respecting .gitignore. It uses the grep crate to match each file, reads matching files into memory, applies the given regex substitution one match at a time, and uses the diff crate to present the resulting changes as patches for human review.

Full documentation

See fastmod --help.

License

fastmod is Apache-2.0-licensed.

GitHub

https://github.com/facebookincubator/fastmod
Comments
  • 1. Windows support?

    The documentation states:

    fastmod is supported on macOS and Linux.

    Is there a reason for this support policy? I tested out fastmod on windows and it seems to work fine. I also skimmed the code base for any unixisms and the only one I could find was assuming $EDITOR was set/defaulting to vim.

    Reviewed by bbatha at 2018-04-19 21:47
  • 2. Publish fastmod to Docker Hub, so that it can be used in container-based codemodding frameworks

    I am wondering if it would be possible publish fastmod to Docker Hub in some official capacity, so that it could be used verbatim in the Sourcegraph batch changes spec?

    This whole batch changes codemodding framework is container-based, similar to how Google Cloud Build configs work:

    https://docs.sourcegraph.com/batch_changes/quickstart#write-a-batch-spec

    Is this something Facebook ever does for other open source tools? I realize this can be done unofficially fairly easily, but an official image would be nice.

    For example, https://github.com/comby-tools/comby/ is published to https://hub.docker.com/r/comby/comby (well, no wonder, since the main author is at Sourcegraph :) )

    Reviewed by kkom at 2021-04-16 15:46
  • 3. Skip to next match when user doesn't change

    If the user selects 'n' (to skip a match), the next comparison only starts one character later. If the beginning of the patch pattern, is, e.g. '( +)' this may continue to find the same match many times. I've added logic that starts the search from the end of the previous match, if the user skips the last change. This is not very well tested, so feel free to review it.

    Reviewed by steven807 at 2018-04-19 23:53
  • 4. Can fastmod also modify hidden files?

    In our codebase we have some committed .env files (it's a separate question whether that's a good idea or not...).

    ➜  fastmod-dot-env ls -al
    total 16
    drwxr-xr-x   4 konradkomorowski  staff  128 25 May 14:52 .
    [email protected] 25 konradkomorowski  staff  800 25 May 14:50 ..
    -rw-r--r--   1 konradkomorowski  staff    4 25 May 14:50 .foo
    -rw-r--r--   1 konradkomorowski  staff    4 25 May 14:52 foo
    ➜  fastmod-dot-env cat .foo 
    bar
    ➜  fastmod-dot-env cat foo 
    bar
    

    I noticed that fastmod will ignore those:

    ➜  fastmod-dot-env fastmod --accept-all bar baz
    ➜  fastmod-dot-env cat .foo 
    bar
    ➜  fastmod-dot-env cat foo 
    baz
    

    things work fine if I'm directly targeting that file:

    ➜  fastmod-dot-env fastmod --accept-all bar baz .foo 
    ➜  fastmod-dot-env cat .foo 
    baz
    

    But sometimes I don't know what the filenames will be (as in the case of the codemod we were running), and we are 100% confident that we want to just replace all substrings.

    I tried options like -g **/* and -g */** but that didn't work.

    What's the motivation for this behavior? Is there a workaround that I'm not aware of?

    PS: here's my version

    ➜  fastmod-dot-env fastmod --version
    fastmod 0.4.1
    
    Reviewed by kkom at 2021-05-25 13:59
  • 5. Panic "out of bounds of"

    $ cat b
    #!/bin/bash
    t () {
        echo bla;
        return $?
    }
    
    # Next command should have been: "fastmod '\$\?' '$?;' b" instead, but:
    
    $ fastmod '\$?' '$?;' b
    thread 'main' panicked at 'byte index 18446744073709551615 is out of bounds of `#!/bin/bash
    t () {
        echo bla;
        return $?
    }
    `', src/libcore/str/mod.rs:2131:9
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    
    $ RUST_BACKTRACE=1 fastmod '\$?' '$?;' b
    thread 'main' panicked at 'byte index 18446744073709551615 is out of bounds of `#!/bin/bash
    t () {
        echo bla;
        return $?
    }
    `', src/libcore/str/mod.rs:2131:9
    stack backtrace:
       0: backtrace::backtrace::libunwind::trace
                 at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
       1: backtrace::backtrace::trace_unsynchronized
                 at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
       2: std::sys_common::backtrace::_print_fmt
                 at src/libstd/sys_common/backtrace.rs:77
       3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
                 at src/libstd/sys_common/backtrace.rs:59
       4: core::fmt::write
                 at src/libcore/fmt/mod.rs:1052
       5: std::io::Write::write_fmt
                 at src/libstd/io/mod.rs:1426
       6: std::sys_common::backtrace::_print
                 at src/libstd/sys_common/backtrace.rs:62
       7: std::sys_common::backtrace::print
                 at src/libstd/sys_common/backtrace.rs:49
       8: std::panicking::default_hook::{{closure}}
                 at src/libstd/panicking.rs:204
       9: std::panicking::default_hook
                 at src/libstd/panicking.rs:224
      10: std::panicking::rust_panic_with_hook
                 at src/libstd/panicking.rs:472
      11: rust_begin_unwind
                 at src/libstd/panicking.rs:380
      12: core::panicking::panic_fmt
                 at src/libcore/panicking.rs:85
      13: core::str::slice_error_fail
                 at src/libcore/str/mod.rs:0
      14: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeTo<usize>>::index::{{closure}}
      15: fastmod::index_to_row_col
      16: fastmod::fastmod
      17: fastmod::main
      18: std::rt::lang_start::{{closure}}
      19: std::rt::lang_start_internal::{{closure}}
                 at src/libstd/rt.rs:52
      20: std::panicking::try::do_call
                 at src/libstd/panicking.rs:305
      21: __rust_maybe_catch_panic
                 at src/libpanic_unwind/lib.rs:86
      22: std::panicking::try
                 at src/libstd/panicking.rs:281
      23: std::panic::catch_unwind
                 at src/libstd/panic.rs:394
      24: std::rt::lang_start_internal
                 at src/libstd/rt.rs:51
      25: main
      26: __libc_start_main
      27: <unknown>
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    
    Reviewed by ghuls at 2020-04-21 15:40
  • 6. Add release workflow using GitHub Actions

    This workflow is the same as @lunaryorn's Mdcat -- kudos to him !

    It is triggered when a release is created with a tag named 'v something', e.g. v1.1.0.

    You can see the results on my own fork here.

    The 3 builds for Linux, MacOS and Windows run fine, but I've only tested the Linux one.

    This PR also creates a CHANGELOG.md file, to be included in the released packages.

    Reviewed by ngirard at 2020-03-06 11:10
  • 7. Add a thread count option

    This PR adds an option to configure the amount of threads that the directory iterator will use. It doesn't close any issues but rather removes a "TO DO" left in the code.

    Reviewed by gsquire at 2019-10-05 21:04
  • 8. Find a way to match on complex, multiline patterns

    Hi and thank you for open sourcing fastmod!

    I was trying to use it to replace

    try {
      await thing.start()
    } catch (e) {
      console.error(e)
    }
    

    with just

    await thing.start()
    

    I assume giving enough effort with the multiline regex I could solve it, but I was wondering there would be interest to have a parameter that would accept a simpler pattern in a file?

    Something like fastmod -p pattern-file "replacement \${1}"

    pattern-file could contain something like

    try {
      $1
    } catch (e) {
      console.error(e)
    }
    

    Everything else would remain the same 😄

    Reviewed by felipesere at 2018-07-23 18:07
  • 9. --extensions can't be used to match files with no extension, like BUCK or Makefile

    The solution to this is probably to support -g/--glob like ripgrep, since we use the ignore crate to implement --extensions anyway and we could just implement -g with passthrough to ignore.

    Reviewed by swolchok at 2018-04-19 23:37
  • 10. Bump regex from 1.4.5 to 1.5.5

    Bumps regex from 1.4.5 to 1.5.5.

    Changelog

    Sourced from regex's changelog.

    1.5.5 (2022-03-08)

    This releases fixes a security bug in the regex compiler. This bug permits a vector for a denial-of-service attack in cases where the regex being compiled is untrusted. There are no known problems where the regex is itself trusted, including in cases of untrusted haystacks.

    1.5.4 (2021-05-06)

    This release fixes another compilation failure when building regex. This time, the fix is for when the pattern feature is enabled, which only works on nightly Rust. CI has been updated to test this case.

    1.5.3 (2021-05-01)

    This releases fixes a bug when building regex with only the unicode-perl feature. It turns out that while CI was building this configuration, it wasn't actually failing the overall build on a failed compilation.

    1.5.2 (2021-05-01)

    This release fixes a performance bug when Unicode word boundaries are used. Namely, for certain regexes on certain inputs, it's possible for the lazy DFA to stop searching (causing a fallback to a slower engine) when it doesn't actually need to.

    [PR #768](rust-lang/regex#768) fixes the bug, which was originally reported in ripgrep#1860.

    1.5.1 (2021-04-30)

    This is a patch release that fixes a compilation error when the perf-literal feature is not enabled.

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    Reviewed by dependabot[bot] at 2022-06-06 21:18
  • 11. Bump crossbeam-utils from 0.8.3 to 0.8.8

    Bumps crossbeam-utils from 0.8.3 to 0.8.8.

    Release notes

    Sourced from crossbeam-utils's releases.

    crossbeam-utils 0.8.8

    • Fix a bug when unstable loom support is enabled. (#787)

    crossbeam-utils 0.8.7

    • Add AtomicCell<{i*,u*}>::{fetch_max,fetch_min}. (#785)
    • Add AtomicCell<{i*,u*,bool}>::fetch_nand. (#785)
    • Fix unsoundness of AtomicCell<{i,u}64> arithmetics on 32-bit targets that support Atomic{I,U}64 (#781)

    crossbeam-utils 0.8.6

    • Re-add AtomicCell<{i,u}64>::{fetch_add,fetch_sub,fetch_and,fetch_or,fetch_xor} that were accidentally removed in 0.8.0 0.7.1 on targets that do not support Atomic{I,U}64. (#767)
    • Re-add AtomicCell<{i,u}128>::{fetch_add,fetch_sub,fetch_and,fetch_or,fetch_xor} that were accidentally removed in 0.8.0 0.7.1. (#767)

    crossbeam-utils 0.8.5

    • Add AtomicCell::fetch_update (#704)
    • Support targets that do not have atomic CAS on stable Rust (#698)

    crossbeam-utils 0.8.4

    • Bump loom dependency to version 0.5. (#686)
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    Reviewed by dependabot[bot] at 2022-06-06 19:04
  • 12. Bump thread_local from 1.1.3 to 1.1.4

    Bumps thread_local from 1.1.3 to 1.1.4.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    Reviewed by dependabot[bot] at 2022-06-17 01:39
  • 13. Allow deletion

    when I was trying to remove a matched set of configurations, I realized that fastmod default replacing the config with an empty line instead:

    fastmod 'some-matching-config.*\n.*\n.*\n' ''
    

    It would be nice to have a --delete flag that, instead of replacing existing matches with an empty line, it would just delete the matching lines.

    Reviewed by sluongng at 2021-09-10 11:25
Related tags
🦀️atos for linux by rust - A partial replacement for Apple's atos tool for converting addresses within a binary file to symbols.

atosl-rs ??️ atos for linux by rust - A partial replacement for Apple's atos tool for converting addresses within a binary file to symbols. tested on

Jun 4, 2022
xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement

xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement, but as a companion utility with some more user-friendly feedback and some optimisations that make sense under certain tasks (see below).

Jun 17, 2022
zoxide is a blazing fast replacement for your cd command
zoxide is a blazing fast replacement for your cd command

zoxide A smarter cd command for your terminal zoxide is a blazing fast replacement for your cd command, inspired by z and z.lua. It keeps track of the

Jun 18, 2022
exa is a modern replacement for ls.
exa is a modern replacement for ls.

exa exa is a modern replacement for ls. README Sections: Options — Installation — Development exa is a modern replacement for the venerable file-listi

Jun 22, 2022
procs is a replacement for ps written in Rust.
procs is a replacement for ps written in Rust.

procs is a replacement for ps written in Rust. Documentation quick links Features Platform Installation Usage Configuration Features Output by t

Jun 23, 2022
A drop-in replacement for `dapp` and `seth` in Rust

dapptools.rs Rust port of DappTools dapp example Usage Run Solidity tests Any contract that contains a function starting with test is being tested. Th

Jun 19, 2022
A readline replacement written in Rust

A readline replacement written in Rust Basic example // Create a default reedline object to handle user input use reedline::{DefaultPrompt, Reedline,

Jun 15, 2022
Fls - Ferris-LS, a very bad LS replacement. Besides that, proves that I suck at clean & good code.

FLS A handy ls remake, purely for learning. Why FLS? There's no reason, at all, don't use it. I just want to learn Rust :D Usage Flags: -i = Use icons

Jan 24, 2022
drop-in replacement for libfuzzer
drop-in replacement for libfuzzer

Fazi A reimplementation of libfuzzer in Rust with some improvements Supported Features libFuzzer's mutations SanCov feedback Building without a main()

Jun 13, 2022
Xcode Neovim Replacement-ish.
Xcode Neovim Replacement-ish.

An XCode replacement-ish development environment that aims to be your reliable XCode alternative to develop exciting new [apple] software products ??

Jun 21, 2022
Voila is a domain-specific language launched through CLI tool for operating with files and directories in massive amounts in a fast & reliable way.

Voila is a domain-specific language designed for doing complex operations to folders & files. It is based on a CLI tool, although you can write your V

Jun 5, 2022
🌳 A lightning-fast system fetch tool made with Rust.
🌳 A lightning-fast system fetch tool made with Rust.

?? treefetch A lightning-fast minimalist system fetch tool made in Rust. Even faster than neofetch and pfetch. Made to practice my new Rust skills ??

Jun 24, 2022
A (aspiring to be fast) tool to find duplicate files.

find-duplicates A decenly fast tool to find duplicate files. Handles symbolic and hard links and treats them seperately to duplicates. Quickstart Inst

Jan 21, 2022
A blazingly fast command-line tool for converting Chinese punctuations to English punctuations
A blazingly fast command-line tool for converting Chinese punctuations to English punctuations

A blazingly fast command-line tool for converting Chinese punctuations to English punctuations

May 31, 2022
An open source, programmed in rust, privacy focused tool for reading programming resources (like stackoverflow) fast, efficient and asynchronous from the terminal.

Falion An open source, programmed in rust, privacy focused tool for reading programming resources (like StackOverFlow) fast, efficient and asynchronou

May 4, 2022
A full featured, fast Command Line Argument Parser for Rust

clap Command Line Argument Parser for Rust It is a simple-to-use, efficient, and full-featured library for parsing command line arguments and subcomma

Jun 27, 2022
Alacritty - A fast, cross-platform, OpenGL terminal emulator
Alacritty - A fast, cross-platform, OpenGL terminal emulator

Alacritty is a modern terminal emulator that comes with sensible defaults, but allows for extensive configuration. By integrating with other applications, rather than reimplementing their functionality, it manages to provide a flexible set of features with high performance. The supported platforms currently consist of BSD, Linux, macOS and Windows.

Jun 17, 2022
A hackable, minimal, fast TUI file explorer, stealing ideas from nnn and fzf.
A hackable, minimal, fast TUI file explorer, stealing ideas from nnn and fzf.

xplr A hackable, minimal, fast TUI file explorer, stealing ideas from nnn and fzf. [Quickstart] [Features] [Plugins] [Documentation] [Upgrade Guide] [

Jun 18, 2022
⚡️ Lightning-fast and minimal calendar command line. Written in Rust 🦀
⚡️ Lightning-fast and minimal calendar command line. Written in Rust 🦀

⚡️ Lightning-fast and minimal calendar command line. It's similar to cal. Written in Rust ??

Mar 29, 2022