fastmod is a fast partial replacement for the codemod tool

Related tags

Command-line fastmod
Overview

fastmod

fastmod is a fast partial replacement for codemod. Like codemod, it is a tool to assist you with large-scale codebase refactors, and it supports most of codemod's options. fastmod's major philosophical difference from codemod is that it is focused on improving the use case "I want to use interactive mode to make sure my regex is correct, and then I want to apply the regex everywhere". For this use case, it offers much better performance than codemod. Accordingly, fastmod does not support codemod's --start, --end, or --count options, nor does it support anything like codemod's Python API.

Examples

Let's say you're deprecating your use of the tag. From the command line, you might make progress by running:

(.*?)' \ '${2}' ">
fastmod -m -d /home/jrosenstein/www --extensions php,html \
    '(.*?)' \
    '${2}'

For each match of the regex, you'll be shown a colored diff and asked if you want to accept the change (the replacement of the tag with a tag), reject it, or edit the line in question in your $EDITOR of choice.

NOTE: Whereas codemod uses Python regexes, fastmod uses the Rust regex crate, which supports a slightly different regex syntax and does not support look around or backreferences. In particular, use ${1} instead of \1 to get the contents of the first capture group, and use $$ to write a literal $ in the replacement string. See the regex crate's documentation for details.

A consequence of this syntax is that the use of single quotes instead of double quotes around the replacement text is important, because the bash shell itself cares about the $ character in double-quoted strings. If you must double-quote your input text, be careful to escape $ characters properly!

fastmod also offers a usability improvement over codemod: it accepts files or directories to process as extra positional arguments after the regex and substitution. For instance, the example above could have been rewritten as

(.*?)' \ '${2}' \ /home/jrosenstein/www ">
fastmod -m --extensions php,html \
    '(.*?)' \
    '${2}' \
    /home/jrosenstein/www

This makes it possible to use fastmod to process a list of files from somewhere else if needed. Note, however, that fastmod does its own parallel directory traversal internally, so doing find ... | xargs fastmod ... may be much slower than using fastmod by itself.

Requirements

fastmod is primarily supported on macOS and Linux.

fastmod has also been reported to work reasonably well on Windows. The major portability concerns are 1) the use of $EDITOR with a fallback and 2) the console UI, which is descended from codemod's ncurses-based text coloring & screen clearing code. Windows-specific issues and PRs will be considered as long as they aren't too invasive. For example, if something doesn't work on Windows because a Linux/Mac-specific API was used instead of equivalent POSIX or Rust standard library calls, we would be happy to fix that. On the other hand, we would like to avoid taking a direct winapi dependency or substantially increasing the size of our dependency graph for Windows-only enhancements.

Building fastmod

fastmod is written in (stable) Rust and compiles with Rust's cargo build system. To build:

$ git clone https://github.com/facebookincubator/fastmod.git
$ cd fastmod
$ cargo build --release
$ ./target/release/fastmod --help
...

Installing fastmod

The easiest way to install fastmod is simply cargo install fastmod. If you have built fastmod from source following the directions above, you can install your build with cargo install.

How fastmod works

fastmod uses the ignore crate to walk the given directory hierarchy using multiple threads in parallel while respecting .gitignore. It uses the grep crate to match each file, reads matching files into memory, applies the given regex substitution one match at a time, and uses the diff crate to present the resulting changes as patches for human review.

Full documentation

See fastmod --help.

License

fastmod is Apache-2.0-licensed.

Comments
  • Windows support?

    Windows support?

    The documentation states:

    fastmod is supported on macOS and Linux.

    Is there a reason for this support policy? I tested out fastmod on windows and it seems to work fine. I also skimmed the code base for any unixisms and the only one I could find was assuming $EDITOR was set/defaulting to vim.

    opened by bbatha 10
  • Publish fastmod to Docker Hub, so that it can be used in container-based codemodding frameworks

    Publish fastmod to Docker Hub, so that it can be used in container-based codemodding frameworks

    I am wondering if it would be possible publish fastmod to Docker Hub in some official capacity, so that it could be used verbatim in the Sourcegraph batch changes spec?

    This whole batch changes codemodding framework is container-based, similar to how Google Cloud Build configs work:

    https://docs.sourcegraph.com/batch_changes/quickstart#write-a-batch-spec

    Is this something Facebook ever does for other open source tools? I realize this can be done unofficially fairly easily, but an official image would be nice.

    For example, https://github.com/comby-tools/comby/ is published to https://hub.docker.com/r/comby/comby (well, no wonder, since the main author is at Sourcegraph :) )

    opened by kkom 5
  • Skip to next match when user doesn't change

    Skip to next match when user doesn't change

    If the user selects 'n' (to skip a match), the next comparison only starts one character later. If the beginning of the patch pattern, is, e.g. '( +)' this may continue to find the same match many times. I've added logic that starts the search from the end of the previous match, if the user skips the last change. This is not very well tested, so feel free to review it.

    CLA Signed Merged 
    opened by steven807 4
  • Can fastmod also modify hidden files?

    Can fastmod also modify hidden files?

    In our codebase we have some committed .env files (it's a separate question whether that's a good idea or not...).

    ➜  fastmod-dot-env ls -al
    total 16
    drwxr-xr-x   4 konradkomorowski  staff  128 25 May 14:52 .
    drwx------@ 25 konradkomorowski  staff  800 25 May 14:50 ..
    -rw-r--r--   1 konradkomorowski  staff    4 25 May 14:50 .foo
    -rw-r--r--   1 konradkomorowski  staff    4 25 May 14:52 foo
    ➜  fastmod-dot-env cat .foo 
    bar
    ➜  fastmod-dot-env cat foo 
    bar
    

    I noticed that fastmod will ignore those:

    ➜  fastmod-dot-env fastmod --accept-all bar baz
    ➜  fastmod-dot-env cat .foo 
    bar
    ➜  fastmod-dot-env cat foo 
    baz
    

    things work fine if I'm directly targeting that file:

    ➜  fastmod-dot-env fastmod --accept-all bar baz .foo 
    ➜  fastmod-dot-env cat .foo 
    baz
    

    But sometimes I don't know what the filenames will be (as in the case of the codemod we were running), and we are 100% confident that we want to just replace all substrings.

    I tried options like -g **/* and -g */** but that didn't work.

    What's the motivation for this behavior? Is there a workaround that I'm not aware of?

    PS: here's my version

    ➜  fastmod-dot-env fastmod --version
    fastmod 0.4.1
    
    opened by kkom 3
  • Panic

    Panic "out of bounds of"

    $ cat b
    #!/bin/bash
    t () {
        echo bla;
        return $?
    }
    
    # Next command should have been: "fastmod '\$\?' '$?;' b" instead, but:
    
    $ fastmod '\$?' '$?;' b
    thread 'main' panicked at 'byte index 18446744073709551615 is out of bounds of `#!/bin/bash
    t () {
        echo bla;
        return $?
    }
    `', src/libcore/str/mod.rs:2131:9
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    
    $ RUST_BACKTRACE=1 fastmod '\$?' '$?;' b
    thread 'main' panicked at 'byte index 18446744073709551615 is out of bounds of `#!/bin/bash
    t () {
        echo bla;
        return $?
    }
    `', src/libcore/str/mod.rs:2131:9
    stack backtrace:
       0: backtrace::backtrace::libunwind::trace
                 at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
       1: backtrace::backtrace::trace_unsynchronized
                 at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
       2: std::sys_common::backtrace::_print_fmt
                 at src/libstd/sys_common/backtrace.rs:77
       3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
                 at src/libstd/sys_common/backtrace.rs:59
       4: core::fmt::write
                 at src/libcore/fmt/mod.rs:1052
       5: std::io::Write::write_fmt
                 at src/libstd/io/mod.rs:1426
       6: std::sys_common::backtrace::_print
                 at src/libstd/sys_common/backtrace.rs:62
       7: std::sys_common::backtrace::print
                 at src/libstd/sys_common/backtrace.rs:49
       8: std::panicking::default_hook::{{closure}}
                 at src/libstd/panicking.rs:204
       9: std::panicking::default_hook
                 at src/libstd/panicking.rs:224
      10: std::panicking::rust_panic_with_hook
                 at src/libstd/panicking.rs:472
      11: rust_begin_unwind
                 at src/libstd/panicking.rs:380
      12: core::panicking::panic_fmt
                 at src/libcore/panicking.rs:85
      13: core::str::slice_error_fail
                 at src/libcore/str/mod.rs:0
      14: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeTo<usize>>::index::{{closure}}
      15: fastmod::index_to_row_col
      16: fastmod::fastmod
      17: fastmod::main
      18: std::rt::lang_start::{{closure}}
      19: std::rt::lang_start_internal::{{closure}}
                 at src/libstd/rt.rs:52
      20: std::panicking::try::do_call
                 at src/libstd/panicking.rs:305
      21: __rust_maybe_catch_panic
                 at src/libpanic_unwind/lib.rs:86
      22: std::panicking::try
                 at src/libstd/panicking.rs:281
      23: std::panic::catch_unwind
                 at src/libstd/panic.rs:394
      24: std::rt::lang_start_internal
                 at src/libstd/rt.rs:51
      25: main
      26: __libc_start_main
      27: <unknown>
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    
    opened by ghuls 3
  • Add release workflow using GitHub Actions

    Add release workflow using GitHub Actions

    This workflow is the same as @lunaryorn's Mdcat -- kudos to him !

    It is triggered when a release is created with a tag named 'v something', e.g. v1.1.0.

    You can see the results on my own fork here.

    The 3 builds for Linux, MacOS and Windows run fine, but I've only tested the Linux one.

    This PR also creates a CHANGELOG.md file, to be included in the released packages.

    CLA Signed 
    opened by ngirard 3
  • Add a thread count option

    Add a thread count option

    This PR adds an option to configure the amount of threads that the directory iterator will use. It doesn't close any issues but rather removes a "TO DO" left in the code.

    CLA Signed 
    opened by gsquire 3
  • Find a way to match on complex, multiline patterns

    Find a way to match on complex, multiline patterns

    Hi and thank you for open sourcing fastmod!

    I was trying to use it to replace

    try {
      await thing.start()
    } catch (e) {
      console.error(e)
    }
    

    with just

    await thing.start()
    

    I assume giving enough effort with the multiline regex I could solve it, but I was wondering there would be interest to have a parameter that would accept a simpler pattern in a file?

    Something like fastmod -p pattern-file "replacement \${1}"

    pattern-file could contain something like

    try {
      $1
    } catch (e) {
      console.error(e)
    }
    

    Everything else would remain the same 😄

    opened by felipesere 3
  • --extensions can't be used to match files with no extension, like BUCK or Makefile

    --extensions can't be used to match files with no extension, like BUCK or Makefile

    The solution to this is probably to support -g/--glob like ripgrep, since we use the ignore crate to implement --extensions anyway and we could just implement -g with passthrough to ignore.

    opened by swolchok 3
  • Feature request: flags to exclude directories / respect .gitignore

    Feature request: flags to exclude directories / respect .gitignore

    Problem

    This came up when I was trying to do a version bump this way:

    fastmod --hidden --ignore-case '(poetry.*)1\.2\.2' '${1}1.3.2'
    

    I set the --hidden flag in order to also match asdf's .tool-versions file (needs to be included in the version bump).

    However, I found fastmod matching files in the .git/ folder - which is something I definitely do not want to mess with (it was matching some of the previous commits where I performed the version bump to 1.2.2):

    ./.git/.graphite_cache_persist:128
            }
          ],
          [
            "poetry",
            {
              "validationResult": "VALID",
              "parentBranchName": "main",
              "parentBranchRevision": "3deb75c540a0daeb29986e3ef3cc36e07b8f07b0",
              "branchRevision": "fce724219a729bfb03b8bdd93035872c56f26dfd",
              "children": [],
              "prInfo": {
                "title": "upgrade poetry to 1.2.2",
    -           "body": "Run this to catch places where our current poetry version is mentioned after the string \"poetry\":\n\n```\nfastmod --hidden --ignore-case '(poetry.*)1\\.2\\.1' '${1}1.2.2'\n```\n",
    +           "body": "Run this to catch places where our current poetry version is mentioned after the string \"poetry\":\n\n```\nfastmod --hidden --ignore-case '(poetry.*)1\\.2\\.1' '${1}1.3.2'\n```\n",
                "number": 280,
                "url": "https://app.graphite.dev/github/pr/exponential-hq/expo/280",
                "base": "main",
                "state": "OPEN",
                "reviewDecision": "REVIEW_REQUIRED",
                "draft": false
              }
            }
          ]
        ]
      }
    Accept change (y = yes [default], n = no, e = edit, A = yes to all, E = yes+edit, q = quit)?
    

    (In this case it was a graphite file, rather than a vanilla git file, but I don't think it matters.)

    I suspect that I could somehow use the --glob flag to exclude the directories I wanted, but it's not obvious to me how to do it in an elegant and reliable way.

    Potential solution 1

    Wdyt about having a flag to explicitly exclude some directories?

    Similar to how comby has --exclude and --exclude-dirs: https://comby.dev/docs/cheat-sheet

    Potential solution 2

    Or have a mode where only files tracked by git would be modified by fastmod? I presume that .git is implicitly ignored by git when doing source control, so that could be also implicitly assumed as well.

    I know that Facebook uses Eden now, but maybe you can consider being aware of git too, as nod to the open source community.

    opened by kkom 2
  • Bump thread_local from 1.1.3 to 1.1.4

    Bump thread_local from 1.1.3 to 1.1.4

    Bumps thread_local from 1.1.3 to 1.1.4.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    CLA Signed dependencies 
    opened by dependabot[bot] 2
Releases(v0.4.3)
  • v0.4.3(Aug 10, 2022)

  • v0.4.2(Apr 23, 2021)

    Bug fixes and minor improvements in this release:

    • Improve compatibility with more terminal color schemes (e.g., Solarized).
    • Add the --hidden flag to allow searching in hidden files.
    • Fix a crash due to poor handling of non-ASCII characters.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Oct 15, 2020)

  • v0.4.0(Apr 8, 2020)

    This release primarily contains performance improvements: fastmod now walks the directory hierarchy using multiple threads in parallel outside of --accept-all mode. It also now uses the grep crate to check whether files match.

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jan 14, 2020)

    In this release:

    • Fixed crash when replacing entire contents of a file.
    • Implemented new advancement policy from issue #3. Advancement after accepting or rejecting a replacement will now skip to the end of the matched text rather than retrying a match 1 character later.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.6(Jul 25, 2019)

  • v0.2.5(Jun 27, 2019)

  • v0.2.4(Jun 26, 2019)

    In this release:

    • Code cleanup (eprintln! instead of writeln! to stderr)
    • Don't print a message about fast mode when --accept-all is passed. This allows fastmod to be quieter when using --accept-all.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.3(Mar 28, 2019)

    This is a polish/maintenance release. Changes:

    • Update to new versions of regex and scopeguard dependencies, and run cargo update to refresh versions in Cargo.lock.
    • Apply cargo clippy fixes.
    • Migrate to Rust 2018.

    This is also the first crates.io-published version to include the single commit from v0.2.2, which added the -F/--fixed-strings option.

    Source code(tar.gz)
    Source code(zip)
  • v0.2.2(Mar 28, 2019)

  • v0.2.1(Jul 30, 2018)

  • v0.2.0(May 8, 2018)

    Fixed in this release:

    • Panic when the regex matches but substitution did not result in changes (@dreid, #5).
    • Logic for deleting consecutive matches was buggy (501a8b9484177b869124df93238bf45a85ccad78).
    • --version did not print the version.
    • Broken documentation link in --help (@natansh, #8)

    New feature:

    • Glob support (like ripgrep's -g/--glob and --iglob options) (@gsquire, #2 & #6)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(May 3, 2018)

Owner
Facebook Incubator
We work hard to contribute our work back to the web, mobile, big data, & infrastructure communities. NB: members must have two-factor auth.
Facebook Incubator
🦀️atos for linux by rust - A partial replacement for Apple's atos tool for converting addresses within a binary file to symbols.

atosl-rs ??️ atos for linux by rust - A partial replacement for Apple's atos tool for converting addresses within a binary file to symbols. tested on

everettjf 60 Dec 29, 2022
xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement

xcp is a (partial) clone of the Unix cp command. It is not intended as a full replacement, but as a companion utility with some more user-friendly feedback and some optimisations that make sense under certain tasks (see below).

Steve Smith 310 Jan 5, 2023
Automatically verify your [Partial]Eq/Ord invariants

Reltester Relation tester is a small testing utility for automatically checking the correctness of PartialEq, PartialOrd, Eq, and Ord implementations.

Filippo Neysofu Costa 18 Mar 24, 2023
epNFTs, the first partial program-owned NFT Standard powered by Instruction Introspection and Transfer Hooks!

epNFT Standard: A Comprehensive Guide Introduction to epNFTs Welcome to the epNFT-standard repository, where we explore the first program-owned NFT st

epPlex 6 Mar 1, 2024
zoxide is a blazing fast replacement for your cd command

zoxide A smarter cd command for your terminal zoxide is a blazing fast replacement for your cd command, inspired by z and z.lua. It keeps track of the

Ajeet D'Souza 8.7k Dec 31, 2022
exa is a modern replacement for ls.

exa exa is a modern replacement for ls. README Sections: Options — Installation — Development exa is a modern replacement for the venerable file-listi

Benjamin Sago 20.3k Jan 8, 2023
procs is a replacement for ps written in Rust.

procs is a replacement for ps written in Rust. Documentation quick links Features Platform Installation Usage Configuration Features Output by t

null 3.6k Dec 30, 2022
A drop-in replacement for `dapp` and `seth` in Rust

dapptools.rs Rust port of DappTools dapp example Usage Run Solidity tests Any contract that contains a function starting with test is being tested. Th

Georgios Konstantopoulos 5k Jan 1, 2023
A readline replacement written in Rust

A readline replacement written in Rust Basic example // Create a default reedline object to handle user input use reedline::{DefaultPrompt, Reedline,

JT 292 Jan 9, 2023
Fls - Ferris-LS, a very bad LS replacement. Besides that, proves that I suck at clean & good code.

FLS A handy ls remake, purely for learning. Why FLS? There's no reason, at all, don't use it. I just want to learn Rust :D Usage Flags: -i = Use icons

florida 6 Jan 24, 2022
drop-in replacement for libfuzzer

Fazi A reimplementation of libfuzzer in Rust with some improvements Supported Features libFuzzer's mutations SanCov feedback Building without a main()

Lander Brandt 56 Nov 2, 2022
Xcode Neovim Replacement-ish.

An XCode replacement-ish development environment that aims to be your reliable XCode alternative to develop exciting new [apple] software products ??

null 272 Dec 30, 2022
A more convenient `#[target_feature]` replacement

A more convenient #[target_feature] replacement To get good performance out of SIMD everything on the SIMD codepath must be inlined. With how SIMD is

Koute 3 Apr 10, 2023
Simpler and more powerful replacement for `find`

FindFile (FF) An simple, ergonomic, and powerful replacement for find. Note: this repo is under active development The syntax is (mostly) figured out,

Sam Westerman 4 Jun 20, 2023
A lightweight, no-fuss, drop-in replacement for Rudderstack

Welcome to Stilgar! Stilgar is a lightweight, no-fuss, drop-in replacement for Rudderstack. Key features: Seamlessly compatible with all Rudderstack c

Withings 4 Jul 21, 2023
A modern, maintained replacement for ls

eza eza is a modern, maintained replacement for ls, built on exa. README Sections: Options — Installation — Development eza is a modern, maintained re

eza 13 Aug 7, 2023
Drop-in replacement for the Actix Web HTTP Logger middleware

actix-contrib-logger Logger middleware for the Actix Web framework. Actually it's a copy & paste from the official Logger middleware (original source

Mariano Ruiz 5 Aug 28, 2023
Patch binary file using IDA signatures and defined replacement bytes in YAML.

fabricbin Patch binary file using IDA signatures and defined replacement bytes in YAML. Install: cargo install --git https://github.com/makindotcc/fab

makin 3 Oct 24, 2023
PyO3 bindings and Python interface to skani, a method for fast fast genomic identity calculation using sparse chaining.

?? ⛓️ ?? Pyskani PyO3 bindings and Python interface to skani, a method for fast fast genomic identity calculation using sparse chaining. ??️ Overview

Martin Larralde 13 Mar 21, 2023