Codemod - Codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention

Related tags

Command-line codemod
Overview

codemod

PyPI downloads Travis CI Code Health

Overview

codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention.

Example: Let's say you're deprecating your use of the tag. From the command line, you might make progress by running:

(.*?)' \ '\2'">
codemod -m -d /home/jrosenstein/www --extensions php,html \
    '(.*?)' \
    '\2'

For each match of the regex, you'll be shown a colored diff, and asked if you want to accept the change (the replacement of the tag with a tag), reject it, or edit the line in question in your $EDITOR of choice.

Install

In a virtual environment or as admin user

pip install codemod

or system wide with sudo

sudo -H pip install codemod

Usage

The last two arguments are a regular expression to match and a substitution string, respectively. Or you can omit the substitution string, and just be prompted on each match for whether you want to edit in your editor.

Options (all optional) include:

-m
  Have regex work over multiple lines (e.g. have dot match newlines).  By
  default, codemod applies the regex one line at a time.
-d
  The path whose ancestor files are to be explored.  Defaults to current dir.
-i
  Make your search case-insensitive
--start
  A path:line_number-formatted position somewhere in the hierarchy from which
  to being exploring, or a percentage (e.g. "--start 25%") of the way through
  to start.  Useful if you're divvying up the substitution task across
  multiple people.
--end
  A path:line_number-formatted position somewhere in the hierarchy just
  *before* which we should stop exploring, or a percentage of the way
  through, just before which to end.
--extensions
  A comma-delimited list of file extensions to process. Also supports Unix
  pattern matching.
--include-extensionless
  If set, this will check files without an extension, along with any
  matching file extensions passed in --extensions
--accept-all
  Automatically accept all changes (use with caution)
--default-no
  Set default behavior to reject the change.
--editor
  Specify an editor, e.g. "vim" or "emacs".  If omitted, defaults to $EDITOR
  environment variable.
--count
  Don't run normally.  Instead, just print out number of times places in the
  codebase where the 'query' matches.
--test
  Don't run normally.  Instead, just run the unit tests embedded in the
  codemod library.

You can also use codemod for transformations that are much more sophisticated than regular expression substitution. Rather than using the command line, you write Python code that looks like:

import codemod
codemod.Query(...).run_interactive()

See the documentation for the Query class for details.

Background

Announcement by Justin Rosenstein on Facebook Notes, circa December 2008

Part of why most code -- and most software -- sucks so much is that making sweeping changes is hard.

Let's say that a month ago you wrote a function that you -- or your entire company -- have been using frequently. And now you decide to change its name, or change the order of its parameters, or split it up into two separate functions and then have half the call sites use the old one and half the call sites use the new one, or change its return type from a scalar to a structure with additional information. IDEs and standard *nix tools like sed can help, but you typically have to make a trade-off between introducing errors and introducing tedium. The result, all too often, is that we decide (often unconsciously) that the sweeping change just isn't worth it, and leave the undesirable pattern untouched for future versions of ourselves and others to grumble about, while the pattern grows more and more endemic to the code base.

What you really want is to be able to describe an arbitrary transform -- using either regexes in the 80% case or Python code for more complex transformations -- that matches for lines (or sets of lines) of source code and converts them to something more desirable, but then have a tool that will show you each of the change sites one at a time and ask you either to accept the change, reject the change, or manually intervene using your editor of choice.

So, while at Facebook, I wrote a script that does exactly that. codemod.py a nifty little utility/library to assist with codebase refactors that can be partially automated but still require human oversight and occasional intervention. And, thanks to help from Mr. David Fetterman, codemod is now open source. Check it out (so to speak):

git clone git://github.com/facebook/codemod.git
(previously svn checkout https://codemod.svn.sourceforge.net/svnroot/codemod/trunk codemod)

It's one of those tools where, the more you use it, the more you think of places to use it -- and the more you realize how much you were compromising the quality of your code because reconsidering heavily-used code patterns sounded just too damn annoying. I use it pretty much every day.

Dependencies

  • python2

Credits

Copyright (c) 2007-2008 Facebook.

Created by Justin Rosenstein.

Licensed under the Apache License, Version 2.0.

Comments
  • Released to PyPI, Added

    Released to PyPI, Added "codemod" as console program, fix errors and styles.

    • Fix #62 - Better setup.py and released to PyPI - PyPI
    • Fixed many errors, PEP8 errors, code smells and styling problems
    • Added Py.test as test_runner for doctests make test
    • Added flake8 to check for styling problems
    • Added Landscape.io to check code health (reached 100%) - Code Health
    • Build passing in travisCI using py.test Travis CI - https://travis-ci.org/facebook/codemod/builds/98046530

    If merging:

    1. I need a PiPY username to transfer the ownership
    2. Somebody with repo access needs to register in landscape.io and change badge url
    CLA Signed 
    opened by rochacbruno 6
  • Fixed python3 issues

    Fixed python3 issues

    Actullay refactored the code so that it can run both on python3 and python2 The changes include:

    • print_function
    • use of range
    • wrapping map function with list
    • some flake8 styling issues
    CLA Signed 
    opened by shantanu404 5
  • Proposed fix for issue #40: add option to include files w/o extensions

    Proposed fix for issue #40: add option to include files w/o extensions

    Create new command line option called --include-extensionless files that will override the path_filter where a path without an extension is encountered.

    opened by glenpike 5
  • implement pep-0394

    implement pep-0394

    make it explicit that codemod needs python2, as it does not run with python3. according to pep-0394, python distributors are allowed to provide either python 2 or 3 as python command, and should provide python2 and python3 commands to refer to python v2 or 3 respectively.

    see http://www.python.org/dev/peps/pep-0394/

    without this fix, codemod doesn't run on platforms where python is python3, such as Arch Linux

    opened by Dieterbe 5
  • Modularize base.py for readability

    Modularize base.py for readability

    ~This is downstream of https://github.com/facebook/codemod/pull/89~

    Partially addresses https://github.com/facebook/codemod/issues/64. Though I think things can be much cleaner still, I didn't want to diverge too far without merging first.

    The code is mostly the same, but individual classes have been moved into their own classes. Additionally, I created two "helper" modules that include general use functions. In order to avoid a circular dependency I removed the run_interactive method from the Query class and update the associated documentation.

    CLA Signed 
    opened by keyan 4
  • Extensions glob matching

    Extensions glob matching

    This PR adds pattern matching to the --extensions argument. It also changes the script to not require either one of --extensions or --include-extensionless in order to run. Instead if --extensions is not provided it will always default to the Unix pattern *. Includes some code cleanup to support this change.

    Open to hearing your thoughts on this. I'm guessing the potentially controversial bit will be defaulting to * and not requiring the above args.

    The only strange behavior I noticed with this change is that if you are invoking the program with a wildcard extension you need to add a comma for argparse to work:

    codemod --extensions *  '<match>' '<change>'            -->        fails
    codemod --extensions *, '<match>' '<change>'            -->        works fine
    

    Not sure how important this is if we keep the default * argument though.

    This resolves https://github.com/facebook/codemod/issues/30 and https://github.com/facebook/codemod/issues/56. Also, unrelated but https://github.com/facebook/codemod/issues/59 should be closed as it is resolved.

    CLA Signed 
    opened by keyan 4
  • Add --default-no flag

    Add --default-no flag

    Addresses issue #50 by adding an option to set default to no. This may not be as extensible as the change you had envisioned @modocache?

    Thanks for the great tool, I've been using it here at Venmo to clean up some areas of the codebase and it has been invaluable!

    CLA Signed 
    opened by keyan 4
  • Usage should indicate that one of '--extensions' and '--include-extensionless' is required

    Usage should indicate that one of '--extensions' and '--include-extensionless' is required

    Currently, if I run codemod.py foo bar, I get the following:

    usage: codemod.py [-h] [-m] [-d D] [-i] [--start START] [--end END]
                      [--extensions EXTENSIONS] [--include-extensionless]
                      [--exclude-paths EXCLUDE_PATHS] [--accept-all]
                      [--editor EDITOR] [--count] [--test]
                      [match] [subst]
    

    In most utilities I've used, the usage string does not use square brackets around required parameters.

    opened by DanielHeath 4
  • Add an option to perform case-insensitive search

    Add an option to perform case-insensitive search

    Adding an -i flag to perform case-insensitive search. This has been particularly helpful for (S)CSS refactoring, tracking down stray hex codes without reliably knowing if they’re upper or lower case.

    CLA Signed 
    opened by daneden 4
  • Documentation, plus Ruby added to default extensions.

    Documentation, plus Ruby added to default extensions.

    Thanks for open-sourcing codemod - it's a great tool.

    I've collated the source code documentation and the original announcement into a README.md which will be displayed on the GitHub project page.

    Also, .rb and .erb files are added to the default extensions to support Ruby projects.

    Cheers! Paul

    opened by pda 4
  • Use re.MULTILINE instead of re.DOTALL

    Use re.MULTILINE instead of re.DOTALL

    Not sure which use cases this change breaks, but I found I need the other mode to also catch trailing newlines when removing rcsids from old C code with this invocation: codemod -m '^.+?RCSID.+?$\n*'

    Or maybe we could add a command line option to choose the multiple line mode?

    CLA Signed 
    opened by snickl 3
  • Use Github actions to replace Travis for CI

    Use Github actions to replace Travis for CI

    Travis CI hasn't been active on this repository for a long time and it looks like it was set up with travis-ci.org (that's going to shut down on December 31st, 2020)

    This PR intends to replace Travis CI with GitHub Actions instead, it replaces all Travis CI operations with a GitHub workflow and fixes a few PEP8 warnings. Newly added workflows are working fine here: https://github.com/mraarif/codemod/pull/1

    CLA Signed 
    opened by mraarif 3
  • --count result is incorrect

    --count result is incorrect

    Hi,

    It seems that the --count option returns the correct result minus one. For example if I have only one matching string, it returns zero matches, or if I have 30 matches, it returns 29.

    Thanks for building for this very useful tool!

    opened by kir0ul 0
  • Use GitHub Actions instead

    Use GitHub Actions instead

    This PR replaces TravisCI with GitHub Actions. ​The number of available builds for TravisCI has been decreasing lately, making a lot of sense to use GitHub Actions. ​This PR also fixes all flake8 errors at the same time. Well, it ignores the W503, but that's not a big problem at all. See: https://github.com/facebook/codemod/pull/120/commits/4ecce39f4720daf247bfbe36a488731d78c60f92#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R10-R16

    CLA Signed 
    opened by smorimoto 3
  • Doesn't recognize .d.ts extension

    Doesn't recognize .d.ts extension

    Trying to run codemod on .d.ts files, but it doesn't find any.

    I'm running; codemod --extensions 'd.ts,' '(export { default as (\w*) }.*)' 'export const \2: unknown; //\1 'a.

    opened by wildeyes 0
Releases(20151117)
Owner
Meta Archive
These projects have been archived and are generally unsupported, but are still available to view and use
Meta Archive
fastmod is a fast partial replacement for the codemod tool

fastmod is a fast partial replacement for codemod. Like codemod, it is a tool to assist you with large-scale codebase refactors, and it supports most of codemod's options.

Facebook Incubator 1.4k Dec 29, 2022
Cross-platform CLI Rimworld Mod manager. Still under development

rwm Inspired by Spoons rmm. This is a cross-platform Mod Manager for RimWorld intended to work with macOS, linux and Windows Up to now, you must have

Alejandro O 7 Sep 5, 2022
Components of Fornjot that are no longer actively maintained. Pull requests still welcome!

Fornjot - Extra Components About These are extra components from the Fornjot repository, that are no longer actively maintained. Fornjot's goal was to

Hanno Braun 4 Jun 6, 2023
a simple program that you can scrap, is shit and really simple but is cool.

if you want to run it you need to have installed curl by default scrap youtube, but you can change it, also change the number of threads and run: carg

pai 7 Oct 15, 2021
Store your transfer.sh links, so you can remember them later and know when they will expire, but now written in Rust.

Transfer.sh helper Rusted The idea of the script is to store your transfer.sh links and simplify its usage, so you can remember them later and know wh

Reinaldo Rozato Junior 10 Nov 30, 2022
Format codebase in documentation 🦤

Gelatyx Format codebase in documentation ?? Features Format language code block inside documentation files Check mode. Ask Gelatyx is the documentatio

azzamsa 3 Oct 24, 2022
Ethereum transaction simulator leveraging Foundry's codebase

Enso Transaction ?? Simulator ?? A simple API which simulates a given transaction request. ?? API ?? POST /api/v1/simulate Simulates a single transact

null 162 Jun 4, 2023
Local-first high performance codebase index engine designed for AI

CodeIndex CodeIndex is a local-first high performance codebase index engine designed for AI. It helps your LLM understand the structure and semantics

Jipiti AI 9 Aug 30, 2023
Mercy is a public Rust crate created to assist with building cybersecurity frameworks, assessment tools, and numerous other projects

Mercy ?? Documentation Mercy is a public Rust crate created to assist with building cybersecurity frameworks, assessment tools, and numerous other pro

Umiko Security 2 Nov 27, 2022
Mercy is a public Rust crate created to assist with building cybersecurity frameworks, assessment tools, and numerous other projects

Mercy ?? Documentation Mercy is a public Rust crate created to assist with building cybersecurity frameworks, assessment tools, and numerous other pro

CyberSuki 2 Nov 27, 2022
A collection of tools for i3 that assist in window, workspace and output operations.

i3-valet A collection of tools for i3 that assist in window, workspace and output operations. i3-valet can be run directly from the command line or as

Erich Heine 15 Jan 8, 2023
This is choose, a human-friendly and fast alternative to cut and (sometimes) awk

Choose This is choose, a human-friendly and fast alternative to cut and (sometimes) awk Features terse field selection syntax similar to Python's list

Ryan Geary 1.4k Jan 7, 2023
zigfi is an open-source stocks, commodities and cryptocurrencies price monitoring CLI app, written fully in Rust, where you can organize assets you're watching easily into watchlists for easy access on your terminal.

zigfi zigfi is an open-source stocks, commodities and cryptocurrencies price monitoring CLI app, written fully in Rust, where you can organize assets

Aldrin Zigmund Cortez Velasco 18 Oct 24, 2022
Sero is a web server that allows you to easily host your static sites without pain. The idea was inspired by surge.sh but gives you full control.

sero Lightning-fast, static web publishing with zero configuration and full control ?? Table Of Contents ?? Table Of Contents ?? Tools ❓ About The Pro

Dmitry Miasnenko 6 Nov 13, 2023
Grep with human-friendly search output

hgrep: Human-friendly GREP hgrep is a grep tool to search files with given pattern and print the matched code snippets with human-friendly syntax high

Linda_pp 345 Jan 4, 2023
A simple, human-friendly, embeddable scripting language

Mica Language reference · Rust API A simple, human-friendly scripting language, developed one feature at a time. Human-friendly syntax inspired by Rub

Mica programming language 32 Dec 30, 2022
Fuzzy Index for Python, written in Rust. Works like error-tolerant dict, keyed by a human input.

FuzzDex FuzzDex is a fast Python library, written in Rust. It implements an in-memory fuzzy index that works like an error-tolerant dictionary keyed b

Tomasz bla Fortuna 8 Dec 15, 2022
Human numeric sorting program — does what `sort -h` is supposed to do!

hns — Human Numeric Sort v0.1.0 (⏫︎2022-09-20) © 2022 Fredrick R. Brennan and hns Authors Apache 2.0 licensed, see LICENSE. man page Packages hns_0.1.

Fredrick Brennan 7 Sep 25, 2022
Display file sizes in human-readable units

hsize Display file sizes in human-readable units $ hsize 1000 1000000 5000000 1.00 KB 1.00 MB 5.00 MB $ hsize -p 5 1048576 12345678 1.04858 MB 12.345

Ryan 2 Nov 21, 2022