Breaking your Rust code for fun and profit

Overview

Breaking your Rust code for fun & profit

this is an architecture-preview, not all components are there

This is a mutation testing framework for Rust code.

Mutation Testing

A change (mutation) in the program source code is most likely a bug of some kind. A good test suite can detect changes in the source code by failing ("killing" the mutant). If the test suite is green even if the program is mutated (the mutant survives), the tests fail to detect this bug.

Mutation testing is a way of evaluating the quality of a test suite, similar to code coverage. The difference to line or branch coverage is that those measure if the code under test was executed, but that says nothing about whether the tests would have caught any error.

How mutagen works

mutagen's core functionality is implemented via a procedural macro that transforms the source code. Known patterns of code are replaced by mutators with identical behavior unless activated. Activating a mutator at runtime alters its behavior - having the effect of a mutation.

The procedural macro has access to the bare AST. Information about inferred types, implemented traits, control flow, data flow or signatures of other functions are not available during the execution of procedural macros. Therefore, the mutations must be possible without additional type-information.

In order to be fast, it is necessary that the compilation of the test suite is performed only once. To achieve this, all mutations are baked into the code once and are selected at runtime via an environment variable. This means the mutations are not allowed to produce code that fails to compile.

This project is basically an experiment to see what mutations we can still apply under those constraints.

Using mutagen

Note: The version of mutagen (0.2.0) referenced in this README is not yet released on crates.io. To install and use an earlier, released version, you can follow the instructions on crates.io mutagen crate.

You need Rust nightly to compile the procedural macro.

Add the library mutagen as a dev-dependency to your Cargo.toml referencing this git repository:

[dev-dependencies]
mutagen = {git = "https://github.com/llogiq/mutagen"}

To use the attribute #[mutate], you need to import it.

#[cfg(test)]
use mutagen::mutate;

Now you can advise mutagen to mutate any function or method by prepending #[cfg_attr(test, mutate)]. The use of cfg_attr ensures the #[mutate] attribute will only be active in test mode. The repository contains an example that shows how mutagen could be used.

Running mutagen

Install cargo-mutagen, which can be done by running cargo install cargo-mutagen. Run cargo mutagen on the project under test for a complete mutation test evaluation.

The mutants can also be run manually: cargo test will compile code and write the performed mutations to target/mutagen/mutations. This file contains ids and descriptions of possible mutations. Then, the environment variable MUTATION_ID can be used to activate a single mutation as defined by the mutations file. The environment variable can be set before calling the test suite, i.e. MUTATION_ID=1 cargo test, MUTATION_ID=2 .., etc. For every mutation count at of least one, the test suite should fail

You can run cargo mutagen -- --coverage in order to reduce the time it takes to run the mutated code. When running in this mode, it runs the test suite at the beginning of the process and checks which tests are hitting mutated code. Then, for each mutation, instead of running the whole test suite again, it executes only the tests that are affected by the current mutation. This mode is especially useful when the test suite is slow or when the mutated code affects a little part of it.

If you referenced mutagen in your cargo.toml via the git repository as noted in the Using Mutagen section, you will probably want to install the development version of cargo-mutagen. To install the development version, run cargo install in the mutagen-runner dir of this repository. Running cargo install --force might be necessary to overwrite any existing cargo-mutagen binary.

A Word of Warning

mutagen will change the code you annotate with the #[mutate] attribute. This can have dire consequences in some cases. However, functions not annotated with #[mutate] will not be altered.

Do not use #[mutate] for code that can cause damage if buggy. By corrupting the behavior or sanity checks of some parts of the program, dangerous accidents can happen. For example by overwriting the wrong file or sending credentials to the wrong server.

Use #[mutate] for tests only. This is done by always annotating functions or modules with #[cfg_attr(test, mutate)] instead, which applies the #[mutate] annotation only in test mode. If a function is annotated with plain #[mutate] in every mode, the mutation-code is baked into the code even when compiled for release versions. However, when using mutagen as dev-dependency, adding a plain #[mutate] attribute will result in compilation errors in non-test mode since the compiler does not find the annotation.

Use mutagen as dev-dependency, unless otherwise necessary. This ensures that no code from mutagen is part of your released library or application.

Limitations of Mutations

No mutations will be introduced in unsafe-blocks and unsafe functions. Such mutations would probably break some invariants. Moreover, mutations in unsafe code could lead to undefined behavior that cannot be observed by any testcase.

const and static expressions cannot be mutated. They are evaluated at compile-time and mutagen can only affect code that can alter its behavior at run-time. Array lengths and global constants are examples of const expressions.

Patterns cannot be mutated. Mutations are introduced by injecting calls to mutagen-internal functions, which cannot be placed inside patterns.

Contributing

Issues and PRs welcome! See CONTRIBUTING.md on how to help.

Comments
  • proposal and preview for mutagen v0.2

    proposal and preview for mutagen v0.2

    As discussed in #142, this is a proposal for a modular architecture, which is the base for mutagen v0.2. I reviewed all parts of the code and tried to move code that works fine to the new system.

    This is a minimal implementation to demonstrate the effectiveness of the architecture. The steps towards releasing mutagen v0.2 are:

    • [x] implement more mutators and document their functionality
    • [x] generate meaningful reports

    Are there any other major tasks for this PR?

    opened by samuelpilz 38
  • Mutagen architecure proposal

    Mutagen architecure proposal

    Hi, I've been working on re-structuring your project mutagen to be more production ready. https://github.com/power-fungus/mutagen-preview

    I spent the last weeks reading into the current project and re-writing parts of mutagen using the same interface, but with a different architecture. It's not completed yet but should give a rough outline of what the finished project will look like: the backbone and glue code already exists and will probably not change too widely.

    @llogiq I would like to know if you are interested in my approach and if you would consider merging my proposal in your project. If so, I will finish this preview into production quality in the next months. Also I am open to any discussion about interface and design decisions.

    I would like to have this rewrite published (once finished) under the "brand" of mutagen:

    • The interface is the same
      • mutators are introduced using the attribute #[mutate]
      • mutations are selected at run-time using the environment variable MUTATION_ID (renamed for clarity)
      • The target code is compiled only once
    • the rewrite is true to the original goals of mutagen
    • the used technologies is the same to the code in the proc-macro-attribute branch
      • new procedural macro attribute
      • uses syn and quote

    For details about the architecture, see: https://github.com/power-fungus/mutagen-preview/tree/master/docs Main Benefits are:

    • Debugging of the code transformation will be easier due to modular transformation
    • The behavior of the mutations can be tested a standard test suite by faking the global MUTATION_ID variable.
    • it will be possible to add customization via arguments to the #[mutate] attribute
    • community collaboration will be easier since mutators and transformers are small bits of code.
    • mutators can be documented independent from each other.

    @llogiq I would like to read from you soon! :)

    opened by samuelpilz 21
  • Mutation: Switch arguments of same type

    Mutation: Switch arguments of same type

    The plugin already has the analysis. There are but two wrinkles to take care of:

    • [ ] It may make a difference if the args are taken by ref or by value, the analysis pass doesn't take that into account yet
    • [ ] we should take care not to issue the same switch twice. This could probably be done easily by filtering the argument pairs so only pairs where the first name compares lower than the last one are switched.
    enhancement help wanted good first issue 
    opened by llogiq 18
  • `mutagen` does not respect `target` directory configuration

    `mutagen` does not respect `target` directory configuration

    If one has the following ~/.cargo/config file

    [build]
    target-dir = "/home/<user>/.cache/cargo"
    

    I would expect mutagen to put artifacts in the configured directory however target/ inside the crate being tested is used.

    opened by tcharding 11
  • inference improvements

    inference improvements

    Fixes #157. The mutation of the snippet given in the issue used to compile but now fails to compile. This PR reverts to the previously use strategy, with slight alterations.

    Background

    We have 2 functions:

    // does not work in old approach
    #[mutate]
    fn test1(mut ret: i64) -> i64 {
        1 + 2
    }
    // does not typecheck in new approach
    #[mutate]
    fn test1(mut ret: i64) -> i64 {
        let x = 1;
        x + 1
    }
    

    The first snippet did not typecheck correctly in the previous version. This led me to implementing a different approach based on detecting if the left side of an arithmetic operation is a integer literal or derived from some integer literal. This made the first one compile but was not smart enough to detect the second case. I thought I could extend it further, but it turned out that there is not enough information to solve this problem in all cases.

    Previously, the trick was to rewrite a + b into if false {a+b} else { <mutator-code>}. I underestimated the potential of the if false trick, which is an error on my part.

    This PR switches to a modified version of the if false trick: a+b is transformed to {let _tmp = a; if false {_tmp+b} else {<mutator-code based on _tmp and b>}}. This makes both versions compile and also passed several other test cases.

    This PR is a work in progress. Feedback is welcome.

    opened by samuelpilz 10
  • Runner - Support for testsuites with more than one executable

    Runner - Support for testsuites with more than one executable

    At the moment, the runner finds only one test executable.

    The method compile_tests compiles the test and returns a single PathBuf, but it should return a vector of them, and execute each of the found test binaries. Note that we are early-returning when we find the first path.

    With this change, mutagen will be able to run on projects which has both unit, integration and doc tests.

    help wanted good first issue runner 
    opened by gnieto 9
  • Example project to show how mutation testing should be used

    Example project to show how mutation testing should be used

    This is a preliminary version of a pull request to add some example code.

    It is not finished yet but I don't really see how to proceed. But I want to post it here to get some feedback. Some of these questions should be answered in the readme or a how to use guide.

    Open questions

    • The mutate attribute: should I place it above all functions that I want to mutate during test?
    • Can/should I make that conditional on cfg(test)? Not sure if I want mutagen code in the final binary
    • I made the first test always fail to show the println!s It seems to show that mutagen is doing something. Is this the case?
    • I used MU.next() 100 times, is there a max or does it depend on the number or types of mutations that are applied? It does not seem to change much after 10 times..
    • Should there be a #[mutate_test] attribute that you could use instead of #[test] to create tests with mutagen?

    TODO

    • [x] The example code is copied from a project without license. I should write a different example to clear up the license issue. I was lazy
    • [x] The first test tries to show that mutagen is working. I am not sure though if this is the intended purpose or direction. But the intention is to make a start with a manual mutagen runner (and maybe move that into the lib/plugin?
    opened by hmvp 9
  • Run only the tests which contains some mutation

    Run only the tests which contains some mutation

    Attempt to fix issue: https://github.com/llogiq/mutagen/issues/28

    I want to get a little bit of feedback before keep going with that.

    As a first step, all the mutations are being instrumented and are calling ::mutagen::report_coverage(). This method, if an environment variables is set (MUTAGEN_COVERAGE), the first time it will be called, it will write to a specific file that some mutation has been hit.

    On the other side, cargo-mutagen will check all the tests that the binary contains, and will execute all the tests individually, with the mentioned env var. After executing each of the tests, we will check if the file has been created. If it exists, we know that the executed test contains code that it's mutated.

    Finally, when running mutations, we will run only the tests that contains mutations (unfortunatelly, we will need to run the test one-by-one, which means that we will spawn a process per test, instead of a process per testsuite). Then, coverage flag it's only useful when the ratio of tests which executes mutated code is low.

    To test with coverage, after updating to the new version of the plugin on the target project, and after reinstalling the runner:

    cargo mutagen -- --coverage
    

    Pending tasks would be (maybe new issues can be opened):

    • Improve the "ping" system with something better than a file
    • ~~Maybe we can include the mutation number to report_coverage method, in order to know which are the mutations that took place on the given test. If we do, we can filter more accuretly which tests should be executed on each mutation~~
    • Maybe we can think about use clap to parse arguments, add subcommands, help, ...
    • ~~Document the cargo mutagen command~~
    opened by gnieto 8
  • Add off-by-one mutation

    Add off-by-one mutation

    As suggested on https://github.com/llogiq/mutagen/issues/5, it would be nice to mutate numeric constants adding or subtracting one unit to numeric constants.

    At the moment, this is very conservative on the limits to add or substract on default integer constants. I've did it like this to avoid compilation issue, as we do not have information regarding the type of the expression. So, in the default case, it subtracts one only if it's > 0 and it only adds one if the value is < 256.

    If the constant has a type annotation (for example, 0i32), It adjust the minimum and the maximum depending on the type hint. Note that it does not check 128 bits numbers, as they are only available on nightly.

    It chains the mutations in order to record only the needed mutations. So, if the code finds a "0" on an unsigned type, it will only record the "add one" mutation, but not the "sub one" (as it would overflow and panic).

    After running it on the example project, it emits mutations like:

        let mut count = {
            if mutagen::now(2usize) {
                0 + 1
            } else {
                0
            }
        };
    

    or

                  ord < {
                        if mutagen::now(12usize) {
                            65 + 1
                        } else {
                            {
                                if mutagen::now(11usize) {
                                    65 - 1
                                } else {
                                    65
                                }
                            }
                        }
                    }
    

    And finally: I didn't add tests. I thought on adding another test on the example project that covers some of the edge cases and I've also thought on unit test the int_constant_can_subtract_one or int_constant_can_add_one, but I'm not sure which is the expected way of testing this library, so, if you have some idea, I can add the kind of tests you think will be more useful.

    opened by gnieto 8
  • Add str literal mutation

    Add str literal mutation

    If you're open to new mutations, I created a mutator for string literals. It replaces empty strings with a single character string and for non-empty strings it does a few things: append and prepend a single character as well as replacing the string with an empty string. I'd welcome any feedback. Thanks!

    opened by samgoldman 7
  • Mutate while let Some(_) loops

    Mutate while let Some(_) loops

    We can mutate while let Some($ident) = $expr { $block } loops to extend $expr to if ::mutagen::now($n) { None } else { $expr }. We already have similar extensions for ifs, so this should be simple to implement.

    enhancement help wanted good first issue 
    opened by llogiq 6
  • Integration with cargo-fuzz

    Integration with cargo-fuzz

    A lot of the tests I have are using libfuzzer/cargo-fuzz to create test curposes, would be nice if I could run those under mutagen, obviously I could write a test function that runs them and then run mutagen, but I have a lot of fuzzers and it will require adding a lot of fuzzing logic code into my crate's tests suite, so it would be nice if I could plug cargo-mutagen into the fuzzer so it will run the fuzzing corpuses under mutations

    opened by elichai 0
  • RUSTSEC-2021-0139: ansi_term is Unmaintained

    RUSTSEC-2021-0139: ansi_term is Unmaintained

    ansi_term is Unmaintained

    | Details | | | ------------------- | ---------------------------------------------- | | Status | unmaintained | | Package | ansi_term | | Version | 0.12.1 | | URL | https://github.com/ogham/rust-ansi-term/issues/72 | | Date | 2021-08-18 |

    The maintainer has adviced this crate is deprecated and will not receive any maintenance.

    The crate does not seem to have much dependencies and may or may not be ok to use as-is.

    Last release seems to have been three years ago.

    Possible Alternative(s)

    The below list has not been vetted in any way and may or may not contain alternatives;

    See advisory page for additional details.

    opened by github-actions[bot] 0
  • Using cargo mutagen in CI pipeline

    Using cargo mutagen in CI pipeline

    Hey!

    I wonder what the options are to use mutagen in CI pipelines. I could not find any support about it such as GitHub actions or so.

    The simplest I could do is to run cargo mutagen in a CI pipeline task and based on the result I could stop or continue the pipeline run. What would be the easiest way to achieve this, what do you think?

    Of course, I am also up to create/submit a PR and change the code regarding, if there is not yet a solution for it.

    Thank you!

    opened by dmoka 7
  • Support output in

    Support output in "mutation elements" format

    There is a format of output for mutation tools called "mutation elements" [1]. Format supported by tools that used for visualization [2] and [3]. It would be nice to support this format in mutagen.

    1. https://github.com/stryker-mutator/mutation-testing-elements
    2. https://www.npmjs.com/package/mutation-testing-elements
    3. https://github.com/ligurio/py-mutation-testing-elements
    opened by ligurio 0
  • Using mutagen breaks type inference in the existing program

    Using mutagen breaks type inference in the existing program

    Given the following lib.rs:

    fn foo() {}
    
    #[cfg(test)]
    mod tests {
        #[test]
        fn it_works() {
    	let x = [0u8];
    	assert_eq!((&x[..]).as_ref(), []);
        }
    }
    

    Running cargo check --tests succeeds. However, if we add an attribute, #[cfg_attr(test, mutagen::mutate)], to fn foo() {}, then cargo check --tests fails:

    error[E0282]: type annotations needed
     --> src/lib.rs:9:2
      |
    9 |     assert_eq!((&x[..]).as_ref(), []);
      |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type
      |
      = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    

    Version information

    Using Mutagen pulled from this repository at https://github.com/llogiq/mutagen/commit/94321e51a7c70c48fcc246e1d6ed13bdead23f39.

    $ cargo version
    cargo 1.43.0-nightly (bda50510d 2020-03-02)
    
    opened by joshlf 4
  • Mutation incorrectly infers integer literals

    Mutation incorrectly infers integer literals

    Minimally reproducible example:

    use mutagen::mutate;
    
    #[mutate]
    fn hi(mut ret: i64) -> i64 {
        let sh = if ret > 5 {
            0
        } else {
            8
        };
        ret += sh - 1;
        ret
    }
    
    fn main() {
        println!("Hello, world! {}", hi(500));
    }
    

    Without the macro it compiles fine. with the macro it returns the following:

    error[E0277]: cannot add-assign `i32` to `i64`
      --> src/main.rs:11:9
       |
    11 |     ret += sh - 1;
       |         ^^ no implementation for `i64 += i32`
       |
       = help: the trait `std::ops::AddAssign<i32>` is not implemented for `i64`
    

    mutagen dep: git+https://github.com/llogiq/mutagen.git#99d9e12bf4e28e3a734b36f216650cec09a54c6c

    opened by elichai 4
Owner
null
How-to: Sanitize your Rust code!

rust-san How-to: Sanitize your Rust code! Intro How to use the sanitizers? Examples AddressSanitizer Out of bounds access Use after free LeakSanitizer

Jorge Aparicio 359 Dec 22, 2022
Travis CI and AppVeyor template to test your Rust crate on 5 architectures and publish binary releases of it for Linux, macOS and Windows

trust Travis CI and AppVeyor template to test your Rust crate on 5 architectures and publish binary releases of it for Linux, macOS and Windows Featur

Jorge Aparicio 1.2k Dec 30, 2022
Advanced Fuzzing Library - Slot your Fuzzer together in Rust! Scales across cores and machines. For Windows, Android, MacOS, Linux, no_std, ...

LibAFL, the fuzzer library. Advanced Fuzzing Library - Slot your own fuzzers together and extend their features using Rust. LibAFL is written and main

Advanced Fuzzing League ++ 1.2k Dec 29, 2022
๐Ÿ‡ Fuzzing Rust code with American Fuzzy Lop

afl.rs Fuzzing Rust code with AFLplusplus What is it? Fuzz testing is a software testing technique used to find security and stability issues by provi

Rust Fuzzing Authority 1.3k Jan 5, 2023
TestDrive automatically scrapes input/output data from BOJ(Baekjoon Online Judge) and runs tests for your executable binary file!

?? TestDrive What does it do? TestDrive automatically scrapes input/output data from BOJ(Baekjoon Online Judge) and runs tests for your executable bin

Hyeonseok Jung 3 Mar 5, 2022
Code for comparing CDN speeds!

How to run speed test. the image to use The image you should probably use is: cf_219kb.png cf_219kb.png is an image that won't be compressed by Jetpac

Speed Test Demon 26 Nov 10, 2022
Playwright is a rust library to automate Chromium, Firefox and WebKit built on top of Node.js library.

?? Playwright for Rust Playwright is a rust library to automate Chromium, Firefox and WebKit built on top of Node.js library. Installation [dependenci

octaltree 132 Jan 6, 2023
A fast Rust-based safe and thead-friendly grammar-based fuzz generator

Intro fzero is a grammar-based fuzzer that generates a Rust application inspired by the paper "Building Fast Fuzzers" by Rahul Gopinath and Andreas Ze

null 203 Nov 9, 2022
A series of test cases to evaluate async Rust on the nrf52840 in terms of power usage and ergonomics.

A series of test cases to evaluate async Rust on the nrf52840 in terms of power usage and ergonomics. This is an experiment that uses unstable features only available on nightly rust.

Tweede golf 1 Oct 15, 2021
A tiny, super simple and portable benchmarking library.

benchmark-simple A tiny benchmarking library for Rust. Trivial to use Works pretty much everywhere, including WebAssembly (WASI, but also in-browser)

Frank Denis 3 Dec 26, 2022
Very minimalist tmux status bar that displays used memory and CPU usage.

woot-bar Ultra minimalist status bar that displays used memory and CPU usage woot-bar is made for tmux but it is compatible with anything that eats st

Nicolas Gryman 3 Dec 27, 2022
Notes and whatnot!

Noted CLI & TUI application to take and track notes. Generate Coverage (with cargo-llvm-cov): LCOV: cargo llvm-cov --all-features --workspace --lcov -

Tony B 2 May 3, 2022
Otto: a unified approach to CRDTs and OT

Otto: a unified approach to CRDTs and OT This repo contains tests for otto. otto enables any boring Rust data structure (without Rc, RefCell etc.) to

Tably 15 Dec 23, 2022
A simple CLI for creating and managing Solana Lookup Tables.

LUT - A simple CLI for creating and managing Solana Lookup Tables Commands lut create Creates a new LUT using the default keypair in the Solana config

Metaplex Foundation 5 Oct 29, 2022
Collect crash reports, triage, and estimate severity.

CASR: Crash Analysis and Severity Report CASR โ€“ collect crash reports, triage, and estimate severity. It is based on ideas from exploitable and apport

Ivannikov Institute for System Programming of the Russian Academy of Sciences 91 Feb 14, 2023
Easy-to-use grammar-based black-box fuzzer. Has found dozens of bugs in important targets like Clang, Deno, and rustc.

tree-crasher tree-crasher is an easy-to-use grammar-based black-box fuzzer. It parses a number of input files using tree-sitter grammars, and produces

Langston Barrett 5 Mar 28, 2023
Componentize.js on node and wasmtime

This is a test implementation of bytecodealliance/componentize-js Pre-requisites Node.js and npm are required to build and run this project. Cargo mus

Natalia Venditto 6 May 14, 2023
Handle some lichess.org/tournament load with Rust, while learning Rust

lila-http Take some of the HTTP load away from lila. WIP! Arena tournaments Clients connected to a tournament page request new data about the tourname

Lichess 22 Jan 2, 2023
HTTP mocking library for Rust.

httpmock HTTP mocking library for Rust. Documentation ยท Crate ยท Report Bug ยท Request Feature ยท Changelog Features Simple, expressive, fluent API. Many

Alexander Liesenfeld 320 Dec 21, 2022