As discussed in #142, this is a proposal for a modular architecture, which is the base for mutagen v0.2. I reviewed all parts of the code and tried to move code that works fine to the new system.

This is a minimal implementation to demonstrate the effectiveness of the architecture. The steps towards releasing mutagen v0.2 are:

• [x] implement more mutators and document their functionality
• [x] generate meaningful reports

Are there any other major tasks for this PR?

Hi, I've been working on re-structuring your project mutagen to be more production ready. https://github.com/power-fungus/mutagen-preview

I spent the last weeks reading into the current project and re-writing parts of mutagen using the same interface, but with a different architecture. It's not completed yet but should give a rough outline of what the finished project will look like: the backbone and glue code already exists and will probably not change too widely.

@llogiq I would like to know if you are interested in my approach and if you would consider merging my proposal in your project. If so, I will finish this preview into production quality in the next months. Also I am open to any discussion about interface and design decisions.

I would like to have this rewrite published (once finished) under the "brand" of mutagen:

• The interface is the same
  • mutators are introduced using the attribute #[mutate]
  • mutations are selected at run-time using the environment variable MUTATION_ID (renamed for clarity)
  • The target code is compiled only once
• the rewrite is true to the original goals of mutagen
• the used technologies is the same to the code in the proc-macro-attribute branch
  • new procedural macro attribute
  • uses syn and quote

For details about the architecture, see: https://github.com/power-fungus/mutagen-preview/tree/master/docs Main Benefits are:

• Debugging of the code transformation will be easier due to modular transformation
• The behavior of the mutations can be tested a standard test suite by faking the global MUTATION_ID variable.
• it will be possible to add customization via arguments to the #[mutate] attribute
• community collaboration will be easier since mutators and transformers are small bits of code.
• mutators can be documented independent from each other.

@llogiq I would like to read from you soon! :)

The plugin already has the analysis. There are but two wrinkles to take care of:

• [ ] It may make a difference if the args are taken by ref or by value, the analysis pass doesn't take that into account yet
• [ ] we should take care not to issue the same switch twice. This could probably be done easily by filtering the argument pairs so only pairs where the first name compares lower than the last one are switched.

If one has the following ~/.cargo/config file

[build]
target-dir = "/home/<user>/.cache/cargo"

I would expect mutagen to put artifacts in the configured directory however target/ inside the crate being tested is used.

Fixes #157. The mutation of the snippet given in the issue used to compile but now fails to compile. This PR reverts to the previously use strategy, with slight alterations.

Background

We have 2 functions:

// does not work in old approach
#[mutate]
fn test1(mut ret: i64) -> i64 {
    1 + 2
}
// does not typecheck in new approach
#[mutate]
fn test1(mut ret: i64) -> i64 {
    let x = 1;
    x + 1
}

The first snippet did not typecheck correctly in the previous version. This led me to implementing a different approach based on detecting if the left side of an arithmetic operation is a integer literal or derived from some integer literal. This made the first one compile but was not smart enough to detect the second case. I thought I could extend it further, but it turned out that there is not enough information to solve this problem in all cases.

Previously, the trick was to rewrite a + b into if false {a+b} else { <mutator-code>}. I underestimated the potential of the if false trick, which is an error on my part.

This PR switches to a modified version of the if false trick: a+b is transformed to {let _tmp = a; if false {_tmp+b} else {<mutator-code based on _tmp and b>}}. This makes both versions compile and also passed several other test cases.

This PR is a work in progress. Feedback is welcome.

At the moment, the runner finds only one test executable.

The method compile_tests compiles the test and returns a single PathBuf, but it should return a vector of them, and execute each of the found test binaries. Note that we are early-returning when we find the first path.

With this change, mutagen will be able to run on projects which has both unit, integration and doc tests.

This is a preliminary version of a pull request to add some example code.

It is not finished yet but I don't really see how to proceed. But I want to post it here to get some feedback. Some of these questions should be answered in the readme or a how to use guide.

Open questions

• The mutate attribute: should I place it above all functions that I want to mutate during test?
• Can/should I make that conditional on cfg(test)? Not sure if I want mutagen code in the final binary
• I made the first test always fail to show the println!s It seems to show that mutagen is doing something. Is this the case?
• I used MU.next() 100 times, is there a max or does it depend on the number or types of mutations that are applied? It does not seem to change much after 10 times..
• Should there be a #[mutate_test] attribute that you could use instead of #[test] to create tests with mutagen?

TODO

• [x] The example code is copied from a project without license. I should write a different example to clear up the license issue. I was lazy
• [x] The first test tries to show that mutagen is working. I am not sure though if this is the intended purpose or direction. But the intention is to make a start with a manual mutagen runner (and maybe move that into the lib/plugin?

Attempt to fix issue: https://github.com/llogiq/mutagen/issues/28

I want to get a little bit of feedback before keep going with that.

As a first step, all the mutations are being instrumented and are calling ::mutagen::report_coverage(). This method, if an environment variables is set (MUTAGEN_COVERAGE), the first time it will be called, it will write to a specific file that some mutation has been hit.

On the other side, cargo-mutagen will check all the tests that the binary contains, and will execute all the tests individually, with the mentioned env var. After executing each of the tests, we will check if the file has been created. If it exists, we know that the executed test contains code that it's mutated.

Finally, when running mutations, we will run only the tests that contains mutations (unfortunatelly, we will need to run the test one-by-one, which means that we will spawn a process per test, instead of a process per testsuite). Then, coverage flag it's only useful when the ratio of tests which executes mutated code is low.

To test with coverage, after updating to the new version of the plugin on the target project, and after reinstalling the runner:

cargo mutagen -- --coverage

Pending tasks would be (maybe new issues can be opened):

• Improve the "ping" system with something better than a file
• ~~Maybe we can include the mutation number to report_coverage method, in order to know which are the mutations that took place on the given test. If we do, we can filter more accuretly which tests should be executed on each mutation~~
• Maybe we can think about use clap to parse arguments, add subcommands, help, ...
• ~~Document the cargo mutagen command~~

As suggested on https://github.com/llogiq/mutagen/issues/5, it would be nice to mutate numeric constants adding or subtracting one unit to numeric constants.

At the moment, this is very conservative on the limits to add or substract on default integer constants. I've did it like this to avoid compilation issue, as we do not have information regarding the type of the expression. So, in the default case, it subtracts one only if it's > 0 and it only adds one if the value is < 256.

If the constant has a type annotation (for example, 0i32), It adjust the minimum and the maximum depending on the type hint. Note that it does not check 128 bits numbers, as they are only available on nightly.

It chains the mutations in order to record only the needed mutations. So, if the code finds a "0" on an unsigned type, it will only record the "add one" mutation, but not the "sub one" (as it would overflow and panic).

After running it on the example project, it emits mutations like:

    let mut count = {
        if mutagen::now(2usize) {
            0 + 1
        } else {
            0
        }
    };

or

              ord < {
                    if mutagen::now(12usize) {
                        65 + 1
                    } else {
                        {
                            if mutagen::now(11usize) {
                                65 - 1
                            } else {
                                65
                            }
                        }
                    }
                }

And finally: I didn't add tests. I thought on adding another test on the example project that covers some of the edge cases and I've also thought on unit test the int_constant_can_subtract_one or int_constant_can_add_one, but I'm not sure which is the expected way of testing this library, so, if you have some idea, I can add the kind of tests you think will be more useful.

If you're open to new mutations, I created a mutator for string literals. It replaces empty strings with a single character string and for non-empty strings it does a few things: append and prepend a single character as well as replacing the string with an empty string. I'd welcome any feedback. Thanks!

We can mutate while let Some($ident) = $expr { $block } loops to extend $expr to if ::mutagen::now($n) { None } else { $expr }. We already have similar extensions for ifs, so this should be simple to implement.

A lot of the tests I have are using libfuzzer/cargo-fuzz to create test curposes, would be nice if I could run those under mutagen, obviously I could write a test function that runs them and then run mutagen, but I have a lot of fuzzers and it will require adding a lot of fuzzing logic code into my crate's tests suite, so it would be nice if I could plug cargo-mutagen into the fuzzer so it will run the fuzzing corpuses under mutations

ansi_term is Unmaintained

| Details | | | ------------------- | ---------------------------------------------- | | Status | unmaintained | | Package | ansi_term | | Version | 0.12.1 | | URL | https://github.com/ogham/rust-ansi-term/issues/72 | | Date | 2021-08-18 |

The maintainer has adviced this crate is deprecated and will not receive any maintenance.

The crate does not seem to have much dependencies and may or may not be ok to use as-is.

Last release seems to have been three years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• anstyle
• nu-ansi-term
• yansi

See advisory page for additional details.

Hey!

I wonder what the options are to use mutagen in CI pipelines. I could not find any support about it such as GitHub actions or so.

The simplest I could do is to run cargo mutagen in a CI pipeline task and based on the result I could stop or continue the pipeline run. What would be the easiest way to achieve this, what do you think?

Of course, I am also up to create/submit a PR and change the code regarding, if there is not yet a solution for it.

Thank you!

There is a format of output for mutation tools called "mutation elements" [1]. Format supported by tools that used for visualization [2] and [3]. It would be nice to support this format in mutagen.

1. https://github.com/stryker-mutator/mutation-testing-elements
2. https://www.npmjs.com/package/mutation-testing-elements
3. https://github.com/ligurio/py-mutation-testing-elements

Given the following lib.rs:

fn foo() {}

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
	let x = [0u8];
	assert_eq!((&x[..]).as_ref(), []);
    }
}

Running cargo check --tests succeeds. However, if we add an attribute, #[cfg_attr(test, mutagen::mutate)], to fn foo() {}, then cargo check --tests fails:

error[E0282]: type annotations needed
 --> src/lib.rs:9:2
  |
9 |     assert_eq!((&x[..]).as_ref(), []);
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type
  |
  = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

Version information

Using Mutagen pulled from this repository at https://github.com/llogiq/mutagen/commit/94321e51a7c70c48fcc246e1d6ed13bdead23f39.

$ cargo version
cargo 1.43.0-nightly (bda50510d 2020-03-02)

Minimally reproducible example:

use mutagen::mutate;

#[mutate]
fn hi(mut ret: i64) -> i64 {
    let sh = if ret > 5 {
        0
    } else {
        8
    };
    ret += sh - 1;
    ret
}

fn main() {
    println!("Hello, world! {}", hi(500));
}

Without the macro it compiles fine. with the macro it returns the following:

error[E0277]: cannot add-assign `i32` to `i64`
  --> src/main.rs:11:9
   |
11 |     ret += sh - 1;
   |         ^^ no implementation for `i64 += i32`
   |
   = help: the trait `std::ops::AddAssign<i32>` is not implemented for `i64`

mutagen dep: git+https://github.com/llogiq/mutagen.git#99d9e12bf4e28e3a734b36f216650cec09a54c6c