Fixture-based test framework for Rust

Last update: Dec 24, 2022

Overview

Fixture-based test framework for Rust

Introduction

rstest uses procedural macros to help you on writing fixtures and table-based tests. To use it, add the following lines to your Cargo.toml file:

[dev-dependencies]
rstest = "0.13.0"

Fixture

The core idea is that you can inject your test dependencies by passing them as test arguments. In the following example, a fixture is defined and then used in two tests, simply providing it as an argument:

use rstest::*;

#[fixture]
pub fn fixture() -> u32 { 42 }

#[rstest]
fn should_success(fixture: u32) {
    assert_eq!(fixture, 42);
}

#[rstest]
fn should_fail(fixture: u32) {
    assert_ne!(fixture, 42);
}

Parametrize

You can also inject values in some other ways. For instance, you can create a set of tests by simply providing the injected values for each case: rstest will generate an independent test for each case.

use rstest::rstest;

#[rstest]
#[case(0, 0)]
#[case(1, 1)]
#[case(2, 1)]
#[case(3, 2)]
#[case(4, 3)]
fn fibonacci_test(#[case] input: u32, #[case] expected: u32) {
    assert_eq!(expected, fibonacci(input))
}

Running cargo test in this case executes five tests:

running 5 tests
test fibonacci_test::case_1 ... ok
test fibonacci_test::case_2 ... ok
test fibonacci_test::case_3 ... ok
test fibonacci_test::case_4 ... ok
test fibonacci_test::case_5 ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

If you need to just providing a bunch of values for which you need to run your test, you can use #[values(list, of, values)] argument attribute:

use rstest::rstest;

#[rstest]
fn should_be_invalid(
    #[values(None, Some(""), Some("    "))]
    value: Option<&str>
) {
    assert!(!valid(value))
}

Or create a matrix test by using list of values for some variables that will generate the cartesian product of all the values.

Use Parametrize definition in more tests

If you need to use a test list for more than one test you can use rstest_reuse crate. With this helper crate you can define a template and use it everywhere .

use rstest::rstest;
use rstest_reuse::{self, *};

#[template]
#[rstest]
#[case(2, 2)]
#[case(4/2, 2)]
fn two_simple_cases(#[case] a: u32, #[case] b: u32) {}

#[apply(two_simple_cases)]
fn it_works(#[case] a: u32, #[case] b: u32) {
    assert!(a == b);
}

See rstest_reuse for more dettails.

Magic Conversion

If you need a value where its type implement FromStr() trait you can use a literal string to build it:

# use rstest::rstest;
# use std::net::SocketAddr;
#[rstest]
#[case("1.2.3.4:8080", 8080)]
#[case("127.0.0.1:9000", 9000)]
fn check_port(#[case] addr: SocketAddr, #[case] expected: u16) {
    assert_eq!(expected, addr.port());
}

You can use this feature also in value list and in fixture default value.

Async

rstest provides out of the box async support. Just mark your test function as async and it'll use #[async-std::test] to annotate it. This feature can be really useful to build async parametric tests using a tidy syntax:

use rstest::*;

#[rstest]
#[case(5, 2, 3)]
#[should_panic]
#[case(42, 40, 1)]
async fn my_async_test(#[case] expected: u32, #[case] a: u32, #[case] b: u32) {
    assert_eq!(expected, async_sum(a, b).await);
}

Currently only async-std is supported out of the box. But if you need to use another runtime that provide it's own test attribute (i.e. tokio::test or actix_rt::test) you can use it in your async test like described in Inject Test Attribute.

To use this feature, you need to enable attributes in the async-std features list in your Cargo.toml:

async-std = { version = "1.5", features = ["attributes"] }

If your test input is an async value (fixture or test parameter) you can use #[future] attribute to remove impl Future<Output = T> boilerplate and just use T:

use rstest::*;
#[fixture]
async fn base() -> u32 { 42 }

#[rstest]
#[case(21, async { 2 })]
#[case(6, async { 7 })]
async fn my_async_test(#[future] base: u32, #[case] expected: u32, #[future] #[case] div: u32) {
    assert_eq!(expected, base.await / div.await);
}

Inject Test Attribute

If you would like to use another test attribute for your test you can simply indicate it in your test function's attributes. For instance if you want to test some async function with use actix_rt::test attribute you can just write:

use rstest::*;
use actix_rt;
use std::future::Future;

#[rstest]
#[case(2, async { 4 })]
#[case(21, async { 42 })]
#[actix_rt::test]
async fn my_async_test(#[case] a: u32, result: #[case] #[future] u32) {
    assert_eq!(2 * a, result.await);
}

Just the attributes that ends with test (last path segment) can be injected.

Use `#[once]` Fixture

If you need to a fixture that should be inizialized just once for all tests you can use #[once] attribute. rstest call your fixture function just once and return a reference to your function result to all your tests:

#[fixture]
#[once]
fn once_fixture() -> i32 { 42 }

#[rstest]
fn single(once_fixture: &i32) {
    // All tests that use once_fixture will share the same reference to once_fixture() 
    // function result.
    assert_eq!(&42, once_fixture)
}

Complete Example

All these features can be used together with a mixture of fixture variables, fixed cases and bunch of values. For instance, you might need two test cases which test for panics, one for a logged in user and one for a guest user.

use rstest::*;

#[fixture]
fn repository() -> InMemoryRepository {
    let mut r = InMemoryRepository::default();
    // fill repository with some data
    r
}

#[fixture]
fn alice() -> User {
    User::logged("Alice", "2001-10-04", "London", "UK")
}

#[rstest]
#[case::authed_user(alice())] // We can use `fixture` also as standard function
#[case::guest(User::Guest)]   // We can give a name to every case : `guest` in this case
                              // and `authed_user`
#[should_panic(expected = "Invalid query error")] // We whould test a panic
fn should_be_invalid_query_error(
    repository: impl Repository, 
    #[case] user: User, 
    #[values("     ", "^%$#@!", "....")]
    query: &str
) {
    repository.find_items(&user, query).unwrap();
}

This example will generate exactly 6 tests grouped by 2 different cases:

running 6 tests
test should_be_invalid_query_error::case_1_authed_user::query_1 ... ok
test should_be_invalid_query_error::case_2_guest::query_2 ... ok
test should_be_invalid_query_error::case_2_guest::query_3 ... ok
test should_be_invalid_query_error::case_1_authed_user::query_2 ... ok
test should_be_invalid_query_error::case_1_authed_user::query_3 ... ok
test should_be_invalid_query_error::case_2_guest::query_1 ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Is that all? Not quite yet!

A fixture can be injected by another fixture and they can be called using just some of its arguments.

#[fixture]
fn user(#[default("Alice")] name: &str, #[default(22)] age: u8) -> User {
    User::new(name, age)
}

#[rstest]
fn is_alice(user: User) {
    assert_eq!(user.name(), "Alice")
}

#[rstest]
fn is_22(user: User) {
    assert_eq!(user.age(), 22)
}

#[rstest]
fn is_bob(#[with("Bob")] user: User) {
    assert_eq!(user.name(), "Bob")
}

#[rstest]
fn is_42(#[with("", 42)] user: User) {
    assert_eq!(user.age(), 42)
}

As you noted you can provide default values without the need of a fixture to define it.

Finally if you need tracing the input values you can just add the trace attribute to your test to enable the dump of all input variables.

#[rstest]
#[case(42, "FortyTwo", ("minus twelve", -12))]
#[case(24, "TwentyFour", ("minus twentyfour", -24))]
#[trace] //This attribute enable traceing
fn should_fail(#[case] number: u32, #[case] name: &str, #[case] tuple: (&str, i32)) {
    assert!(false); // <- stdout come out just for failed tests
}

running 2 tests
test should_fail::case_1 ... FAILED
test should_fail::case_2 ... FAILED

failures:

---- should_fail::case_1 stdout ----
------------ TEST ARGUMENTS ------------
number = 42
name = "FortyTwo"
tuple = ("minus twelve", -12)
-------------- TEST START --------------
thread 'should_fail::case_1' panicked at 'assertion failed: false', src/main.rs:64:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

---- should_fail::case_2 stdout ----
------------ TEST ARGUMENTS ------------
number = 24
name = "TwentyFour"
tuple = ("minus twentyfour", -24)
-------------- TEST START --------------
thread 'should_fail::case_2' panicked at 'assertion failed: false', src/main.rs:64:5


failures:
    should_fail::case_1
    should_fail::case_2

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

In case one or more variables don't implement the Debug trait, an error is raised, but it's also possible to exclude a variable using the #[notrace] argument attribute.

You can learn more on Docs and find more examples in tests/resources directory.

Changelog

See CHANGELOG.md

License

Licensed under either of

Apache License, Version 2.0, (LICENSE-APACHE or license-apache-link)
MIT license LICENSE-MIT or license-MIT-link at your option.

Comments

I want multiple tests to share a single set of cases
Problem

I want multiple tests to share a single set of cases

Suggestion: Use a const vector/array as cases

let cases = [ (0, "zero"), (1, "one"), ]; #[rstest_from(cases)] fn mytest (case: (i32, &str)) { // ... }
opened by KSXGitHub 17
Add async fixtures
Would fix #86

TODO:

[x] Actual implementation

[ ] Good error when async fixtures are used in non-async-tests.

[x] Add test with impl Trait output of a fixture (i.e., nested impl Trait); not sure whether that can work.

[ ] Test with partials
opened by rubdos 14
Mutable Fixture
I would like to confirm the mechanics of fixtures and how I can create them so that changes made to the value carries on to other rests.

I am using a reqwest::Client as the output of the fixture and reusing it in all my tests.

Each test will check if the client is authenticated and will perform the handshake if not authenticated

The expected outcome is that I should see only a single attempt to log in and all subsequent tests merely reuse the bearer auth key produced from the test that happened to make the first request

It looks like the fixture function is evaluated for each test meaning that each test gets a new client instead.
opened by candronikos 11
Struggling with equally named templates
I am trying to use rstest-reuse and it works great. Until I added a second template, with the same name, in a different module of my crate.

It looks like it is creating a macro which gets exported, and thus "pollutes" the root namespace of the crate.

So I tried to define the template in a central location, and re-use it from different sub-modules. Also without success as the apply macro panics, as it expects an Ident as argument.

Importing the template into the module's namespace using use also doesn't work:

error: macro-expanded `macro_export` macros from the current crate cannot be referred to by absolute paths --> src/tests/mqtt/mod.rs:17:5 | 17 | use crate::mqtt_integration_versions; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = note: `#[deny(macro_expanded_macro_exports_accessed_by_absolute_paths)]` on by default = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release! = note: for more information, see issue #52234 <https://github.com/rust-lang/rust/issues/52234>

I am not sure what the right approach here is.
opened by ctron 10
Use just `[rstest]` to generate test.

My macro knowledge is pretty poor so please excuse my ignorance, but I think it'd be conceptually and syntactically to use have [rstest] which may also be optionally parametrized. What do you think?

opened by svenstaro 10
Wrong panic message when `#[timeout(...)]` is not expired
In rstests with the #[timeout(...)] macro attribute, the panic messages in tests in which the timeout is not reached are replaced by Timeout ... expired.

The original panics are still readable because the timeout thread panicked, but it's very confusing to see a Timeout ... expired although the timeout didn't expire. Also, it's not possible to match against the original panic string for example with #[should_panic = "my assertion"]

Here's a short example:

The test code showing the wrong behavior

#[rstest] #[should_panic = "my error message"] #[timeout(Duration::from_secs(60))] fn test_me() { panic!("my error message"); }

The test output of cargo test for the test_me test function:

---- test_me stdout ---- thread '<unnamed>' panicked at 'my error message', tests/test_me.rs:8:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'test_me' panicked at 'Timeout 60s expired', /home/lenny/.cargo/registry/src/github.com-1ecc6299db9ec823/rstest-0.16.0/src/timeout.rs:16:29 note: panic did not contain expected string panic message: `"Timeout 60s expired"`, expected substring: `"my error message"` failures: test_me

The correct test output shouldn't show the panic message of the <unnamed> thread and should show thread 'test_me' panicked at 'my error message' instead of thread 'test_me' panicked at 'Timeout 60s expired'.

If interested, I can submit a pr fixing this in the execute_with_timeout_sync function in rstest/rstest/src/timeout.rs.
opened by Joining7943 9
An option to disable "TEST START" println?

https://github.com/la10736/rstest/blob/4e1069b7468d0d49a5d78c47f1336fd88d41e472/rstest_macros/src/render/mod.rs#L268

When running in IDEs (like CLion) that automatically capture output in the test runner, there's tons of --- TEST START --- printlns. It's not mission-critical, but very annoying 😢

Looking at the code, wonder why are they needed at all? I found one place where this is checked, in rstest's tests:

https://github.com/la10736/rstest/blob/e12494bf082ef2c82353e4e88ba74c2c30d1addd/rstest/tests/rstest/mod.rs#L259

But then again, I guess there's other ways to check this, without having to dump this to stdout on every test? (Or as another option, if it's really required, could make it a non-default crate feature; or yet another proc-macro attr, although that's an overkill)

If these printlns could be removed, conditionally or unconditionally, would be much appreciated.

opened by aldanor 9
Is it possible to use fixture values in case or value lists?
Hey, I realize I have a somewhat weird case but bear with me.

#[fixture] pub fn rng() -> StdRng { let seed: [u8; 32] = rand::thread_rng().gen(); let rng = StdRng::from_seed(seed); rng } #[rstest] #[case(make_password(&mut rng))] #[tokio::test] async fn my_test( mut rng: StdRng, #[case] password: String, ) -> Result<()> {}

So I want to use the rng() fixture in my_test() but also use the resulting rng value in make_password which provides a value to my case.

This currently doesn't work and I suppose it's not currently intended to work. Is there any chance this could be made to work?
opened by svenstaro 9
Add timeouts
The option to timing out the test should be a parameter. E.g.,:

#[timeout(Duration::from_millis(1000))] fn test_apple() { }

When async is detected, a timeout future can be wrapped around the test itself, terminating if it times out.
opened by tbraun96 8
parameterizing fixtures
With rstest, I can parameterize a test case using #[rstest(foo => [...])] or #[rstest(foo, case(...), ...)] syntax. But I don't see any way to parameterize a fixture. What I would like to do would be to define a fixture that can return multiple values, and use that to parameterize several test cases, like this:

#[fixture(foo => [1, 2, 3])] fn inputs(foo: i32) -> i32 { foo } #[rstest] fn test_a(inputs: i32) { ... } #[rstest] fn test_b(inputs: i32) { ... }

I would then expect to see six separate test cases get run, like

running 6 tests test test_a_inputs_1 ... ok test test_a_inputs_2 ... ok test test_a_inputs_3 ... ok test test_b_inputs_1 ... ok test test_b_inputs_2 ... ok test test_b_inputs_3 ... ok

Would such a thing be possible?
opened by asomers 8
Backtrace loses code location in macro
My apologies if this is already in another issue and I've just missed it. I did look, but I might lack the vocabulary as a newbie.

I have used rstest (which I am very excited about) to replace a loop over a list (https://github.com/msarahan/version-compare/blob/polymorph_parts/src/test/test_version_set.rs#L14 and https://github.com/msarahan/version-compare/blob/polymorph_parts/src/version_compare.rs#L94) with parametrized tests (https://github.com/msarahan/version-compare/blob/rstest/src/test/macros.rs#L44 and https://github.com/msarahan/version-compare/blob/rstest/src/version_compare.rs#L89)

In doing so, my backtraces seem to lose any relevant line number references. They are replaced with lines in the macro instead:

8: std::panicking::begin_panic_fmt at src\libstd/panicking.rs:339 9: version_compare::version_compare::tests::compare at src/lib.rs:1 10: version_compare::version_compare::tests::compare::case_11 at src\test/macros.rs:94 11: version_compare::version_compare::tests::compare::case_11::{{closure}} at src\test/macros.rs:46 12: core::ops::function::FnOnce::call_once at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54\src\libcore\ops/function.rs:235

Is there any way to preserve the reference to the original code location in my tests?
opened by msarahan 8
Update toml_edit requirement from 0.15.0 to 0.16.2
Updates the requirements on toml_edit to permit the latest version.

Commits

38593de chore: Release

aa1f926 docs: Update changelog

b0ad923 Merge pull request #409 from epage/test

87ee5f2 test(parser): Check more overflow cases

ce27cd7 Merge pull request #408 from epage/key

b084184 fix(parser): Avoid stackoverflow with long dotted keys

9174af0 Merge pull request #406 from epage/nom8

d3ebe17 chore: Upgrade to latest nom8

e51f83f chore: Release

9e3491d docs: Update changelog

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies rust
opened by dependabot[bot] 0
Prepend `super::` to function call?
When seeing if I could make reasonable use of rstest for time, I quickly ran into an issue. If a case parameter is the same as the function name, compilation will fail. For example,

#[rstest] #[case(Sunday, Monday)] fn next(#[case] day: Weekday, #[case] next: Weekday) { assert_eq!(day.next(), next); }

This fails because next is both a case parameter and the function name, and it is ambiguous in the expansion:

#[cfg(test)] fn next(day: Weekday, next: Weekday) { assert_eq!(day.next(), next); } #[cfg(test)] mod next { use super::*; #[test] fn case_1() { let day = Sunday; let next = Monday; next(day, next) } }

However, if next(day, next) is instead super::next(day, next), compilation would succeed. Given that the module is generated by rstest, we know that this path is always correct. Would it be possible to add this prefix to support the edge case?

Thanks for maintaining this crate!
opened by jhpratt 2

Use fixtures in in cases

I have an idea for using fixture in the case macro:

struct TestDependency;  // complex object 

#[fixture]
fn dependency_built_in_one_way() -> TestDependency {
    ...  // long setup
}

#[fixture]
fn dependency_built_in_another_way() -> TestDependency {
    ... // long setup
}

#[rstest]
fn testing_specific_feature(dependency_built_in_one_way: TestDependency) {
    ... // testing something only on the first object
}

// A test that should test a feature on both complex configurations of the dependency
#[rstest]
#[case(dependency_built_in_one_way)]
#[case(dependency_built_in_another_way)]
fn my_test(#[case] dependency: TestDependency) {
    ...
}

That would be useful to me. I could do something similar with cases on the fixture, but then I wouldn't be able to only use one of the built variants in certain tests.

opened by fhennig 2

Allow awaiting futures automatically in parameters
Async fixture are kind of hard to use currently as reported in #157 . This PR adds a syntax to make tests with async fixtures easier to write.

Before:

#[rstest] async fn it_can_return_its_ip(#[future] client: Client) { let client = client.await; client.doThis(); client.doThat(); }

After:

#[rstest] async fn it_can_return_its_ip(#[await_future] client: Client) { client.doThis(); client.doThat(); }

This PR is ready code-wise, but it still lacks some tests and documentation. I wanted to get feedback for this approach before going further, please tell me what you think :)
opened by blastrock 13
Omit case_n for named cases of parameterized tests
Consider the following parameterized test for a trivial function square:

fn square(i: i32) -> i32 { i * i } #[rstest] #[case::negative_value(-5, 25)] #[case::zero(0, 0)] #[case::positive_value(3, 9)] #[case::large_value(10000, 100000000)] fn squares_the_value(#[case] value: i32, #[case] expected: i32) { assert_eq!(square(value), expected); }

I've given each test case an expressive name to make the generated test names more helpful. The output is as follows:

test repro::squares_the_value::case_1_negative_value ... ok test repro::squares_the_value::case_2_zero ... ok test repro::squares_the_value::case_3_positive_value ... ok test repro::squares_the_value::case_4_large_value ... ok

The generated test name contains both a numbered case_n and the manual case names. To me, the case_n part doesn't add any benefit. For more complex tests with longer names, I find that it makes the test names harder to read.

Would it make sense to omit case_n from the generated names of parameterized tests if the corresponding case has an explicit name?
opened by DanielSWolf 5
Ability to generate test cases from file path

Hey, thanks for creating this! I think a feature to generate tests for each file matching a certain glob would be great.

An existing crate that does this is test-generator, but building something similar here would add a lot of value. For example, a user would be able to create a matrix test where the first parameter is every path matching a certain glob, and the second parameter is configurable via the #[values()] attribute.

opened by ajeetdsouza 1

Releases(0.16.0)

0.16.0(Nov 27, 2022)

Use values expression to define test names.
Source code(tar.gz)
Source code(zip)
0.15.0(Jun 27, 2022)

Fix bug on timeout tests that doesn't accept values that without copy trait.
Source code(tar.gz)
Source code(zip)
0.14.0(Jun 19, 2022)

Feature gate async-timeout
Source code(tar.gz)
Source code(zip)
0.13.0(May 15, 2022)

Timeout

Introduce #[timeout] attribute
Source code(tar.gz)
Source code(zip)
0.12.0(Dec 12, 2021)

Introduce #[once] fixture attribute
Source code(tar.gz)
Source code(zip)
0.11.0(Aug 1, 2021)

Bug fixing
Source code(tar.gz)
Source code(zip)
0.10.0(May 16, 2021)

Introduce fixture renaming and fix generic types partial impl cleaner
Source code(tar.gz)
Source code(zip)
0.9.0(May 2, 2021)

Introduce #[future] atribute
Source code(tar.gz)
Source code(zip)
0.8.0(Apr 25, 2021)

Introduce Magic Conversion for literal string and change the syntax for default fixture arguments attribute from #[default = arbitrary rust expression] (unstable syntax) to #[default(arbitrary rust expression)].
Source code(tar.gz)
Source code(zip)
0.7.0(Mar 21, 2021)

Introduce the new syntax that leverage on function and arguments attributes for all features.
Source code(tar.gz)
Source code(zip)
0.6.4(Jun 21, 2020)

Document rstest_reuse to apply a tests list to multiple test function
Source code(tar.gz)
Source code(zip)
0.6.3(Apr 19, 2020)

Introduce fixture default args
Source code(tar.gz)
Source code(zip)
0.6.2(Apr 19, 2020)

Inject test attribute
Source code(tar.gz)
Source code(zip)
0.6.1(Apr 5, 2020)

Introducing async support
Source code(tar.gz)
Source code(zip)
0.6.0(Mar 5, 2020)

Remove deprecated rstest_parametrize and rstest_matrix. Remove _ before call fixture and accept per case attributes.
Source code(tar.gz)
Source code(zip)
0.5.3(Jan 23, 2020)

Fixed a false unused mut warning regression
Source code(tar.gz)
Source code(zip)
0.5.2(Dec 29, 2019)

Fixed an annoying bug that raise unused attribute for should_panic rstest functions.
Source code(tar.gz)
Source code(zip)

Owner

Michele d'Amico

GitHub

Test for rust-based plugin system for swc

rust-dylib-test Steps Run cargo build in plugin_a Ensure that plugin_a dynamically links to runtime/common by otool -L plugin_a/target/debug/libplugin

1 Apr 6, 2022

Travis CI and AppVeyor template to test your Rust crate on 5 architectures and publish binary releases of it for Linux, macOS and Windows

trust Travis CI and AppVeyor template to test your Rust crate on 5 architectures and publish binary releases of it for Linux, macOS and Windows Featur