I've begun adding async support to Cucumber.

Async test running functionality

• [x] Running async tests
• [x] Basic async steps
• [x] Regex async steps
• [x] Regex typed async steps
• [x] Ability to configure whether scenarios should run in parallel

Async ergonomics

• [x] Add support for async tests
• [x] Add support for async regex tests
• [x] Ability to select executor to use
• [x] Re-implement nice printed output
• [x] Document usage

I was wondering if it's something that we can add to the framework. We already have a function to repeat failed tests output but we don't have a function to run failed tests again. It's common to have flaky tests that sometimes can be solved by running them again.

For example :

#[tokio::main]
async fn main() {
    AnimalWorld::cucumber()
        .replay_failed(2)
        .run_and_exit("tests/features/book/output/terminal_repeat_failed.feature")
        .await;
}

Where 2 is the maximum number of times the tests should be run again in case of test failures. Let's say we have the tests A, B, C and D :

1. During the first run only A passes
2. Then B, C and D are run again (1 replay left if we have new test failures)
3. Only B passes, we run again C and D (0 replay left)
4. D fails again, this one is certainly more than just unstable

Regarding the output we have two choices :

• Print all test executions : it's transparent but can be repetitive when the tests fail multiple times (like D in this example)
• Or just print the result once the last replay is done (which can be the maximum, here 2, or a previous run if all tests are passing)

I believe that adding j/xunit output support would go a long way to streamlining the tests, as those are the formats supported by IDE's and CI tools, so it would make cucumber-rust fit in like a glove with how devs work right now, rather than relying on reading the text output.

I will probably give it a whack myself, as it's of high interest to me, but certainly, welcome any pointers and suggestions!

I've a step that's used after both a given and then.

I couldn't find the and keyword and figured that I've to use given (etc.) respectively. In this case this leads to having to write the step definition twice (with given and then), which isn't ideal.

Right now if you pass test-thread argument to cargo test, cucumber will fail:

     Running `/home/humb1t/workspace/soramitsu/iroha/target/debug/deps/cucumber-09561164203f68fb --test-threads=1 --skip=cucumber`
error: Found argument '--test-threads' which wasn't expected, or isn't valid in this context

Command: cargo test --workspace --verbose -- --test-threads=1

It will be great to support this feature or at list ignore it instead of failure.

Hello,

First of all, thanks for the work on this awesome crate.

If I understand correctly, currently there is no way to disable printing the World in error messages. Whenever there is a failed test or panic, the whole World struct is printed. In some use cases this object can be quite large, making the errors difficult to navigate.

Would it be possible to add a command line flag that controls this behaviour?

This pull request is a draft, waiting for the official release of clap v3. However, if needed, It can already be reviewed as the api shouldn't drastically change.

The doc comments on top of structs deriving clap::Args or clap::Parser is still an issue with Clap v3. So I let the workaround in place.

Closes #155

I expected to access the values in example table easier, like scenario outline can be execute for each row of the example table and in those scenarios we can access the values easier, like:

builder.given_example("a number <num>", |mut world, step, vars| {
  let string_num = vars.value("num");
  world.num_param = string_num::parse::<i32>::parse().unwrap();
  world
})
...

Or do we need to collect all examples in the World, but how? The example test is not implement for that.

Resolves #58, #134, #140

Synopsis

In v0.10.0 CLI was preserved as is with error messages for unsupported options. But we want to rework it for more flexibility.

Solution

Redesign CLI and make it composable.

Checklist

• Created PR:
  • [x] In draft mode
  • [x] Name contains Draft: prefix
  • [x] Name contains issue reference
  • [x] Has assignee
• [x] Documentation is updated (if required)
• [x] Tests are updated (if required)
• [x] Changes conform code style
• [x] CHANGELOG entry is added (if required)
• [x] FCM (final commit message) is posted
  • [x] and approved
• [x] Review is completed and changes are approved
• Before merge:
  • [x] Milestone is set
  • [x] PR's name and description are correct and up-to-date
  • [x] Draft: prefix is removed
  • [x] All temporary labels are removed

Pre-history and motivation

We hardly use cucumber_rust in our production projects having ~800 scenarios and recently have started to suffer from its limitations: no concurrent runner, inability to customize its parts without changing upstream code.

This PR is our long-going effort to make the cucumber_rust extensible, customizable, while ergonomic and rational out-of-the-box for most cases. It represents an almost complete redesign and rewrite of the library with introducing some new features along and fixing many existing issues. It was battle-tested in our code base for quite a while, and showed significant improvements of test execution times. All the documentation, images and the Book are made up-to-date with the changes.

New architecture

The new design of the library splits it to the 3 main components in the following pipeline:

Parser -> Runner -> Writter

where:

• Parser is responsible for sourcing Scenarios for execution.
• Runner is responsible for executing the stream of Scenarios.
• Writer is responsible for outputting execution results.

All the 3 components may be customized or changed to a custom implementation, which solves any questions about implementing custom output, or custom execution order, or anything else.

Still the library provides reasonable defaults:

• parser::Basic - reads from .feature files and supports # language: comment.
• runner::Basic - executes all the scenarios concurrently by default (max 64 at the same time), and customizable to run fully (max 1 concurrent) or partially sequential (via @serial tag).
• writer::Basic - simply outputs execution results to STDOUT, while writer::Normalized and writer::Summarized are out-of-the-box wrappers introducing output order normalization and writing summary of execution (for anyone implementing custom Writter to no bother about these problems).

Output

We've also stripped from the default output the filenames and line numbers for successful steps, as from our experience they make the output unreasonable bloated and hard-to-read. We still keep them for failing steps, though.

Codegen

macros feature is now enabled by default, as provides more ergonomic and understandable way to write step matchers.

Code style

rustc lints of the project are somewhat hardened and clippy::pedantic lints are checked on CI now. We believe that tightening code style is beneficial for open source libraries and future contributions, despite it may mandate some strange or verbose situations in code.

Releasing and CI

Release process now is fully automated via GitHub actions. Just prepare the repository for a release, commit and push the appropriate version tag.

⚠️ ACTION REQUIRED: You should set the CRATESIO_TOKEN to the project's GitHub secrets to release via CI.

CI now fully checks and tests the project, including the Book.

Resolved issues

• #56: writer::Basic has really minimal output for success steps now.

• #62: All the GIFs are ap-to-date now/

• #63: We've tried our best to document everything pretty well, and enabled Clippy and rustc lints to keep an eye on that.

• #64: Possible by using regex now.

• #71: Fixed and not applicable anymore.

• #80: Parsing errors are now propagated to the Writer.

• #86: Just fixed.

• #88: Fixed and not applicable anymore.

• #102: Now possible with Cucumber::language().

• #103: Can be done by Cucumber::filter_run_and_exit().

• #119: writer::Basic pretty-prints only if terminal was detected.

• #123: Now possible with writer::Summarised::fail_on_skipped().

• #126: Now uses # language: comment to deduce language dynamically.

• (partially) #50 and #127: While not provided out-of-the-box, may be implemented via custom Writer.

Contribution and maintaining.

We understand that this PR brings a huge impact to the library and may be undesirable by its author. If the later is true, we're OK to go on with our own fork and maintain it. However, from the Rust ecosystem perspective, it would be better to not introduce any ecosystem splits and disperse contributors efforts. That's why we also ask @bbqsrc to add me (@tyranron) and @ilslv to project maintainers, so we can continue to maintain the project and contribute to it more easily and actively. Ideally, we see the project to migrate to some GitHub org (like cucumber-rs or smthng), where new maintainers may step-in to the project more easily in the future.

I couldn't find any references to them, in documentation, examples or even implementation code.

For the record, Cucumber Expressions are the usual way to write step definitions in most languages, it's slightly less powerful than Regex but much more readable.

For example consider:

I have {float} cucumbers in my belly

vs

I have ([+-]?([0-9]+([.][0-9]*)?|[.][0-9]+)) cucumbers in my belly

(Float regex matcher taken from Stackoverflow)

I believe we can support better logging by integrating support for brilliant tracing crate, as current solution described in #177 is quite unintuitive and requires additional setup.

We can implement custom tracing::Subscriber and pass Feature, Scenario and Step context inside tracing::Span. So we can retrieve this information and pass logging event as a new kind of cucumber::Events for logging them, when appropriate time comes.

Unresolved questions, that require additional investigations:

1. spawning a future, that will log after a Step has been finished may lead to strange output behaviour

#[given(...)]
fn delayed_log(w: &mut World) {
    spawn(
        async {
            sleep(Duration::from_secs(1).await;
            tracing::info!("Delayed");
        }
        .instrument(tracing::Span::current())
    )
}

Moreover, spawned Futures by default don't inherit Span from context they were spawned from. See https://github.com/tokio-rs/tracing/issues/394 for more info.

2. Possible integrations with existing tracing ecosystem.

Priority attribute tells the executor how early need to add scenario to the execution queue.

In some cases there is no way to tell the executor to run scenarios in specific order. For example: Scenarios that will be executing for a long time must be added to execution queue before others.

Design proposal:

@priority [integer]
Scenario: my slow scenario

Scenarios with lower priority value will be added to execution queue earlier (or vice versa).

Revealed from https://github.com/cucumber-rs/gherkin/discussions/28#discussion-3257377

Cucumber Messages is a message protocol for representing results and other information from Cucumber - Replaces json and junit formatters.

Stability/Performance:

With Cucumber Messages, several messages containing smaller pieces of information are emitted continuously to a stream, avoiding high memory consumption and enabling real-time processing of results.

Additional functionality (motivation):

The protocol aims to decouple various components of the Cucumber platform so that:

• Each component only needs to know about a subset of messages

• Gherkin is decoupled from the Cucumber execution component

  • This is part of a strategy to support other formats such as Markdown and Excel

@ilslv what do you think about this? We have the similar thing in our implementation, but hand-baked. Should we support it as a bare-bone of our implementation, or rather have just yet another Writer for it?