Hey

I build the tests in a docker stage and execute them in the next. I run into

│ Running tests in sequence                                                                                                                                                                                                  │
│   Extracting 1 binaries to /tmp/nextest-archive-bb9dDJ                                                                                                                                                                     │
│    Extracted 3 files to /tmp/nextest-archive-bb9dDJ in 1.58s                                                                                                                                                               │
│ Error:                                                                                                                                                                                                                     │
│    0: error building test list                                                                                                                                                                                             │
│    1: running ''/tmp/nextest-archive-bb9dDJ/target/debug/deps/integration_tests-9cdfa58268cee69f --list --format terse'' failed                                                                                            │
│    2: No such file or directory (os error 2)                                                                                                                                                                               │
│                                                                                                                                                                                                                            │
│   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                                                                                                                                         │
│                                 ⋮ 3 frames hidden ⋮                                                                                                                                                                        │
│    4: cargo_nextest::dispatch::App::build_test_list::h0b139060a3d7e2cb                                                                                                                                                     │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│    5: cargo_nextest::dispatch::CargoNextestApp::exec::hc26ab86754cfce02                                                                                                                                                    │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│    6: cargo_nextest::main::h1c3419e2de29e9b3                                                                                                                                                                               │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│    7: std::sys_common::backtrace::__rust_begin_short_backtrace::h29484de5e2a0cf31                                                                                                                                          │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│    8: std::rt::lang_start::{{closure}}::ha37d78b2499ee6c9                                                                                                                                                                  │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│    9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::had4f69b3aefb47a8                                                                                                               │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:259                                                                                                                              │
│   10: std::panicking::try::do_call::hf2ad5355fcafe775                                                                                                                                                                      │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492                                                                                                                                  │
│   11: std::panicking::try::h0a63ac363423e61e                                                                                                                                                                               │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456                                                                                                                                  │
│   12: std::panic::catch_unwind::h18088edcecb8693a                                                                                                                                                                          │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137                                                                                                                                      │
│   13: std::rt::lang_start_internal::{{closure}}::ha7dad166dc711761                                                                                                                                                         │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:128                                                                                                                                         │
│   14: std::panicking::try::do_call::hda0c61bf3a57d6e6                                                                                                                                                                      │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492                                                                                                                                  │
│   15: std::panicking::try::hbc940e68560040a9                                                                                                                                                                               │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456                                                                                                                                  │
│   16: std::panic::catch_unwind::haed0df2aeb3fa368                                                                                                                                                                          │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137                                                                                                                                      │
│   17: std::rt::lang_start_internal::h9c06694362b5b80c                                                                                                                                                                      │
│       at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:128                                                                                                                                         │
│   18: main<unknown>                                                                                                                                                                                                        │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│   19: __libc_start_main<unknown>                                                                                                                                                                                           │
│       at <unknown source file>:<unknown line>                                                                                                                                                                              │
│   20: _start<unknown>                                                                                                                                                                                                      │
│       at <unknown source file>:<unknown line>

I dug around the code and think the default runner is not found. But I did not understand what the default runner actually is. I tried overriding it but to no avail. Do you have any hints how to get it to run? When I just run /tmp/nextest-archive-bb9dDJ/target/debug/deps/integration_tests-9cdfa58268cee69f --list --format terse i get the list of tests successfully :)

Thanks, Noah

Implements https://github.com/nextest-rs/nextest/issues/44

This is still work in progress, but I just wanted to be sure I was on the right tracks from your point of view.

I have the basic feature working:

# builds and list binaries
> cargo nextest list-bins --message-format json > binaries.json
# runs without building
> cargo nextest run --test-list binaries.json

Still to do:

• [x] add path remapping
• [x] add tests

At the moment, I'm still creating the graph when running, I'm wondering if I should save it (or at least the needed part). This could allow me to use the runner without cargo installed (which could be great for my use case, but not indispensable for the moment).

As you are up apparently, I push this to open the discussion.

Following what you said, I was thinking maybe something along the line of:

DSL

• package(name)
• deps(package_name)
• rdeps(package_name)
• test(partial_name)
• not(expr)
• expr && expr / expr and expr
• expr || expr / expr or expr
• (expr)

Like -E "package(driver) or test(use_driver_)", -E "package(utility_package) and rdeps(utility_package)"

Currently the entire target directory is needed when reusing build on another machine. This is quite a setback for us because previously we only need to transfer a specific test binary under target/debug/deps/ generated by cargo test --no-run.

I'm wondering if it's possible to reduce the binaries that must be shared between two machines, and make this information easy enough to use. Or better yet, package the necessary test artifacts by nextest itself.

This would be especially useful for test partitioning. For our use case, test cases from one large repo are sharded to tens of partitions. With this feature, each partition wouldn't have to download all the test binaries.

Currently, the only way to install cargo-nextest is with the pre-built binaries, or by using cargo install. It would be nice to add other binary installation mechanisms as well.

To consider:

• [x] ~Release signing~ (now tracked at https://github.com/nextest-rs/nextest/issues/369)
• [x] Integrate into cargo binstall
• [x] ~Point people to cargo quickinstall~ (we already do our own binary releases now)
• [x] Homebrew?

cargo-nextest 0.9.10 introduces experimental support for reusing the build across machines and invocations. This issue tracks stabilizing this option:

• [x] How does this interact with #82?
• [x] Verify option names.
• [x] Ensure that the "cargo present on destination" and "cargo not present on destination" scenarios both work well.
• [x] Is the current path remapping support sufficient?

cc @Guiguiprim

Consider adding a partition() operator to filter expressions, and possibly deprecating the current --partition flag.

This is in principle possible to do, since (assuming a specific state of source code) all partition assignments for tests are completely deterministic. However, the current partition scheme for count: uses mutable state, which is too complicated -- instead, we can simply give each test a numeric ID and simply use modulo N for the count partition operator.

Strictly speaking, this doesn't block the release of filter expressions. However, if we do it before releasing filter expressions, we can ban using --partition and -E simultaneously. If we implement this after the release, we'll have to scan the query to look for partition operators, raising the complexity of the implementation -- so there's some advantages to doing it now.

This might be a weird request, but:

To optimize CI resources, improve build reproducibility and other things, I have the following setup:

• One CI step: cross compilation for windows from a linux CI runner (using nix-docker image)
• An other CI step: run the tests in a windows VM

Currently I build the tests with cargo test --no-run and then execute them without cargo test in the VMs (with an ugly script).

If it was possible to:

• cargo nixtest --no-run in the CI runner
• copy the target folder (or at least the relevant part) in the VM
• cargo nixtest --no-build in the VM

It would be really awesome.

As is, I don't know if it would be easily doable. If it is not to complex and an acceptable feature, I might be open in doing it under mentoring.

Hi, first of all awesome crate! :D On just a dual core system it seems to be 10-15% faster than default cargo test. :)

While playing around with it, I wondered if it would be useful to have some kind of --recursive flag that does not only run test in the current directory but also on all other crates of the current workspace.

I just noticed there is a Runner option:

--no-capture              Run tests serially and do not capture output

Why tests run serially when --no-capture flag is enabled? I think it would be more convenient if nextest can run tests parallely when --no-capture is specified. And the --nocapture for cargo test is parallelism.

Can I do some work to make nextest run parallely when --no-capture flags is enabled?

Implement fixed and exponential backoffs between retries. Sample configuration:

retries = { count = 5, backoff = "exponential", delay = "1s", max-delay = "10s", jitter = true }

The code was heavily inspired by the again crate, but the iterator is not public so I had to copy it. There are no tests yet and no good documentation. Reporting: adds the delay as follows: 1/3 RETRY in 1s. The jitter currently picks a random number from the interval (0, calculated_delay), which might not be ideal.

Add a way to display test groups as they currently apply to tests.

In the future we'll add support for displaying other kinds of configuration, but hopefully this will be useful for now.

Bumps self_update from 0.32.0 to 0.34.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps quick-xml from 0.26.0 to 0.27.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I see that here https://nexte.st/book/slow-tests.html - it is possible to set a limit for what is marked as slow. I think it would also be nice to show the slowest N (10?) tests without having to know the limit up front. This would be pretty easy to implement externally after #20 (just pipe to sort), but I bet it would look a lot nicer if internal.

Bumps console from 0.15.2 to 0.15.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This is quite tricky to do because there are a number of different permutations involved. Putting up this WIP to get back to at some point in the future (or if someone else would like to take it from there).

TODO:

• [ ] Work out all the permutations (make a table for a methodical approach)
• [ ] Figure out if symlinks impact this
• [ ] Add tests

Fixes #695.