See #145 for what that looks like. There are currently no concrete examples unless you count the "Users" section, but those are prod crates that are big and complicated.

The company I work for is using trycmd and we noticed our CI timing out on Windows. After some investigation, it appears writing more than 4096 bytes to stdout will result in trycmd deadlocking on Windows. This is presumably related to #45572 in Rust (std::process::Command hangs if piped stdout buffer fills) and makes sense since Windows uses a much smaller buffer than Linux.

A reproducible test case on Windows:

// src/main.rs
fn main() {
    // Change to 4096 and no deadlock appears
    for _ in 0..4097 {
        print!("a");
    }
}

A simple trycmd is all that it takes because we aren't looking for anything to actually pass or fail.

// tests/bug.rs
#[test]
fn trycmd_bug() {
    let t = trycmd::TestCases::new();
    t.cases("tests/bug.md");
}

In bug.md:

$ trycmd-bug
a

On windows, my tests are failing because [EXE] shows up: https://github.com/SUPERCILEX/ftzz/actions/runs/3161445260/jobs/5147037527. I thought it was supposed to be removed? In general, I don't understand variables. The docs are very unclear. Are they replacing input or output? Is the value a search string? Or is the var replaced with the value? Tried both and neither did anything. Examples are needed of what happens to the output or input when a variable is added.

0.3.1 of Snapbox was a breaking change for me—as I'm using it to compare some generated TOML here. The \ in these cases are not path separators, so should not be normalized.

If this is intentional, I can just replace my usage of assert_eq_path with a custom Assert that disables normalization—but I wanted to report the issue just in case it was unintentional.

Thanks!

It took me several hours to understand how this library worked... the docs need a lot of love.

• In the readme and intro docs, a test is added with that looks through "tests/cmd/*.trycmd" but no example is shown of a trycmd so I had no idea these files actually needed to exist (I thought TRYCMD=dump would create them and was very confused for a while). An example of the trycmd file is needed.
• It also needs to be mentioned that md files are interpreted the same way.
• The example here makes no sense. It looks like that rust code will be compiled on the fly (it isn't) and where is my-cmd coming from? And how does the output make any sense? It says print hello world but then only hello is printed. This example needs to be completely reworked: it should clearly and explicitly be shown that you have a basic rust binary with a main function that prints something, an md/trycmd file that calls that binary (no need for input params), and a test that has the trycmd file as a case.
• Eliding content needs an example because I still can't figure out what it does: https://github.com/assert-rs/trycmd/issues/140

I'm not sure if it's ok to trim whitespace in all output (since it'd be nice to have a tailing newline in some stdout file), so I'm thinking only trailing lines in readmes should be trimmed.

When debugging a test failure in cargo, I came across match_contains

//! Routines for comparing and diffing output.
//!
//! # Patterns
//!
//! Many of these functions support special markup to assist with comparing
//! text that may vary or is otherwise uninteresting for the test at hand. The
//! supported patterns are:
//!
//! - `[..]` is a wildcard that matches 0 or more characters on the same line
//!   (similar to `.*` in a regex). It is non-greedy.
//! - `[EXE]` optionally adds `.exe` on Windows (empty string on other
//!   platforms).
//! - `[ROOT]` is the path to the test directory's root.
//! - `[CWD]` is the working directory of the process that was run.
//! - There is a wide range of substitutions (such as `[COMPILING]` or
//!   `[WARNING]`) to match cargo's "status" output and allows you to ignore
//!   the alignment. See the source of `substitute_macros` for a complete list
//!   of substitutions.
//!
//! # Normalization
//!
//! In addition to the patterns described above, the strings are normalized
//! in such a way to avoid unwanted differences. The normalizations are:
//!
//! - Raw tab characters are converted to the string `<tab>`. This is helpful
//!   so that raw tabs do not need to be written in the expected string, and
//!   to avoid confusion of tabs vs spaces.
//! - Backslashes are converted to forward slashes to deal with Windows paths.
//!   This helps so that all tests can be written assuming forward slashes.
//!   Other heuristics are applied to try to ensure Windows-style paths aren't
//!   a problem.
//! - Carriage returns are removed, which can help when running on Windows.

crates/cargo-test-support/src/compare.rs

Summary

While trying to use insert_var to replace data for cargo --help's output, in order to add UI testing to cross for our integrationg tests, I noticed that I was unable to use insert_var to actually replace data for very specific corner cases. Here's a sample project reproducing the failing test case. The minimum, failing example test string is:

const DATA: &str = r#"'
:"#;

Detailed Information

The test case uses:

$ trycmd-test
[REPLACE]

And the test script is:

const DATA: &str = r#"'
:"#;

#[test]
fn cli_tests() {
    trycmd::TestCases::new()
        .case("tests/cmd/failed-replacement.md")
        .insert_var("[REPLACE]", DATA)
        .unwrap();
}

Where main.rs is:

fn main() {
    println!("{DATA}");
}

const DATA: &str = r#"'
:"#;

It will fail with the following output:

running 1 test
Testing tests/cmd/failed-replacement.md:2 ... failed
Exit: success

---- expected: stdout
++++ actual:   stdout
   1      - [REPLACE]
        1 + '
        2 + :
stderr:

Update snapshots with `TRYCMD=overwrite`
Debug output with `TRYCMD=dump`
test cli_tests ... FAILED

Note that all of the following actually work:

const EX1: &str = r#"':"#;
const EX2: &str = r#"X's USAGE:"#;

It seems to require the following regex pattern: '.*\n.*:. If the newline is not present, everything works as expected.

Original Use

The original failing output, and what we were originally trying to match, is as follows:

Rust's package manager

USAGE:
    cargo [+toolchain] [OPTIONS] [SUBCOMMAND]

OPTIONS:
    -V, --version               Print version info and exit
        --list                  List installed commands
        --explain <CODE>        Run `rustc --explain CODE`
    -v, --verbose               Use verbose output (-vv very verbose/build.rs output)
    -q, --quiet                 Do not print cargo log messages
        --color <WHEN>          Coloring: auto, always, never
        --frozen                Require Cargo.lock and cache are up to date
        --locked                Require Cargo.lock is up to date
        --offline               Run without accessing the network
        --config <KEY=VALUE>    Override a configuration value (unstable)
    -Z <FLAG>                   Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for
                                details
    -h, --help                  Print help information

Some common cargo commands are (see all commands with --list):
    build, b    Compile the current package
    check, c    Analyze the current package and report errors, but don't build object files
    clean       Remove the target directory
    doc, d      Build this package's and its dependencies' documentation
    new         Create a new cargo package
    init        Create a new cargo package in an existing directory
    add         Add dependencies to a manifest file
    run, r      Run a binary or example of the local package
    test, t     Run the tests
    bench       Run the benchmarks
    update      Update dependencies listed in Cargo.lock
    search      Search registry for crates
    publish     Package and upload this package to the registry
    install     Install a Rust binary. Default location is $HOME/.cargo/bin
    uninstall   Uninstall a Rust binary

See 'cargo help <command>' for more information on a specific command.

Closes #141. Content elision still needs work, but I'll leave that to #140.

I think it's now clear how to use the library. I unfortunately couldn't figure out how to include the simple.md file into the README which means their contents are duplicated. Very annoying but oh well.

I added the lib and this test:

#[test]
fn readme_test() {
    trycmd::TestCases::new().case("README.md");
}

Running the test gives Testing README.md ... ignored with no explanation of why. Here's the repo in question: https://github.com/SUPERCILEX/ftzz

I'm brand new to trycmd, and would love to use it to snapshot test various combinations of args passed to a binary I'm building. However, despite the reputation of trycmd doing exactly this, I can't find a single example here of how to do that. Reading the example provided in examples/, there seems to be no binary involved. Even though I'll probably figure this out by looking at how various opensource projects use it on GitHub, I think providing an end-to-end working example of a binary that takes an argument, accepts input from stdin, and writes output to stdout, being snapshot tested end to end, would help a lot of others onboard to the project.

A potential example program might be a simple CLI app that reads stdin, reverses each line, and spits out the reversed lines to stdout.

I'd like to be able to show examples in my documentation that use standard input. My usual way to document this is something like:

$ echo "some simple input" | mycommand -
the output

... so that users can copy-paste it and get the same result (and then modify as needed). It would be great to be able to test that these examples work using trycmd.

Moved from https://github.com/assert-rs/trycmd/issues/168#issuecomment-1323763887

I use clap, and I put examples in the documentation comments for my commands/subcommands like:

enum Commands {
    /// Counts from START to STOP.
    ///
    /// $ mycli counter 1 3
    /// 1
    /// 2
    /// 3
    Counter {
        start: usize,
        stop: usize,
    }
}

From discussion in https://github.com/clap-rs/clap/discussions/4500

One alternative is to pull a markdown file into your doc comment

e.g. #[doc = include_str!("../../README.md")]

I was wondering if there was any functionality that would allow using command output to bind a variable, such that it could be used to test a subsequent command? For example, given a command to generate some key and list generated keys:

Generate two keys:

$ gen-key
yD0zydG611bl6RzwLutdnGd7zfgj6fdNDDreI786
$ gen-key
0zydG611bl6RzwLutdnGd7zfgj6fdNDDreI786UH

Then list all your keys:

$ list-keys
yD0zydG611bl6RzwLutdnGd7zfgj6fdNDDreI786
0zydG611bl6RzwLutdnGd7zfgj6fdNDDreI786UH

One way I imagine this could work is if we could bind variables in the output, for example:

$ gen-key
[KEY1]
$ gen-key
[KEY2]
$ list-keys
[KEY1]
[KEY2]

The first [KEY1] and [KEY2] would bind new variables. The second [KEY1] and [KEY2] (in list-keys) would expect the value to equal the bound variables.

Without any way to use command output in the test, it requires the tests to be implemented partly in rust and partly in .md files, which is cumbersome, whenever there is output that can't be known in advance.

Thoughts? I'm happy to help implement something like this if it makes sense for this project.

I'm testing a tool that requires for example creating a git repository and cd'ing into it, something like:

mkdir foo
cd foo
git init .
my-cmd
...

It seems like it's possible to "register" some of these commands using register_bin, but some commands like cd are actual shell built-ins. What is the correct way to write this test in such a way that it can be read as realistic example as well?

Windows has the extended path length syntax to work around legacy file path restrictions. These paths start with the string \\?\ followed by a normal path; more details can be found here. The problem is that Rust's std::fs::canonicalize creates these kinds of paths on Windows (which makes sense given that paths over 260 characters are otherwise disallowed), but trycmd can't handle them. If my binary under test's output, on Windows, includes something like:

\\?\C:\Path\To\Working\Directory\Some\File.txt

trycmd will transform this into:

//?/[CWD]/Some/File.txt

However, on Posix-like systems, my binary's output looks like this:

/Path/To/Working/Directory/Some/File.txt

as the behaviour of fs::canonicalize is different (on all systems, canonicalization is handled by the operating system, in the case of Posix, via realpath). Therefore, trycmd "compresses" that into

[CWD]/Some/File.txt

Depending on whether the test was blessed on Windows or a Posix-like system, the tests will therefore fail on the other family of systems.

A workaround is of course to discard extended length path syntax in my binary before printing the path or file name, but as stated above that can lead to other problems. I'd like trycmd to be able to handle this; i.e. collapse the leading extended path syntax specifier into [CWD] if it is found before the normal working directory path.

I have a program that produces a number of lines of output that do not need to be in any particular order. I suppose I could add an option to my program to sort output and then use that in my tests, but in my case, that would require quite a bit of restructuring. It would be nice if there was a way to have trycmd sort lines of output before comparing.