insta: a snapshot testing library for Rust

Would you take a PR to change the inline test indentation from using ⋮ to just removing the rectangle of white space, similar to https://github.com/janestreet/ppx_expect#whitespace?

I had started on that PR but thought it worth checking first...

At the moment, insta library pulls quite a few deps. Specifically, in my Cargo.lock I see

dependencies = [
 "chrono 0.4.7 (registry+https://github.com/rust-lang/crates.io-index)",
 "ci_info 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
 "console 0.7.7 (registry+https://github.com/rust-lang/crates.io-index)",
 "difference 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
 "failure 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
 "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
 "pest 2.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
 "pest_derive 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
 "ron 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
 "serde 1.0.99 (registry+https://github.com/rust-lang/crates.io-index)",
 "serde_json 1.0.40 (registry+https://github.com/rust-lang/crates.io-index)",
 "serde_yaml 0.8.9 (registry+https://github.com/rust-lang/crates.io-index)",
 "uuid 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)",
]

insta is dev-dependency, so those transtive deps don't leak into the final executable, but they still affect compile time and the size of the ./target dir, making overall CI time larger.

I think I understand why each of those dependencies is there, but, at the same time, some of them look like that could have been avoided or made optional.

Specifically:

• json, yaml and ron are kind of the same thing, perhaps serialziation formats can be opt-in?
• pestis used for redactions, but it's a heavy dependency, could redactions be made a feature?
• failure slightly out of place in the library, perhaps sticking with std::error will work?
• ci_info looks pretty entreprisy, Cargo just checks two env vars: https://github.com/rust-lang/cargo/blob/1dd02155c988707b3e12606f52b29a90c76520dd/src/cargo/util/mod.rs#L86-L89
• chrono is also heavy, and, I think, it is only used to format time in snap files. In this case, I don't know how to improve this, except for removing the time field altogether.

We are trying to debug why a setup that keeps the tests in a separate repository is failing only on the CI system and not locally and insta is not helping figure out why the test fails.

I've got some tests using Insta, to test the output of templating for my webapp. On macOS these work just great, with filenames that are relevant to the test in question. (e.g. tests__no_templates.snap, tests__simple_template.snap, etc).

When I'm running the exact same build on my CI setup, which runs on Linux instead, this fails. And it fails because the filenames it is looking for are tests__main.snap, tests__main-2.snap, etc.

In both cases I'm using the latest nightly Cargo cargo 1.41.0-nightly (626f0f40e 2019-12-03) and Rust rustc 1.42.0-nightly (0de96d37f 2019-12-19), and I'm using Insta 0.12.0, with a Cargo.lock file so CI should be using the identical version. I'm also doing all of the assertions with the assert_debug_snapshot! macro from the crate.

Cheers

I'm building a couple tools using Insta, and I want to package up the code I need to properly transform my data into a snapshottable form, and then run Insta's snapshot assertion functions on that data.

However, Insta puts those snapshot files relative to where the utility functions are defined, not relative to the test that's actually invoking them!

• My utility function is in src/tree_view.rs
• The code using the utility is in src/commands/serve.rs in a mod named test.
• The snapshots from that test end up in src/snapshots with a name like tree_view__my_snapshot_name.snap.

I would expect the snapshots to instead be put into src/commands/snapshots and have a name like serve__my_snapshot_name.snap.

What happened?

• cargo insta test error

cargo insta test   
    Finished test [unoptimized + debuginfo] target(s) in 0.37s
     Running tests/main.rs (target/debug/deps/main-655db0c9f854f61a)
error: Found argument '-q' which wasn't expected, or isn't valid in this context

        If you tried to supply `-q` as a value rather than a flag, use `-- -q`

• cargo test would work

I did a quick check and found that parameters were added, But I don't know if that's it cli

  proc.arg("--");
  proc.arg("-q");

Reproduction steps

1. git clone --branch insta_with_cucumber https://github.com/huang12zheng/insta_issue
2. cargo insta test

Insta Version

insta = "1.15.0"

rustc Version

1.59

What did you expect?

pass test Is the '-q' needed?

Fixes #137

Previously the columns delimiting the replacement were being interpreted as byte offsets into their lines, rather than char offsets. This fix just updates the prefix/suffix construction to use a chars iterator.

I wasn't sure on how/where to add tests for this, as this code is only run as part of cargo-insta, and the existing tests seem to only check the snapshot generation/equivalency, rather than inline application.

Supporting are two action runs, before and after this change:

• before: https://github.com/c-spencer/insta-unicode-repro/actions/runs/351125741
• after: https://github.com/c-spencer/insta-unicode-repro/actions/runs/351158192

I've also tested locally on that repository and the patch is applied correctly (i.e. without duplicated parts of the previous snapshot in the suffix, as was happening for me and as described in #42 previously).

Seems to be a re-occurrence of #42

This is with cargo-insta and insta 1.1.0

Some snapshots are updating incorrectly as per the original issue (adding additional content past the end of the snapshot string), have also run into some panics as below where it's ending up trying to splice in the middle of a char.

Source code of the test causing segfault:

        adb!(
            ty("forall N:N. forall A:*. (A -> 1 + 1) -> Vec N A -> some K:N. Vec K A"),
            @"(∀ν : ℕ. (∀α : ⋆. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))"
        );

Snapshot file and log/backtrace when interacting with updating:

{"run_id":"1604428650-748676057","line":166,"new":{"module_name":"hale__sac__parser__tests","snapshot_name":"type_parsing-5","metadata":{"source":"src/sac/parser.rs","expression":"ty(\"forall N:N. forall A:*. forall B:*.(A -> B) -> Vec N A -> Vec N B\")"},"snapshot":"(∀ν : ℕ. (∀α : 🟊. (∀β : 🟊. ((α → β) → (Vec ν α → Vec ν β)))))"},"old":{"module_name":"hale__sac__parser__tests","metadata":{},"snapshot":"(∀ν : ℕ. (∀α : ⋆. (∀β : ⋆. ((α → β) → (Vec ν α → Vec ν β)))))"}}
{"run_id":"1604428650-748676057","line":170,"new":{"module_name":"hale__sac__parser__tests","snapshot_name":"type_parsing-6","metadata":{"source":"src/sac/parser.rs","expression":"ty(\"forall N:N. forall A:*. (A -> 1 + 1) -> Vec N A -> some K:N. Vec K A\")"},"snapshot":"(∀ν : ℕ. (∀α : 🟊. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))"},"old":{"module_name":"hale__sac__parser__tests","metadata":{},"snapshot":"(∀ν : ℕ. (∀α : ⋆. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))"}}

-old snapshot
+new results
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ty("forall N:N. forall A:*. (A -> 1 + 1) -> Vec N A -> some K:N. Vec K A")
────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────
    1       │-(∀ν : ℕ. (∀α : ⋆. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))
          1 │+(∀ν : ℕ. (∀α : 🟊. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))
────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────

  a accept   keep the new snapshot
  r reject   keep the old snapshot
  s skip     keep both for now
thread 'main' panicked at 'byte index 82 is not a char boundary; it is inside '∃' (bytes 80..83) of `            @"(∀ν : ℕ. (∀α : ⋆. ((α → (1 + 1)) → (Vec ν α → (∃κ : ℕ. Vec κ α)))))"`', /home/chris/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/str/mod.rs:2052:47
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:490
  11: rust_begin_unwind
             at src/libstd/panicking.rs:388
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:101
  13: core::str::slice_error_fail
             at src/libcore/str/mod.rs:0
  14: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}
  15: cargo_insta::inline::FilePatcher::set_new_content
  16: cargo_insta::cargo::SnapshotContainer::commit
  17: cargo_insta::cli::process_snapshots
  18: cargo_insta::cli::run
  19: cargo_insta::main
  20: std::rt::lang_start::{{closure}}
  21: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  22: std::panicking::try::do_call
             at src/libstd/panicking.rs:297
  23: std::panicking::try
             at src/libstd/panicking.rs:274
  24: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  25: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  26: main
  27: __libc_start_main
  28: _start

Currently when moving files around the old snapshots with the old filenames aren't deleted.

reproduce

1. Create a snapshot in a file foo.rs
2. Rename that file to bar.rs
3. run the tests again and accept the snapshots
4. now there are two snapshots, the unused snap and the one used for bar.rs

Closes https://github.com/mitsuhiko/insta/issues/78

Is this the right design? A good case is the insta__test__embedded.snap change below. I suspect those cases are rare; though it's a breaking change where they do exist.

I need to adds a whatsnew

Hi! I just upgraded to insta 0.11 and started taking advantage of the Settings struct to partition my snapshots a little more.

My tests are roughly of this shape:

let mut settings = insta::Settings::new();

let snapshot_path = Path::new(env!("CARGO_MANIFEST_DIR")).join("serve-test-snapshots");
settings.set_snapshot_path(snapshot_path);

settings.bind(|| {
    // Do the actual test

    assert_yaml_snapshot!(info, {
        ".sessionId" => "[session id]"
    });
});

When I get a snapshot failure, the .snap.new file is created in the correct place. The output tells me to run cargo insta review.

I run cargo insta review --all instead because I'm running in a workspace, but Insta tells me I have no snapshots to review:

lgreathouse /c/projects/rojo (master)
$ cargo insta review --all
done: no snapshots to review

...but I know it's lying, since my tests are still failing!

This issue also causes cargo insta test to fail, since it cannot find the snapshots that were just created. This means that the snippet recommended in the changelog to upgrade to 0.11.0 doesn't finish!

What happened?

Thanks for the recent changes in 1.26! I've updated PRQL with the latest changes, they're great.

The last thing I'm having trouble with is forcing updates — Windows seems to produce different metadata, which causes its tests to fail. Specifically, Windows doesn't seem to show input_file in glob! tests. Even if we skipped those tests, anyone contributing on Windows would be submitting PRs with huge diffs.

I looked around the insta code, but couldn't find why that might happen (looks like there isn't much platform-specific code in insta, so it's a puzzle, and unfortunately I don't have a Windows machine, so it's difficult to debug).

Here's an example: https://github.com/PRQL/prql/actions/runs/3861652619/jobs/6582754606#step:3:2726. (Note I made the tests pass there to see the diff, but otherwise it fails; e.g. here)

Any thoughts on why that might be happening? If not I can look more.

Reproduction steps

No response

Insta Version

1.26

rustc Version

rustc 1.65.0 (897e37553 2022-11-02)

What did you expect?

No response

Insta and cargo-insta currently have very limited coverage for behavioral flags which makes it easy to accidentally regress. This relies on manual testing at the moment.

Refs #331, #326, #330, #329.

What happened?

I've used insta for a while now and the "jump to definition" feature from the VSCode extension has never worked for me.
Jumping to definition brings me to insta own source code.
I do get additional insta commands, so the extension is active.

Reproduction steps

A new rust project. cargo install cargo-insta
cargo.toml:

[dev-dependencies]
insta = { version = "1", features = ["yaml"] }

Code that should show the Jump to definition:

insta::assert_snapshot!("foo");

Insta Version

insta = "1.23.0"

rustc Version

rustc 1.65.0 (897e37553 2022-11-02)

What did you expect?

Jumping to definition should work like in the README.

I often have a need for comparing snapshots with some tolerance. I write a lot of numerical code, and although for some code I want to really be exact, for many things I need to be able to provide some amount of "tolerance", typically in the form of a relative or absolute error.

There are several reasons for this:

• I don't want tests to break just because I reordered a floating-point summation.
• The result is non-deterministic due to e.g. parallel computation.
• In some cases I simply want to give quite a bit of leeway for the result and I'm more interested in the "trend" of the result (e.g. by plotting graphs and verifying that they satisfy expectations). Here I might for example only want insta tests to fail if some numbers deviate by more than 1% compared to the current reference value (which has been manually confirmed).

I'm not sure exactly how such functionality could look like in insta, as I'm not familiar with the inner workings, but I assume it would not work with Debug output, and could only work with serialized snapshots comparisons, such as JSON. This also raises a question of granularity: What if you have different "tolerances" for different numbers?

My motivation for writing this is to see if there is more interest in having this kind of functionality in insta. If so, we can flesh it out more together.

I have a few thoughts about https://github.com/mitsuhiko/insta/commit/745b45b5b47e5c2b60ff06ab9bf4af2138dc6d5a.

Firstly, the error message saying Insta does not allow inline snapshot assertions in loops isn't quite right. The new panic can happen without any loops:

fn helper(i:i64) {
   assert_snapshot!(i.to_string(), @"1");
}

#[test]
fn a_test() {
    helper(1);
    helper(2);
}

Here's how I encountered this situation.

Secondly, a friendlier behavior would be to not have an error merely because the snapshot at the same line was encountered twice. Rather, insta could error out only if , when this snapshot is repeatedly encountered, insta sees a different string compared to it than it saw before. Then, tests that were correct before wouldn't start failing because of upgrading insta.

In other words, the example above would have insta abort and report an error. But if we replace helper with

fn helper(i:i64) {
   do_something_complicated(i);
   assert_snapshot!(mission_report(), @"Total success!");
}

the test should be valid. Ideally (but not if it's too much trouble to implement), if mission_report() is modified to report "Complete success" both times, insta could handle that kind of change and allow the user to use the normal cargo insta review functionality to fix the test.

It's hard for me to tell how much effort it would take to make this happen. Is this the kind of thing you would accept a PR for? (No promises at this point, though) Do you think it's not worth the effort?

I have just migrated from Microsoft VSCode to VSCodium and could not find the insta snapshots extension in the OpenVSX registy.

Could you publish your extension there? The official VSCode registry may be used by Microsoft's official version according to their Terms of service, which is why libre forks of VSCode like VSCodium are using OpenVSX.