It would be great to have a way to disable the intermediate shell for very fast commands where we need a resolution that is better than 3 ms (typical standard deviation of the shell spawning time).

This new mode would obviously not be able to benchmark complex commands like seq 1000 | factor, but we could use ~~https://crates.io/crates/shellwords~~ https://crates.io/crates/shell-words to be able to run commands like my_command --foo "some strong" --bar.

By default, Hyperfine will attach Stdio::null() to the stdout of the command it runs. Some programs, like greps, will detect this, alter how they execute and run more quickly as a result. For example, in the case of GNU grep, if it detects that stdout is /dev/null, then it will behave as if the -q/--quiet flag were present. This means GNU grep can exit as soon as a match is found. This is a perfectly legitimate optimization since its only observable behavior is its exit code, and its exit code is only influenced by whether a match exists or not (assuming no errors or interrupts occur).

What this means is that, e.g., time grep foo file | wc -l and hyperfine 'grep foo file' aren't quite doing the same thing. That's no fault of Hyperfine, however, it would be nice to have a way to easily work around it. As of today, here are some work-arounds:

• Use the --show-output flag, which captures stdout. But this also prints stdout, which can be undesirable if there is a lot of output.
• In the case of grep specifically, use a query that is not expected to match and give Hyperfine the --ignore-failure flag.

A better work-around might be to have some other flag, maybe --capture-output, that causes Hyperfine to capture stdout but then do nothing with it.

I was motivated to write this feature request based on an errant benchmark found in the wild: https://old.reddit.com/r/commandline/comments/mibsw8/alternative_to_grep_less/gt90lmm/

Re: #288

Hopefully this is analogous to sharkdp/fd#66. Changes to ci/before_deploy.bash mirror sharkdp/fd when copying shell completions across.

This is the first PR I've submitted with Rust -- still learning OSS and Rust, so any feedback and I'll fix up what I can.

The CLI output has a really great summary formatted like:

Summary
  'curl google.com' ran
    2.93 ± 2.29 times faster than 'wget google.com'

The relative speed provided (2.93) is, in my opinion, a major take-a-way when reading speed tests. Unfortunately, this information is not (directly) part of the exported data for markdown, json and csv.

Would you be willing to accept a PR adding information about the relative speed to the exported data?

I'm using a slightly complicated for loop combined with hyperfine to compare the relative speeds of different versions of grep. The lines beginning with tee simply redirect the contents of both STDOUT and STDERR - in this case, all output from hyperfine - to a plain text file using tee.

for i in {1..15}; \
do
(hyperfine -m 10 --show-output -p 'sync64 -nobanner' 'LC_ALL=C grep -i "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt') 2>&1 |
tee -a "../Tests/grep Test 5 (Hyperfine Full Directory w. Sync).txt"; \
(hyperfine -m 10 --show-output -p 'sync64 -nobanner' 'LC_ALL=C grep -Fi "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt') 2>&1 |
tee -a "../Tests/grep -F Test 5 (Hyperfine Full Directory w. Sync).txt"; \
(hyperfine -m 10 --show-output -p 'sync64 -nobanner' 'rg -i "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt') 2>&1 |
tee -a "../Tests/ripgrep Test 5 (Hyperfine Full Directory w. Sync).txt"; \
done;

However, it fails with the following output:

Benchmark #1: LC_ALL=C grep -i "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt

Flushing: B C D 'LC_ALL' is not recognized as an internal or external command, operable program or batch file. Error: Command terminated with non-zero exit code. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong. Benchmark #1: LC_ALL=C grep -Fi "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt

Flushing: B C D 'LC_ALL' is not recognized as an internal or external command, operable program or batch file. Error: Command terminated with non-zero exit code. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong. Benchmark #1: rg -i "ajndoandajskaskaksnaodnasnakdnaosnaond" *.txt

Flushing: B C D *.txt: The filename, directory name, or volume label syntax is incorrect. (os error 123) Error: Command terminated with non-zero exit code. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

Each command in the loop passed through hyperfine fails with an error, with two of them being distinct errors here. The first two suggest that hyperfine doesn't support commands prefixed with the LC=ALL_C environment variable, and the second appears to have trouble with the glob in *.txt. Does hyperfine not support these, or am I using it/them incorrectly?

It'd be nice to be able to show other statistics such as the 95th percentile or median runtime. HdrHistogram can record this with relatively little overhead, and there's a pretty good official Rust implementation here (I'm one of the maintainers).

As commented on #336, I've made an initial patch to allow one to disable the shell intermediation, and just parse the command as a list of words (perhaps with quoting).

As @sharkdp mentioned, I currently use --shell '' to disable the shell, but that is quite ugly UX; however I wanted to keep the patch as small as possible.

Also, given that Rust doesn't easily allow cross-compile I can't cargo check --target ...windows..., so although I've tried to get the code right on Windows most likely there is a compile error.

As mentioned in the issue at least in terms of outputs, it seems that hyperfine already handles the shell overhead and subtracts it from the outputs. Thus this patch is mostly to speed-up the overall execution by not passing through the shell. It shouldn't change the measured values.

When using hyperfine, I am missing the possibility of specifying a range or floating point values on --parameter-scan:

hyperfine --parameter-scan i 1 10 2 "sleep {i}"

or

hyperfine --parameter-scan i 0.1 1.0 "sleep {i}"

I'd like to measure the median instead of the mean. Currently, I have a perl script that reads the JSON output and calculates the quartiles. But that's not very convenient. Do you think it's good to output the median (instead of, or in addition to) the mean?

Here are some points:

• If cron fires during a benchmark, the mean might be off
• It seems many academic papers prefer the median.

PS: Thanks for hyperfine. It's much more convenient than time.

I don't know what algorithm hyperfine uses to determine how many samples to draw for determining the shell startup time, but when I use my custom "shell" that sends the code to my (local) REST server to be evaled, hyperfine takes a few seconds to determine the shell startup time (the commands execute fast themselves). I think that if hyperfine just caches this measurement for later reuse, it would make this workflow that much faster.

Hi;

I am using hyperfine to test some of my software against some other software and I would like to know if it's possible to get a feature that would allow me (or us) to merge different markdown files into one; or to have the possibility to generate one single markdown file instead that would have two columns referencing the data for each tests (would make that easier if you are trying to compare two different test set's results)

(I am not sure if my wall of text is clear and understand, please let me know and I will try to explain what I have on my mind a little bit better)

Not sure what the UI should look like, but if you only use parameter values in a prepare command, you currently have no way of knowing which benchmarks used which parameters.

Bumps assert_cmd from 2.0.5 to 2.0.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps serde from 1.0.147 to 1.0.152.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I am benchmarking the mold linker on an in-house application, using hyperfine 1.15.0 (from conda-forge) to drive the benchmarks.

While the wall-clock times reported seem correct, the "User" and "Sys" times seem way too low (and also are not impacted by the number of threads used). Example:

Benchmark 1: mold180 (1 threads)
  Time (mean ± σ):     24.302 s ±  0.477 s    [User: 0.392 s, System: 0.234 s]
  Range (min … max):   23.836 s … 24.789 s    3 runs

Benchmark 2: mold180 (2 threads)
  Time (mean ± σ):     14.871 s ±  0.597 s    [User: 0.385 s, System: 0.246 s]
  Range (min … max):   14.351 s … 15.523 s    3 runs

The benchmark invokes the linker by calling ninja MyBinary (with --setup and --prepare used to make sure ninja does not much else than invoking the linker)

Specs of the machine used to run the benchmark:

• OS: Ubuntu 20.04 with kernel 5.4
• CPU: Dual socket Intel Xeon Gold 5218
• RAM: 128GB

$ cargo install hyperfine

error[E0432]: unresolved import `winapi::um::handleapi`
 --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\hyperfine-1.15.0\src\timer\windows_timer.rs:9:9
  |
9 |         handleapi::CloseHandle,
  |         ^^^^^^^^^ could not find `handleapi` in `um`

For more information about this error, try `rustc --explain E0432`.

I got this error on a workflow run today even though everything was working perfectly fine a few days ago when it last ran.

After some digging around, the winapi crate turned out to have a flag for handleapi, enabling that in the Cargo.toml file fixes the issue.

I'm not experienced enough to really understand how this broke now considering how both this and the winapi crates have not been updated that recently but it was suggested to me in the Rust Discord that some lockfile somewhere was perhaps requiring an older version of one of these crates which did not need the explicit feature requirement.

This was tested locally by cloning the repo and installing it with cargo install --path ./hyperfine which only worked after the change I made in the pull request.