hyperfine
A command-line benchmarking tool.
Demo: Benchmarking fd and find:
Features
- Statistical analysis across multiple runs.
- Support for arbitrary shell commands.
- Constant feedback about the benchmark progress and current estimates.
- Warmup runs can be executed before the actual benchmark.
- Cache-clearing commands can be set up before each timing run.
- Statistical outlier detection to detect interference from other programs and caching effects.
- Export results to various formats: CSV, JSON, Markdown, AsciiDoc.
- Parameterized benchmarks (e.g. vary the number of threads).
- Cross-platform
Usage
Basic benchmark
To run a benchmark, you can simply call hyperfine <command>.... The argument(s) can be any shell command. For example:
hyperfine 'sleep 0.3'
Hyperfine will automatically determine the number of runs to perform for each command. By default, it will perform at least 10 benchmarking runs. To change this, you can use the -m/--min-runs option:
hyperfine --min-runs 5 'sleep 0.2' 'sleep 3.2'
Warmup runs and preparation commands
If the program execution time is limited by disk I/O, the benchmarking results can be heavily influenced by disk caches and whether they are cold or warm.
If you want to run the benchmark on a warm cache, you can use the -w/--warmup option to perform a certain number of program executions before the actual benchmark:
hyperfine --warmup 3 'grep -R TODO *'
Conversely, if you want to run the benchmark for a cold cache, you can use the -p/--prepare option to run a special command before each timing run. For example, to clear harddisk caches on Linux, you can run
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
To use this specific command with Hyperfine, call sudo -v to temporarily gain sudo permissions and then call:
hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' 'grep -R TODO *'
Parameterized benchmarks
If you want to run a benchmark where only a single parameter is varied (say, the number of threads), you can use the -P/--parameter-scan option and call:
hyperfine --prepare 'make clean' --parameter-scan num_threads 1 12 'make -j {num_threads}'
This also works with decimal numbers. The -D/--parameter-step-size option can be used to control the step size:
hyperfine --parameter-scan delay 0.3 0.7 -D 0.2 'sleep {delay}'
This runs sleep 0.3, sleep 0.5 and sleep 0.7.
Shell functions and aliases
If you are using bash, you can export shell functions to directly benchmark them with hyperfine:
$ my_function() { sleep 1; }
$ export -f my_function
$ hyperfine my_function
If you are using a different shell, or if you want to benchmark shell aliases, you may try to put them in a separate file:
echo 'my_function() { sleep 1 }' > /tmp/my_function.sh
echo 'alias my_alias="sleep 1"' > /tmp/my_alias.sh
hyperfine 'source /tmp/my_function.sh; eval my_function'
hyperfine 'source /tmp/my_alias.sh; eval my_alias'
Export results
Hyperfine has multiple options for exporting benchmark results: CSV, JSON, Markdown (see --help text for details). To export results to Markdown, for example, you can use the --export-markdown option that will create tables like this:
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
find . -iregex '.*[0-9]\.jpg$' |
2.275 ± 0.046 | 2.243 | 2.397 | 9.79 ± 0.22 |
find . -iname '*[0-9].jpg' |
1.427 ± 0.026 | 1.405 | 1.468 | 6.14 ± 0.13 |
fd -HI '.*[0-9]\.jpg$' |
0.232 ± 0.002 | 0.230 | 0.236 | 1.00 |
The JSON output is useful if you want to analyze the benchmark results in more detail. See the scripts/ folder for some examples.
Installation
On Ubuntu
Download the appropriate .deb package from the Release page and install it via dpkg:
wget https://github.com/sharkdp/hyperfine/releases/download/v1.11.0/hyperfine_1.11.0_amd64.deb
sudo dpkg -i hyperfine_1.11.0_amd64.deb
On Fedora
On Fedora, hyperfine can be installed from the official repositories:
dnf install hyperfine
On Alpine Linux
On Alpine Linux, hyperfine can be installed from the official repositories:
apk add hyperfine
On Arch Linux
On Arch Linux, hyperfine can be installed from the official repositories:
pacman -S hyperfine
On NixOS
On NixOS, hyperfine can be installed from the official repositories:
nix-env -i hyperfine
On Void Linux
Hyperfine can be installed via xbps
xbps-install -S hyperfine
On macOS
Hyperfine can be installed via Homebrew:
brew install hyperfine
Or you can install using MacPorts:
sudo port selfupdate
sudo port install hyperfine
On FreeBSD
Hyperfine can be installed via pkg:
pkg install hyperfine
On OpenBSD
doas pkg_add hyperfine
With conda
Hyperfine can be installed via conda from the conda-forge channel:
conda install -c conda-forge hyperfine
With cargo (Linux, macOS, Windows)
Hyperfine can be installed via cargo:
cargo install hyperfine
Make sure that you use Rust 1.43 or higher.
From binaries (Linux, macOS, Windows)
Download the corresponding archive from the Release page.
Alternative tools
Hyperfine is inspired by bench.
Integration with other tools
Chronologer is a tool that uses hyperfine to visualize changes in benchmark timings across your Git history.
Make sure to check out the scripts folder in this repository for a set of tools to work with hyperfine benchmark results.
Origin of the name
The name hyperfine was chosen in reference to the hyperfine levels of caesium 133 which play a crucial role in the definition of our base unit of time — the second.
License
Copyright (c) 2018-2020 The hyperfine developers
hyperfine is distributed under the terms of both the MIT License and the Apache License 2.0.
See the LICENSE-APACHE and LICENSE-MIT files for license details.
![[FR] Allow caching of the shell startup time](https://avatars.githubusercontent.com/u/36224762?v=4)
3.1k Jan 8, 2023
631 Jan 3, 2023
6 Oct 10, 2022
4 Feb 12, 2022
45 Apr 10, 2023
14.1k Jan 6, 2023
5 Aug 20, 2022
103 Dec 8, 2022
3 May 6, 2022
4 Dec 15, 2022
3 Dec 26, 2022
7 Oct 3, 2022
0 Feb 2, 2022
2 Jul 13, 2023
3 Nov 2, 2023