#363 This is closed but I think it's important to re-open it because it seems necessary to reuse an async runtime instead of always doing async_std::task::block_on in b.iter

I'd like to get criterion back into a usable state, and I'd like to improve the user experience around it.

Here's my plan so far, input is appreciated and PR are also welcomed.

Immediate actions: criterion 0.1.0

Basically, release criterion with its current public API, refactor its internals, further improvements shouldn't break the API.

• [x] land new timing loops #26
• [x] make compilable on current nightly
• [x] refactor: use the new Instant/Duration APIs (when stable?)
• [x] fix travis
• [x] CI: test on mac. see #70
• [x] CI: test on windows. see #70
• [x] documentation: explain output/plots, add examples, how to use in a cargo project
• [x] fix Appyevor
• [x] release dependencies
• [x] start making releases.

Short term: criterion 0.1.x

• [x] refactor: reduce usage of unstable features.
  • Throw away or put simd behind a cargo feature, it's unstable and only works with x86_64
  • criterion won't work on stable anytime soon because it depends on test::black_box
• [x] colorize output messages
• [x] developer documentation: explain the math
• [x] refactor: proper error handling in internals. bubble up errors and report them instead of panicking/unwrapping
• [x] clippy
• [x] rustfmt

Future: criterion 0.2

If we are allowed breaking changes, how can we improve the user experience? Some ideas:

• [x] simplify the public API
  • #26 adds several methods, perhaps all the similarly named methods can be replaced with a function that takes enums or some builder pattern.
• [ ] expose errors to user instead of panicking/exiting
• [ ] expose more internals, e.g. return an intermediate struct with the benchmark results instead of directly writing the results to disk.
• [ ] move away from gnuplot, produce a web report with interactive plots
• [x] better integration with cargo, output files to target directory, plots could live in target/doc

If you used criterion in the past, I'd like to hear about your experience.

• In general, What worked well? And, What didn't work for you?
• How helpful/clear/confusing were the generated plots/the output message?
• Was the performance regression detection reliable, or did you get false positives?
• What functionality do you think is missing?
• Any suggestion to improve the user experience?

The example has all code put in benches/. In practical use this will want to call functions from src/. It is non-trivial to call these functions (a simple use wont work).

The quickstart example should show:

• How to prefix the use statements (crate::, project_name::, etc.)
• What the visibility of items need to be (pub or pub(crate) etc).
• How to make it work when the project only has a main.rs.

That is, start with src/main.rs like so:

fn fibonacci(n: u64) -> u64 {
    match n {
        0 => 1,
        1 => 1,
        n => fibonacci(n-1) + fibonacci(n-2),
    }
}

pub fn main() {
    println!("The 1000th fibnonacci is {}", fibonacci(1000));
}

and explain what needs to be done to Criterion-bench that fibonacci function (without moving it).

I always struggle to get this working when I want to use Criterion in a new project. It would help me if the quickstart example covers this.

Hi guys, thanks for this great library!

This seems very simple, so it may be my mistake. I am running the following benchmark from /benches/benchmarks.rs:

fn main(){
    
    // make Criterion listen to command line arguments
    let mut c = Criterion::default().configure_from_args();
    
    c.bench(
        "name",
        Benchmark::new("dosomething", move |b| {
            // set up things            
            b.iter(|| 
            // do thing
            )
        })
        // Limit the measurement time and the sample size
        // to make sure the benchmark finishes in a feasible amount of time.
        .measurement_time(Duration::new(100, 0)).sample_size(20)
    );
}

Running cargo bench works, and faithfully runs the above benchmark.

I want to use the --save-baseline command line argument, however, and provide it way the docs suggest:

cargo bench -- --save-baseline master

This produces the following error:

Finished release [optimized] target(s) in 0.13s
Running target/release/deps/abclib-60df177b6b7c338b
error: Unrecognized option: 'save-baseline'
error: bench failed

Initiating the benchmark using the criterion_main!(*) macros instead does not change this.

What is going wrong?

I am running all this on stable. Versions:

cargo 1.28.0 (96a2c7d16 2018-07-13)
stable-x86_64-apple-darwin
rustc 1.28.0 (9634041f0 2018-07-30)

This allows us an option print bencher type format on CLI

$ cargo bench -- -R bencher
   Compiling criterion v0.3.1 (/Users/pksunkara/Coding/clap-rs/criterion.rs)
   Compiling clap v3.0.0-beta.1 (/Users/pksunkara/Coding/clap-rs/clap)
    Finished bench [optimized] target(s) in 1m 15s
     Running target/release/deps/01_default-439bccc0e739a77b
Gnuplot not found, using plotters backend
test build_empty_app ... bench:        135 ns/iter (+/- 0)                                 
test parse_empty_app ... bench:      1,940 ns/iter (+/- 10)  

When running the latest version from git, I get the following panic once all the benchmarks have run:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:355:21
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:71
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   2: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:227
             at src/libstd/panicking.rs:491
   3: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:398
   4: rust_begin_unwind
             at src/libstd/panicking.rs:325
   5: core::panicking::panic_fmt
             at src/libcore/panicking.rs:95
   6: core::panicking::panic
             at src/libcore/panicking.rs:59
   7: <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next
   8: <criterion::html::Html as criterion::report::Report>::final_summary
   9: <criterion::report::Reports as criterion::report::Report>::final_summary

This re-occurs each time, even if I do a cargo clean.

I might just have missed the how-to in the documentation but trying something like:

rustup run nightly cargo bench benches::my_benchmark::benchmark_1 does not seem to filter out benchmarks.

If the feature does not yet exists it would be incredibly useful as, technically, optimization often focus on a single function at the time.

I hope I remembered to add everything I needed here.

I both added an iter function and the possibility to benchmark multiple functions. Do you think both should be included?

Example of where I use this: https://github.com/faern/forkjoin-benchmarking/blob/master/src/main.rs#L108

@japaric

Changes:

• The Criterion::can_plot function has been removed.
• The Criterion::bench_function_over_inputs function has been removed.
• The Criterion::bench_functions function has been removed.
• The Criterion::bench function has been removed.
• --feature csv_output is now required for csv output.
• --feature html_reports is now required for html output.
• Added a --discard-baseline flag.
• rayon and plotters are optional (and default) dependencies.
• Status messages ('warming up', 'analyzing', etc) are printed to stderr, benchmark results are printed to stdout.
• Terminal escape codes are handled by the anes crate.
• Accept subsecond durations as command-line options.
• Formally support WASI (and automatically test for wasi regressions).
• Add --quiet flag (for printing a single line per benchmark).
• Replace serde_cbor with ciborium.
• Upgrade to clap v3.
• Bump regex to version 1.5.
• Bump MSRV to 1.56.1.

I've tried analyzing my benchmark with valgrind, and I've experienced a crash. Seems like criterion was the culprit (to be precise, running the benchmark with the standard harness made the crash go away), so I wanted to bring this to your attention. Seems to me that analyzing a benchmark with profiling tools is something people generally would want to do.

Or maybe I've simply made a mistake? Anyways, details can be found here, if there's anything I should to to investigate or help fix it, please let me know :)

I'm getting a panic which i pointing to this line: https://github.com/japaric/criterion.rs/blob/master/src/plot/mod.rs#L440

I assume it's because some how i've got a lot of zeroes coming from somewhere that's not supposed to be zero but thought i'd let you know

     Running C:\Users\DavidHewson\Documents\Repositories\rust\la05\target\release\deps\criterion_benchmark-40c30aef1aa85bcb.exe
invert                  time:   [0.0000 ps 0.0000 ps 0.0000 ps]
                        change: [   NaN%    NaN%    NaN%] (p = 0.00 < 0.05)
                        Change within noise threshold.
thread 'main' panicked at 'index out of bounds: the len is 500 but the index is 18446744073709551615', C:\Users\David\.cargo\registry\src\github.com-1ecc6299db9ec823\criterion-0.2.2\src\plot\mod.rs:440:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: bench failed

The problem:

Currently, there exists new to create from a function name and parameter, and from_parameter for creating just from a parameter. IntoBenchmarkId lets strings become a BenchmarkId with just a function name, yet it isn't publicly exported.

Why is this needed:

Some methods, such as bench_with_input on Criterion, take a BenchmarkId instead of any IntoBenchmarkId. This could be changed to take a generic, but I think even if that happens it makes sense to add this functionality anyways.

Desired solution:

A from_function or similar function to create a BenchmarkId with just a function name

The --profile-time option of criterion is very convenient for profiling, but in some cases you don't want to automatically scale the number of iterations, such as when you compare profile of different benchmarks executions.

In such a case, having the same number of iterations for all benchmarks is very convenient, as it allows to easily compare profiling metrics such as instruction count (and other performance counters).

I think that this could be achieved by simply adding a command-line parameter --profile-iter that would behave identically to --profile-time, except that it would set a fixed number of benchmark iterations instead of fixed time.

Besides, it would also be helpful if --profile-time could print the number of iterations it ran.

I was trying to benchmark some of my code and could not get the HTML report generated. I wasn't sure why up until I checked a previous project I worked on which specified the html_reports feature.

I feel like mentioning this in here would probably help any newcomers to the project. I could make a small PR for this, just let me know what you think of this!

I would add something like

"Criterion.rs can generate an HTML report displaying the results of the benchmark under target/criterion/reports/index.html. Note that for this to happen you need to include the html_reports like so criterion = { version = "0.4.0", features = [ "html_reports" ]}.`"

Issue

I'm unable to run specific groups of benches with the --bench option.

Example

mod day01 and mod day02 contain similarly defined criterion_groups.

criterion_group!(benches, bench_part_1, bench_part_2,);

Both groups are mentioned in the criterion_main

criterion_main!(
    day01::benches,
    day02::benches,
)

Expectation: mentioning either criterion_group or module should run that bench alone.

Actual:

❯ cargo criterion --bench bench::day01
error: no bench target named `bench::day01`.
Available bench targets:
    bench
    day01
    day02

❯ cargo criterion --bench day01

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

cargo criterion correctly runs all benches. Is there a different way to specify certain benches alone should be run?

std::hint::bench_black_box will be stable in 1.66 (2022-12-15 stable date) and I believe it may supercede the need for criterion:: black_box in some ways. It may be worth upating the docs to indicate this, and possible tradeoffs if any.

Docs: https://doc.rust-lang.org/std/hint/fn.black_box.html

I'm trying to benchmark some database code and found that setup the database is pretty complicated.

I need to create a new database on disk to run the benchmark, but can't see how to do it once per benchmark without doing it inside it.

The problem is that in the setup sections there are no parameters to see which run it is, so I can't use a pool or similar to split the db creation.

If I set up the DB outside the bench, it gets "copied" and all the benches run against the same DB on disk.

 #[derive(Debug)]
    struct Data {
        a: i32,
        b: u64,
        c: String,
    }

    impl Data {
        pub fn new(a: i32) -> Self {
            let b = (a + 13153) as u64;
            Self { a, b, c: b.to_string() }
        }
    }

    #[derive(Copy, Clone)]
    enum Runs {
        Tiny = 100,
    }

    impl Runs {
        pub fn range(self) -> Range<u16> {
            let x = self as u16;
            0..x
        }

        pub fn data(self) -> impl Iterator<Item = Data> {
            let x = self as u16;
            (0..x).into_iter().map(|x| Data::new(x as i32))
        }
    }

    mod bench_sqlite {
        use super::*;
        use rusqlite::{Connection, Transaction};

        fn build_db() -> ResultTest<Connection> {
            let tmp_dir = TempDir::new("sqlite_test")?;
            let db = Connection::open(tmp_dir.path().join("test.db"))?;
            db.execute_batch(
                "PRAGMA journal_mode = WAL;
                PRAGMA synchronous = normal;",
            )?;

            db.execute_batch(
                "CREATE TABLE data (
                a INTEGER PRIMARY KEY,
                b BIGINT NOT NULL,
                c TEXT);",
            )?;

            Ok(db)
        }

        pub(crate) fn insert_tx_per_row(run: Runs) -> ResultTest<()> {
            let db = build_db()?; // <-- HOW AVOID THIS?
            for row in run.data() {
                db.execute(
                    &format!("INSERT INTO data VALUES({} ,{}, '{}');", row.a, row.b, row.c),
                    (),
                )?;
            }
            Ok(())
        }
    }

    fn bench_insert_tx_per_row(c: &mut Criterion) {
        let mut group = c.benchmark_group("insert row");
        let run = Runs::Tiny;
        group.throughput(Throughput::Elements(run as u64));

        group.bench_function(BenchmarkId::new(SQLITE, 1), |b| {
            b.iter(|| bench_sqlite::insert_tx_per_row(run))
        });
        group.bench_function(BenchmarkId::new(PG, 1), |b| {
            b.iter(|| bench_pg::insert_tx_per_row(run))
        });

        group.finish();
    }

    criterion_group!(benches, bench_insert_tx_per_row);
    criterion_main!(benches);```