A unix "time" like benchmarking tool on steroids

Related tags

Profiling rust unix benchmark cargo rust-crate
Overview

benchie

Build Status Crate API Rust Version Platform License

Usage

Binary

Once Rust is installed (see step 1 in "Toolchain Setup"), you can easily install the latest version of benchie with:

$ cargo install benchie --locked
$ benchie ./program --option 4

Library

Usage

Add this to your Cargo.toml:

[dependencies]
benchie = "0"

Toolchain Setup

benchie can be build and tested on all major platforms.

  1. Bootstrap Rust v1.60.0 from https://rustup.rs and make sure:
  • you install it with one of the supported host triples and
  • add it to your path

Build and Test from Source

  1. Test your toolchain setup by compiling benchie:
$ cargo build --locked
  1. Execute tests:
$ cargo test --locked

License

Licensed under the MIT license.

Comments
  • Add a comment field

    Add a comment field

    A user should be able to add a comment field to his benchmark like so: benchie --comment "This is bubble sort with 200 iterations" ./program input.txt

    opened by ChristianMoesl 1
  • .benchie directory should be the root of the Repository

    .benchie directory should be the root of the Repository

    benchie should search for a Git repository, and if it finds one, it should use a direct child directory for its data (e.g. <repo>/.benchie/... If it doesn't find a Git repository in it's current path, it should use the current directory (e.g. <current_dir>/.benchie.

    feature size: small 
    opened by ChristianMoesl 0
  • Add detailed documentation how benchie can be used

    Add detailed documentation how benchie can be used

    This documentation should be in README.md, which is usually the point, where users have the first contact with benchie. The following topics should be described:

    • [ ] How to install benchie
    • [ ] How to use benchie
    • [ ] How to use benchie with output parsing
    • [ ] How to use benchie show?
    documentation 
    opened by ChristianMoesl 0
  • Fix issues in `benchie show`

    Fix issues in `benchie show`

    This is looking weird. Probably it makes more sense to truncate values. Screen Shot 2022-05-07 at 14 32 33

    Moreover, some additional statistics would be nice: • number of total benchmarks, etc.

    opened by mwidmoser 0
  • Refactor Storage Interface

    Refactor Storage Interface

    Save benchmark as giant hash map of key value pairs. e.g.:

    .benchie/data.json:

    {
  "benchmarks": [
    {
      "runtime": {
        "type": "Float",
        "value": 2.1
      },
      "algo": {
        "type": "String",
        "value": "bubblesort"
      }
    }
  ]
}
    opened by ChristianMoesl 0
  • Display 2-dimensional table in `benchie show`

    Display 2-dimensional table in `benchie show`

    benchie show --row commit --col command runtime Here, "runtime" is what we want to show.

           | ls -l | sleep 3 | cat file.txt
-------|--------------------------------
390f0d | 0.5   |          |
390f0d | 0.6   |          |
560f0c | 0.23  |          |
780f0c | 0.12  |          |

    Blocked by #16

    opened by mwidmoser 0
  • Display 1-dimensional table in `benchie show`

    Display 1-dimensional table in `benchie show`

    benchie show --row commit runtime Here commit is the row and runtime is the "metric"

    commit | runtime
-------|---------
390f0d | 0.5
390f0d | 0.6
560f0c | 0.23
780f0c | 0.12
    opened by mwidmoser 0
  • Parse predefined Output

    Parse predefined Output

    User program can print output to stdout like

    @benchie <key> <value>
    • to have custom parameters or metrics like counters, etc.
    • benchie does not measure anything
    • stores key-value pairs: try to parse the value as float -- on failure, store as string
    prio: low 
    opened by mwidmoser 0
  • Filtered output

    Filtered output

    Add option to show filtered output like: benchie show --filter commit=09er211 --filter command="ls -la" --filter key=value

    For now, filterable fields should be: commit, command, tag, where tag is a key-value pair.

    opened by ChristianMoesl 0
  • Add ability to tag benchmarks

    Add ability to tag benchmarks

    A benchmark should be tag-able e.g. like so: benchie --tag key=value ./program input.txt

    • multiple tags per benchmark are allowed
    • store key-value pairs: try to parse the value as float -- on failure, store as string
    opened by ChristianMoesl 0
  • CSV export

    CSV export

    Implement an additional flag for benchie show, which renders the table as a CSV to the stdout. If this option is enabled, printing of warnings or other info messages should be prevented.

    feature size: medium 
    opened by ChristianMoesl 2
  • Order in `benchie show`

    Order in `benchie show`

    1-d and 2-d table views yield different orderings (of rows and columns) each time when benchie show is called. It would be helpful if this is consistent.

    bug prio: low 
    opened by mwidmoser 0
  • Checkpointing

    Checkpointing

    User program can print so-called checkpoints to stdout like

    @benchie checkpoint start <key>

@benchie checkpoint end <key>

    Here benchie measures runtime between the checkpoints.

    prio: low 
    opened by mwidmoser 0
Releases(0.5.0)

  • 0.5.0(Jun 20, 2022)

  • 0.4.0(Jun 10, 2022)

    What's Changed

    • benchie saves data in Git repository root (#39) by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/40

    Full Changelog: https://github.com/benchie-io/benchie/compare/0.3.1...0.4.0

    Source code(tar.gz)
    Source code(zip)

  • 0.3.1(May 16, 2022)

    What's Changed

    • Fix git information discovery issue for detached states by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/36

    Full Changelog: https://github.com/benchie-io/benchie/compare/0.3.0...0.3.1

    Source code(tar.gz)
    Source code(zip)

  • 0.3.0(May 11, 2022)

    What's Changed

    • fix "show" subcommand in CLI by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/22
    • extract and save Git information of repository by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/9
    • Display 1-dimensional table in benchie show (closes #16) by @mwidmoser in https://github.com/benchie-io/benchie/pull/24
    • user can add custom tags to benchmarks by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/23
    • display useful information about all stored benchmarks (#15) by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/25
    • Display 2-dimensional table in benchie show (closes #17) by @mwidmoser in https://github.com/benchie-io/benchie/pull/27
    • add filters based on KV pairs to show command (#12) by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/26
    • add OS information to benchmarka (#5) by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/29
    • Parse cmd output by @mwidmoser in https://github.com/benchie-io/benchie/pull/30
    • truncate long example values in show command (#28) by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/31
    • colorize warning messages in terminal by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/33
    • Add slim documentation (closes #32) by @mwidmoser in https://github.com/benchie-io/benchie/pull/34

    Full Changelog: https://github.com/benchie-io/benchie/compare/0.2.0...0.3.0

    Source code(tar.gz)
    Source code(zip)

  • 0.2.0(Apr 30, 2022)

    What's Changed

    • add crash reporter with Github link generator by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/7
    • measure execution time exactly like "time" by @ChristianMoesl in https://github.com/benchie-io/benchie/pull/8
    • use hashmap to store key value pairs in benchmark (#19) by @mwidmoser in https://github.com/benchie-io/benchie/pull/20

    Full Changelog: https://github.com/benchie-io/benchie/compare/0.1.0...0.2.0

    Source code(tar.gz)
    Source code(zip)

  • 0.1.0(Apr 23, 2022)

Owner
benchie
benchie
GitHub
Statistics-driven benchmarking library for Rust

Criterion.rs Statistics-driven Microbenchmarking in Rust Getting Started | User Guide | Master API Docs | Released API Docs | Changelog | | Criterion.

Brook Heisler 3.1k Jan 8, 2023
Experimental one-shot benchmarking/profiling harness for Rust

Iai Experimental One-shot Benchmark Framework in Rust Getting Started | User Guide | Released API Docs | Changelog Iai is an experimental benchmarking

Brook Heisler 409 Dec 25, 2022
A stopwatch library for Rust. Used to time things.

rust-stopwatch This is a simple module used to time things in Rust. Usage To use, add the following line to Cargo.toml under [dependencies]: stopwatch

Chucky Ellison 77 Dec 11, 2022
An intrusive flamegraph profiling tool for rust.

FLAME A cool flamegraph library for rust Flamegraphs are a great way to view profiling information. At a glance, they give you information about how m

null 631 Jan 3, 2023
Benchmark tool for comparing with other runtimes.

Monoio Benchmark TCP ping-pong(not echo) is a common benchmark for network applications. We will use 1K ping-pong to test performance of different run

null 6 Oct 10, 2022
It is not about Keanu Reeves but a benchmark tool.

benchman Features Focus on one-shot benchmark RAII-style Statistics (Average, Median, 95% and 99% percentile) Colored output Tagging Nesting Motivatio

Akira Hayakawa 4 Feb 12, 2022
A http server benchmark tool written in rust 🦀

rsb - rust benchmark rsb is a http server benchmark tool written in rust. The development of this tool is mainly inspired by the bombardier project, a

Michael 45 Apr 10, 2023
Common vulnerability scanning on steroids ☄️

Hogg ?? An experimental passive website scanner. Hogg acts as a proxy between you and your DNS server and scans every website you visit for common vul

Yallxe 13 Jan 8, 2023
A command-line benchmarking tool

hyperfine 中文 A command-line benchmarking tool. Demo: Benchmarking fd and find: Features Statistical analysis across multiple runs. Support for arbitra

David Peter 14.1k Jan 6, 2023
A command-line benchmarking tool

hyperfine 中文 A command-line benchmarking tool. Demo: Benchmarking fd and find: Features Statistical analysis across multiple runs. Support for arbitra

David Peter 14.1k Jan 4, 2023
Benchmarking web frameworks written in rust with rewrk tool.

Web Framework Benchmarks Benchmarking web frameworks written in rust with rewrk tool.

null 103 Dec 8, 2022
ObfusEval is the benchmarking tool to evaluate the reliability of the code obfuscating transformation.

ObfusEval ObfusEval is the benchmarking tool to evaluate the reliability of the code obfuscating transformation. The following two metrics related the

Software Engineering Lab @ NAIST 4 Dec 15, 2022
A bash-like Unix shell written in Rust

Cicada Unix Shell Cicada is a simple Unix shell written in Rust. Documents Install cicada Environment Variables Cicada Builtins Completion RC File His

Hugo Wang 921 Dec 28, 2022
the file filesystem: mount semi-structured data (like JSON) as a Unix filesystem

ffs: the file filesystem ffs, the file filessytem, let's you mount semi-structured data as a fileystem---a tree structure you already know how to work

Michael Greenberg 176 Dec 31, 2022
A RISC-V and unix-like operating system developed just for fun.

Orca 虎鲸 A RISC-V and unix-like operating system developed just for fun. Schedule [*] bootloader (use RustSBI-qemu) [ ] linker & stack [ ] memory alloc

Mr.Z 4 May 29, 2022
Safe Unix shell-like parameter expansion/variable substitution via cross-platform CLI or Rust API

Safe Unix shell-like parameter expansion/variable substitution for those who need a more powerful alternative to envsubst but don't want to resort to

Isak Wertwein 4 Oct 4, 2022
Superviseur - A simple process supervisor for UNIX-like systems

A simple process supervisor for UNIX-like systems. Currently supports non-containerized services. Containerized services and wasm services will be supported in the future.

Tsiry Sandratraina 25 Mar 28, 2023
Get unix time (nanoseconds) in blazing low latency with high precision

RTSC Get unix time (nanoseconds) in blazing low latency with high precision. About 5xx faster than SystemTime::now(). Performance OS CPU benchmark rts

Ⅲx 8 Jul 14, 2022
Statistics-driven benchmarking library for Rust

Criterion.rs Statistics-driven Microbenchmarking in Rust Getting Started | User Guide | Master API Docs | Released API Docs | Changelog | | Criterion.

Brook Heisler 3.1k Jan 8, 2023
Experimental one-shot benchmarking/profiling harness for Rust

Iai Experimental One-shot Benchmark Framework in Rust Getting Started | User Guide | Released API Docs | Changelog Iai is an experimental benchmarking

Brook Heisler 409 Dec 25, 2022