Splits test files into multiple groups to run tests in parallel nodes

Fumiaki MATSUSHIMA

Last update: Dec 12, 2022

Related tags

Text processing split-test

Overview

split-test

split-test splits tests into multiple groups based on timing data to run tests in parallel.

Installation

Download binary from GitHub releases page:

https://github.com/mtsmfm/split-test/releases

Usage

split-test command outputs test groups to stdout depends on its executing time.

Let's say we have three test files spec/a_spec.rb, spec/b_spec.rb, spec/c_spec.rb and the following test report report/report.xml.

<testsuites>
<testsuite classname="all" name="all" time="18.0">
<testcase classname="a" name="a" file="./spec/a_spec.rb" time="5.0"></testcase>
<testcase classname="b" name="b" file="./spec/b_spec.rb" time="10.0"></testcase>
<testcase classname="c" name="c" file="./spec/c_spec.rb" time="3.0"></testcase>
</testsuite>
</testsuites>

If you run them in series, it'll take 18 seconds.

You can use split-test to split them into two groups and run in parallel.

For node 0:

$ split-test --junit-xml-report-dir report --node-index 0 --node-total 2 --tests-glob 'spec/**/*_spec.rb'

You'll get the following result on stdout:

/path/to/spec/a_spec.rb
/path/to/spec/c_spec.rb

For node 1:

$ split-test --junit-xml-report-dir . --node-index 1 --node-total 2 --tests-glob 'spec/**/*_spec.rb'

You'll get:

/path/to/spec/b_spec.rb

Please be sure to increment --node-index arg.

You can use --debug option to make sure how it's grouped:

$ split-test --junit-xml-report-dir . --node-index 1 --node-total 2 --tests-glob 'spec/**/*_spec.rb' --debug
[2021-01-09T02:55:04Z DEBUG split_test] {"/path/to/spec/b_spec.rb": 10.0, "/path/to/spec/c_spec.rb": 3.0, "/path/to/spec/a_spec.rb": 5.0}
[2021-01-09T02:55:04Z DEBUG split_test] node 0: recorded_total_time: 8
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/a_spec.rb
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/c_spec.rb
[2021-01-09T02:55:04Z DEBUG split_test] node 1: recorded_total_time: 10
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/b_spec.rb
/path/to/spec/b_spec.rb

Pass the result to test command to run grouped tests:

$ rspec $(split-test --junit-xml-report-dir report --node-index 0 --node-total 2 --tests-glob 'spec/**/*_spec.rb' --debug)

GitHub Actions

on: push

jobs:
  # Download test-report and save as test-report-tmp to use the exactly same test report across parallel jobs.
  download-test-report:
    runs-on: ubuntu-latest
    steps:
      # Use dawidd6/action-download-artifact to download JUnit Format XML test report from another branch
      # https://github.com/actions/download-artifact/issues/3
      - uses: dawidd6/action-download-artifact@v2
        with:
          branch: main
          name: test-report
          workflow: ci.yml
          path: report
        # Use continue-on-error to run tests even if test-report is not uploaded
        continue-on-error: true
      - uses: actions/upload-artifact@v2
        with:
          name: test-report-tmp
          path: report

  test:
    needs: download-test-report
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node_index: [0, 1, 2]
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
          ruby-version: 3.0.0
      - uses: actions/download-artifact@v2
        with:
          name: test-report-tmp
          path: report-tmp
        # Use continue-on-error to run tests even if test-report is not uploaded
        continue-on-error: true
      - run: |
          curl -L --out split-test https://github.com/mtsmfm/split-test/releases/download/v0.3.0/split-test-x86_64-unknown-linux-gnu
          chmod +x split-test
      - run: bin/rspec --format progress --format RspecJunitFormatter --out report/rspec-${{ matrix.node_index }}.xml $(./split-test --junit-xml-report-dir report-tmp --node-index ${{ matrix.node_index }} --node-total 3 --tests-glob 'spec/**/*_spec.rb' --debug)
      - uses: actions/upload-artifact@v2
        with:
          name: test-report
          path: report
          if-no-files-found: error
        # Upload test-report on main branch only to avoid conflicting test report
        if: github.ref == 'refs/heads/main'

You can find working example on https://github.com/mtsmfm/split-test-example

Note

split-test is inspired by circleci tests split command.

split-test assumes test report has additional attribute file or filepath.

<testsuites>
<testsuite>
<testcase classname="spec.0_spec" name="0 is expected to eq 0" file="./spec/0_spec.rb" time="0.000373"></testcase>
</testsuite>
</testsuites>

To be exact it isn't JUnit standard.

A tool to calculate mean and standard deviation from multiple tests using PageSpeed Insights API.

psi-sample PSI Test tool is an open source tool to assist web developers that runs Page Speed Insight test manually! Installing To install the psi-tes

14 Sep 10, 2022

Black-box integration tests for your REST API using the Rust and its test framework

restest Black-box integration test for REST APIs in Rust. This crate provides the [assert_api] macro that allows to declaratively test, given a certai

10 Nov 23, 2022

Parser and test runner for testing compatable common Ethereum full node tests against Polygon Zero's EVM.

EVM Test Parses and runs compatible common Ethereum tests from ethereum/tests against Polygon Zero's EVM. Note: This repo is currently very early in d

3 Nov 4, 2022

Single-reader, multi-writer & single-reader, multi-verifier; broadcasts reads to multiple writeable destinations in parallel

Bus Writer This Rust crate provides a generic single-reader, multi-writer, with support for callbacks for monitoring progress. It also provides a gene

26 Feb 7, 2022

Download a file using multiple threads in parallel for faster download speeds.

multidl Download a file using multiple threads in parallel for faster download speeds. Uses 0 external dependencies. Usage Usage: multidl [--help] ADD

2 Sep 12, 2021

A PAM module that runs multiple other PAM modules in parallel, succeeding as long as one of them succeeds.

PAM Any A PAM module that runs multiple other PAM modules in parallel, succeeding as long as one of them succeeds. Development I created a VM to test

8 Apr 23, 2024

Android Device Pool - A tool to run device tests against a pool of devices.

adp (Android Device Pool) What is this? A tool to run device tests against a pool of devices. It will 'checkout' a device to run your tests against an

1 Oct 21, 2021

A simple, opinionated way to run containers for tests in your Rust project.

rustainers rustainers is a simple, opinionated way to run containers for tests. TLDR More information about this crate can be found in the crate docum

4 Nov 23, 2023

1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel

massh 1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel. The binary crates are CLI and GUI "frontends" for the li

2 Oct 16, 2022

Rust bindings to Linux Control Groups (cgroups)

cgroups-fs Native Rust library for managing Linux Control Groups (cgroups). This crate, curently, only supports the original, V1 hierarchy. You are we

28 Nov 21, 2022

A simple web-app allowing you to batch archive groups of repositories from a given organization

ice-repos My goal here is to build a simple web-app allowing you to batch archive groups of repositories from a given organization, using Rust+Yew. As

6 Nov 5, 2022

Small program which groups images based on the GPS position.

gps-cluster This small program will take some pictures in input, and based on the metadata on every image, it will group them by their GPS position, i

2 Sep 12, 2022

Library for Unix users and groups in Rust.

uzers-rs Adoption and continuation of the unmaintained ogham/rust-users crate. Big shout-out to its creator Benjamin Sago. This is a library for acces

8 Aug 25, 2023

Library for Unix users and groups in Rust.

uzers-rs Adoption and continuation of the unmaintained ogham/rust-users crate. Big shout-out to its creator Benjamin Sago. This is a library for acces

8 Sep 18, 2023

atttribute macro for running a flaky test multiple times

flaky_test This attribute macro will register and run a test 3 times, erroring only if all three times fail. Useful for situations when a test is flak

23 Mar 23, 2022

Collects accurate files while running in parallel through directories. (Simple, Fast, Powerful)

collectfiles Collects accurate files while running in parallel through directories. (Simple, Fast, Powerful) | Docs | Latest Note | [dependencies] col

2 Jun 1, 2022

Parallel iteration of FASTA/FASTQ files, for when sequence order doesn't matter but speed does

Rust-parallelfastx A truly parallel parser for FASTA/FASTQ files. Principle The input file is memory-mapped then virtually split into N chunks. Each c

8 Oct 24, 2022

Coppers is a custom test harnass for Rust that measures the energy usage of your test suite.

Coppers Coppers is a test harness for Rust that can measure the evolution of power consumptions of a Rust program between different versions with the

175 Dec 4, 2022

A simple tool to test sqlx with postgres. It will automatically create a database and drop it after the test.

sqlx-db-tester This a tool to test sqlx with postgres. It only supports tokio runtime at this moment. How to use it You should first create a TestDb d

4 Nov 29, 2022

Comments

actions returns -out as an ambiguous option which result to exit code 2

We started to get the following exit status when running split test and caused CI to not run. It is possible that GitHub changed ubuntu-latest library a bit and caused some update to the curl. It would be great if the team can check it out.

Run curl -L --out split-test https://github.com/mtsmfm/split-test/releases/download/v1.0.0/split-test-x86_64-unknown-linux-gnu
[10](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:11)
curl: option --out: is ambiguous
[11](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:12)
curl: try 'curl --help' or 'curl --manual' for more information
[12](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:13)
Error: Process completed with exit code 2.

I do not have public repo available to share the fix and where it caused the issue but can share some screenshots. Hope it helps!

opened by nanophate 1

Splits test files into multiple groups to run tests in parallel nodes

Related tags

Overview

split-test

Installation

Usage

GitHub Actions

Note

You might also like...

A tool to calculate mean and standard deviation from multiple tests using PageSpeed Insights API.

Black-box integration tests for your REST API using the Rust and its test framework

Parser and test runner for testing compatable common Ethereum full node tests against Polygon Zero's EVM.

Single-reader, multi-writer & single-reader, multi-verifier; broadcasts reads to multiple writeable destinations in parallel

Download a file using multiple threads in parallel for faster download speeds.

A PAM module that runs multiple other PAM modules in parallel, succeeding as long as one of them succeeds.

Android Device Pool - A tool to run device tests against a pool of devices.

A simple, opinionated way to run containers for tests in your Rust project.

1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel

Rust bindings to Linux Control Groups (cgroups)

A simple web-app allowing you to batch archive groups of repositories from a given organization

Small program which groups images based on the GPS position.

Library for Unix users and groups in Rust.

Library for Unix users and groups in Rust.

atttribute macro for running a flaky test multiple times

Collects accurate files while running in parallel through directories. (Simple, Fast, Powerful)

Parallel iteration of FASTA/FASTQ files, for when sequence order doesn't matter but speed does

Coppers is a custom test harnass for Rust that measures the energy usage of your test suite.

A simple tool to test sqlx with postgres. It will automatically create a database and drop it after the test.

Comments

actions returns -out as an ambiguous option which result to exit code 2

Releases(v1.0.0)

v1.0.0(Jul 31, 2021)

v0.3.0(Jan 2, 2021)

v0.2.0(Dec 30, 2020)

v0.1.0(Dec 30, 2020)

Owner

Fumiaki MATSUSHIMA

This tool is for those who often want to search for a string deeply into a directory in recursive mode, but not with the great tool: grep, ack, ripgrep .........一个工具最大的价值不是它有多少功能，而是它能够让你以多快的速度达成所愿......

bottom encodes UTF-8 text into a sequence comprised of bottom emoji

Find and replace text in source files

Find files (ff) by name, fast!

⏮ ⏯ ⏭ A Rust library to easily read forwards, backwards or randomly through the lines of huge files.

Difftastic is an experimental structured diff tool that compares files based on their syntax.

Subtitles-rs - Use SRT subtitle files to study foreign languages

A command line tool for renaming your ipa files quickly and easily.

ABQ is a universal test runner that runs test suites in parallel. It’s the best tool for splitting test suites into parallel jobs locally or on CI

The main repo for manta relay nodes and manta parachain nodes