Splits test files into multiple groups to run tests in parallel nodes

Overview

split-test

split-test splits tests into multiple groups based on timing data to run tests in parallel.

Installation

Download binary from GitHub releases page:

https://github.com/mtsmfm/split-test/releases

Usage

split-test command outputs test groups to stdout depends on its executing time.

Let's say we have three test files spec/a_spec.rb, spec/b_spec.rb, spec/c_spec.rb and the following test report report/report.xml.

<testsuites>
<testsuite classname="all" name="all" time="18.0">
<testcase classname="a" name="a" file="./spec/a_spec.rb" time="5.0"></testcase>
<testcase classname="b" name="b" file="./spec/b_spec.rb" time="10.0"></testcase>
<testcase classname="c" name="c" file="./spec/c_spec.rb" time="3.0"></testcase>
</testsuite>
</testsuites>

If you run them in series, it'll take 18 seconds.

You can use split-test to split them into two groups and run in parallel.

For node 0:

$ split-test --junit-xml-report-dir report --node-index 0 --node-total 2 --tests-glob 'spec/**/*_spec.rb'

You'll get the following result on stdout:

/path/to/spec/a_spec.rb
/path/to/spec/c_spec.rb

For node 1:

$ split-test --junit-xml-report-dir . --node-index 1 --node-total 2 --tests-glob 'spec/**/*_spec.rb'

You'll get:

/path/to/spec/b_spec.rb

Please be sure to increment --node-index arg.

You can use --debug option to make sure how it's grouped:

$ split-test --junit-xml-report-dir . --node-index 1 --node-total 2 --tests-glob 'spec/**/*_spec.rb' --debug
[2021-01-09T02:55:04Z DEBUG split_test] {"/path/to/spec/b_spec.rb": 10.0, "/path/to/spec/c_spec.rb": 3.0, "/path/to/spec/a_spec.rb": 5.0}
[2021-01-09T02:55:04Z DEBUG split_test] node 0: recorded_total_time: 8
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/a_spec.rb
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/c_spec.rb
[2021-01-09T02:55:04Z DEBUG split_test] node 1: recorded_total_time: 10
[2021-01-09T02:55:04Z DEBUG split_test] /path/to/spec/b_spec.rb
/path/to/spec/b_spec.rb

Pass the result to test command to run grouped tests:

$ rspec $(split-test --junit-xml-report-dir report --node-index 0 --node-total 2 --tests-glob 'spec/**/*_spec.rb' --debug)

GitHub Actions

on: push

jobs:
  # Download test-report and save as test-report-tmp to use the exactly same test report across parallel jobs.
  download-test-report:
    runs-on: ubuntu-latest
    steps:
      # Use dawidd6/action-download-artifact to download JUnit Format XML test report from another branch
      # https://github.com/actions/download-artifact/issues/3
      - uses: dawidd6/action-download-artifact@v2
        with:
          branch: main
          name: test-report
          workflow: ci.yml
          path: report
        # Use continue-on-error to run tests even if test-report is not uploaded
        continue-on-error: true
      - uses: actions/upload-artifact@v2
        with:
          name: test-report-tmp
          path: report

  test:
    needs: download-test-report
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node_index: [0, 1, 2]
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
          ruby-version: 3.0.0
      - uses: actions/download-artifact@v2
        with:
          name: test-report-tmp
          path: report-tmp
        # Use continue-on-error to run tests even if test-report is not uploaded
        continue-on-error: true
      - run: |
          curl -L --out split-test https://github.com/mtsmfm/split-test/releases/download/v0.3.0/split-test-x86_64-unknown-linux-gnu
          chmod +x split-test
      - run: bin/rspec --format progress --format RspecJunitFormatter --out report/rspec-${{ matrix.node_index }}.xml $(./split-test --junit-xml-report-dir report-tmp --node-index ${{ matrix.node_index }} --node-total 3 --tests-glob 'spec/**/*_spec.rb' --debug)
      - uses: actions/upload-artifact@v2
        with:
          name: test-report
          path: report
          if-no-files-found: error
        # Upload test-report on main branch only to avoid conflicting test report
        if: github.ref == 'refs/heads/main'

You can find working example on https://github.com/mtsmfm/split-test-example

Note

split-test is inspired by circleci tests split command.

split-test assumes test report has additional attribute file or filepath.

<testsuites>
<testsuite>
<testcase classname="spec.0_spec" name="0 is expected to eq 0" file="./spec/0_spec.rb" time="0.000373"></testcase>
</testsuite>
</testsuites>

To be exact it isn't JUnit standard.

You might also like...
A tool to calculate mean and standard deviation from multiple tests using PageSpeed Insights API.

psi-sample PSI Test tool is an open source tool to assist web developers that runs Page Speed Insight test manually! Installing To install the psi-tes

Black-box integration tests for your REST API using the Rust and its test framework

restest Black-box integration test for REST APIs in Rust. This crate provides the [assert_api] macro that allows to declaratively test, given a certai

Parser and test runner for testing compatable common Ethereum full node tests against Polygon Zero's EVM.

EVM Test Parses and runs compatible common Ethereum tests from ethereum/tests against Polygon Zero's EVM. Note: This repo is currently very early in d

Single-reader, multi-writer & single-reader, multi-verifier; broadcasts reads to multiple writeable destinations in parallel

Bus Writer This Rust crate provides a generic single-reader, multi-writer, with support for callbacks for monitoring progress. It also provides a gene

Download a file using multiple threads in parallel for faster download speeds.

multidl Download a file using multiple threads in parallel for faster download speeds. Uses 0 external dependencies. Usage Usage: multidl [--help] ADD

A PAM module that runs multiple other PAM modules in parallel, succeeding as long as one of them succeeds.

PAM Any A PAM module that runs multiple other PAM modules in parallel, succeeding as long as one of them succeeds. Development I created a VM to test

Android Device Pool - A tool to run device tests against a pool of devices.

adp (Android Device Pool) What is this? A tool to run device tests against a pool of devices. It will 'checkout' a device to run your tests against an

A simple, opinionated way to run containers for tests in your Rust project.

rustainers rustainers is a simple, opinionated way to run containers for tests. TLDR More information about this crate can be found in the crate docum

1 library and 2 binary crates to run SSH/SCP commands on a
1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel

massh 1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel. The binary crates are CLI and GUI "frontends" for the li

Rust bindings to Linux Control Groups (cgroups)

cgroups-fs Native Rust library for managing Linux Control Groups (cgroups). This crate, curently, only supports the original, V1 hierarchy. You are we

A simple web-app allowing you to batch archive groups of repositories from a given organization

ice-repos My goal here is to build a simple web-app allowing you to batch archive groups of repositories from a given organization, using Rust+Yew. As

Small program which groups images based on the GPS position.

gps-cluster This small program will take some pictures in input, and based on the metadata on every image, it will group them by their GPS position, i

Library for Unix users and groups in Rust.

uzers-rs Adoption and continuation of the unmaintained ogham/rust-users crate. Big shout-out to its creator Benjamin Sago. This is a library for acces

Library for Unix users and groups in Rust.

uzers-rs Adoption and continuation of the unmaintained ogham/rust-users crate. Big shout-out to its creator Benjamin Sago. This is a library for acces

atttribute macro for running a flaky test multiple times

flaky_test This attribute macro will register and run a test 3 times, erroring only if all three times fail. Useful for situations when a test is flak

Collects accurate files while running in parallel through directories. (Simple, Fast, Powerful)

collectfiles Collects accurate files while running in parallel through directories. (Simple, Fast, Powerful) | Docs | Latest Note | [dependencies] col

Parallel iteration of FASTA/FASTQ files, for when sequence order doesn't matter but speed does

Rust-parallelfastx A truly parallel parser for FASTA/FASTQ files. Principle The input file is memory-mapped then virtually split into N chunks. Each c

Coppers is a custom test harnass for Rust that measures the energy usage of your test suite.
Coppers is a custom test harnass for Rust that measures the energy usage of your test suite.

Coppers Coppers is a test harness for Rust that can measure the evolution of power consumptions of a Rust program between different versions with the

A simple tool to test sqlx with postgres. It will automatically create a database and drop it after the test.

sqlx-db-tester This a tool to test sqlx with postgres. It only supports tokio runtime at this moment. How to use it You should first create a TestDb d

Comments
  • actions returns -out as an ambiguous option which result to exit code 2

    actions returns -out as an ambiguous option which result to exit code 2

    We started to get the following exit status when running split test and caused CI to not run. It is possible that GitHub changed ubuntu-latest library a bit and caused some update to the curl. It would be great if the team can check it out.

    Run curl -L --out split-test https://github.com/mtsmfm/split-test/releases/download/v1.0.0/split-test-x86_64-unknown-linux-gnu
    [10](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:11)
    curl: option --out: is ambiguous
    [11](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:12)
    curl: try 'curl --help' or 'curl --manual' for more information
    [12](https://github.com/yasslab/railsguides.jp_web/actions/runs/3653925624/jobs/6174061904#step:11:13)
    Error: Process completed with exit code 2.
    

    I do not have public repo available to share the fix and where it caused the issue but can share some screenshots. Hope it helps!

    Screen Shot 2022-12-09 at 12 53 19 Screen Shot 2022-12-09 at 12 53 45
    opened by nanophate 1
Owner
Fumiaki MATSUSHIMA
Fumiaki MATSUSHIMA
bottom encodes UTF-8 text into a sequence comprised of bottom emoji

bottom encodes UTF-8 text into a sequence comprised of bottom emoji (with , sprinkled in for good measure) followed by ????. It can encode any valid UTF-8 - being a bottom transcends language, after all - and decode back into UTF-8.

Bottom Software Foundation 345 Dec 30, 2022
Find and replace text in source files

Ruplacer Find and replace text in source files: $ ruplacer old new src/ Patching src/a_dir/sub/foo.txt -- old is everywhere, old is old ++ new is ever

Tanker 331 Dec 28, 2022
Find files (ff) by name, fast!

Find Files (ff) Find Files (ff) utility recursively searches the files whose names match the specified RegExp pattern in the provided directory (defau

Vishal Telangre 310 Dec 29, 2022
⏮ ⏯ ⏭ A Rust library to easily read forwards, backwards or randomly through the lines of huge files.

EasyReader The main goal of this library is to allow long navigations through the lines of large files, freely moving forwards and backwards or gettin

Michele Federici 81 Dec 6, 2022
Difftastic is an experimental structured diff tool that compares files based on their syntax.

Difftastic is an experimental structured diff tool that compares files based on their syntax.

Wilfred Hughes 13.9k Jan 2, 2023
Subtitles-rs - Use SRT subtitle files to study foreign languages

Rust subtitle utilities Are you looking for substudy? Try here. (substudy has been merged into the subtitles-rs project.) This repository contains a n

Eric Kidd 268 Dec 29, 2022
A command line tool for renaming your ipa files quickly and easily.

ipa_renamer A command line tool for renaming your ipa files quickly and easily. Usage ipa_renamer 0.0.1 A command line tool for renaming your ipa file

Noah Hsu 31 Dec 31, 2022
ABQ is a universal test runner that runs test suites in parallel. It’s the best tool for splitting test suites into parallel jobs locally or on CI

?? abq.build   ?? @rwx_research   ?? discord   ?? documentation ABQ is a universal test runner that runs test suites in parallel. It’s the best tool f

RWX 13 Apr 7, 2023
The main repo for manta relay nodes and manta parachain nodes

Manta Manta is a privacy preserving DeFi stack on Polkadot/Substrate. The code currently hasn't been properly security audited (work in progress), use

Manta Network 99 Dec 14, 2022