ABQ is a universal test runner that runs test suites in parallel. It’s the best tool for splitting test suites into parallel jobs locally or on CI

Related tags

Command-line abq
Overview

abq

🌐 abq.build   🐦 @rwx_research   πŸ’¬ discord   πŸ“š documentation

ABQ is a universal test runner that runs test suites in parallel. It’s the best tool for splitting test suites into parallel jobs locally or on CI. ABQ is implemented in Rust with bindings available for several test frameworks.

To use ABQ, check out the documentation on getting started.

Demo

Here's a demo of running an RSpec test suite, and then using abq to run it in parallel. ABQ invokes any test command passed to it, so you can continue using your native test framework CLI with any arguments it supports.

abq-demo.svg

Test Frameworks

:octocat: rwx-research/rspec-abq   :octocat: rwx-research/jest-abq   :octocat: rwx-research/playwright-abq   :octocat: rwx-research/pytest-abq

ABQ is currently compatible with

Open source sponsorship is available for anybody interested in implementing bindings for other test frameworks. Get in touch on discord or by emailing [email protected]

Development

For notes on working on abq, see development.md

Comments
  • Revive manylinux builds, and drop mimalloc

    Revive manylinux builds, and drop mimalloc

    Builds development abq binaries on a succifiently-old glibc system, suitable for dynamic linking of newer glibcs on EC2 and Github Actions.

    Drops configuration of mimalloc as the global allocator. Instead, prefer the default system or libc allocator. By default, rustc will defer to the system allocator.

    opened by ayazhafiz 3
  • Persist results in s3

    Persist results in s3

    Implements persistence of results to a remote persistence task.

    Results are only persisted to a remote when

    • there are no pending results-persistence tasks left in the queue
    • the initial manifest is already complete

    This is to avoid starvation by way of continuously persisting to the remote.

    opened by ayazhafiz 2
  • Cancel runs if no progress is made in the manifest

    Cancel runs if no progress is made in the manifest

    Presently, active runs can reach a state where all associated workers die, and no progress is made on the test suite, but the test suite sticks around in the queue memory. Since such runs may not be returned to at all, we'd like to diminish the amount of pressure they might place on a running queue.

    This series of patches addresses the problem by running a job every hour that checks whether test runs have had any progress in their manifest. If either

    • there is no manifest associated with the run after an hour, or
    • no more items have been popped off the manifest since the last time progress was checked

    then the run will be cancelled. If progress has been made, or the run was already done, the run is left untouched. If progress has been made and the run is not yet done, a job to check the progress again later is re-enqueued.

    In the future, we'll likely want to adjust the behavior to not outright cancel a job, but to admit some way to re-launch the job from the last failure state.

    opened by ayazhafiz 2
  • Integrate dumping manifests to remote as part of manifest persistence

    Integrate dumping manifests to remote as part of manifest persistence

    Now, when a queue persists a manifest to local disk after the initial manifest has been handed out, it will also persist that manifest to a remote (if any).

    Needs #4 first.

    opened by ayazhafiz 2
  • Scaffold S3 remote persistence interface

    Scaffold S3 remote persistence interface

    The new RemotePersistence trait provides an interface for loading and storing files on in remote location. The present implementation includes one implementation of this interface for AWS's S3 service. The RemotePersistence trait will optionally back the file-based persisters of results and manifests.

    Support for S3 is added as a feature, so that folks self-compiling ABQ do not have to compile-in S3. By default, RWX will build ABQ binaries with support for S3.

    Note that the implementation adds a dependency on AWS's Rust SDK, which is presently a "developer preview" and not stable. However, anecdotal evidence suggests (1, 2) the dependency is usable. We also do not have much other choice with regards to AWS S3 clients.

    opened by ayazhafiz 2
  • Build releases as musl-only

    Build releases as musl-only

    ABQ queues can suffer high memory utilization when there are a high number of concurrent requests, which have large payload sizes, presumably due to the ephemeral nature of allocations and increased heap fragmentation. The best behavior for a synthetic, but pathological workload appears to be with a statically-linked musl build, using the default musl allocator.

    image image image image
    opened by ayazhafiz 1
  • manylinux build release jemalloc

    manylinux build release jemalloc

    • Build development binaries on manylinux containers
    • Drop all references to mimalloc
    • Build releases against manylinux builds as well
    • Install wget for typos as well
    • Update artifacts_dir
    • Add a missing comment
    • Use npm in cli tests over yarn
    • Fix doc comment
    • Build a release build of manylinux
    • Build jemalloc release
    opened by ayazhafiz 1
  • manylinux build release

    manylinux build release

    • Build development binaries on manylinux containers
    • Drop all references to mimalloc
    • Build releases against manylinux builds as well
    • Install wget for typos as well
    • Update artifacts_dir
    • Add a missing comment
    • Use npm in cli tests over yarn
    • Fix doc comment
    • Build a release build of manylinux
    opened by ayazhafiz 1
  • Add no-op and custom remote persisters, and their CLI interfaces

    Add no-op and custom remote persisters, and their CLI interfaces

    The CLI now supports two modes for remote persistence - either via a custom executable, or via configuring S3 for storage. If neither is specified, remote persistence will not be used.

    See individual commits for details. In a follow-up, these paths will begin to be exercised. For now, this is stubbed out to get a feel for the interface.

    opened by ayazhafiz 1
  • Dependency Dashboard

    Dependency Dashboard

    This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

    Awaiting Schedule

    These updates are awaiting their schedule. Click on a checkbox to get an update now.

    • [ ] Update 8BitJonny/gh-get-current-pr digest to 2215326
    • [ ] Update all non-major dependencies (clap, futures, mockito, moka, reqwest, ruby, rustls, serde, serde_derive, serde_json, syn, tempfile, tokio, tokio-rustls)
    • [ ] Update Rust crate serial_test to v2

    Detected dependencies

    bundler
    Gemfile
    • rspec undefined
    • rspec-abq undefined
    cargo
    Cargo.toml
    • serde 1.0.158
    • serde_derive 1.0.158
    • serde_json 1.0.94
    • rustls 0.20.8
    • rustls-pemfile 1.0.2
    • tokio 1.26.0
    • tokio-rustls 0.23.4
    • futures 0.3.27
    • parking_lot 0.12.1
    • moka 0.10.0
    • fnv 1.0.7
    • signal-hook 0.3.15
    • signal-hook-tokio 0.3.1
    • rand 0.8.5
    • rand_chacha 0.3.1
    • tracing 0.1.37
    • tracing-subscriber 0.3.16
    • tracing-appender 0.2.2
    • thiserror 1.0.40
    • anyhow 1.0.70
    • indoc 2.0.1
    • clap 4.1.13
    • atty 0.2.14
    • num_cpus 1.15.0
    • termcolor 1.2.0
    • indicatif 0.17.3
    • async-trait 0.1.68
    • static_assertions 1.1.0
    • derive_more 0.99.17
    • insta 1.29.0
    • paste 1.0.12
    • fs4 0.6.3
    • tempfile 3.4.0
    • tracing-test 0.2.4
    • ntest 0.9.0
    • serial_test 1.0.0
    • nix 0.26.2
    • regex 1.7.3
    • uuid 1.3.0
    crates/abq_cli/Cargo.toml
    • mimalloc 0.1.34
    crates/abq_hosted/Cargo.toml
    • url 2.3.1
    • reqwest 0.11.15
    • mockito 1.0.1
    crates/abq_queue/Cargo.toml
    crates/abq_reporting/Cargo.toml
    crates/abq_runners/generic_test_runner/Cargo.toml
    crates/abq_test_support/abq_reporting_test_utils/Cargo.toml
    crates/abq_test_support/abq_test_utils/Cargo.toml
    crates/abq_test_support/native_runner_simulator/Cargo.toml
    crates/abq_test_support/run_n_times/Cargo.toml
    • syn 2.0.10
    • quote 1.0.26
    crates/abq_test_support/with_protocol_version/Cargo.toml
    • syn 2.0.10
    • quote 1.0.26
    crates/abq_test_support/write_runner_number/Cargo.toml
    crates/abq_test_support/write_worker_number/Cargo.toml
    crates/abq_utils/Cargo.toml
    • flate2 1.0.25
    crates/abq_workers/Cargo.toml
    crates/reporters/dot/Cargo.toml
    crates/reporters/junit_xml/Cargo.toml
    • junit-report 0.8.2
    • strip-ansi-escapes 0.1.1
    crates/reporters/line/Cargo.toml
    crates/reporters/progress/Cargo.toml
    crates/reporters/rwx_v1_json/Cargo.toml
    github-actions
    .github/workflows/bigtest.yml
    • actions/checkout v3
    • actions/upload-artifact v3
    • actions/checkout v3
    • ruby/setup-ruby v1
    • actions/setup-node v3
    • actions/download-artifact v3
    • actions/checkout v3
    • ruby/setup-ruby v1
    • actions/setup-node v3
    • actions/download-artifact v3
    • actions/upload-artifact v3
    • actions/checkout v3
    • actions/download-artifact v3
    • actions/checkout v3
    • 8BitJonny/gh-get-current-pr v2.2.0@f4d16fd424adba1334de83b538f6cac7529d4683
    • thollander/actions-comment-pull-request v2.3.1@632cf9ce90574d125be56b5f3405cda41a84e2fd
    • actions/download-artifact v3
    .github/workflows/build_and_upload.yml
    • actions/checkout v3
    • actions/cache v3.3.1
    .github/workflows/test_and_package_development.yml
    • actions/checkout v3
    • actions/cache v3.3.1
    • actions-rs/cargo v1.0.3
    • actions-rs/cargo v1.0.3
    • actions/setup-node v3
    npm
    bigtest/benchmark_jest/package.json
    • jest 29.5.0
    • @jest/core 29.5.0
    • jest-config 29.5.0
    • jest-runner 29.5.0
    • jest-circus 29.5.0
    testdata/jest/npm-jest-project-with-failures/package.json
    • jest 29.5.0
    • @jest/core 29.5.0
    • jest-config 29.5.0
    • jest-runner 29.5.0
    • jest-circus 29.5.0
    testdata/jest/npm-jest-project/package.json
    • jest 29.5.0
    • @jest/core 29.5.0
    • jest-config 29.5.0
    • jest-runner 29.5.0
    • jest-circus 29.5.0
    ruby-version
    .ruby-version
    • ruby 3.2.1

    • [ ] Check this box to trigger a request for Renovate to run again on this repository
    opened by renovate[bot] 0
Releases(v1.3.2)
  • v1.3.2(Apr 3, 2023)

    ABQ 1.3.2 includes bug fixes and improved output for abq test and abq report.

    • Failure detail is provided in abq report output.
    • Retried tests are listed in summary output.
    • Runner number is included in failed/retried test lists during abq test.
    • Worker number is included in failed/retried test lists during abq report.
    • Fix reporting of native runner failures that are later retried.
    • rwx-v1-json reporter counts native runner failures as otherErrors.
    • Manifest generation output suppressed when it's empty.
    • Document support for pytest.
    Source code(tar.gz)
    Source code(zip)
  • v1.3.1(Apr 3, 2023)

    ABQ 1.3.1 is a patch release with several bug fixes.

    • ABQ now defaults to passing-through output from a test framework when abq test uses a single runner.
    • Retry banners are output when a native test runner is re-launched for ABQ retries.
    • ABQ's help menu is improved and usage patterns are clarified. References to the now-removed are elided.
    • abq report now streams large test results messages as smaller chunks.
    • The maximum-message-size restriction is lifted from intra-ABQ communication.
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Apr 3, 2023)

    ABQ is a universal test runner for parallelizing your test suite. ABQ needs no custom configuration to integrate with your test suite, making integration as easy as installing an ABQ plugin and prepending abq test -- to your existing test command.

    ABQ works locally and on all CI providers. It’s the best tool for splitting test suites into parallel jobs in CI.

    Get started with ABQ at rwx.com/abq.

    Highlights of ABQ include:

    • Optimal parallelization of your test suite using ABQ's message queue strategy of distributing tests.
      • ABQ's distribution strategy is designed to minimize network overhead, even if the message queue is far away.
    • Parallelization of a test suite across any number of machines, with abq test --worker <worker number>.
    • Running multiple parallel test processes in one invocation of abq test via the -n CLI flag.
    • Automated retries of tests via the --retries flag to abq test.
    • Manual retries of abq test nodes, that retry only the tests they ran the first time around.
    • Test result reporting how you want it - ABQ won't interfere with your test framework's configured reporters.
      • ABQ also comes with standardized test result reporters, like --reporter dot, that can be used to enhance the reporters of your underlying test framework.
      • ABQ supports several structured reporters, like JUnit XML and RWX v1 JSON, so that you can export your test results in a machine-readable format without hassle.
    • Aggregation of test results from all abq test nodes via abq report.
    • ABQ is distributed as a standalone binary with all dependencies statically linked. All you need to run abq test, or self-host an ABQ queue, is to download the binary for your platform.
    • Seamless integration with Captain.

    This version of ABQ supports ABQ native runner protocol 0.2. 1.3.0 is the first public release of ABQ.

    Learn more about using ABQ at the docs.

    Source code(tar.gz)
    Source code(zip)
Owner
RWX
RWX builds tools that optimize build and test performance, reliability, and developer experience
RWX
Bam Error Stats Tool (best): analysis of error types in aligned reads.

best Bam Error Stats Tool (best): analysis of error types in aligned reads. best is used to assess the quality of reads after aligning them to a refer

Google 54 Jan 3, 2023
Wikit - A universal dictionary

Wikit - A universal dictionary What is it? To be short, Wikit is a tool which can (fully, may be in future) render and create dictionary file in MDX/M

bugnofree 120 Dec 3, 2022
a universal meta-transliterator that can decipher arbitrary encoding schemas, built in pure Rust

transliterati a universal meta-transliterator that can decipher arbitrary encoding schemas, built in pure Rust what does it do? You give it this: Π‘Π°Ρ€Π»

Catherine Koshka 7 Dec 21, 2022
Universal Windows library for discovering common render engines functions. Supports DirectX9 (D3D9), DirectX10 (D3D10), DirectX11 (D3D11), DirectX12 (D3D12).

Shroud Universal library for discovering common render engines functions. Supports DirectX9 (D3D9), DirectX10 (D3D10), DirectX11 (D3D11), DirectX12 (D

Chase 6 Dec 10, 2022
Truly universal encoding detector in pure Rust - port of Python version

Charset Normalizer A library that helps you read text from an unknown charset encoding. Motivated by original Python version of charset-normalizer, I'

Nikolay Yarovoy 29 Oct 9, 2023
πŸ—½ Universal Node Package Manager

?? NY Universal Node Package Manager node β€’ yarn β€’ pnpm Features Universal - Picks the right package manager for you based on the lockfile in your fol

Kris Kaczor 46 Oct 12, 2023
A universal load testing framework for Rust, with real-time tui support.

rlt A Rust Load Testing framework with real-time tui support. rlt provides a simple way to create load test tools in Rust. It is designed to be a univ

Wenxuan 129 Jul 20, 2024
sg = Steam Guard, access sg verification code locally.

Steam Guard CLI Access your Steam Guard token locally. Credit https://github.com/steevp/UpdogFarmer/blob/master/app/src/main/java/com/steevsapps/idled

Vanilla's Lab 7 Nov 10, 2022
Mac App/CLI that automatically adds project logos to your locally cloned GitHub repos

Download the app Automatically adds project logos to your locally cloned GitHub repos. Youtube Video This repository contains the source code for the

Sam Denty 365 Dec 25, 2022
Download pdbs from symbol servers and cache locally, parse symbol paths from env vars

symsrv This crate lets you download and cache pdb files from symbol servers, according to the rules from the _NT_SYMBOL_PATH environment variable. It

Markus Stange 6 Sep 15, 2022
Attempt to summarize text from `stdin`, using a large language model (locally and offline), to `stdout`

summarize-cli Attempt to summarize text from stdin, using a large language model (locally and offline), to stdout. cargo build --release target/releas

null 4 Aug 23, 2023
CLI tool that make it easier to perform multiple lighthouse runs towards a single target and output the result in a plotable format.

Lighthouse Aggregator CLI tool that make it easier to perform multiple lighthouse runs towards a single target and output the result in a "plotable" f

Polestar 1 Jan 12, 2022
being Ariel's best friend!

Sebastian seΒ·basΒ·tian - sΙͺˈbΓ¦stΙͺΙ™n A simple tool used to access UniMi services -- mainly ariel, but not only -- via CLI. Important: state First of all

null 2 Sep 2, 2022
Solutions for exact and optimized best housing chains in BDO using popjumppush and MIP.

Work in progress. About This project is an implementation of the pop_jump_push algorithm. It uses graph data from the MMORPG Black Desert Online's tow

Thell 'Bo' Fowler 3 May 2, 2023
Best-effort attempt at rewriting Reticulum in Rust.

reticulum-rs WIP rewrite of Reticulum in Rust explicitly targeted at the ESP32 (no tokio, avoids memory-inefficient data structures. etc). Not ready f

Ellen Poe 8 Sep 25, 2023
An implementation of ksplang, the best programming language

A stack-based programming language with 33 instructions, independently designed by 33 people. Introduction This is an implementation of ksplang, the b

KSP MFF UK 8 Feb 24, 2024
comfy is a flexible command script manager / runner written in Rust

comfy is a cross-platform command script manager / runner tool, which allows you to run commands in the command line itself, but being these predefined in a portable and universal .comfy file.

daCoUSB 17 Nov 12, 2021
Shellcode Runner/Injector in Rust using NTDLL functions directly with the ntapi Library

RustSCRunner Shellcode Runner/Injector in Rust using NTDLL functions directly with the ntapi Library. Surprisingly this is my first ever Rust project

null 86 Dec 18, 2021
A wayland native, highly customizable runner.

anyrun A wayland native krunner-like runner, made with customizability in mind. Features Style customizability with GTK+ CSS More info in Styling Can

null 18 Jan 22, 2023