Builds development abq binaries on a succifiently-old glibc system, suitable for dynamic linking of newer glibcs on EC2 and Github Actions.

Drops configuration of mimalloc as the global allocator. Instead, prefer the default system or libc allocator. By default, rustc will defer to the system allocator.

Implements persistence of results to a remote persistence task.

Results are only persisted to a remote when

• there are no pending results-persistence tasks left in the queue
• the initial manifest is already complete

This is to avoid starvation by way of continuously persisting to the remote.

Presently, active runs can reach a state where all associated workers die, and no progress is made on the test suite, but the test suite sticks around in the queue memory. Since such runs may not be returned to at all, we'd like to diminish the amount of pressure they might place on a running queue.

This series of patches addresses the problem by running a job every hour that checks whether test runs have had any progress in their manifest. If either

• there is no manifest associated with the run after an hour, or
• no more items have been popped off the manifest since the last time progress was checked

then the run will be cancelled. If progress has been made, or the run was already done, the run is left untouched. If progress has been made and the run is not yet done, a job to check the progress again later is re-enqueued.

In the future, we'll likely want to adjust the behavior to not outright cancel a job, but to admit some way to re-launch the job from the last failure state.

Now, when a queue persists a manifest to local disk after the initial manifest has been handed out, it will also persist that manifest to a remote (if any).

Needs #4 first.

The new RemotePersistence trait provides an interface for loading and storing files on in remote location. The present implementation includes one implementation of this interface for AWS's S3 service. The RemotePersistence trait will optionally back the file-based persisters of results and manifests.

Support for S3 is added as a feature, so that folks self-compiling ABQ do not have to compile-in S3. By default, RWX will build ABQ binaries with support for S3.

Note that the implementation adds a dependency on AWS's Rust SDK, which is presently a "developer preview" and not stable. However, anecdotal evidence suggests (1, 2) the dependency is usable. We also do not have much other choice with regards to AWS S3 clients.

ABQ queues can suffer high memory utilization when there are a high number of concurrent requests, which have large payload sizes, presumably due to the ephemeral nature of allocations and increased heap fragmentation. The best behavior for a synthetic, but pathological workload appears to be with a statically-linked musl build, using the default musl allocator.

• glibc allocator

• jemalloc debug allocator

• jemalloc

• musl only allocator

• Build development binaries on manylinux containers
• Drop all references to mimalloc
• Build releases against manylinux builds as well
• Install wget for typos as well
• Update artifacts_dir
• Add a missing comment
• Use npm in cli tests over yarn
• Fix doc comment
• Build a release build of manylinux
• Build jemalloc release

• Build development binaries on manylinux containers
• Drop all references to mimalloc
• Build releases against manylinux builds as well
• Install wget for typos as well
• Update artifacts_dir
• Add a missing comment
• Use npm in cli tests over yarn
• Fix doc comment
• Build a release build of manylinux

The CLI now supports two modes for remote persistence - either via a custom executable, or via configuring S3 for storage. If neither is specified, remote persistence will not be used.

See individual commits for details. In a follow-up, these paths will begin to be exercised. For now, this is stubbed out to get a feel for the interface.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• [ ] Update 8BitJonny/gh-get-current-pr digest to 2215326
• [ ] Update all non-major dependencies (clap, futures, mockito, moka, reqwest, ruby, rustls, serde, serde_derive, serde_json, syn, tempfile, tokio, tokio-rustls)
• [ ] Update Rust crate serial_test to v2

Detected dependencies

• [ ] Check this box to trigger a request for Renovate to run again on this repository