This series of commits introduces the use of arbitrary and honggfuzz-rs in driving property tests, as discussed briefly by @Shnatsel and myself in #2. The code is now split between the library bits -- the Op, Arbitrary definition etc -- and the interpreter loop, now in the fuzz target called hash_map. If you have honggfuzz-rs installed you can run the fuzz target like:

> cargo hfuzz run hash_map

Installation instructions for honggfuzz can be found at the above project link.

In terms of finding bugs, this was effective at finding the panic crash resulting from @llogiq's introduction of Op::Reserve. The problem, as discussed in 9596448, is that we can make very large capacity requests of the HashMap and it will be unable to allocate memory to meet that request, causing a panic. HashMap does not publish its overhead per pair so we cannot determine if a reservation will fail ahead of time. The check we have in place mostly guards against this but fuzzing managed to turn up truly large reservation failures. It was neat.

The downside to this approach is there's no tooling support for it: no shrinking, failure reports are given as the byte inputs. Dropping into a debugger works well enough but it's not the best.

This commit updates the README to reflect the changes made to use model-based fuzzing, as opposed to straight QuickCheck. This resolves

Signed-off-by: Brian L. Troutwine [email protected]

Now that clusterfuzz is open-source we can resolve a long-standing issue of bughunt-rust: not having enough CPU time. We should port tests to be runnable in clusterfuzz, setup a build pipeline etc.

Specific tickets to follow.

With this series of commits I've now enabled clusterfuzz support for the project, running at https://bughunt.appspot.com/. The use of AFL is now removed in favor of cargo-fuzz/libFuzzer for easy compatibility with clusterfuzz. This has made the README instructions more straightforward, which hopefully will help new contributors to the project.

For reasons related, I think, to the travis org/com distinction the branch push builds -- done on org -- fail while the branch builds -- done on com -- succeed. This is a result of the secrets being pegged to com. I'll see after this is merged if I can sort out which domain the project will always run on.

The clusterfuzz setup consumes actual funds, though for now we're operating out of credits. This project will need to sort out funding at some point, if the clusterfuzz setup proves worth keeping.

This commit introduces three new operations against HashMap: ShrinkToFit, CheckCapacity and Clear. The consequences of these operations are documented in the commit itself. I've also updated the travis CI runner to run more tests of larger size by default. This will need to be dialed backward as the number of models grow.

Signed-off-by: Brian L. Troutwine [email protected]

This commit introduces a basic Travis CI configuration into the project. QC tests will just fly right on by but our clippy and formatting warnings will be checked. The developer should still run their models on their local machines for long periods.

Signed-off-by: Brian L. Troutwine [email protected]

Any bug searching technique lives and dies by its ability to find, well, bugs. To judge how well our approach is doing in this project we need a list of known defects. That is:

• details / bug issue number
• [optional] method to reproduce
• impacted version(s) of Rust

From this we can start to build a table of Time to Discovery for the given issue.

As detailed in this post hunting for bugs is hampered right now because of Rust's panic strategy around allocation failures. We should introduce techniques to avoid these panics, related issues of edge-cases being masked by OOM panics.

Opening an issue because github doesn't provide a better communication mechanism. Hopefully some of these points are actionable.

Downside is, the std data structures we're testing don't have any sanitizers turned on etc on account of the project is run against the usual Rust releases.

libdislocator can catch many (but not all) memory errors for black-box binaries, assuming they use the system allocator.

Rust sanitizer integration lists steps to rebuild stdlib with sanitizer support, but calls it "better backtraces". We might want to get in touch and check how exactly sanitizers can be enabled for stdlib primitives.

This project will happily take fuzz code. Or, if someone could contrive to combine a fuzzer with a shrinking step this project will jump on that in a heartbeat.

Both AFL and libfuzzer shrink the inputs as they go along. They have Rust integration as afl.rs and cargo-fuzz respectively. Both have a one-shot testcase minimization command, which doesn't usually do a great job. However, AFL also comes with a crash exploration mode lets you explore alternative crashing inputs and minimize the testcase to your heart's content.

Also, Rust QuickCheck README mentions proptest as an alternative to QuickCheck that improves on shrinking and links to some comparisons of the two.