Since slice::split gives an iterator of slices, I wanted to see if I could write zero copy with it. No degrading performance in the current benchmark.

Because slice::split gives slices without the delimiter I moved the delimiter printing into the closure handler.

Just wanna show a fellow Rustacean another style. Happy coding! 👍

Using a C library complicates the compilation, since it requires that the Clang compiler be present.

xxHash has already been implemented in Rust. If there is a good reason to not use this crate, then it would be great to provide precompiled binaries of huniq.

Hi, everywhere in the README it says and shows how huniq sorts too. But in the very first paragraph of the README file, right after the heading, there it says "Note that huniq does not sort the input, it just removes duplicates." Is this maybe a left-over sentence from an older version? thx

fixes #17

I haven't done the benchmarking yet. The speedups should show especially in repeated runs with -c for inputs with large amounts of unique lines.

Also includes small cleanups for Cargo.lock, and uses std::hash::BuildHasherDefault. If I misunderstood the purpose of the custom implementation of BuildDefaultHasher I can drop the commit, it just looked identical to me.

The title is what I set out to do but I set up my editor so it auto-formats the source and before I realized I made all my fixes based on rustfmt formatted code 😞

I decided to at least separate out the formatting change to its own commit. If you don't like it let me know.

The command fc-list : family | huniq could not produce the expected result as fc-list : family | sort -u, and I can confirm that the piped lines are terminated by 0x0a, as the example of echo -e "foo\nbar\nfoo\nbaz" | huniq.

This replaces the custom split_read_zerocopy with bstr's own minimally-copying record reader, which simplifies the code somewhat without sacrificing performance in any of my measurements (98% - 115% observed performance against baseline)

This also removes the sysconf dependency, which caused a compile error here on FreeBSD.

This also adds a --no-trailing-delimiter flag requested in #14, since this code needed to be moved anyway.

Also switch use of write with write_all, since the former can succeed with partial writes.

Documenting some problems I've had building this. In order of appearance.

OS: macOS Catalina 10.15.3

zsh: command not found: llvm-config

Fix: brew install llvm then export PATH="/usr/local/opt/llvm/bin:$PATH"

./vendor/xxhash/xxhash.h:663:10: fatal error: 'stdlib.h' file not found

Fix: Running xcode-select --install

benchmark.sh: line 86: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Fix: Use zsh ./benchmark.sh instead of bash ./benchmark.sh

uniq 2 huniq2-rust time: illegal option -- f
uniq 2 huniq2-rust usage: time [-lp] command.

Infinite loop. Fix: Run brew install gnu-time then export PATH="/usr/local/opt/gnu-time/libexec/gnubin:$PATH"

Benchmark Result

uniq 1 huniq2-rust 0.09 10324
uniq 1 huniq1-c++  0.76 10824
uniq 1 awk-sys     0.66 23216
uniq 1 shell       1.37 52812

uniq 2 huniq2-rust 0.07 10324
uniq 2 huniq1-c++  1.06 10196
uniq 2 awk-sys     1.19 23224
uniq 2 shell       3.29 101620

uniq 5 huniq2-rust 0.25 10324
uniq 5 huniq1-c++  2.69 11992
uniq 5 awk-sys     3.12 23252
uniq 5 shell       9.75 247944

uniq 10 huniq2-rust 0.28 10344
uniq 10 huniq1-c++  5.13 11200
uniq 10 awk-sys     6.02 23212
uniq 10 shell       20.89 491792

uniq 50 huniq2-rust 1.34 10324
uniq 50 huniq1-c++  24.28 11044
uniq 50 awk-sys     29.55 23216
uniq 50 shell       120.39 2411632

Didn't wanna wait for the rest of it.

Added an example GitHub action that will build and upload the compiled binary as a GitHub release - as seen here: https://github.com/Phasip/huniq/releases/tag/latest

The action executes on every push and can also be manually launched.

When I run printf "1" | huniq -d ",", it prints 1,. When the input doesn't even contain the delimiter, I'd expect huniq to not change the input. But when the input is something like printf "1," | huniq -d ",", then printing 1, makes sense. I don't imagine it'd be that hard to fix this, but it would be really useful. A flag like --no-trailing-delimiter would also work as well.

“The memchr crate provides heavily optimized routines for searching bytes” (quoting their README).

On my machine it improves performance of ~15%. Also it's a minimal change and IMHO it actually simplifies the code.

In other words, use %7d format for numbers.

That is, instead of printing

1 world
12345 hello

print

******1 hello
**12345 world

(I've replaced spaces with *s for clarity).

I've compared speed with AWK, and it's not so much quicker.

huniq

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | huniq | pv -l | wc -l
20.0M 0:00:30 [ 664k/s]
20007130

awk

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | awk '!a[$0]++{print}' | pv -l | wc -l
20.0M 0:00:32 [ 608k/s]
20008984

unique

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | unique | pv -l | wc -l
20.0M 0:00:38 [ 524k/s]
20006340

quniq

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | quniq -max-workers 10 | pv -l | wc -l
20.0M 0:00:45 [ 436k/s]
20013324

runiq

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | runiq - | pv -l | wc -l
20.0M 0:00:35 [ 571k/s]
20007072

perl

$ echo | pee "seq 20000000" "seq 20000000" "seq 20000000" | perl -e 'while(<>){if(!$s{$_}){print $_;$|=1;$s{$_}=1;}}' | pv -l | wc -l
20.0M 0:00:55 [ 363k/s]
20008104

It's only 2 seconds quicker than awk when processing 20 millions lines.

Is there any way to process the lines even quicker?

Requirments

• The benchmarks should be entirely written in rust.

• The benchmarks should be portable and not rely on the presence of platform defined dictionary files.

• The benchmarks should have the ability to be run with specific parameters

  1. Number of input lines
  2. Fraction of duplicates
  3. Distribution of input line length
  4. Char set (binary/text)

• The benchmarks should still be able to run against all the preexisting commands (sort|uniq).

Design

A CLI application should be written that produces a set of random tokens according to the parameters specified on the CLI:

genbench --charset ascii/binary --delim CHAR --number NUM --duplicates PERCENTAGE --short LEN --long LEN

The short/long parameters each indicate the 90% percentile of string lengths, using a gaussian distribution.

For the actual benchmark we should write a benchmark executor that runs each of the implementations with a variety of parameters handed to genbench.

Tests

We can reuse the same strategy for testing by generating test data with genbench and then comparing the output of the full huniq and a super naive, unoptimized huniq implementation. We should specifically make sure, that buffer growing is tested (supply some very long, >20kb strings).