This project now lives on in a rewrite at https://gitlab.redox-os.org/redox-os/parallel

Last update: Nov 20, 2022

Overview

MIT/Rust Parallel: A Command-line CPU Load Balancer Written in Rust

This is an attempt at recreating the functionality of GNU Parallel, a work-stealer for the command-line, in Rust under a MIT license. The end goal will be to support much of the functionality of GNU Parallel and then to extend the functionality further for the next generation of command-line utilities written in Rust. While functionality is important, with the application being developed in Rust, the goal is to also be as fast and efficient as possible.

Note

See the to-do list for features and improvements that have yet to be done. If you want to contribute, pull requests are welcome. If you have an idea for improvement which isn't listed in the to-do list, feel free to email me and I will consider implementing that idea.

Benchmark Comparison to GNU Parallel

Note: Parallel in these benchmarks is compiled with MUSL instead of glibc. This is highly recommended as it reduces memory consumption by half and doubles performance.

Printing 1 to 10,000 in parallel

GNU Parallel

~/D/parallel (master) $ seq 1 10000 | time -v /usr/bin/parallel echo > /dev/null
    User time (seconds): 194.73
    System time (seconds): 66.49
    Percent of CPU this job got: 230%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 1:53.08
    Maximum resident set size (kbytes): 16140

Rust Parallel (Built with MUSL target)

~/D/parallel (master) $ seq 1 10000 | time -v target/release/x86_64-unknown-linux-musl/parallel echo > /dev/null
    User time (seconds): 0.40
	System time (seconds): 2.53
	Percent of CPU this job got: 97%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.01
    Maximum resident set size (kbytes): 1768

Cat the contents of every binary in /usr/bin

GNU Parallel

~/D/parallel (master) $ time -v /usr/bin/parallel cat ::: /usr/bin/* > /dev/null
    User time (seconds): 71.71
    System time (seconds): 27.67
    Percent of CPU this job got: 222%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:44.62
    Maximum resident set size (kbytes): 17576

Rust Parallel (Built with MUSL target)

~/D/parallel (master) $ time -v target/release/x86_64-unknown-linux-musl/release/parallel cat ::: /usr/bin/* > /dev/null
    User time (seconds): 1.07
	System time (seconds): 4.40
	Percent of CPU this job got: 191%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.86
    Maximum resident set size (kbytes): 1844

Logging echo ::: $(seq 1 1000)

GNU Parallel

~/D/parallel (master) $ time -v /usr/bin/parallel --joblog log echo ::: $(seq 1 1000) > /dev/null
    User time (seconds): 21.27
    System time (seconds): 7.44
    Percent of CPU this job got: 238%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.05
    Maximum resident set size (kbytes): 16624

Rust Parallel (Built with MUSL target)

~/D/parallel (master) $ time -v target/x86_64-unknown-linux-musl/release/parallel --joblog log echo ::: $(seq 1 1000) > /dev/null
    User time (seconds): 0.02
    System time (seconds): 0.28
    Percent of CPU this job got: 85%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.36
    Maximum resident set size (kbytes): 1768

Syntax Examples

The following syntax is supported:

parallel 'echo {}' ::: *                          // {} will be replaced with each input found.
parallel echo ::: *                               // If no placeholders are used, it is automatically assumed.
parallel echo :::: list1 list2 list3              // Read newline-delimited arguments stored in files.
parallel echo ::: arg1 ::::+ list :::+ arg2       // Interchangeably read arguments from the command line and files.
parallel echo ::: 1 2 3 ::: A B C ::: D E F       // Permutate the inputs.
parallel echo '{} {1} {2} {3.}' ::: 1 2 file.mkv  // {N} tokens are replaced by the Nth input argument
parallel ::: "echo 1" "echo 2" "echo 3"           // If no command is supplied, the input arguments become commands.
parallel 'cd {}; echo Directory: {}; echo - {}'   // Commands may be chained in the platform\'s shell.
seq 1 10 | parallel 'echo {}'                     // If no input arguments are supplied, stdin will be read.
seq 1 10 | parallel --pipe cat                    // Piping arguments to the standard input of the given command.
#!/usr/bin/parallel --shebang echo                // Ability to use within a shebang line.

Manual

Parallel parallelizes otherwise non-parallel command-line tasks. When there are a number of commands that need to be executed, which may be executed in parallel, the Parallel application will evenly distribute tasks to all available CPU cores. There are three basic methods for how commands are supplied:

A COMMAND may be defined, followed by an which denotes that all following arguments will be used as INPUTS for the command.
If no COMMAND is provided, then the INPUTS will be interpreted as COMMANDS.
If no INPUTS are provided, then standard input will be read for INPUTS.

Parallel groups the standard output and error of each child process so that outputs are printed in the order that they are given, as if the tasks were executed serially in a traditional for loop. In addition, commands are executed in the platform's preferred shell by default, which is sh -c on Unix systems, and cmd /C on Windows. This comes at a performance cost, so it can be disabled with the --no-shell option.

INPUT MODES

Input modes are used to determine whether the following inputs are files that contain inputs or inputs themselves. Files with inputs have each input stored on a separate line, and each line is considered an entire input.When there are multiple collected lists of inputs, each individual input list will be permutated together into a single list.

:::

Denotes that the input arguments that follow are input arguments. Additionally, those arguments will be collected into a new list.

:::+

Denotes that the input arguments that follow are input arguments. Additionally, those arguments will be added to the current list.

::::

Denotes that the input arguments that follow are files with inputs. Additionally, those arguments will be collected into a new list.

::::+

Denotes that the input arguments that follow are files with inputs. Additionally, those arguments will be added to the current list.

INPUT TOKENS

COMMANDs are typically formed the same way that you would normally in the shell, only that you will replace your input arguments with placeholder tokens like {}, {.}, {/}, {//} and {/.}. If no tokens are provided, it is inferred that the final argument in the command will be {}. These tokens will perform text manipulation on the inputs to mangle them in the way you like. Ideas for more tokens are welcome.

{}: Each occurrence will be replaced with the name of the input.
{.}: Each occurrence will be replaced with the input, with the extension removed.
{^abc...}: Each occurrence will be replaced with a custom suffix removed
{/}: Each occurrence will be replaced with the base name of the input.
{/.}: Each occurrence will be replaced with the base name of the input, with the extension removed.
{/^abc...}: Each occurrence will be replaced with the base name of the input, with a custom suffix removed.
{//}: Each occurrence will be replaced with the directory name of the input.
{%}: Each occurrence will be replaced with the slot number.
{#}: Each occurrence will be replaced with the job number.
{##}: Each occurrence will be replaced with the total number of jobs.
{N}: Where N is a number, display the associated job number.
{N.}: Will remove the extension from the Nth job.
{N^abc...}: Defines a custom suffix to remove from the Nth job, if found.
{N/}: Displays the base name (file name) of the Nth job.
{N//}: Displays the directory name of the Nth job.
{N/.}: Displays the base name of the Nth job with the extension removed.
{N/^abc...}: Displays the basename of the Nth job, with a custom suffix removed.

OPTIONS

Options may also be supplied to the program to change how the program operates:

--delay: Delays starting the next job for N amount of seconds, where the seconds can be fractional.
--dry-run: Prints the jobs that will be run to standard output, without running them.
--eta: Prints the estimated time to complete based on average runtime of running processes.
-h, --help: Prints the manual for the application (recommended to pipe it to less).
-j, --jobs: Defines the number of jobs/threads to run in parallel.
--joblog: Logs job statistics to a designated file as they are completed.
--joblog-8601: Writes the start time in the ISO 8601 format: YYYY-MM-DD hh:mm:ss
--memfree: Defines the minimum amount of memory available before starting the next job.
-n, --max-args: Groups up to a certain number of arguments together in the same command line.
--num-cpu-cores: Prints the number of CPU cores in the system and exits.
-p, --pipe: Instead of supplying arguments as arguments to child processes, instead supply the arguments directly to the standard input of each child process.
-q, --quote: Escapes the command argument supplied so that spaces, quotes, and slashes are retained.
-s, --silent, --quiet: Disables printing the standard output of running processes.
--shebang: Grants ability to utilize the parallel command as an interpreter via calling it within a shebang line.
--shellquote: Prints commands that will be executed, with the commands quoted.
--tmpdir: Defines the directory to use for temporary files
--timeout: If a command runs for longer than a specified number of seconds, it will be killed with a SIGKILL.
-v, --verbose: Prints information about running processes.
--version: Prints the current version of the application and it's dependencies.

Useful Examples

Transcoding FLAC music to Opus

ffmpeg is a highly useful application for converting music and videos. However, audio transcoding is limited to a a single core. If you have a large FLAC archive and you wanted to compress it into the efficient Opus codec, it would take forever with the fastest processor to complete, unless you were to take advantage of all cores in your CPU.

parallel 'ffmpeg -v 0 -i "{}" -c:a libopus -b:a 128k "{.}.opus"' ::: $(find -type f -name '*.flac')

Transcoding Videos to VP9

VP9 has one glaring flaw in regards to encoding: it can only use about three cores at any given point in time. If you have an eight core processor and a dozen or more episodes of a TV series to transcode, you can use the parallel program to run three jobs at the same time, provided you also have enough memory for that.

vp9_params="-c:v libvpx-vp9 -tile-columns 6 -frame-parallel 1 -rc_lookahead 25 -threads 4 -speed 1 -b:v 0 -crf 18"
opus_params="-c:a libopus -b:a 128k"
parallel -j 3 'ffmpeg -v 0 -i "{}" $vp9_params $opus_params -f webm "{.}.webm"' ::: $(find -type f -name '*.mkv')

Installation Instructions

There are a number of methods that you can use to install the application. I provide binary packages for AMD64 systems that are available for download:

Gentoo

I have a personal Gentoo layman overlay that provides this application for installation.

Arch Linux

A PKGBUILD is available for Arch Linux users from the AUR.

Everyone Else

rustup target add x86_64-unknown-linux-musl
wget https://github.com/mmstick/parallel/archive/master.zip
unzip master.zip
cd parallel-master
cargo build --release --target x86_64-unknown-linux-musl
sudo install target/x86_64-unknown-linux-musl/release/parallel /usr/local/bin/parallel

Comments

"command error: Resource temporarily unavailable (os error 11)" when processing a lot of files

When invoking Parallel 70,000 files with filenames piped on stdin, its memory usage grows continously up to 0,5Gb of RAM, and when it stops growing Parallel starts spamming the following line:

parallel: command error: Resource temporarily unavailable (os error 11)

Happens both with and without --no-shell argument. Command line to reproduce the issue:

find '/some/folder/with/tens/of/thousands/of/files' -type f | 'target/release/parallel' -j 6 cat '{}' > /dev/null

Parallel built from git with cargo build --release on Linux x86_64, rustc 1.11.0 (9b21dcd6a 2016-08-15)

opened by Shnatsel 16
Performs slower than GNU parallel when transparent huge pages are enabled

The benchmark in README.md as of version 0.5.0 claims time 0:04 for Rust parallel vs 0:54 for GNU parallel. However, this benchmark is misleading because the command line used is completely useless:

seq 1 10000 | time -v parallel echo > /dev/null would never print anything because it lacks parameter substitution. The correct command that actually does something would be seq 1 10000 | time -v parallel echo '{}' > /dev/null

On my machine Rust parallel measures 1:02 vs 0:33 for GNU parallel for the actually useful command.

opened by Shnatsel 15
parallel multiple orders of magnitude slower than gnu parallel

the test below are with 10 000 iterations, too few to see a gain with gnu parallel or rust parallel but it shows the difference between them

I also piped the output to less rather than /dev/null; around 4500 entries, less displays the message "waitng for input"

The test uses --pipe and -q, which perform as expected. I tested the output on a smaller input set

without parallelization

seq 10000 | time -v piper --global hi blue '\d+' red > /dev/null
Command being timed: "piper --global hi blue \d+ red" User time (seconds): 0.06 System time (seconds): 0.00 Percent of CPU this job got: 94% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 7884 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 1123 Voluntary context switches: 1 Involuntary context switches: 2 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

gnu parallel

seq 10000 | time -v parallel -q --pipe piper --global hi blue '\d+' red > /dev/null
Command being timed: "parallel -q --pipe piper --global hi blue \d+ red" User time (seconds): 0.15 System time (seconds): 0.01 Percent of CPU this job got: 94% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.18 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 16448 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 18527 Voluntary context switches: 137 Involuntary context switches: 31 Swaps: 0 File system inputs: 0 File system outputs: 1544 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

rust parallel, check Minor (reclaiming a frame) page faults

seq 10000 | time -v rust-parallel -q --pipe piper --global hi blue '\d+' red > /dev/null
Command being timed: "rust-parallel -q --pipe piper --global hi blue \d+ red" User time (seconds): 116.85 System time (seconds): 9.37 Percent of CPU this job got: 600% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.02 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 7024 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 9793578 Voluntary context switches: 71623 Involuntary context switches: 65275 Swaps: 0 File system inputs: 0 File system outputs: 176 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

hugepage info

cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

opened by nkh 13
Parallel doesn't play well with filenames containing a quote
Hello, it's me again with my filenames problems

I'm gonna take an example to explain the situation.

Say we have two folders with files to move.

mkdir folder1 folder2 touch "folder1/Let'sgo.txt" "folder1/Let'sgo-q1.txt" "folder1/Let'sgo-q2.txt" "folder1/Let'sgo-q3.txt" export file="Let'sgo"

I can move a file containing a quote with mv without any problem:

mv "folder1/${file}.txt" "folder2/."

However if I try to use parallel to do the same thing, I get an error:

seq 1 3 | parallel mv "folder1/${file}-q{}.txt" "folder2/." parallel: reading inputs from standard input mv: missing destination file operand after 'folder1/Letsgo-q1.txt folder2/.' Try 'mv --help' for more information. mv: missing destination file operand after 'folder1/Letsgo-q2.txt folder2/.' Try 'mv --help' for more information. mv: missing destination file operand after 'folder1/Letsgo-q3.txt folder2/.' Try 'mv --help' for more information.

It seems parallel remove the quote from my filename or misinterpret it, despite it being inside an enclosed variable.

If I escape the quote, it works:

seq 1 3 | parallel mv "folder1/Let\'sgo-q{}.txt" "folder2/."

but it's not really a workable solution because I feed filenames as a variable inside a bigger script. I think there shouldn't be any difference of behaviour between a non-parallelized function and a parallelized one.
opened by WyohKnott 7
Name improvement
Hi Michael

I have looked at rust-parallel. I am impressed with the speed and see there is a niche which is well filled by rust-parallel that GNU Parallel is likely never to fill.

However, calling the program 'parallel' may lead to problems you are not aware of. GNU Parallel was started before there was a UNIX tool called 'parallel'. Unfortunately another tool was later distributed as 'parallel', and that has lead to confusions that haunts us even today. The other tool was of course incompatible with GNU Parallel, but people thought they had installed GNU Parallel and were frustrated when the examples did not work. You can read more about that situation on https://www.gnu.org/software/parallel/history.html

The problem would be smaller if rust-parallel was 100% option compatible with GNU Parallel, and that it could serve as a drop-in replacement, but already now I can see rust-parallel does something different when running:

parallel echo ::: a b c :::+ 1 2 3 ::: d e f :::+ 4 5 6 parallel -n 2 echo ::: a b c d parallel --shellquote ::: '*#"'

and I find it highly improbable that you will ever implement the '{= perl expression =}' replacement string:

parallel echo '{= $_ > 2 ? $_=">2" : $_-- =}' ::: 1 2 3 4

Walking through GNU Parallel's tutorial with rust-parallel is also impossible.

On top of this the name clash makes it hard to have both tools installed: Maybe some users like the short startup time that rust-parallel provides for some tasks, while they like the flexibilty of GNU Parallel for others? By using the same name you are making it harder for them to use both GNU Parallel and rust-parallel.

Because of these issues I will be really happy if you change the name 'parallel', as I think in the long term you will avoid creating problems for users where none need be.

I understand you are making the tool to show off what you can do with Rust. Maybe you could show Rust directly in the name? Rust-parallel, rustpar, pa(r)rust?

Do not get me wrong. I am flattered by the imitation and I like competition: A lot of the ideas in GNU Parallel came from other tools, and I hope that rust-parallel will come up with new game-changing ideas, as that will benefit society.
opened by ole-tange 6
disable the notice of "parallel: reading inputs from standard input"

Hi developers,

Can I disable the notice of "parallel: reading inputs from standard input" when reading inputs from the pipe.

This notice makes rust_paralelel different with gun parallel, and it's not compatible with the Linux philosophy of do not do something extra more.

Version: MIT/Rust Parallel 0.11.0

Best regards, Wallace

opened by wavefancy 5

example comparing parallel with gnu-parallel is nowhere close to the times posted

seq 1 10000 | time -v rust-parallel echo > /dev/null

    Command being timed: "rust-parallel echo"
    User time (seconds): 2.03
    System time (seconds): 84.76
    Percent of CPU this job got: 407%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.29
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 27516
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 1220681
    Voluntary context switches: 80355
    Involuntary context switches: 21519
    Swaps: 0
    File system inputs: 0
    File system outputs: 176
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

duplicate invalid

opened by nkh 5

Buggy behaviour when working with accentuated characters

When working with filenames containing accentuated or weird characters, the {} is not replaced correctly.

For exemple: seq 1 10 | parallel echo "Québec-q{}.webm"

gives:

Québec-1}.webm
Québec-2}.webm
Québec-3}.webm
Québec-4}.webm
Québec-5}.webm
Québec-6}.webm
Québec-7}.webm
Québec-8}.webm
Québec-9}.webm
Québec-10}.webm

instead of Québec-q1.webm and so on.

If there's more non-ascii characters, the program segfault:

seq 1 10 | RUST_BACKTRACE=1 parallel echo "Œuf_échaudé-q{}.webm"

gives

parallel: reading inputs from standard input
thread 'main' panicked at 'byte index 18 is not a char boundary; it is inside 'é' (bytes 17..19) of `echo Œuf_échaudé-q{}.webm`', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/str/mod.rs:1771
stack backtrace:
   1:     0x560b9875899a - std::sys::imp::backtrace::tracing::imp::write::h9c41d2f69e5caabf
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
   2:     0x560b98757cee - std::panicking::default_hook::{{closure}}::hcc803c8663cda123
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:351
   3:     0x560b98756fdb - std::panicking::rust_panic_with_hook::hffbc74969c7b5d87
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:367
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:555
   4:     0x560b98756b3f - std::panicking::begin_panic::hc4c5d184a1e3fb7c
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:517
   5:     0x560b98756ac9 - std::panicking::begin_panic_fmt::h34f5b320b0f94559
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:501
   6:     0x560b98765956 - core::panicking::panic_fmt::h1016b85b51d1931f
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/panicking.rs:477
   7:     0x560b98766e4f - core::str::slice_error_fail::h02b27cb27b0f1c1d
                        at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/str/mod.rs:1771
   8:     0x560b9875069f - parallel::main::h6c96215d2b4b63a7
   9:     0x560b9875334f - main
  10:     0x7f1a66c82400 - __libc_start_main
  11:     0x560b9871d649 - _start
  12:                0x0 - <unknown>

opened by WyohKnott 4

Feature request: Command queue via file

Sample use case:

A file containing nuanced/complex commands (e.g.):

cat commands.list

echo "one" && du -hcs ./one echo "two" && du -hcs ./two echo "three" && du -hcs ./three

To be executed in parallel like so:

parallel < commands.list

This would reproduce a useful command-line behavior similar to what's found in recent version of GNU Parallel.

As a work-around, perhaps there's a way to do this in the current version using an alternate syntax?

opened by galiagante 4
Feature Request: joblog option

This would reproduce the --joblog feature seen in current versions of GNU Parallel.

Handy for troubleshooting or for tasks that otherwise require information on the exit values of tasksrun via Parallel

opened by galiagante 4
parallel complains if no arguments are passed

Parallel complains if no arguments are passed. However, it should be fine to write something like:

echo 'echo Hello' | parallel

If I just pick an argument to add (like -n or -q or -v) then the code runs fine as I expect.

opened by ChrisJefferson 4
parallel does not complete and exit its own process

Scenario: We launch two different tasks in parallel. Let's say two bash scripts. One sleeps and echos that is done. The other sleeps and doesn't echo and simply exits.

If the entire bash script does not send any output to console or error and exits, it may not be perceived as completed.

My hunch

The wrong exit code or wrong signal is sent and parallel doesn't capture that signal and never perceives that bash script as completed. As a result all of the parallel did not complete and did not exit its own process.

OR

The empty output or empty error never gets closed because it never got used. The buffer that was reading the output console or error console output never closed or exited so it never perceived that thread to be completed either and hung on reading an empty file expecting to read something. As a result all of the parallel did not complete and did not exit its own process.

opened by omac777 1
Parallel always uses the same /tmp directory -- cannot run two parallel commands in parallel

If I try to run two parallel commands at once, I get weird issues and panics. This is because both commands use the same directory /tmp/parallel for their internal state. Every time a parallel command is ran, a different /tmp directory should be chosen, by appending several random characters to the end of the existing directory name, like other utilities using /tmp directories do.

Alternatively, use /tmp/parallel-${PID} as the tmp directory to avoid collisions.

In practice this bug causes hard to debug issues when shell scripts involving parallel are ran concurrently.

opened by mateon1 1
GNU parallel -X equivalent parameter(s)

When using parallel with grep/ripgrep, without particular options, neither this parallel nor GNU parallel will display the file name (xargs works well). But GNU parallel has an -X that can solve this issue. How can I achieve with this parallel's command line?

BTW, when using xargs with ripgrep, the matched patterns can still be highlighted automatically (--color=auto); while neither GNU parallel nor this parallel can keep the highlighting.

opened by hongxuchen 3
Parallel with Asian characters

parallel seems not to be able to understand Windows directory structure, and thus when I put {/} for basename, it just outputs the whole entire line and when I use {.} it outputs the whole entire line, minus the file's extension. This is under Powershell in Windows 10, not only that, it seems to not be able to understand Asian characters. https://pastebin.com/zqbinCB5

When I use ls -Recurse -file -Path '*.m4a' | select -ExpandProperty FullName | parallel 'echo {.}' it will not understand at all https://pastebin.com/XB4kX2km

opened by 1480c1 0
Security bug

Dear Michael

Attached is a security bug. It has been encrypted for the key IDs 0xB2732D4240C9212C, 0xAAA0BBBF61CC465A, 0xE1FA5A7641DCA481, 0x70A85F710B8EFB78, 0xD4207ED15F3E197D, 0x777000CCB3384721. They all refer to the email address [email protected], so I hope you have the private key to at least one of them.

-----BEGIN PGP MESSAGE----- Version: GnuPG v1

hQIMA7ZBibGYo1SgARAAuHmsHIMu93HOZ8BKqpTbuFXnd9UOI0Ax2jlBN9UyWYQz BvmvJoTfa4Xr8G3gou58QI5uUxUAzQbW8RaDitOX8CyZ+YpndasElpm5w4GREchn RxDHhZFH31jnDM6OwTNaMBvDQRae8JM0Q4ZyTVemsfbUozbpEAx22f22gCUfmj2y rWYPOGNOyIfdSu5vrpldOx0M+Ubae4pWKevDEgYDxJvVy6pirR62qerdu58G5INo hG3TJbXcnNhbVwPd8ELSGC4lr9mbQx/Gw5wlZ+ZEzl6f7hDC5kuJhiH6nwkEYzRV Sh/0fjVuIgtJjqYbgQOxeEaeUFcc1kzGpNAFEeOA3OKav0hN9NvhMyoFWrx4SwOi GboCOonPSd1WmNgClChibKBW+e62cxfsPHOdycHzFU0khcsorcTaNgfW2Epnt1Mw pteKKscNtgd4r9vZYCZGXV8SJkwcrSwIsQGPABdHYvAneFKm3jLrgmi0090zqCUI +htgvOCw0akX0js14DM+rY71fw8ADR+YHIXtZOD/3rK/aY5rkyGYUO8SKgXRtvoP ImqZPi/N+HC2mZ0TUOQEQqXKQCfyJHI20tbECPp4JKf8idItMRkk3+rZ9MvrR7O6 MQct4TdnmSv+KdulCFIQucqh/TrfJFGhyDcp8bgSNX3XAa7sAgGoCjqdFTcaqB2F AgwDolBG8em3ogkBD/0cWCXFE4Q5/obrootqgY8fWx0OYVhXpLt5oYr4uKcTrlki VFiks8qnev+igvA/amTefdRP0PJK8eK5U5RrjaVDBqspjsHtfY5jsuHr8JNhASuK Fabg2Q007GkjoTl6oR7ee3BjmmKTK1M90ggseiYkXG/POsi7OBxP+roiOb8BOnan vgCaaquR93IGzYFl7Lr9KRhg8ESIIPDCbEka9CZ9tPgwCxEH2HFt79bXrF/ab8Yy WT45hO5f1y71gJF3dvqcfaksUdwl/zB1loWZ9AYOsM2Kt0TYt6EAT2jhSDG1+X6p 7fZ1gxPqHhvSFuYfddyiK+bc1f/2W5H8dZfueMM1r2+i+nP8Ok/bvCpxck/UurSb tBX7AIDVTdUgmbmbx804IvPsqHouBq8JdwxceQ9vrjHSIC48F/ibrGV1toE+Pv+c MvoSCPLPRhsndmkMCtWaGfX+0rgeJAJ/CgSNSo5QXH+GJc8HHnWExQY61asQ/zX6 Y8b+myk6eK8e510zEvv/hPLK6a+LcuUj/IFCPQl0pGOhlKw1wr7VLP2DUgboGvA5 d9sO3T4+PXdI5x0ftvHaYcL2tuxb2zjmr2tIy9V0PXPU5VL/mReiuCpQxuJrIqlm EdFpHbM3X5RCS4AnV0Nv4Q1KN2zHRgumO08ve77h771M407fbhy8NFNIS0BdpYUC DAOSI3jNDqUhSgEP/RFU7ETNFpJB4QJhDv339snONqTYeVb3T51gDnl+JIR9/N4L F9hcF5PUAaxMWm/vIL5OqiS2dB78kv+9NCmOD2gpIhW0hX5b0Cs/ba3kpqcI9s63 qrQig6bxVjt/xKqetmwjxN0SVG8rloSTKYjmuUyKJZgeZwFTUq48Fu1qC2c3Bx4J FD10hXjF5QF6Brf8oVaEfEpjACVkxBlI7/czTniXLBJndlVbqe7gDBCzsByWlkce fekghS9I7+PB/gL4+xuq39y7XYo6XVlWuaeYWHBtCyiuPQ0OvtLuN7ilGjxv7kwP n4bhZsJ21AFYtg5wwRPgBuqv6vi3DJCVS/iJnlM+q1uVsuxzHaG0gA2eySiLGJEy X21vsCqdmdEvneATizD3U70XKLLKBU4JCFmQ3sbFksjF7qzfOUQ1nAcbp6fUgUpH 7XwGOmtizWIipJ/9T+BxN6sdj6EFhrEscGCWL/OLMiL7/lF50iuBZ1bCdpxT7Yok 9HSVdnLSI16LzhEFDapeL3lrQ//VOuEFAYTYayGh+S5NzBsbL700WWSLBJsKYk4s +FQ/sgBWLqHT5HIhneEp/lbaUVzTkqIics0k2MsJSkZuXpIdFxXM1AVTnVNQdEWg LIyJQ+HLvl8PggbneUNiP607oLUI4T9Eu7/z+IxhsFvUwplxx3X0s0mIwCi2hQIM A+bLOJ65t0qgARAAtEQZNPlMirt8nPKfG6rFN00vA5GjrCIc8IMbqjus1TWVNNUo fSmpwe57Q+Dm6CEvaYFmT1BQx8G53XeZpYjMqBqdvvCDy/ZJupBIFCZXf24pzKQw iErF9AXd2RFkMdhu6CnylbLgcnhK0AGDtf9yGaM81iDP09OUHg03PTm4k2jqa/zh BklxFVFx6jHHVrUQBgJOdaEKbhnUBUIkB5ZkqQaDikCgjKXUW+Fu1Y3e60PVGNby 9nE0d/aXqZF03F9s19B7a44qvnRAyy014A29uo4qJWu36FjyC/kKtAhCnMZrhRZh IDfdtW+mG3uH+NCW6Ig+AYUhj4KZk9NSGFbsTv37QvzNE/d5oycKEfd72+QrvK15 K8VLHTAKSDdpK1JU6PndADi3XKOxiWsFDbtKic/x/NpC5Bw2aHfx7TeC/39nm88N +gM0jGap84MaQkYpXS2Ifqj07He+XEt0gQE/aNL8J+ziKPPGP5TmH1r4Y260nzZQ NfGVdCWgqohSyKI4+qSJ1y1P7NqUjNm7Y8wu9Etp2M/0qB6QZCt/ATefM/xgSGI+ OZZA8KylDjjF3ClQnRwdsu6kVayYzCxWDw/3OhsB0hIWrjtpgy4b7q0Oy9lQGGow O1VMDnrh+knv0dYjg338oZrrzBBtXajx/4a6knHectDbHLDm+N9ZHoBuTPmFAgwD 5C45kuwZVQoBEACdbMX0StZ1vxeNfSEKGAscpO6DE09/E76eo3lTsJtm+C2g/9Wt 8oX8qFqpBUxOIyd8JnvDZ2CYV2NGV07XZI3xJ9mogAxrlqaOKdVvdNHBBSpmaPal yoXdu8zIwJH4llykG7oMxZYTFoJJCD1SQ3Kyr+pDV4y4yVgjE9NcuwDtekrQGpCh DX47z0T+J1PZt9RpoGv+77fVRTU32sszoNPmQDb9y8cBu1DSZy6wt0Z4Z5WhUFcu nC9to5lPivV2H/REye/G9TtJtOEzJSzsfbo7Aq8/tQE58lIwJrfMDL9Acw5MfYE4 1AWyHWsTKrKW+3dWvy4MhRVA18AKdtqCFoS58yvALWfTIQZDAUwUsCy1VTQnEmGx ysD/+e8fwBTjKDvEGDSSC6FzNgGli4LtYFRYa0QH8fMBVzNsfmTwqsBbxZtxGvXc vXKIkODXO5c+cZBsRxjEHt42eZB8Jfb+X6K0DhRr/M//m07ykVrJTQz5H97dLCx/ 4WkwQ5E9jIOarb+pGvwQQGnz+xCg9O9mxiI4EhZ2Bz7KV3zuRWi0hSTHJYaZsUZ5 5gWmsLwkexl54sfDmCvwlUC0vbMMWn4gqpxUpxDfT7ldxYoh8P3VF1A5DlBh5/tk 52WtEReGnoWul7KYUceck9/aSts8NOqH4Onv1s8C1Zbq2Ua+YeMNgSR5j4UBDANh ftGCYG5IrgEH/0ljpselF08ObxnI0TgeKH70LoHQnW6wnY2Rotb+XXSSENxrVkIN VbUXP+4VYVaEc1LB1aIRuqIBTqKgPKLgOjMWb/CxRObRqKcUcCkxLbGRLL5Mvwvq Flj0dFaDOhdjWnlu3VnY/dyr2oSvwrZU5/V4XNjn71gfiljqWUnYn7tSqz4c2zLh 2uODXY2GfSWxjWTdaZy4HhYG+3lxEuxEyMEVWjMSWEmqpp/A12Z20xw/JinMAniz 9102dfCOSOroF2Bfs0s6BzJ4kayBTy5ChxCd+c4fZhGACejg8NtxrhA605njRLB1 g7BzJUR4M1GX1etzGYB3K8XIOC0SqMD7ywGFBOAD0atFFoiIiIgBJqDA5D5eI+Vz lDjlFd8TV6Dp6YB+tX3Lo8zQL+Z5yKYevi1NdgaxAXKHFyd+N9aNXAmq4fw3sbrQ Cnun9fJjJrxZRI72w8kxAJLnXOA7PiWX9gFBEJ9/onV3MrtHsmqwfjVn1w55ZrDm rKKbhu1k4t1Ky7sAslAZhMFHEH9zep+uYv2zee2Ffnse9S9ymbxN4ZoU+G9t/LMI ZMWM4v1RnspZeOFkdmal1kBb37a4u6eSBxyr/3I/WSO1MnTlrKP2XxqjgtD+yEmN fDUuBLG8E4+nIauAUyCXNPssdEJTkQ+Wx79ffyL6MG6oUUjgwxezYvKlAg/MD5tf 1y2yhMGV3+yk3ImmhIVHKHpYy283kpgNYhqjiXCgQJXRuTLDSIQIcEGaXunukKvb TQh1yePshfCkaWHU1g0lLYDJvfzu0SZcachUv29nMdmWIylJ5TpRCTHn7K06GxE0 6DKsUpm2aB3s2LZspKROpvXaXObXPvLSrvGxmPyKZenxQtoKqEzwYv6JWepSfhjQ qq5sQAYdjw/DH8BbywgizoQOj2GuYH4pnjgmOEwZe6l25Ejk3wPLV0ag2dok0Sft N98oq7KrYavkrEsj/PURcvxXgQXhCam1H9kgO7dvwnj61MvOvczjuYMMEDbKbmYK 3VUoxtKKNfan6Ly9XQhGpaVZpbTVs1ZTPd9Qp4nmVAxv4DiPquLRGpxqjxM1CS+P jdww+p31H/+lcW0fyCM15+jbi+j5D3OqJnyILh299LTDVh0/typfp3JAPWchW5+/ PVzj2diib12jeirnXGhjlMmXjfrn3PNNfJk43yPGoXl7plnqMHz7GknVSXcnxw5n 62C4GA0Ed6gJYopv/t8k4DkcZENqsVgjS9LT6avCUDSmFDCBHzxSODeHdXIPskHp oofDy/EjAq84AbETvSpjzSSK6g7IQpQRBAeumBWrYzAu3UPrs/yMBzMSb3QnafPv TE4owShmgVnh5XUvgJq2O8Sxl/aPrqo3c3lQgwrLg/a/UVu2ADDe/Exk4EOEsu1p a4/r67fRS/z8c/TJLDjoy4cVzT8CpH8o/2BogdcMZKbA9XfSXjUmvBASN4GvlclB 2qJ5GFKvMBkK6nZN/iE1wdwLxjuS59oCnQBMpy5h5IAHTJAk5PQfFrUgjgiyoMAV 5v0+7ZilCjszVQN4hik5I7UywVP4B3/wqj0NdBNk903guaMH1//CmY71e4b7h1Nt umIl1Qe+f1AStkPdFunTJ8R15qu2qakzO/Ah4nuCRa5Dq/WjhKouSfMIXVu/6ixd VSenJcMFbvyebgekJPn3KBgHX5vCoBYhVHLuQUN62vgDy+SlpijMNp8tZLnk8f7M VFNILn9+OOhKaTVj19o7tjefbeBlxAqF/mGgrudmK/PFcOyzf3lVEgj+Sk7QzOcV vEwWn+9j5yCO0MUetdGZs9pnAJ+FRzeLapntwsGhVUsVhtjtgemNEFQWp/n0pZwd JXBpfOcZ6Eqp2Oqf39dQE9SP8bOiIHl59q7XuDwsRmO4+N0WEpSYL9yQoVEH82DG xlWPuCNp8iDdxnCwKQUwAHXZOD/tD/j9fObRKANpX92ypGN2wQoVaQMFr2eAkMX1 mBwFZjx01Ach1VNgt9PdeGIoCGRPRhR17cmHLuvS6wHPs0fKklzsGw47ovzd8mvV F+yWhADA7gFH+jkAFUKGqI6LyEp3B/3rBAKhsNaH5DxJVEzhIthOA9C+Z8IJP/IA VjNTcpKpZjbcEmwgpYjFhiAo3iUlMStUOYPmrrftQrHDWDGW9465Pn4BzxCVIT8x +V5+m7FjD9vb+XaVbR6pPWM/4FF7YL/IChhpKgL7xIsPGvfcnFrt0HavlABQperI +xOfs5TEQ9XDLRXHPX65uQv+f0TUJb/J9mHFAgnB+K0Qurhlz+lRpT5QvqrT2Slr jJ3nEi21ydY7mSXNmSakyUYqpLw6qsCgB3U1DWTtrilbZYhczlCmdoF4AvIkqvvM tQzV78HlOdEVMZEl9dbuV+pfEd1U0qXO2NPryxneBtzAkYitS27MEs3/ZfHY05/0 ldppv8krpF7iLVv+qtvT+cWE7xF9Ca8Qk73853ZWc0ovuFqExrFB6pz4RgF34oUK YO6ynkT+VeJ2gi09Y9lIWLsVSO7JiNOSFSxESdi+tdsxct8hp8a1Oct7VtVr8R9F aC81/R9b3o/OQqKZ4UfAybitwHjRUK+crSq2z0FSDEDwO4x9hr9uaduZ2O2gxI6R yr6eaIl5iqYh3MjvCRqQxT7xDQkoomXS+o2zKT3nmcess9wStrtSKXJlkb/nx/47 p/wqY8rgjuLyImSgt0QmleFGqvkv0AZGEXRJMhZX/4cPATXz0JkzYfZ1ZQnG9No7 1rrS59LPJt/OHEavMSu/xhLRVIdmvZsK6VWXFnzf1yz0Y06IkfH6eLZ/V5/ido7J zkXhrcCzHwNtpADRjqx1EPjd/ZCA223PBexjArkOqEFoFMZG9IGUkY0LW2ZEpWar VtaJ8vZj/qAiM0b5catFTgi/u6kVs8T0wlp7c1pf+X9IHFHuV7bl3Ybakbd/IVNc uOh2t3Md3qEx2jccWy3NkSkqXEXS82rFxvS9AEPeh0qQ/EPGk2GCThwHlwiAepzZ oMPwGWCjng/OIo6fZ3UxJ8uVYhphzAD3dJVf1o1s/wi22ua0ITiVsr+rytQcRG0B gcbvKkk63BjhpsIVKfTNmLLPB6lcGivIm6DAjlJxU8DyH7FXBpGQAiymaBJxcCKr A8/AoFuLlgT+lpjVOPjqA8fQlLP1iIZ393GdkZVJ+J2MJyRQVpOJMdkMQtvQxzP/ SGc4985TO+HK0b9Z03lTq5Jucp5sQ7ZfcXOMUGoWBFCt3DrdVA9Fq9gl/Yl+m67A KZKkENLlaG2cUnLpz9rsQzRGG5yVsWG1Z1c3iuJ2wowIzIWaYFk7mPYkqm3KLA4k pweNDaSZHH8oNlN3+u4wilbXfU7rQey/uk75D90suhLh4n9uLX6l/RjvXddb/Zkv xfKEwh+v4RLxrlHWDa4O3uceMPutA4iknZUETWdAxSFuBvELZlAuHs3vMpg2T8SI Op7iLYba/VfcSSTOFLl2JDlIPaFK4z+yTfxvDwzQQcdlh48kEtg/OB0SziuIiJ8j 1ZGbTFX9hubSY/snKqsqrbimCLKav9U47hTNtpe2WHUGO6VT/2WD1Eg/eIRW9QKY ttZdr+BZKP5ybp4E7fi8WbisXAyYnn7kPgSd4hjucZycDWprHghdTPb25DGG2Gt3 P2hjmuZPRQ36AIhpW7NvPgyfqnjQeHhy7qm4+GI77XBUvUh7P5LBBORgJLev23Ob evKBQZQ5G8RubyD4rEJwk33CGzByFcs4xQyXJVD+LBTAff31GvB7jtgVAL5wU6JZ A+QMbVeSNHEma3vE/GUfsfH1am4XZ9cO6UC9rU/zlb7XBKpPUv1W3Rk7uenkLd/8 9JxQCHDwLRE5CBCtUgBx4igCNMyQh2ogA0ZDL9nSGJYVVdU71TIq8dE9KuMqPOWY pvjmrW1CXNLQcNNOBqpGQpOD+f68vrtxr4luFSjSI7OhPWQk7HhZnkdQOzXi9LAw PSOG8alX3cX4vawrw3QSMf8goT1C8YVpNoSKDvNJFtK8oRnQQYTVrEvrBAxeDJY5 dhkFYhhzUujwCP0ayZts8YL2yj0ShSAAjBuKE0Uo0JGfjusHTi7CpEwZ4XTeiTEG 4kQU4cYNxxHGgslbztcR7hpsgB8EtOX4aULLLYGdcxwILawVkB+2ivUXEnXwsDxl xIHDVl155Dpl3ymd2iAnkzxJRaC5FjwpUxjmdIUW5NJFrZqw9UqgPMi4HpBCGAiA ab9m066lk9PvEzf6A6m7MW/qDY2BE2VKjaKyDlFTEGvUk5xckvstGI2EPPUDJWVF wZRkcNYIe/JtMv7t15q27bXy4fVtRw8eSlVSbwPnIZCdGE3KAT0nAheta09HYE/7 IXVnJtKcGxtPdxuplsb0SE4bAG0ZIV6xQDNUwwwzIHPvoGbouWvOhoi3ULFNrt+8 RJhT/2HEU14BLQ5i50PG86hS1TsQuEUSTcq+PrfKL/95FKz8Ol0FJ1XBKsB4+PU6 snTTdZJmhP1Ccu9iWzgCqLjQnCFirK+sr+hE4/lA48pWipFR+ojpuggKXSwjxj5A wlIOO0X3UQo4tl+TH5KYamTdRfqZmcPFaagWgke+mI6Rb40moA7t5RPXllp9E111 Ps52U1ItPcpWrDV/eamVsEe2g4N4BQBeCfkxzvg9jLNN8vWajka8y6xbUxqsphde vI+YxxSkX7XVk22ahgWcjyucoKYFWulRAK4xEL4MnbLgjC8+U4bYNFHIet0Dmpdj 6yOZNbadx6il6tOHVK/f6W/AdE1OTpesQuUNnd9ktCG5JXtzgYKqu1MjQ44phy0c 5oW1HXgR8kRUerDmpyl6AaLj/lsbuPLvOWt7/jVNYrTmClaEMT/NURZFvujjQKp2 VbjMfx8v/alJEBaOZn4Zt6BPExgqPiIvBkkQXOo/iwpPzVrPzZCky+2xezgEvBas BGk8QNhNAO+DfNgmaOfRKUH/0tbJkQxbXeJQEGzzsCT4vj1F9XJX4AP3W5pMIuxd ZuD2h4O7OMADQogXTnHqHtwzZjcEfcAaoDJmnIQD5ZIaFoFghp8m2b+3L6P0zZuJ 84LLFZ29pMMRSLfNtN5cEwCZJ820Ao/tbve+pZTnL6hKnYb15ggEU8DTgsesK7qv qZB6RVRS+iEx66vrxwPNzNDZRADeHrHO5b8qcmgWZ2nsRzq3+1ID4OT8kTI9G9Z8 8XKCoqkc+NQKtPojZJeHw1sHpMzFyEWQ6EZaql+xdvrTEJlkUbJUTH72k7smAmAN kEssAPkbkYWK2hiT1ZJa/jz0mRF3oR1v6d81q9fW5YzQdmPjrEta8ymqg3PGlqxj ufomgAyJ5UmT6sq0JzL4ZSJ/5bPtJkeIqnJxRNBop/idFtZLQIkAOC/plqK2XNFY XzJl1rEi26bibuo0LQLinTYxDE4OH3251AdRHKMm9NHJ7Ep2TcYb1ngt4uvWAfFL wDpuT7AL64+ZNzO6+voOhjF30wEKzReWPTLLAYnTyy/bmudnwWaOtW7JmnyxZD0T lFPIRPv+aapjpbvQNhzrK08/s4KE0mYwIWtyvRX5nUC5AuzGhb4S/gshm3GUsUO5 pNV2tw5oKIaYTDUup09EewU5g6eLYMokdTpz3VtI25bv4qrHBAHfSoQz1N4qbHol 3U3HiYsj/PGE4LHHl+2gPfdMvUmr3x9NwNFvr1Q5aNu2oEn+1/yhiJmhLSIBh3LG v80wCgZQoYf6Oh4kyS/zYfKmccYgXFjslABDqDK3ELcXRuxeQdUAzhggaUWCqiaP EKH5L0h78+T4+as3 =ZfDC -----END PGP MESSAGE-----

Regards,

Ole Tange

opened by ole-tange 8

Releases(0.11.3)

0.11.3(Jun 9, 2017)

The exit status will now be the number of errors that have occurred during processing.
Source code(tar.gz)
Source code(zip)
0.11.2(Jun 9, 2017)
On Linux, a check will be performed on transparent_hupages and issue a warning if it is set to always instead of madvise

The Ion shell is now a preferred shell, with Dash coming as second to Ion

The ArgumentSplitter was improved -- copied from my Ion shell implementation

Invalid arguments will issue warnings instead of crashing the program

If an empty line is passed to standard input, it will be ignored

All Rust dependencies have been updated to their latest versions

Source code(tar.gz)
Source code(zip)
0.11.1(Feb 1, 2017)
Crates.io now supports categories, so this project has been updated to be listed in a category.

Support for a new token, {^suffix}, that allows defining your own custom suffix to remove from inputs

Source code(tar.gz)
Source code(zip)
0.11.0(Jan 21, 2017)
Additions since 0.10.7

Proper support for quoting has been implemented

Unneeded UTF-8 re-validations have been disabled

The -j / --jobs parameter now supports +/- modifiers

Additions since the 0.10.0

The --tempdir / --tmpdir parameters were implemented

The joblog now supports a --joblog-8601 parameter for writing the start time using the ISO 8601 standard

A numtoa crate was created for optimizing integer to byte array conversions

This crate is 10% faster at base 10 integer conversions than itoa

A handful of dummy arguments were added to appease existing parallel scripts

Code comments were added throughout the codebase

Proper support for quoting has been implemented

Unneeded UTF-8 re-validations have been disabled

The -j / --jobs parameter now supports +/- modifiers

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(196.59 KB)
0.10.7(Jan 18, 2017)

If someone passes a silly argument like -j0 or --jobs 0, this argument will now be ignored so as to not set the number of jobs to run to 0.
Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(191.85 KB)
0.10.6(Jan 18, 2017)

Even though the last release was only half an hour ago, I felt that this feature deserved another version bump due to it's importance in established scripts that are using the existing implementation of parallel.

The --no-notice, --line-buffer, --group and --ungroup arguments are now supported arguments that do nothing. They have been implemented merely to prevent the program from crashing due to being supplied with invalid arguments.
Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(191.24 KB)
0.10.5(Jan 18, 2017)
An issue was discovered with the tokenizer.

That issue was then fixed with the following commit

The unprocessed and processed logs weren't being properly truncated at the beginning of the program, so that was also fixed.

The numtoa crate was updated to version 0.0.5 which has more performance optimizations

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(191.31 KB)
0.10.4(Jan 16, 2017)

An error would occur if the tempdir was not created. With this change, temp directories will be created if they do not already exist.
Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(191.41 KB)
0.10.3(Jan 16, 2017)
Implemented the ISO 8601 format for the start time column in joblog as an option

Malformed columns were fixed in the joblog file

ETA was optimized with the itoa crate

The codebase was improved with more code comments

Some refactoring to improve the overall quality

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(188.18 KB)
0.10.2(Jan 14, 2017)
I forgot to remove unnecessary println statements

There will also be less heap allocations for each job run

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(194.25 KB)
0.10.1(Jan 13, 2017)
Some potentially unsafe joblog-related code was removed.

The --tmpdir / --tempdir parameter has been implemented.

Much filepath-related code has been deprecated and thus removed.

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(194.42 KB)
0.10.0(Jan 12, 2017)
Implemented the joblog parameter which may be invoked with --joblog FILE

Properly implemented the shellquote parameter and removed the quote parameter

DiskBufferWriter was replaced with BufWriter<File> for performance reasons

A misc module was added which contains useful traits such as NumToA and Digits.

Integer to string optimizations are performed to eliminate heap allocations.

Instead of converting numbers into strings with the to_string() method, numbers will be converted and stored into a shared byte array via a numtoa() method

A ton of refactoring work to make the source code easier to manage

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(201.70 KB)
0.9.0(Jan 4, 2017)
New Features Added

memfree: Only execute the next task when the available memory is above a certain threshold.

shebang: Adds ability to use the parallel application as an interpreter in a shebang line.

eta: Track the average runtime of processes and print ETA statistics.

timeout: When a process is running too long, kill the process.

delay: Delay launching processes for a specified time when set.

Known Issues

The timeout parameter has no effect on processes running within a shell.

The -n parameter does not work correctly with {N} tokens.

It's no longer possible to compile a Windows binary from a Linux host (sys-info conflict)

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(193.32 KB)
0.8.0(Dec 30, 2016)
This release makes the implementation another step closer to the GNU implementation.

The :::+ and ::::+ modes have been fixed.

The -n and --no-shell flags have been removed as they are no longer required.

The -n and --max-args flags have been added.

The {#^} token has been renamed to {##} to match the GNU implementation

The {#} token now starts counting from 1 instead of 0.

A number of source code improvements has been made to the original parser. Many more improvements could be made to this part of the application though.

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(151.84 KB)
parallel_amd64_windows.zip(299.29 KB)
0.7.0(Dec 27, 2016)
Implemented ArgumentSplitter which splits arguments better when not using a shell.

Handles both double and single quotes unlike the previous implementation

NOTE: Does not yet support environment variables

Implemented temporary disk buffering for command outputs

The memory-buffered implementation has been replaced with this disk-based version.

Half as fast as the memory-buffered approach but uses significantly less memory.

NOTE: At a later date I will re-add support for memory-buffering

Improved permutation speed when permutating multiple lists of arguments.

The permutate crate was updated to version 0.2.0

Allocations will no longer occur after the first iteration, as successive permutations will now re-use the buffer created by the first permutation.

Source code(tar.gz)
Source code(zip)
parallel_amd64_linux.tar.xz(149.38 KB)
parallel_amd64_windows.zip(294.07 KB)
0.6.5(Dec 23, 2016)
The problem with programming past midnight is that you make mistakes. The following issues are now corrected:

The shell is now disabled/enabled properly when it is not needed

Dash is only searched for and enabled if the shell is enabled

A command needs a shell if it has a | pipe

Source code(tar.gz)
Source code(zip)
parallel_linux_amd64_musl.tar.xz(148.01 KB)
parallel_windows_amd64_gnu.zip(291.14 KB)
0.6.4(Dec 23, 2016)
Heap allocations have been drastically mitigated, improving performance and reducing memory consumption

Jemalloc is disabled by default now, for a 100% performance boost

If the command argument does not require a shell, then the shell will be disabled

If dash is installed, dash will be used by default when the shell is enabled

File paths are no longer recalculated

Source code(tar.gz)
Source code(zip)
0.6.2(Nov 12, 2016)

As simple as adding support for --quote was, this little update will also add support for --shellquote
Source code(tar.gz)
Source code(zip)
0.6.1(Nov 12, 2016)

This is just a small update that adds an option for the --quote mode from GNU Parallel. I had to change the -q flag which was previously assigned to --quite to be the short version of --quote, and have added the -s flag along with --silent as a replacement. However, --quite will remain.
Source code(tar.gz)
Source code(zip)
0.6.0(Nov 12, 2016)
Adds support for piping inputs to commands via the new --pipe option

Removes support for ungrouped mode until further notice

Adds shell detection so that Parallel will attempt to use your preferred shell of choice

Non-text inputs are fixed so that problems will no longer occur when taking binary data or garbage inputs

All uses of the try! macro have been replaced by the new ? operator

A bit of refactoring and sprucing up the existing source code

Source code(tar.gz)
Source code(zip)
0.5.0(Sep 24, 2016)

Source code(tar.gz)
Source code(zip)
0.4.1(Sep 7, 2016)

Resolves panic when no arguments are provided, and adds a help message for when parsing errors occur.
Source code(tar.gz)
Source code(zip)
parallel_0.4.1_amd64.deb(182.15 KB)
parallel_0.4.1_amd64.tar.xz(168.59 KB)
0.4.0(Sep 7, 2016)

This change adds support for permutating inputs when there are multiple input lists provided to the program. Input modes have been updated to work precisely as they do in GNU Parallel, with ::: creating a new list with new input arguments, :::+ appending arguments to the current list, :::: creating a new list with inputs from files, and ::::+ appending inputs from files to the current list.

Update: Found a bug, will update release with fixed packages soon. Update2: Updated release here
Source code(tar.gz)
Source code(zip)
parallel_0.4.0_amd64.deb(181.72 KB)
parallel_0.4.0_amd64.tar.xz(168.07 KB)
0.3.0(Sep 4, 2016)

With this new major release, Rust Parallel has gained the ability of supporting {N} tokens, which is a placeholder for the Nth input in the input list. The support for this feature was made possible through recursion and pattern matching in the tokenizer() function. The Token::Number token was added, which contains both a number and a Token itself. Pattern matching is done to ensure that the embedded token is not an invalid Token, such as putting a Token::Number inside of a Token::Number.

Additionally, I've made a number of bug fixes along with a newly-added unit test.
Source code(tar.gz)
Source code(zip)
parallel_0.3.0_amd64.deb(177.31 KB)
parallel_0.3.0_amd64.xz(163.05 KB)
0.2.3(Sep 2, 2016)
This change will modify the parsing code to use an enum for switching between modes, and the new input mode being added is support for files that contain inputs as input arguments. This will also enable support for the abili

seq 1 10 > list parallel 'echo {}' ::: starting :::: list ::: done
Source code(tar.gz)
Source code(zip)
parallel_0.2.3_amd64.deb(165.33 KB)
parallel_0.2.3_amd64.tar.xz(161.41 KB)
0.2.2(Sep 2, 2016)

Source code(tar.gz)
Source code(zip)
parallel_0.2.2_amd64.deb(164.27 KB)
parallel_0.2.2_amd64.tar.xz(159.78 KB)
0.2.1(Sep 2, 2016)

This change will fix the test units, add a new test unit, and allow for the parsing of arguments when using --no-shell mode. Before, when using --no-shell, arguments with spaces would be interpreted as completely separate arguments. Now, arguments can contain spaces if they are double quoted, or if the space follows a backslash.
Source code(tar.gz)
Source code(zip)
parallel_0.2.1_amd64.deb(164.54 KB)
parallel_0.2.1_amd64.tar.xz(265.11 KB)
0.2.0(Sep 1, 2016)
Added ability to disable the platform's shell with -n or --no-shell.

Added ability to print verbose information with -v or --verbose.

When grouping is enabled, the child processes will have their standard output and error piped.

Messages will be printed to standard output/error on demand as soon as they become available.

Source code(tar.gz)
Source code(zip)
parallel_0.2.0_amd64.deb(173.48 KB)
parallel_0.2.0_amd64.tar.xz(169.03 KB)
0.1.1(Aug 27, 2016)
Runs commands within the operating system's shell to allow for multiple commands within a command:

parallel 'cd {}; echo ==={}===; ls' ::: $(find -type d)
Source code(tar.gz)
Source code(zip)
parallel_0.1.1_amd64.deb(166.04 KB)
parallel_0.1.1_amd64.tar.xz(161.94 KB)
0.1.0(Aug 27, 2016)

Source code(tar.gz)
Source code(zip)
parallel_0.1.0_amd64.deb(168.28 KB)
parallel_0.1.0_amd64.tar.xz(164.15 KB)