I managed to decouple the CLI from the main worker which I now call main_worker. There are some issues associated with it that are described in test.rs, such as the trait bounds etc.

Copying the description of the test_in_dirs function:

Runs all tests in the tests directory. This assumes that the directory tests contains multiple directories, each with at least two files: command and output. The command is executed and compared with the output (assert). The test is run from the top-level project directory but arguments in the command ending with .json and .tsv are automatically prefixed with the path to the subdirectory. The first parameter of the command is the file that's redirected to the worker (imagine prefixing the whole command with a <). To add more tests, simply copy one of the existing folders.

Pros of this approach:

• Very easy to add more simple tests (simply create a new directory)

Cons of this approach:

• When running cargo test, all these tests from all folders are under the same name test_in_dirs and count as one test.
• It is also hard to see what test failed.

There are three options:

1. Accept this code as is
2. Find a way to resolve the cons but using the current structure
3. Turn the written test code into an easily assertible function that takes in a string of commands (written in code, each in individual functions) and a path to expected output.

I personally lean towards 3 because it easily fixes all the current cons. What I don't like about it is that the command to run rjp is written in code. From the perspective of tests that seems rather like data and should therefore be in a separate file. But maybe there's an intermediate solution that combines the current directory structure with the more modular functions.

I currently added two test from your comment in #3 and did not think much about their validity. There are more issues with this code, such as panicking whenever rjp panics but I think that for the initial shot on testing it's ok. Apologies it took me so long to get to this.

Let me know whether option 3 is a good way to go and whether you have some more remarks. It should be fairly easy to implement it.

I'm still in the process of transforming the code to be testable (first step is detaching the CLI from the code entry point). @ales-t do you have some test cases which you used during development that could guide me in this?

Despite the readme processor descriptions being properly organized using markdown headings, I find them hard to navigate. Putting them in tables saves on vertical space (less heading padding) and makes it easier to find the commands.

This PR reorganizes the project into directories and modules for easier management. It may not be the most elegant solution but gets rid of the flat directory structure.

There's no real content in this PR as it is mostly cosmetics.

• README typos
• Begin work on making code testable (separation of CLI from entrypoint)
• Lint every file with rustfmt
• Accept suggestions by clippy

Since the current test files are very small and cargo bench rounds the runtime to 0.0s, I was thinking about running the test on larger files (notably on (1) very long lines, (2) lots of lines or (3) both). The issue is that nobody really wants to have 0.5GB files in the repository.

Since their content is rather arbitrary, maybe we could just generate random data (with fixed seed) and have them be generated only locally? A few lines of a Python script would do but maybe it'd be purer adding a simple binary crate to this package.

In all cases I'm not sure if there's an elegant way to link the dependency other than erroring on tests that the files were not found and that the user probably forgot to generate them.

In jq one can write the following (complex) query jq '[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]}]'. I find this templating quite intuitive and useful but at the same time it seems like there's a lot of work behind this (mostly the parsing).

Currently rjp is based around simple operations and piping multiple rjp calls together so I'm not sure that this would even fit into this schema. If it was implemented, it would replace extract, rename and select.

Relevant to #7, we should definitely transition from the handwritten argument parser to something like clap. I'm however wary of implementing it now because version 3.0 should come out soon:tm: and I had issues by including the beta in Cargo.toml.

Currently, rjp will stop when encountering any error, such as:

• Select/rename not finding the required fields in an instance.
• serde failing to parse an input line.
• Join not finding the keys to join on in an instance.
• Merge when the stream lengths are mismatched.
• ...

Oftentimes, JSON lines files are noisy and contain lines with problems. It would be helpful to have some way to request "tolerant" behavior. For example, rename_field would not change an instance if the input fields are not found. Alternatively, problematic instances may be skipped in the output stream.

It's currently not clear to me how the various processors should behave in this tolerant setting, or whether there should be more ways (for instance --skip-bad-instances, --stop-on-bad-instance, --keep-bad-instances?).

There are multiple ways in which we could do paralelization. Assuming that each line can be processed independently (which may not be true in the future if we make more processors), then with workers A B C the paralelization may look as follows (assume 12 lines):

1. ABC ABC ABC ABC each worker is fed exactly one line and when all three are finished, they are sent to the buffer. This is the easiest solution but may not even be worth it given that thread spawning is not zero cost.
2. AABBCC AABBCC another step is to increase the number of lines fed to each worker (here 2). There is certainly a threshold number from which it becomes beneficial to use threading (that number probably being larger than 1).
3. AAA BBB CCC this chunks the whole data into equal parts and feeds them to the workers. It creates the largest chunks and is the fastest from paralelization perspective but requires that the whole data is first loaded into memory which may not be viable for large files. (In a sense this is an extreme case of 3)
4. ABC AAABB CAC this is the most complicated solution (though probably the fastest one). It considers the processing dynamically and e.g. if A is finished and B and C are working on some long and difficult lines, then A can get next lines assigned.

I think that 2 is the best option because it can be parametrized via two parameters from the CLI. Maybe I missed some approach.