It currently panics when you give it a file with one object per line, like

{"user_id": null, "user_email": null, "user_name": null, "email_domain": null, "org": null, "method": "POST", "path": "/logged_in", "status_code": 200, "latency": 0.09175515174865723, "timestamp": "2023-09-05T21:38:34+0000", "response_len": 450, "properties": {}}
{"user_id": 2, "user_email": "[email protected]", "user_name": "Tim", "email_domain": "fr.ai", "org": null, "method": "GET", "path": "/datasets/web", "status_code": 200, "latency": 0.0032494068145751953, "timestamp": "2023-09-05T21:38:35+0000", "response_len": 672, "properties": {}}
{"user_id": 2, "user_email": "[email protected]", "user_name": "Tim", "email_domain": "fr.ai", "org": null, "method": "GET", "path": "/datasets/file", "status_code": 200, "latency": 0.3335745334625244, "timestamp": "2023-09-05T21:38:35+0000", "response_len": 85195, "properties": {}}
{"user_id": null, "user_email": null, "user_name": null, "email_domain": null, "org": null, "method": "POST", "path": "/logged_in", "status_code": 200, "latency": 0.04735732078552246, "timestamp": "2023-09-05T22:13:40+0000", "response_len": 459, "properties": {}}
{"user_id": 3, "user_email": "[email protected]", "user_name": "Justin", "email_domain": "fr.ai", "org": null, "method": "GET", "path": "/datasets/web", "status_code": 200, "latency": 0.002116680145263672, "timestamp": "2023-09-05T22:13:42+0000", "response_len": 672, "properties": {}}

An easy way to handle this might be to ignore everything after the first complete object, e.g. this works fine:

❯ cat usage_log.json | head -n1 | drivel describe

But realistically, I'd want it treated as if the top level is an implicit array

Use a tool like https://opensource.axo.dev/cargo-dist/book/introduction.html to automatically build common binaries and make them available as a release whenever a tag is pushed.

Using some basic heuristics, we can decrease the amount of regex matching that we do. This leads to a significant (15-20% in some local benchmarks) speedup for large data sets, depending on how many strings are matches for the specific stringtypes.

Closes #5

Adds a command line option to infer enums from strings, with the ability for a user to specify what ratio of unique strings they want to consider the "cutoff" for thinking of a string as an enum, as well as a minimum sample size.

Closes #8 by introducing rayon to parallelise array schema inference as well as value production.

Benchmarking shows that jemalloc is required in order to see a perf benefit for operating on large (100+ MB) JSON files. Without changing allocator, there is a point at which multi-threading becomes slower than single threaded on Linux due to the allocator working overtime.

When working on seriously large JSON files (100s of MBs to GBs), schema inference starts to become a little bit slow. The same is true for producing a large number of values.

We can leverage a crate like rayon to utilise multiple CPU cores for the inference and production logic, where things can be parallelised. For infererring, for instance, we can parallelise inferring the type of each array element and folding the merge function over them. For producing, we can parallelise produce all values in the root array in parallel.

For some data sets, it might not be useful to generate random strings based on the sample of characters observed. Instead, it might be better to treat strings with multiple values as enums, where one of the enum variants is picked at random whenever a value is produced.

Description generated by Claude 3

Currently, drivel infers the schema from the provided JSON input and generates data based on that inferred schema. While this is useful, there are cases where users may want to define their own schema and have drivel generate data based on that schema.

We should add a new feature that allows users to provide a schema file (in a format to be determined, possibly JSON Schema or a custom format) that describes the desired data structure. drivel should then generate data that conforms to this user-defined schema.

This feature would be beneficial in scenarios where:

• The user wants to generate data with a specific structure that may not be easily inferred from a single JSON example.
• The user wants to generate data with certain constraints or patterns that are not present in the example JSON.
• The user wants to generate data for a schema that they have designed beforehand, without needing to provide a JSON example.

To implement this feature, we would need to:

• Design a schema format (or adopt an existing one) that allows users to define the desired data structure, including field names, types, constraints, and relationships between fields.
• Add a new command-line flag (e.g., --schema-file) that allows users to specify the path to their schema file.
• Modify the produce mode to check for the presence of a user-defined schema file. If present, use that schema for data generation instead of inferring the schema from the input JSON.
• Update the documentation to describe how to use this new feature, including examples of the schema format and how to generate data from a user-defined schema.

This feature would greatly enhance the flexibility and usefulness of drivel, allowing users to generate data for a wider variety of use cases and scenarios.