When using pqrs cat --json the output still contains the file headers for each file, making it much less useful for quickly converting a file or a bunch of files to JSON.

I feel like the headers should not be present at all when outputting JSON or CSV. There could be an additional flag to add them.

Feature request!

Is it possible for merge to merge files without decompressing and recompressing them?

My usecase:

My parquet generator makes 1GB row groups (in memory), and writes them to individual parquet files. <40MB on disc, one row group per file. (It does this because it can't be bothered to deal with schema variations, another problem.).

I'd like to concatenate these files; take the row group out of any that have the same schema, and make one big file with multiple row groups, with exactly the same schema?

The current merge implementation can do this, but needs >>200GB of memory to merge 8GB of parquet files, which is not ideal.

Fixes Issue #20 by moving the cat "header" to stderr instead of stdout to allow for piping.

Feel free to reject this if you think that a --quiet flag might be a better option.

joserfjunior@Clodovil Downloads % pqrs cat FND_USER_CLEAN.parquet --json

############################ File: FND_USER_CLEAN.parquet ############################

thread 'main' panicked at 'No such local time', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/chrono-0.4.19/src/offset/mod.rs:173:34 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Stumbled upon this project and it looks like a real missing link in parquet tooling.

For the schema subcommand, it would be nice if there was an optional way to output the data in a structured way (eg json) for consumption by other tools. Something like pqrs schema data.parquet --json. Ideally both the simple and detailed flags, but even capturing only the non-detailed data would be handy.

This merge request has only have:

• arrow 12
• parquet 12
• clap 3.0 + rewrite commands for clap::Parser. I have removed trait PQRSCommand in favor of structs.
• commands cat and head uses Json/Csv writers from arrow.
• fmt + clippy fixes

All tests passes.

Reason for this MR is it will be easier to add new sub-commands.

I have to convert formats from time to time and being able to just pipe the outputs is super helpful.

Unfortunately having the extra 6 lines means having a (slightly) more complicated command which is harder to share among my team.

pqrs cat -j foo.parquet | tail -n +6 |jq -c . - | gzip -c > foo.ndjson.gz

Adding a --quiet/-q flag to mute any non-data output would be neat, or possibly just moving that type of output to stderr instead (ie. changing lines 110-112 from println! to eprintln)?

Hi, we're very happily using pqrs now and found two small issues with it:

1. head does not support directories:

#> pqhead data.parquet 
Error: ParquetError(General("underlying IO error: Is a directory (os error 21)"))

2. It panics when used in a pipe:

#> pqcat data.parquet | head

###########################################################################################################################################################################################################
File: data.parquet/d66ac6554cc44c3cbfaa56b75fa446e4.parquet
###########################################################################################################################################################################################################

[...]
thread 'main' panicked at 'failed printing to stdout: Broken pipe (os error 32)', library/std/src/io/stdio.rs:935:9

Hi, I just tried your tool and I find it super useful :+1: However, I noticed that one cannot simply cat whole directories:

# pqrs cat vcf.parquet 
Error: ParquetError(General("underlying IO error: Is a directory (os error 21)"))

However, this works: pqrs cat vcf.parquet/**/*.parquet

It would be a nice convenience function if pqrs could handle this case.

Cannot be used alongside --detailed flag currently. Will add support to view detailed json in a future PR. Closes https://github.com/manojkarthick/pqrs/issues/25.

Directory support:

• Add supports for directories as a parameter to the cat command.
• When a directory is provided, the directory is traversed recursively and all the non-hidden, non-symlink files are displayed.

CSV support:

• Adds support for displaying Parquet files in CSV format
• CSV output format is not supported if the Parquet schema contains Struct or Byte fields.
• If one of the input files for the cat command does not support CSV format, an error message is displayed and skipped. The contents of the other files will still be displayed.

Version bump: Bumping the version of the package to v0.2.0

When calling head or cat on a single largish file, pqrs v0.2.2 (and previous versions) attempts to open lots of files? Why? I only want to read one line at a time and translate .parquet to .csv.:

$ pqrs cat --csv live_int8.parquet > live.csv

#######################
File: live_int8.parquet
#######################

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet-12.0.0/src/util/io.rs:82:50
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I would love to pipe the contents of pqrs cat --json <filename into jq or other tools. Currently the header from pqrs cat prevents me from doing this.

######################################################
File: ../myfolder/myfile.parquet
######################################################

Is there a chance we can either remove this entirely or remove it specifically when used with the --json flag?

My current version:

❯ pqrs --version
pqrs 0.2.0

Going forward will this tool only be available on nightly?

  Installing pqrs v0.2.2
error: failed to compile `pqrs v0.2.2`, intermediate artifacts can be found at `/tmp/cargo-installWmgUiA`

Caused by:
  failed to download `once_cell v1.15.0`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/amooren/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.15.0/Cargo.toml`

Caused by:
  feature `edition2021` is required

  this Cargo does not support nightly features, but if you
  switch to nightly channel you can add
  `cargo-features = ["edition2021"]` to enable this feature

Hi,

I was looking for a command line util to read Parquet files and found your project. I was wondering, would you be interested in contributions to the project, e.g. open issues, new features, etc?

I'm a software engineer (mostly backend for professional work), and recently took an interest in learning Rust. Practice is the best way to learn so let me know what you think!

Thanks, have a nice day

Robo

Hi, with pqrs v0.2.1 I get the following error when trying to read a parquet file written with Polars:

Error: ParquetError(General("Could not parse metadata: bad data"))

Pandas can read it without issues. Could it be that there is some missing feature flag for the parquet reader? E.g. some missing compression library?