Pure rust implementation of jq

Last update: Jan 4, 2023

Related tags

Utilities xq

Overview

XQ

JQ reimplemented purely in Rust.

Caution

This program is under development. You probably want to use the original implementation of jq, or pure Go implementation of it called gojq built by itchyny instead.

Current state

Most of the jq components that require syntactical support are, hmm, implemented and tested against queries taken from the jq user manual. Exceptions are imports and module headers. Many builtin functions are missing, include those require intrinsic implementation. Need more unit tests. Most of the CLI options are missing.

Goals

Learn jq.

Ideas

Since we use serde to deserialize input / serialize output, we should be able to support any format which has serde implementation without too much effort, except those require scheme (protobuf, thrift, etc.).

As a library, we should be able to make anything that implements Serialize as the input type, and anything implements Deserialize as the output type. So in theory, we should be able to write a library that can handle following;

#[derive(Serialize)]
struct GitHubRepo {
    user_name: String,
    repo_name: String,
    stars: usize,
}
#[derive(Deserialize)]
struct User {
    name: String,
    total_stars: usize,
}
let repos: Vec<GitHubRepo> = fetch_repos();
let users: Vec<User> = xq::somefunction_slurp<GitHubRepo, User>(repos, r#"
    group_by(.user_name)[]
    | { name: .[0].user_name, total_stars: ([.[].stars] | add) }
"#).collect();

I mean, it's not something that one should really use. It's just something possible and interesting... isn't it?

Install and run

Are you sure you want to use this??? It's not a daily-usable stage. Though if you want to try, you can install it via

$ cargo install xq
$ cat whatever.json | xq 'query goes here'

. Or to get the latet version, please clone this repository to your local, and run

$ cargo install --path path-to-cloned-dir
$ cat whatever.json | xq 'query goes here'

. Alternatively,

$ cat whatever.json | cargo run -- 'query goes here'

in the cloned directory to try out them without installing.

Uninstall

$ cargo uninstall xq

if you've installed via cargo install. Also remove the cloned directory if you've cloned the repository.

Acknowledgements

Although this isn't a direct translation at all, I referred to jq built by Stephen Dolan and gojq built by itchyny. Thank you for the interesting product!

Credits

Test cases in tests/from_manual are taken from the jq user manual, distributed under CC BY 3.0, Copyright (C) 2012 Stephen Dolan. Please refer to mod.rs there for more detail.

Author

Mi_Sawa

LICENSE

MIT. Please refer to LICENSE file.

Comments

Compile failed

after cargo install, I got the following messages, it maybe caused by breaking change of clap 3.x

error[E0432]: unresolved import `clap::Clap`
 --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:2:25
  |
2 | use clap::{AppSettings, Clap, ValueHint};
  |                         ^^^^ no `Clap` in the root

error: cannot determine resolution for the derive macro `Clap`
 --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:9:10
  |
9 | #[derive(Clap, Debug)]
  |          ^^^^
  |
  = note: import resolution is stuck, try simplifying macro imports

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:10:3
   |
10 | #[clap(author, about, version)]
   |   ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:11:3
   |
11 | #[clap(long_version(option_env!("LONG_VERSION").unwrap_or(env!("CARGO_PKG_VERSION"))))]
   |   ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:12:3
   |
12 | #[clap(setting(AppSettings::ColoredHelp))]
   |   ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:15:7
   |
15 |     #[clap(default_value("."))]
   |       ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:19:7
   |
19 |     #[clap(
   |       ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:30:7
   |
30 |     #[clap(short, long)]
   |       ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error: cannot find attribute `clap` in this scope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:34:7
   |
34 |     #[clap(short('v'), long("verbose"), parse(from_occurrences))]
   |       ^^^^
   |
   = note: `clap` is in scope, but it is a crate, not an attribute

error[E0599]: no function or associated item named `parse` found for struct `Args` in the current sc
ope
  --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/xq-0.1.7/src/bin/main.rs:54:28
   |
13 | struct Args {
   | ----------- function or associated item `parse` not found for this
...
54 |     let args: Args = Args::parse();
   |                            ^^^^^ function or associated item not found in `Args`
   |
   = help: items from traits can only be used if the trait is implemented and in scope
   = note: the following traits define an item `parse`, perhaps you need to implement one of them:
           candidate #1: `Parser`
           candidate #2: `nom::internal::Parser`
           candidate #3: `nom::sequence::Tuple`

Some errors have detailed explanations: E0432, E0599.
For more information about an error, try `rustc --explain E0432`.
error: failed to compile `xq v0.1.7`, intermediate artifacts can be found at `/tmp/cargo-installlxSc
4q`

Caused by:
  could not compile `xq` due to 10 previous errors

opened by hh9527 3

Deep nesting of string interpolation slows parser

Nesting string interpolation extremely slows the parser (not execution because adding invalid token to the query does not exit quickly).

❯ time xq -n '"\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\(1+2)")")")")")")")")")")")")")")")")")")")")")"'
"3"
xq -n   16.01s user 0.02s system 99% cpu 16.034 total

❯ time jq -n '"\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\("\(1+2)")")")")")")")")")")")")")")")")")")")")")"'
"3"
jq -n   0.00s user 0.00s system 65% cpu 0.010 total

opened by itchyny 3

"\u007f" is printed without escaped
Not sure where the difference come from, but I noticed following behavior.

❯ jq -n '"\u007f"' "\u007f" ❯ xq -n '"\u007f"' "" ❯ xq -n '"\u007f"' | xxd -p 227f220a
opened by itchyny 2
Implement trigonometric and hyperbolic functions

This PR adds trigonometric functions, inverse trigonometric functions, hyperbolic functions, and inverse hyperbolic functions. In tests, I used floor to round the results because we cannot get some floating-point number exactly (for example, we can't test 2 | sin equals to 0.9092974268256817 due to deserializing error). Also note that current testing macro does not allow to test nan value (and maybe infinite as well).

opened by itchyny 2

Optional object indexing suffix differences

Using optional object indexing as term suffix seems to have different semantics.

❯ jq -n '[1,{x:2},3,{x:4}] | [.[].x?]'
[
  2,
  4
]

❯ xq -n '[1,{x:2},3,{x:4}] | [.[].x?]'
[]

❯ xq --version
xq 0.2.3-a7d4cfbf962f3570b326dd97ecf288365d396ed4

Interestingly, adding | will fix the behavior.

❯ xq -n '[1,{x:2},3,{x:4}] | [.[]|.x?]'
[
  2,
  4
]

bug

opened by itchyny 2

Binding variable after comma operator does not work

❯ jq -n '[1, 2 as $x | $x + 1]'
[
  1,
  3
]

❯ xq -n '[1, 2 as $x | $x + 1]'
Error: compile query

Caused by:
    ParseError(UnrecognizedToken(Loc { line: 0, col: 6, byte_idx: 6 }, "Keyword(\"as\", As)", ["\"!=\"", "\"%\"", "\"%=\"", "\")\"", "\"*\"", "\"*=\"", "\"+\"", "\"+=\"", "\",\"", "\"-\"", "\"-=\"", "\"/\"", "\"//\"", "\"//=\"", "\"/=\"", "\":\"", "\";\"", "\"<\"", "\"<=\"", "\"=\"", "\"==\"", "\">\"", "\">=\"", "\"]\"", "\"and\"", "\"catch\"", "\"elif\"", "\"else\"", "\"end\"", "\"or\"", "\"then\"", "\"|\"", "\"|=\"", "InterpolationEnd"]))

❯ xq --version
xq 0.1.9-dcd6c4a024b54b3b2d0dc7ed90a349d42c11aa67

This looks like a regression due to switching to lalrpop.

❯ xq -n '[1, 2 as $x | $x + 1]'
[
  1,
  3
]

❯ xq --version
xq 0.1.9-95558180fabd53143a91d72e226780939461f8c5

opened by itchyny 2

Fix stringify in string interpolation

This PR fixes the stringify function used in string interpolation. I'm not sure it's ok to change the behavior of stringify, but it is still used only for this purpose. Note that the default format is @text, and tostring filter in jq.

opened by itchyny 2
Fix path tracking on value arguments
I received a bug report to gojq (https://github.com/itchyny/gojq/issues/186). This bug is reproducible in xq as well. I fixed the bug.

$ xq -n 'def f($x): .y; path(f(.x))' [ "x", "y" ] $ jq -n 'def f($x): .y; path(f(.x))' [ "y" ]
opened by itchyny 1
Improve implementation of unique_by filter

Unsure what happened in #147 (I believe GitHub automatically changes the base branch when it get merged but failed now?), this is a PR for master branch.

opened by itchyny 1
Change indexing string by out of range to emit null

While the current implementation is same as gojq v0.12.7, I changed in the HEAD version to emit null for indexing string by out of range. This is more consistent with array indexing and behaviors of other languages; in JavaScript "abcde"[10] results in undefined, not "". What do you think?

opened by itchyny 1
Implement indexing by array and object
I implemented indexing array by array (=indices), array by object (=slicing), string by object (=slicing).

❯ jq -nc '[0,1,0,2,0] | .[[0]]' [0,2,4] ❯ jq -nc '[0,1,0,2,0] | .[{start:1, end:4}]' [1,0,2] ❯ jq -nc '"abcde" | .[{start:1, end:4}]' "bcd"
opened by itchyny 1
CLI support for multiple inputs
This feature, in particular, is quite important to me: the ability to merge multiple documents in one query. It is a major feature of jq, and one missing from both xq and dasel

But it calls for a design choice:

Stay compatible with jq CLI and add --slurp-file and inputs, but quid of mixed format inputs?

Implement a xq way to do it (and possibly break xq specific options to be more streamlined?)
opened by Larandar 0
feat(tracing): move to tracing library
This is IMO the first step toward #119; the ability to have structured logs will for sure be useful for it. As a bonus, I updated clap to 4.0 and fixed clippy pestering about variables in format! directly.

feat(tracing): move to tracing library

chore(deps): update clap to 4.0

chore(clippy): fix clippy warnings
opened by Larandar 0
Handle large integer values losslessly?
Currently these are formatted with scientific notation (in some cases lossily). For example (from a user JSON object from the Twitter API):

$ xq .id < twitter-test.json 1.470944601309528e18 $ jq .id < twitter-test.json 1470944601309528000 $ gojq .id < twitter-test.json 1470944601309528072

Is this intentional? I'm currently using gojq instead of jq specifically because of how it handles values like this, and the lossless approach seems like it would generally be the least likely to cause issues for users.
opened by travisbrown 4
Implement `--stream`
To really implement this without expand JSON in memory, we need one of followings

generator feature so that we can yield to pass value into the VM while keeping the intermediate state of the parser,

spawn a thread that reads input, and send the result into std::sync::mpsc::sync_channel(0) (or 1 so the reader thread can read one element ahead) to do a similar with above, or

implement a JSON parser on our own (and perhaps yaml as well).
opened by MiSawa 5
Limit string repeat
fuzz found

$ cargo fuzz fmt xq artifacts/xq/minimized-from-35b822bd4dbd5db9bca8f9b3c812cec01a6503fd Output of `std::fmt::Debug`: ( "tostring*8E18 ", "8 ", )

Essentially this is "a" * 8E18 that cause an allocation with a huge amount of bytes causing an memory allocation failure. Though we can't fully prevent memory allocation issue anyway...
opened by MiSawa 0