A JSON Query Language CLI tool built with Rust 🦀

Davy Duperron

Last update: Jan 1, 2023

Related tags

Encoding JSON rust cli json utility tools tool cargo rustlang devops-tools

Overview

JQL

A JSON Query Language CLI tool built with Rust 🦀

📜 Core philosophy

📦 Stay lightweight
🎮 Keep its features as simple as possible
🧠 Avoid redundancy
💡 Provide meaningful error messages
↔️ Eat JSON as input, process, output JSON back

🚀 Installation

Cargo

cargo install jql

Archlinux

The AUR package is maintained by @frol.

yay -S jql

Homebrew

brew install jql

🛠️ Usage

If you find some of the following examples confusing, please have a look at The JavaScript Object Notation (JSON) Data Interchange Format.

Root selection

"This is a valid JSON text with one value"

jql '.' example.json

"This is a valid JSON text with one value"

Child selection

{
  "some": {
    "property": "yay!"
  }
}

jql '"some"."property"' example.json

"yay!"

Index selection

{
  "primes": [7, 11, 13]
}

jql '"primes".[0]' example.json

Please note that the following is also valid:

jql '"primes"[0]"' example.json

You can also select a set of indexes:

jql '"primes".[2,0]' example.json

[13, 7]

Range selection

{
  "cats": [{ "first": "Pixie" }, { "second": "Kitkat" }, { "third": "Misty" }]
}

jql '"cats".[1:2]' example.json

[
  {
    "second": "Kitkat"
  },
  {
    "third": "Misty"
  }
]

Please note that you can reverse it:

jql '"cats".[2:1]' example.json

[
  {
    "third": "Misty"
  },
  {
    "second": "Kitkat"
  }
]

Bonus, you can do it again to get it back:

jql '"cats".[2:1].[1:0]' example.json

[
  {
    "second": "Kitkat"
  },
  {
    "third": "Misty"
  }
]

Please note that you can still access the children:

jql '"cats".[2:1].[0]."third"' example.json

"Misty"

You can also use the start or the end position as a range selector:

jql '"cats".[1:]' example.json

[
  {
    "second": "Kitkat"
  },
  {
    "third": "Misty"
  }
]

jql '"cats".[:1]' example.json

[
  {
    "first": "Pixie"
  },
  {
    "second": "Kitkat"
  }
]

Array selection

{
  "primes": [7, 11, 13]
}

jql '"primes".[]' example.json

[7, 11, 13]

Please note that this is basically an alias for a full range selection:

jql '"primes".[0:2]' example.json

Property selection

{
  "object": { "a": 1, "b": 2, "c": 3 }
}

jql '"object".{"a","c"}' example.json

{
  "a": 1,
  "c": 3
}

Property selection can also be used with indexes and ranges. Please note that in this case a remapping/transformation is applied to the JSON data:

{
  "alpha": "red",
  "beta": "green",
  "gamma": "blue"
}

jql '{[2,0,1]}' example.json

{
  "2": "blue",
  "0": "red",
  "1": "green"
}

jql '{[1:2]}' example.json

{
  "1": "green",
  "2": "blue"
}

This is pretty unusual but it might help in some scenarios when e.g. one wants to extract some properties out of a complex JSON structure based on their order:

{
  "some-property": [
    {
      "key1": [
        {
          "subkey1": "value"
        }
      ],
      "key2": 123
    },
    {
      "key3": [
        {
          "subkey3": "value"
        }
      ],
      "key4": "something"
    }
  ]
}

jql '.."some-property"|{[0]}|"0"' example.json

[
  {
    "subkey1": "value"
  },
  {
    "subkey3": "value"
  }
]

Multi-selection

{
  "one": [1, 2, 3],
  "two": 2,
  "three": 3
}

jql '"one".[2:0],"two","three"' example.json

[[3, 2, 1], 2, 3]

Filter

{
  "laptops": [
    {
      "laptop": {
        "brand": "Apple",
        "options": ["a", "b", "c"]
      }
    },
    {
      "laptop": {
        "brand": "Asus",
        "options": ["d", "e", "f"]
      }
    }
  ]
}

jql '"laptops"|"laptop"' example.json

[
  {
    "brand": "Apple",
    "options": ["a", "b", "c"]
  },
  {
    "brand": "Asus",
    "options": ["d", "e", "f"]
  }
]

You can also combine a filter with a child selection, a multi-selection and ranges at the same time:

jql '"laptops"|"laptop"."brand"' example.json

["Apple", "Asus"]

jql '"laptops".[1:0]|"laptop"."brand","laptops"|"laptop"."brand"' example.json

[
  ["Asus", "Apple"],
  ["Apple", "Asus"]
]

Please note that you can combine filters to achieve the same result:

jql '"laptops".[1:0]|"laptop"|"brand","laptops"|"laptop"|"brand"' example.json

[
  ["Asus", "Apple"],
  ["Apple", "Asus"]
]

Flatten arrays

{
  "dna": [[[[["c", "a", "c"]]]], "g", "t", [[["a", ["t"]]]]]
}

jql '.."dna"' example.json

["c", "a", "c", "g", "t", "a", "t"]

Truncate

The truncate selector ! can be used to stop walking the children's values and to explore an unknown JSON file / structure. Each children is then transformed into a JSON primitive for convenience, e.g.:

primitive	value	result
object	`{ "a": 1, "b": 2, "c": 3 }`	`{}`
array	`[1, 2, 3]`	`[]`
string	`"foo"`	`"foo"`
number	`666`	`666`
null	`null`	`null`

{
  "foo": {
    "a": null,
    "b": "bar",
    "c": 1337,
    "d": {
      "woot": [1, 2, 3]
    }
  }
}

jql '.!' example.json

{
  "foo": {}
}

jql '"foo"!' example.json

{
  "a": null,
  "b": "bar",
  "c": 1337,
  "d": {}
}

Special characters

{
  ".valid": 1337,
  "": "yeah!",
  "\"": "yup, valid too!"
}

jql '".valid"' example.json

jql '""' example.json

"yeah!"

jql '"\""' example.json

"yup, valid too!"

💻 Shell integration

How to save the output

output.json">

jql '"foo"."bar"' input.json > output.json

How to read from stdin

cat example.json | jql '"foo"."bar"'

Available flags 🤖

Help

jql -h
jql --help

Check

The command will return a matching exit code based on the validity of the JSON content or file provided. No selector is needed in this case!

jql --c example.json
jql --check example.json

Please note that this flag is exclusive.

Version

jql -V
jql --version

Inlining the JSON output

jql -i '"some"."selector"' example.json
jql --inline '"some"."selector"' example.json

Raw output

Use the raw-output flag on a string selection to directly return the raw string without JSON double-quotes:

echo "{\"foo\":\"bar\"}" | jql --raw-output '"foo"'
bar
echo "{\"foo\":\"bar\"}" | jql -r '"foo"'
bar

Streaming

Use the stream flag to read a stream of JSON lines:

while true; do echo '{"foo": 2}'; sleep 1; done | jql '.!' --stream

while true; do echo '{"foo": 2}'; sleep 1; done | jql '.!' -s

Please note that this option is only about reading valid JSON output streamed line by line (e.g. Docker logs with the --follow flag). This is not an option to read an incomplete streamed content (e.g. a very large input)!

🍿 Library

This crate is both a binary (the CLI tool) and a library that can be directly used https://docs.rs/crate/jql/.

⚡ Performance

Some benchmarks comparing a set of similar functionalities provided by this tool and jq are available here.

Comments

Conditional filters
A feature to conditionally filter items of an array would be very useful! This would be akin to jq's select function.

My feeling is that the jql syntax is stronger than jq in it's simplicity, and thus should not adopt the "function" type signature (select(...)). I propose the following:

Given the below JSON

[ { "key": "give-me-this", "another-key": 42 }, { "key": "dont-give-me-this", "another-key": 13 }, { "key": "give-me-this", "another-key": 69 } ]

one could run this query:

$ jql '.|{"key" == "give-me-this"}' file.json [ { "key": "give-me-this", "another-key": 42 }, { "key": "give-me-this", "another-key": 69 }, ]

Inequalities would obviously be specified with !=. Other conditions could be supported as well, such as a way of checking if an array contains a specific element.

What do you think about this proposal? Also, thank you very much for making this amazing crate, it has been indispensable as a library in making mdzk happen!
enhancement
opened by kmaasrud 14

Cannot process some json as expected

I can make it work with jq, but not with jql :sob:, do you thinks it's possible or jql miss some feature that could make this possible?

Working jq command

jq -r '[ .[][][] ]'

Input

[
  {
    "key1": [
      {
        "subkey1": "value"
      }
    ]
  },
  {
    "key2": [
      {
        "subkey2": "value"
      }
    ]
  }
]

Expected output

[
  {
    "subkey1": "value"
  },
  {
    "subkey2": "value"
  }
]

enhancement

opened by shouze 12

Benchmark and query question

I edited the performance.sh with a benchmark to test jql an jq with a large, approximately 5.9 MB, json file (First 1000 Microsoft repos on github).

#!/usr/bin/env bash

REPORT=$(pwd)/PERFORMANCE.md
PERFORMANCE_TMP_DIR=$(pwd)/performance_tmp
MD_FILES="$PERFORMANCE_TMP_DIR/"*".md"
MIN_RUNS=1000
JSON=$(pwd)/ms.json
JSON_TMP=$(pwd)/tmp.json

# Remove export file if present.
rm -f $REPORT

# Create the directory.
mkdir $PERFORMANCE_TMP_DIR

# Get a large json file
if [ ! -f $JSON ]; then
    for PG in {1..10}; do
        echo -ne "\rGettings microsoft github repos ... ${PG}0% "
        curl -S -s -H "Accept: application/vnd.github.v3+json" \
        https://api.github.com/orgs/microsoft/repos?per_page=100\&page=$PG \
        -o $JSON_TMP
        cat $JSON_TMP >> $JSON 
        rm $JSON_TMP
    done
    echo -e "\nms.json is $(du -h ${JSON} | | awk '{print $1}')"
fi

Bench1="'{\"foo\": \"bar\"}'"
Bench2="'[1, 2, 3]'"
Bench3="'[1, [2], [[3]]]'"

# Run the benchmarks.
hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/OBJECT.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench1 | jq '.foo'" \
    "echo $Bench1 | jql '.\"foo\"'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/ARRAY_INDEX.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench2 | jq '.[0]'" \
    "echo $Bench2 | jql '.[0]'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/ARRAY_FLATTEN.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench3 | jq 'flatten'" \
    "echo $Bench3 | jql '...'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/LARGE_JSON.md" \
    --min-runs $MIN_RUNS \
    "cat $JSON | jq -r '.[] | .name, .url, .language, .stargazers_count, .watchers_count'" \
    "cat $JSON | jql '.|{\"name\",\"url\", \"language\", \"stargazers_count\", \"watchers_count\"}'"

# Merge all the markdown files into the performance one.
for md_file in $MD_FILES; do (cat "${md_file}"; echo) >> $REPORT; done

# Remove the directory.
rm -R -f $PERFORMANCE_TMP_DIR

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative | |:---|---:|---:|---:|---:| | cat /home/doc/Code/Bench/ms.json \| jq -r '.[] \| .name, .url, .language, .stargazers_count, .watchers_count' | 104.0 ± 9.8 | 89.2 | 121.3 | 1.00 | | cat /home/doc/Code/Bench/ms.json \| jql '.\|{"name","url", "language", "stargazers_count", "watchers_count"}' | 2317.6 ± 178.6 | 1801.3 | 2653.9 | 22.29 ± 2.71 |

jql ended up being much slower so I figure I probably did something wrong. How can I make this a fairer comparison?

opened by Th3Whit3Wolf 12

Crash when piping to a pager like less, bat
jql crashes for me when piping the output to a pager like less or bat and the output is not fully read (because it's longer than the viewport). I'm running version 2.9.3. installed via brew on macos (latest version).

This works as it only provides very few lines of output for my example file:

RUST_BACKTRACE=1 jql '.!' tmp.json | less

However, when outputting a bit more, it crashes when I quit the pager without scrolling to the end:

RUST_BACKTRACE=1 jql '.' tmp.json | less thread 'main' panicked at 'failed printing to stdout: Broken pipe (os error 32)', library/std/src/io/stdio.rs:1021:9 stack backtrace: 0: _rust_begin_unwind 1: std::panicking::begin_panic_fmt 2: jql::render_output 3: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll 4: jql::main note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. [1] 60860 abort RUST_BACKTRACE=1 jql '.' tmp.json | 60861 done less

When I scroll the less to the end (e.g. by pressing G key) or when I pipe to cat which also reads everything, there's no issue.

Please let me know when you need more information.
bug
opened by bb 11
Automate releases
Re.: #140

Publishes a release on pushed version tags and builds binaries for that release. The workflow uses rust-build.action, which means the supported triples are:

x86_64-pc-windows-gnu

x86_64-unknown-linux-musl

x86_64-apple-darwin

Optimally, aarch64-* should be supported as well, but this has proven hard to cross-compile (after a lot of trial and error.) Either way, building for x86 is a step in the right direction, and hopefully rust-build.action will support more triples in the future.
opened by kmaasrud 10
Access children of an array without using an array index
Hi I am trying to use jql to parse some elasticsearch query outputs but it seems its unable to fetch children of an array without an index.

Example:

curl localhost:9200/index/_search | jql '"hits"."hits".[]."_source"'

The output from the curl looks like this:

{ "hits": { "hits": [ { "_id": "b541465c-fd6a-11e8-becd-8c859022e616", "_index": "index", "_score": 1.0, "_source": { "key": "val" }, "_type": "index_type" } ] }

jql will return the entire object in the array instead of just the _source field. Though when I use an index it seems to work:

curl localhost:9200/index/_search | jql '"hits"."hits".[0]."_source"'

Thanks, awesome library, hopefully I might have sometime to fix this.
enhancement
opened by LucioFranco 8
Flattened/Minified json file cannot be parsed
I have a very big json file that was flattened / minified to a single line, and I would like to explore this document, however jq don't work and neither does jql.

$ jql --raw-output '.' aggregate-raw.json Invalid JSON file or content

That would be nice if JQL could process this document.
bug
opened by bric3 7
An example of working with arrays in the docs?

The readme for this project is quite good and I've been been very productive with it so far.

However, I'm having problems getting jql to print only the values. The closest I can get is screened below.

My question is, how does one start by selecting all the objects in an array then filtering out by keys, then displaying the children?

I haven't been able to figure it out from looking at the docs. [ { "txid": "15a10e97de1dca95dfeb3d8d06dac87dbb494eab8acc00abca0f96c140b97a73", "vout": 0, "address": "tb1qd2cvqk30f8kw3hj53wlj053dd9amnhhefg02gx", "label": "", "scriptPubKey": "00146ab0c05a2f49ece8de548bbf27d22d697bb9def9", "amount": 0.001, "confirmations": 61, "spendable": true, "solvable": true, "desc": "wpkh([b9a5ac96/0'/0'/2']03d94b78001ec7bdfffe41cca947e77219342272bb62946af779311358f835e57f)#3e78xplt", "safe": true }, { "txid": "393e1b08b7d2afd21f41f23025f5d79a06a7d12bc95a598de8b3c2f0340293a0", "vout": 1, "address": "tb1qd2cvqk30f8kw3hj53wlj053dd9amnhhefg02gx", "label": "", "scriptPubKey": "00146ab0c05a2f49ece8de548bbf27d22d697bb9def9", "amount": 0.001, "confirmations": 61, "spendable": true, "solvable": true, "desc": "wpkh([b9a5ac96/0'/0'/2']03d94b78001ec7bdfffe41cca947e77219342272bb62946af779311358f835e57f)#3e78xplt", "safe": true }, { "txid": "faa64c516c90302c624c7f7ffc26366376ac12813b1cc6e4eb8a192e3e0f08e4", "vout": 0, "address": "tb1qtlefx6f36vyx4l4g2wyvas4ghyuynn46xptyya", "label": "", "scriptPubKey": "00145ff2936931d3086afea85388cec2a8b93849ceba", "amount": 0.0022, "confirmations": 61, "spendable": true, "solvable": true, "desc": "wpkh([b9a5ac96/0'/0'/1']03a430c19febf6d59786fe600ff185e2e8c4efa81212ead8f5a741a9427090e766)#9lt0k576", "safe": true } ]

Final Goal; $ bt listunspent | jql -r 'something something something' 15a10e97de1dca95dfeb3d8d06dac87dbb494eab8acc00abca0f96c140b97a73 0 0.001 393e1b08b7d2afd21f41f23025f5d79a06a7d12bc95a598de8b3c2f0340293a0 1 0.001 faa64c516c90302c624c7f7ffc26366376ac12813b1cc6e4eb8a192e3e0f08e4 0 0.0022

opened by keblek 7
Windows build always gets 'Empty group' in all cases but `.` (once again)

#96 was closed prematurely. @yamafaktory you closed #96 and didn't notice echo '{ "primes": [7, 11, 13] }' | jql '"primes"' has been tested by me with same result. And what about CMD results? Powershell on Linux may behave differently. Please take a closer look.

opened by cpkio 6

ranges on top-level arrays are not handeled properly

data-input

(shortened from https://api.github.com/orgs/tauri-apps/repos?per_page=100&type=sources):

[
  {
    "id": 196701619,
    "node_id": "MDEwOlJlcG9zaXRvcnkxOTY3MDE2MTk=",
    "name": "tauri",
    "full_name": "tauri-apps/tauri",
    "html_url": "https://github.com/tauri-apps/tauri",
    "open_issues_count": 55,
    "forks": 51,
    "open_issues": 55,
    "watchers": 1629,
    "default_branch": "dev"
  },
  {
    "id": 206315614,
    "node_id": "MDEwOlJlcG9zaXRvcnkyMDYzMTU2MTQ=",
    "name": "examples",
    "full_name": "tauri-apps/examples",
    "html_url": "https://github.com/tauri-apps/examples",
    "open_issues_count": 7,
    "forks": 4,
    "open_issues": 7,
    "watchers": 6,
    "default_branch": "master"
  }
]

example of the bug

jql '[0]."name"' org.json
"tauri"

jql '[0:]."name"' org.json
Node "name" not found on parent range [0:]

bug

opened by dym-sh 6

State the performance

I suggest you to state whether jql is fast or not, and if the throughput is one of the goals of the project. As simple as it sounds, it may drive adoption.

Publishing benchmarks result may also help.

Good luck!
enhancement

opened by tacone 5
Sort keys of objects
The sorting flag from jq is very helpful when dealing with JSON APIs.

-S sort keys of objects on output;

There is an example:

❯ echo '{ "foo": 1, "bar": 2, "nested": { "will": 1, "sort": 2, "as": 3, "well": 4 } }' | jq -S . { "bar": 2, "foo": 1, "nested": { "as": 3, "sort": 2, "well": 4, "will": 1 } }

Can we have this feature as well? Also, as a new rustacean I really appreciate your work. I had some issue to link lib-jq on my Mac, so this crate will save me a lot of time.
enhancement
opened by jameschenjav 9