Jq - Command-line JSON processor

Stephen Dolan

Last update: Jan 4, 2023

Related tags

Encoding JSON jq

Overview

jq

jq is a lightweight and flexible command-line JSON processor.

, Unix: , Windows:

If you want to learn to use jq, read the documentation at https://stedolan.github.io/jq. This documentation is generated from the docs/ folder of this repository. You can also try it online at jqplay.org.

If you want to hack on jq, feel free, but be warned that its internals are not well-documented at the moment. Bring a hard hat and a shovel. Also, read the wiki: https://github.com/stedolan/jq/wiki, where you will find cookbooks, discussion of advanced topics, internals, release engineering, and more.

Source tarball and built executable releases can be found on the homepage and on the github release page, https://github.com/stedolan/jq/releases

If you're building directly from the latest git, you'll need flex, bison (3.0 or newer), libtool, make, automake, and autoconf installed. To get regexp support you'll also need to install Oniguruma or clone it as a git submodule as per the instructions below. (note that jq's tests require regexp support to pass). To build, run:

git submodule update --init # if building from git to get oniguruma
autoreconf -fi              # if building from git
./configure --with-oniguruma=builtin
make -j8
make check

To build without bison or flex, add --disable-maintainer-mode to the ./configure invocation:

./configure --with-oniguruma=builtin --disable-maintainer-mode

(Developers must not use --disable-maintainer-mode, not when making changes to the jq parser and/or lexer.)

To build a statically linked version of jq, run:

make LDFLAGS=-all-static

After make finishes, you'll be able to use ./jq. You can also install it using:

sudo make install

If you're not using the latest git version but instead building a released tarball (available on the website), then you won't need to run autoreconf (and shouldn't), and you won't need flex or bison.

To cross-compile for OS X and Windows, see docs/Rakefile's build task and scripts/crosscompile. You'll need a cross-compilation environment, such as Mingw for cross-compiling for Windows.

Cross-compilation requires a clean workspace, then:

# git clean ...
autoreconf -i
./configure
make distclean
scripts/crosscompile <name-of-build> <configure-options>

Use the --host= and --target= ./configure options to select a cross-compilation environment. See also "Cross compilation" on the wiki.

Send questions to https://stackoverflow.com/questions/tagged/jq or to the #jq channel (http://irc.lc/freenode/%23jq/) on Freenode (https://webchat.freenode.net/).

Comments

Generate array always for specified xml element

Generate array always for specified xml element "aitem" even with single set .
It must open square braces as an array like aitem[{}]
Please also specify expression to write in /.jq file as a def, to maintain many single set items to be an array.

<root>
    <aitem>
        <name>abc</name>
        <value>123</value>
    </aitem>
    <bitem>
        <name>bbbb</name>
        <value>2222</value>
    </bitem>
    <bitem>
        <name>BB</name>
        <value>22</value>
    </bitem>
</root>

Normal Output aitem{} bitem[{}{}]

 {
  "root": {
    "aitem": {
      "name": "abc",
      "value": "123"
    },
    "bitem": [
      {
        "name": "bbbb",
        "value": "2222"
      },
      {
        "name": "BB",
        "value": "22"
      }
    ]
  }
}

Expected Output , aitem[{}] bitem[{}{}] even its single set aitem must become array

{
  "root": {
    "aitem": [
    {
      "name": "abc",
      "value": "123"
    }
    ],
    "bitem": [
      {
        "name": "bbbb",
        "value": "2222"
      },
      {
        "name": "BB",
        "value": "22"
      }
    ]
  }
 }

opened by ajit4sbi 5

is it possible to insert key value from 2nd array into 1st

Not sure if this is possible.

  "questions": [
    {
      "season": 1,
      "episode": 1,
      "question": "what is the name of the character played by Michael Richards?"
    }
  ],
  "episodes": [
    {
      "season": 1,
      "episode": 1,
      "title": "The Seinfeld Chronicles"
    },
    {
      "season": 1,
      "episode": 2,
      "title": "The Stake Out"
    }
  ]

looking to insert 2nd array title, if it matches the season and episode of 1st array

so it would add "title": "The Seinfeld Chronicles" to the questions array for season1 and episode1.

opened by chrismccoy 1

Can't chain generic object indexes

Describe the bug When selecting subfields (e.g. .foo.bar), using the generic notation (e.g. .["foo"].["bar"]) throws a syntax error.

To Reproduce Given the input (sample.json):

{
  "title": "Title 1",
  "media:content": {
    "media:credits": "media:content.media:credits 1"
  }
}

I want to pull .media-content.media:credits to the top level and relabel the field to "credits" like this:

$ cat sample.json | jq '{title, credits: .["media:content"].["media:credits"]}'

Expected output

{
  "title": "Title 1",
  "credits": "media:content.media:credits 1"
}

Actual output

jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
{title, credits: .["media:content"].["media:credits"]}
jq: 1 compile error

Environment (please complete the following information):

OS and Version: macOS Ventura 13.1
jq version: 1.6

Additional context If the keys can be used in identifier-like syntax, like this, then it works: (sample2.json)

{
  "title": "Title 1",
  "content": {
    "credits": "content.credits 1",
  },
}

$ cat sample2.json | jq '{title, credits: .content.credits}'

{
  "title": "Title 1",
  "credits": "content.credits 1"
}

But even then , it still doesn't work with the generic quoted syntax chained after the first selector

$ cat sample2.json | jq '{title, credits: .["content"].["credits"]}'  # Error
$ cat sample2.json | jq '{title, credits: .content.["credits"]}'      # Error
$ cat sample2.json | jq '{title, credits: .["content"].credits}'      # Works

Addendum: While writing this, I did finally discover a workaround using the pipe operator:

$ cat sample2.json | jq '{title, credits: .["content"] | .["credits"]}'.            # Works
$ cat sample.json | jq '{title, credits: .["media:content"] | .["media:credits"]}'. # Works

But I'm filing the bug report anyways because, according to the docs, .foo.bar should be equivalent to .["foo"].["bar"]. (and likewise, .["foo"].["bar"] should be equivalent to .["foo'] | .["bar"]

Thanks for the tool!

opened by partap 4

Name of MacOS binary ends in "-amd64"

On your download page https://stedolan.github.io/jq/download/, the name of the MacOS binary for v1.6 is "jq-osx-amd64". Apple has never used AMD CPUs, so what architecture is this binary actually for? If it really is for Macs, you should change it's misleading name.

opened by Twangist 2
Unrecognized options --nul-output, --binary on Windows

jq issues "Unknow option" when trying to pass either --nul-output or --binary to the executable.

` C:>jq --version jq-1.6

C:>jq --binary /cygdrive/c/Programs/cygwin64/bin/jq: Unknown option --binary Use /cygdrive/c/Programs/cygwin64/bin/jq --help for help with command-line options, or see the jq manpage, or online docs at https://stedolan.github.io/jq

C:>jq --nul-output /cygdrive/c/Programs/cygwin64/bin/jq: Unknown option --nul-output Use /cygdrive/c/Programs/cygwin64/bin/jq --help for help with command-line options, or see the jq manpage, or online docs at https://stedolan.github.io/jq

C:>jq-win --version jq-1.6

C:>jq-win --binary C:\Software\utils\jq-win.exe: Unknown option --binary Use C:\Software\utils\jq-win.exe --help for help with command-line options, or see the jq manpage, or online docs at https://stedolan.github.io/jq

C:>jq-win --nul-output C:\Software\utils\jq-win.exe: Unknown option --nul-output Use C:\Software\utils\jq-win.exe --help for help with command-line options, or see the jq manpage, or online docs at https://stedolan.github.io/jq `

opened by apsen-github 0
`id` integer field showing the same value in result, when diffent in original json file.
Describe the bug For some json files there is strange behavior for id field. I see the same value in result view, but in input json file the id value are distinct. Here is the result what I see https://ibb.co/7GsQn2g . Original json is below. Result:

$ cat test_failure.json | jq { "sections": [ { "items": [ { "id": 50000000014435120, "price": { "full": 3450 } }, { "id": 50000000014435120, "price": { "full": 3450 } }, { "id": 50000000014435120, "price": { "full": 3450 } }, { "id": 50000000014435120, "price": { "full": 3450 } } ], "name": "cancelled" } ] }

To Reproduce json: { "sections": [ { "items": [ { "id": 50000000014435121, "price": { "full": 3450 } }, { "id": 50000000014435122, "price": { "full": 3450 } }, { "id": 50000000014435123, "price": { "full": 3450 } }, { "id": 50000000014435124, "price": { "full": 3450 } } ], "name": "cancelled" } ] }

command: cat test_failure.json | jq

Expected behavior all id-s are different as in raw json.

Environment (please complete the following information):

Ubuntu 18.04

jq-1.5-1-a5b5cbe

Additional context I see this bug also in Firefox and Google Chrome. I think the use this library for preview of json.
opened by melonaerial 5

Releases(jq-1.6)

jq-1.6(Nov 2, 2018)
New in this release since 1.5:

Destructuring Alternation

New Builtins:

builtins/0

stderr/0

halt/0, halt_error/1

isempty/1

walk/1

utf8bytelength/1

localtime/0, strflocaltime/1

SQL-style builtins

and more!

Add support for ASAN and UBSAN

Make it easier to use jq with shebangs (8f6f28c)

Add $ENV builtin variable to access environment

Add JQ_COLORS env var for configuring the output colors

Bug fixes:

Calling jq without a program argument now always assumes . for the program, regardless of stdin/stdout. (5fe0536)

Make sorting stable regardless of qsort. (7835a72)

Adds a local oniguruma submodule and the ./configure --with-oniguruma=builtin option to make it easier to build with oniguruma support on systems where you can't install system-level libraries. (c6374b6 and 02bad4b)

And much more!

Source code(tar.gz)
Source code(zip)
jq-1.6.tar.gz(1.66 MB)
jq-1.6.zip(1.84 MB)
jq-linux32(2.65 MB)
jq-linux64(3.77 MB)
jq-osx-amd64(843.78 KB)
jq-win32.exe(2.58 MB)
jq-win64.exe(3.36 MB)
jq-1.5(Aug 16, 2015)
Thanks to the 20+ developers who have sent us PRs since 1.4, and the many contributors to issues and the wiki.

The manual for jq 1.5 can be found at https://stedolan.github.io/jq/manual/v1.5/

Salient new features since 1.4:

regexp support (using Oniguruma)!

a proper module system

import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

and

include "foo/bar"; # import foo/bar.jq's defs into the top-level

destructuring syntax (. as [$first, $second, {$foo, $bar}] | ...)

math functions

an online streaming parser

minimal I/O builtions (inputs, debug)

One can now write:

jq -n 'reduce inputs as $i ( ... )'

to reduce inputs in an online way without having to slurp them first! This works with streaming too.

try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

a lexical non-local exit system

One can now say

label $foo | ..... | break $foo

where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

tail call optimization (TCO), which allows efficient recursion in jq

a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next)), many of which internally use TCO

an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

the ability to read module data files

import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

--argjson var '<JSON text>'

Using --arg var bit me too many times :)

--slurpfile var "filename"

Replaces the --argfile form (which is now deprecated but remains for backward compatibility).

support for application/json-seq (RFC7464)

a large variety of new utility functions, many being community contributions (e.g., bsearch, for binary searching arrays)

datetime functions

a variety of performance enhancements

def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

test and build improvements, including gcov support

Lastly, don't forget the wiki! The wiki has a lot of new content since 1.4, much of it contributed by the community.
Source code(tar.gz)
Source code(zip)
jq-1.5.tar.gz(721.98 KB)
jq-1.5.zip(729.88 KB)
jq-linux32(1.52 MB)
jq-linux32-no-oniguruma(1.24 MB)
jq-linux64(2.88 MB)
jq-osx-amd64(634.76 KB)
jq-win32.exe(1.16 MB)
jq-win64.exe(2.21 MB)
jq-1.5rc2(Jul 27, 2015)
Thanks to the 20+ developers who have sent us PRs since 1.4, and the many contributors to issues and the wiki. We're nearing a 1.5 release, finally.

Salient new features since 1.4:

regexp support (using Oniguruma)!

a proper module system

import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

and

include "foo/bar"; # import foo/bar.jq's defs into the top-level

destructuring syntax (. as [$first, $second, {$foo, $bar}] | ...)

math functions

an online streaming parser

minimal I/O builtions (inputs, debug)

One can now write:

jq -n 'reduce inputs as $i ( ... )'

to reduce inputs in an online way without having to slurp them first! This works with streaming too.

try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

a lexical non-local exit system

One can now say

label $foo | ..... | break $foo

where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

tail call optimization (TCO), which allows efficient recursion in jq

a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next)), many of which internally use TCO

an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

the ability to read module data files

import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

--argjson var '<JSON text>'

Using --arg var bit me too many times :)

--slurpfile var "filename"

Replaces the --argfile form (which is now deprecated but remains for backward compatibility).

support for application/json-seq (RFC7464)

a large variety of new utility functions, many being community contributions (e.g., bsearch, for binary searching arrays)

datetime functions

a variety of performance enhancements

def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

test and build improvements, including gcov support

Lastly, don't forget the wiki! The wiki has a lot of new content since 1.4, much of it contributed by the community.
Source code(tar.gz)
Source code(zip)
jq-1.5rc2.tar.gz(697.31 KB)
jq-1.5rc2.zip(739.09 KB)
jq-linux-x86(1.39 MB)
jq-linux-x86_64(3.04 MB)
jq-osx-x86_64(634.89 KB)
jq-win32.exe(1.16 MB)
jq-win64.exe(2.21 MB)
jq-1.5rc1(Jan 1, 2015)
Salient new features since 1.4:

regexp support (using Oniguruma)

an online streaming parser

Included is the ability to control reading of inputs from the jq program, using the new input and inputs builtins.

Finally we can write:

jq -n 'reduce inputs as $i ( ... )' # reduce online!

to reduce inputs without slurping them first. This works with streaming too.

try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

a lexical non-local exit system

Using try/catch to break out of control structures was not a good thing. A lexical mechanism is.

You can now say

label $foo | ..... | break $foo

where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

tail call optimization (TCO), which allows efficient recursion in jq

a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next))

an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

a proper module system

import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

the ability to read module data files

import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

--argjson var '<JSON text>'

Using --arg var bit me too many times :)

--in-place / -i for in-place editing of files

support for application/json-seq.

a variety of new utility functions, many being community contributions

a variety of performance enhancements (e.g., constant folding)

def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

Lastly, don't forget the wiki! It has a lot of new content since 1.4, much of it contributed by the community.
Source code(tar.gz)
Source code(zip)
jq-1.5rc1.tar.gz(630.66 KB)
jq-linux-x86_64-static(3.26 MB)
jq-win32.exe(1.46 MB)
jq-win64.exe(1.29 MB)
jq.1(77.38 KB)
jq-1.4(Aug 8, 2015)

The manual for jq 1.4 can be found at https://stedolan.github.io/jq/manual/v1.4/
Source code(tar.gz)
Source code(zip)
jq-1.4.tar.gz(564.76 KB)
jq-linux-x86(578.96 KB)
jq-linux-x86_64(486.13 KB)
jq-osx-x86(277.36 KB)
jq-osx-x86_64(224.26 KB)
jq-solaris11-32(359.67 KB)
jq-solaris11-64(409.53 KB)
jq-win32.exe(742.20 KB)
jq-win64.exe(918.09 KB)
jq-1.3(Aug 8, 2015)

The manual for jq 1.3 can be found at https://stedolan.github.io/jq/manual/v1.3/
Source code(tar.gz)
Source code(zip)
jq-linux-x86(553.53 KB)
jq-linux-x86_64(602.46 KB)
jq-osx-x86(224.02 KB)
jq-osx-x86_64(252.96 KB)
jq-win32.exe(540.31 KB)
jq-win64.exe(620.72 KB)
jq-1.2(Aug 8, 2015)

jq 1.2
Source code(tar.gz)
Source code(zip)
jq-1.1(Aug 8, 2015)

jq 1.1
Source code(tar.gz)
Source code(zip)
jq-1.0(Aug 8, 2015)

jq 1.0
Source code(tar.gz)
Source code(zip)