Functional command-line JSON processor

Related tags

Command-line rust jq
Overview

rq

rq is a tiny functional language with which you can manipulate JSON. Basically, it is (an insignificant subset of!) jq, written in Rust.

Usage example

NOTE: This project is in its very early stages; lot's of essential functions—and perhaps even syntax—might be missing, and overall I can't guarantee that anything actually works. Use at your own risk :)

Installation

Use cargo install. For nix users, a dev-shell is provided by the flake; one can access it with nix develop. Additionally, you can run rq directly from the git repo:

$ nix run github:slotThe/rq

Usage

Call rq with an expression, and pipe some JSON into it!

$ cat test.json
[{"name": "John Doe", "age": 43, "phones": ["+44 1234567", "+44 2345678"]}]

$ cat test.json | rq '\x -> x.0.phones.1'
+44 2345678

Some more usage examples:

$ cat simple.json
[{"name": "John Doe", "age": 43, "phone": +44 1234567"},{"name":"Alice"},{"name":"Bob", "age":42}]

$ cat simple.json | rq 'map .name'
["John Doe","Alice","Bob"]

$ cat simple.json | rq 'map .age | foldl (+) 0'
85

$ cat simple.json | rq 'filter (get "age" | (>= 42)) | map (\x -> { x.name: x.age })'
[{"John Doe":43},{"Bob":42}]

$ cargo metadata --format-version=1 | rq '.packages | map .name'
[ahash, allocator-api2, anyhow, ariadne, cc, cfg-if, chumsky, hashbrown, libc, once_cell, proc-macro2, psm, quote, rq, stacker, syn, unicode-ident, unicode-width, version_check, winapi, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, yansi, zerocopy, zerocopy-derive]

The expression language

  • Constants: null, false, true, 1, 2.6, "string".

  • Lambdas, which can be written in various ways:

      \x -> x        |x| x        λx → x
    
  • Application is done via whitespace: (\x -> x 1 2) const. This would be akin to

      (|x| x(1, 2))(const)
    

    in pseudo-Rust notation (const is a builtin function).

  • Binary operations:

    • Arithmetic operations, with the usual precedence rules of * and / being preferred over + and -:

        1 + 3 * 5 + 4 - 7    ≡    ((1 + (3 * 5)) + 4) - 7
      

      Additionally + also concatenates strings.

        "furble" + "wurble"    ≡    "furblewurble"
      
    • Comparison operations:

        1 = 3 * 5 + 4 - 7    ≡    1 = (((3 * 5) + 4) - 7)
      
        1 < 4 + 5 = 5        ≡    (1 < (4 + 5)) = 5
      

    The following table details the precedence rules:

    Op Precedence
    *, / 3
    +, - 2
    =, !=, <, <=, >, >= 1
  • If-then-else expressions:

      if 5 = 2 + 3 then "wurble" else 4  ≡  if (5 = (2 + 3)) then "wurble" else 4
                                         ≡  "wurble"
    
  • Arrays: [1, 3, null]. Arrays can contain arbitrary expressions:

      λx → [1, get 0 x, if false then 1 else 5]
    
  • Objects: { "this": 3, "that": null }. Apostrophes can be omitted:

      { this: 3, that: null }    ≡    { "this": 3, "that": null }
    

    In fact, keys can be arbitrary expressions—just make sure they actually evaluate to something sensible!

      { if true then "this" else "thus": 3, that: null }
        ≡  { "this": 3, "that": null }
    
  • Expressions can be type-annotated with :: or , though this is usually not necessary.

      λ> 1 + 2 :: JSON
      3
    
      λ> :e 1 + 2 + (3 ∷ JSON)
      + (+ 1 2) (3 ∷ JSON)
    

Syntactic sugar

  • The get function—with which one can index arrays and objects—can be abbreviated by .:

      λx → x.0.this     ≡    λx → get "this" (get 0 x)
    
      (λx → x.0.this) [{this: 4}]    ≡    4
    

    Additionally, .0 is sugar for (|x| x.0). This composes sanely:

      .0.1.2  ≡  λx → get 2 (get 1 (get 0 x))
    

    Note that this syntax is only available if the to-be-indexed-thing is a variable.

      [1, 2, 3].0     # Parse error!
    
  • Instead of manually composing functions, | may be used instead;

      (get 0 | λx → { x.id: x.name }) [{id: 42, name: "Arthur"}, 4]
        ≡  { 42: Arthur }
    
  • Lambdas can be written taking multiple arguments, in which case they are automatically curried:

      \x y -> x    ≡    \x -> \y -> x    ≡    |x, y| x
    
  • A shadowed variable may be accessed using its De Bruijn index:

      λ> (λx → λx → x@2) 1 2
      variable not in scope: x@2
      λ> (λx → λx → x@1) 1 2
      1
      λ> (λx → λx → x@0) 1 2
      2
      λ> (λx → λx → x  ) 1 2
      2
    
  • Various binary operators can be written in pettier/alternative ways:

    • Multiplication: *, ·
    • Division: /, ÷
    • Equality: =, ==
    • Non-equality: !=, /=,
    • Less-or-equal: <=,
    • Bigger-or-equal: >=,

Standard library

  • Numerical operators:

    (+)  : JSON  JSON  JSON  -- Also works for string concatenation
    (-)  : JSON  JSON  JSON
    (*)  : JSON  JSON  JSON
    (/)  : JSON  JSON  JSON
  • Comparisons:

    Essensially, everything that is not false or null is considered truthy.

    (=)  : JSON  JSON  JSON
    (!=) : JSON  JSON  JSON
    (<)  : JSON  JSON  JSON
    (<=) : JSON  JSON  JSON
    (>)  : JSON  JSON  JSON
    (>=) : JSON  JSON  JSON
  • Higher order functions:

    -- `map f xs` applies `f` to every "value" in `xs`, which may be an
    -- array (in which case value means element), or an object (in which
    -- case it really means value).
    map : (JSON  JSON)  JSON  JSON
    
    -- Like map, `filter p xs` applies `p` to every value of `xs`.
    -- Keep the elements for which the predicate returns truthy.
    filter : (JSON  JSON)  JSON  JSON
    
    -- Left-associative fold over an array or (values of an) object; e.g.,
    --
    --   foldl f init [x₁, x₂, …, xₙ]  ≡  f(f(…f(init, x₁), …), xₙ)
    --
    foldl : (JSON  JSON  JSON)  JSON  JSON  JSON
  • Misc

    -- The identity function.
    id    : ∀a. a  a,
    
    -- Return the first argument
    const : ∀a. ∀b. a  b  a
    
    -- `get i x` gets the i'th thing out of x. I should be (evaluate to) a
    -- number or a string, with x evaluating to array or object, respectively.
    get   : JSON  JSON  JSON

REPL

A REPL is provided for getting familiar with the language; either call rq without arguments, or with a repl positional argument:

$ rq
λ>

By default, expressions will first be type-checked, and then evaluated as far as they can:

λ> 1 + 2
3

λ> |x| x
λx'. x'

λ> \x -> x x
Occurs check: can't construct infinite type: b ≡ b → c

λ> \x -> ids x
variable not in scope: ids

λ> (get 0 | λx → { x.id: x.name }) [{id: 42, name: "Arthur"}, 4]
{ 42: Arthur }

Additionally, the following keywords are available:

  • Pretty-print the expression given (this just runs the parser, followed by the pretty-printer): :e

    λ> :e \x -> x x
    λx. (x x)
    
    λ> :e \x -> get 0 x + 3 * 5 - 7
    λx. (- (+ (get 0 x) (· 3 5)) 7)
    
  • Type-check an expression, and print the type: :t

    λ> :t \f -> \g -> \x -> f x (g x)
    (a → b → c) → (a → b) → a → c
    
    λ> :t \x -> get 0 x + 3 * 5 - 7
    JSON → JSON
    
    λ> :t map
    (JSON → JSON) → JSON → JSON
    
    λ> :t \x -> x x
    Occurs check: can't construct infinite type: b ≡ b → c
    
  • Get information on a builtin function with :i:

    λ> :i <
    e < e' checks whether e is less than e'
    
    λ> :i map
    map f xs applies f to every value in xs, which may be an
    array (in which case »value« means element) or an object
    (in which case it really means value).
    
  • To list all builtin functions, use :l:

    λ> :i map
    +	Add two number, or concatenate two strings.
    -	Subtract two numbers.
    <	e < e' checks whether e is less than e'
    … and so on …
    
  • Debugging: :d

    λ> :d \x -> x 4 "flurble"
    Lam("x", App(App(Var("x"), Const(Num(OrdF64(4.0)))), Const(String("flurble"))))
    
    • Prettier, yet more verbose, output: :dp

        λ> :dp \x -> x x
        Lam(
            "x",
            App(
                Var(
                    "x",
                ),
                Var(
                    "x",
                ),
            ),
        )
      
You might also like...
Pink is a command-line tool inspired by the Unix man command.

Pink is a command-line tool inspired by the Unix man command. It displays custom-formatted text pages in the terminal using a subset of HTML-like tags.

Checkline: checkbox line picker for stdin line input

checkline is a Unix command line interface (CLI) terminal user interface (TUI) that prompts you to check each line of stdin, to pick each line to output to stdout

A full featured, fast Command Line Argument Parser for Rust

clap Command Line Argument Parser for Rust It is a simple-to-use, efficient, and full-featured library for parsing command line arguments and subcomma

Docopt for Rust (command line argument parser).

THIS CRATE IS UNMAINTAINED This crate is unlikely to see significant future evolution. The primary reason to choose this crate for a new project is if

Parse command line arguments by defining a struct.

StructOpt Parse command line arguments by defining a struct. It combines clap with custom derive. Documentation Find it on Docs.rs. You can also check

A command line progress reporting library for Rust
A command line progress reporting library for Rust

indicatif Documentation A Rust library for indicating progress in command line applications to users. This currently primarily provides progress bars

Low-level Rust library for implementing terminal command line interface, like in embedded systems.

Terminal CLI Need to build an interactive command prompt, with commands, properties and with full autocomplete? This is for you. Example, output only

⚡️ Lightning-fast and minimal calendar command line. Written in Rust 🦀
⚡️ Lightning-fast and minimal calendar command line. Written in Rust 🦀

⚡️ Lightning-fast and minimal calendar command line. It's similar to cal. Written in Rust 🦀

Command-Line program that takes images and produces the copy of the image with a thin frame and palette made of the 10 most frequent colors.
Command-Line program that takes images and produces the copy of the image with a thin frame and palette made of the 10 most frequent colors.

paleatra v.0.0.1 Command-Line program that takes an image and produces the copy of the image with a thin frame and palette made of the 10 most frequen

Comments
  • miette error diagnostics

    miette error diagnostics

    This pull request:

    • factors out all the custom error types into a new error module
    • implements help diagnostics for said errors
    • sets up the main app to make it ready to render the miette errors
    • does some custom displaying of the miette errors in case the user chose the repl mode
    • converts some std error types in the one shot mode for compatibility reasons

    This can be used to implement more detailed errors in the future since it'll support things like source spans to point to the error locations.

    Feel free to suggest better help diagnostics btw! 🙈

    opened by RobWalt 3
Owner
Tony Zorman
Mathematics PhD student, Haskell and Emacs devotee, @xmonad maintainer. I like free software. More personal projects at gitlab.com/slotThe
Tony Zorman
tpp (Tera Pre-Processor) is a versatile CLI (Command Line Interface) tool crafted for preprocessing files using the Tera templating engine.

tpp (Tera Pre-Processor) is a versatile CLI (Command Line Interface) tool crafted for preprocessing files using the Tera templating engine. Drawing inspiration from pre-processors like cpp and gpp, tpp is the next evolution with its powerful expressive toolset.

null 3 Nov 23, 2023
A dead simple functional testing tool for command line applications

Pharaoh : build that test pyramid! What it is Pharaoh is a dead simple, no permission needed, functional test runner for command line applications, wr

Kevin Sztern 17 Dec 13, 2021
Small command-line tool to switch monitor inputs from command line

swmon Small command-line tool to switch monitor inputs from command line Installation git clone https://github.com/cr1901/swmon cargo install --path .

William D. Jones 5 Aug 20, 2022
A Supra + Pandoc post-processor for footnote cross-references.

AutoCref AutoCref is a Supra and Pandoc post-processor that turns footnote cross-references in a Word document into automatically updated fields. Abou

null 2 Sep 4, 2022
This is a simple command line application to convert bibtex to json written in Rust and Python

bibtex-to-json This is a simple command line application to convert bibtex to json written in Rust and Python. Why? To enable you to convert very big

null 3 Mar 23, 2022
hj is a command line tool to convert HTTP/1-style text into JSON

hj hj is a command line tool to convert HTTP/1-style text into JSON. This command is inspired by yusukebe/rj, which is a standalone HTTP client that s

FUJI Goro 10 Aug 21, 2022
Rust Crate that allows to do interruptions in console. Will be implemented to functional terminal customization kit.

termpause Rust Crate that allows to do interruptions in console. Will be implemented to functional terminal customization kit. Usage Add this in your

Just Said 4 Sep 21, 2022
A structure editor for a simple functional programming language, with Vim-like shortcuts and commands.

dilim A structure editor for a simple functional programming language, with Vim-like shortcuts and commands. Written in Rust, using the Yew framework,

Joomy Korkut 6 Nov 18, 2022
🛠️ An experimental functional systems programming language, written in Rust and powered by LLVM as a backend.

An experimental functional systems programming language, written in Rust, and powered by LLVM as a backend. ?? Goal: The intent is to create a program

codex 3 Nov 15, 2023
Command-line HTTP client for sending a POST request to specified URI on each stdin line.

line2httppost Simple tool to read lines from stdin and post each line as separate POST request to a specified URL (TCP connection is reused though). G

Vitaly Shukela 3 Jan 3, 2023