hq
jq, but for HTML.
hq reads HTML and converts it into a JSON object based on a series of CSS selectors. The selectors are expressed in a similar way to JSON, but where the values are CSS selectors. For example:
{posts: .athing | [ {title: .titleline > a, url: .titleline > a | @(href)} ] }
This will select all .athing elements, and it will create an array (| [{...}]) of objects for each element selected. Then for each element it will select the text of the titleline > a element, and the href attribute (| @(href)).
The end result is the following structure:
{
"posts": [
{
"title": "...",
"url": "..."
}
]
}
Install
cargo install html-query
Examples
Full hacker news story extraction
{posts: .athing | [{href: .titleline > a | @(href), title: .titleline > a, meta: @sibling(1) | {user: .hnuser, posted: .age | @(title) }}]}
This selects each .athing element, extracts the URL from the href attribute as well as the title. It then selects the sibling .athing element, and extracts the user and post time from that:
{
"posts": [
{
"title": "...",
"url": "...",
"meta": {
"posted": "...",
"user": "..."
}
}
]
}
Special query syntax
Selecting attributes
.foo | @(href)
This will select the href attribute from the first element matching .foo.
Parents
.foo | @parent
This will return the parent element from the first element matching .foo.
Siblings
.foo | @sibling(1)
This will return the sibling element from the first element matching .foo.
829 Dec 28, 2022
80 Dec 21, 2022
1 Dec 11, 2021
79 Sep 29, 2022
1.2k Dec 30, 2022
9 Dec 27, 2022
6.3k Jan 3, 2023
3 Jan 2, 2023
511 Jan 5, 2023
41 Nov 1, 2022
35 Aug 18, 2022
1.4k Jan 1, 2023
267 Dec 11, 2022
13 May 28, 2022
1.7k Dec 26, 2022
15 Feb 16, 2021
1 Apr 10, 2022
2 Dec 5, 2021