Tools for working with Twitter JSON data

Overview

Twitter stream user info extractor

Build status Coverage status

This project lets you parse JSON data from the Twitter API or other sources to extract some basic user information, and then to query that data more or less efficiently, even if you've extracted information about a lot of users.

My local instance has processed a small fraction of the Archive Team Twitter Stream data sets this week, and currently contains 115 million historical screen names for 106 million Twitter accounts (many of which have been deactivated or suspended).

Why would you do this?

I have a lot of one-off Rust projects like this, and I decided to publish this one to show how Rust can be used for little data exploration tasks that might not traditionally be considered its domain.

I'm not a Rust expert and the code is probably mostly uninteresting and not a demonstration of best practices, but that's kind of the point: you can just hack together something fun pretty quickly and the language doesn't really get in the way.

As for the task, there's just a lot of bizarre history in archived Twitter metadata, and it can be fun to explore. Like Rudy Giuliani briefly changing his Twitter handle to @xxxxxxx37583982 in November 2016, for example:

$ target/release/ts-db query-screen-name RudyGiuliani
770781940341288960
$ target/release/ts-db query-user-id 770781940341288960
RudyGiuliani
xxxxxxx37583982

Why did this ghoul-ass motherfucker do this? I have no idea.

It can also actually be useful. Consider for example de-anonymizations like this one, where "Mormonger", a popular homophobic and transphobic figure in the far-right Deseret nationalist movement, was revealed to be a man named Cole Noorda from Salt Lake City, Utah. The published de-anonymization involves a lot of careful research and presentation of evidence, but Mormonger's screen name history gives away the punch line:

$ target/release/ts-db query-screen-name Mormonger
1408886100
$ target/release/ts-db query-user-id 1408886100
Mormonger
colenoorda

He seems to have changed the account's screen name from @Mormonger to @colenoorda very briefly in June 2016, but the Internet Archive remembers.

The other reason I did this is just that I wanted to play around with this Rust library for working with RocksDB, and this data seemed like a good excuse.

License

This project is licensed under the Mozilla Public License, version 2.0. See the LICENSE file for details.

You might also like...
Decode Metaplex mint account metadata into a JSON file.

Simple Metaplex Decoder (WIP) Install From Source Install Rust. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Clone the source: git c

A fast and simple command-line tool for common operations over JSON-lines files

rjp: Rapid JSON-lines processor A fast and simple command-line tool for common operations over JSON-lines files, such as: converting to and from text

A node package based on jsonschema-rs for performing JSON schema validation

A node package based on jsonschema-rs for performing JSON schema validation.

CLI tool to convert HOCON into valid JSON or YAML written in Rust.

{hocon:vert} CLI Tool to convert HOCON into valid JSON or YAML. Under normal circumstances this is mostly not needed because hocon configs are parsed

Typify - Compile JSON Schema documents into Rust types.

Typify Compile JSON Schema documents into Rust types. This can be used ... via the macro import_types!("types.json") to generate Rust types directly i

A easy and declarative way to test JSON input in Rust.

assert_json A easy and declarative way to test JSON input in Rust. assert_json is a Rust macro heavily inspired by serde json macro. Instead of creati

A fast way to minify JSON

COMPACTO (work in progress) A fast way to minify JSON. Usage/Examples # Compress # Input example (~0.11 KB) # { # "id": "123", # "name": "Edua

JSON Schema validation library

A JSON Schema validator implementation. It compiles schema into a validation tree to have validation as fast as possible.

Jq - Command-line JSON processor

jq jq is a lightweight and flexible command-line JSON processor. , Unix: , Windows: If you want to learn to use jq, read the documentation at https://

Owner
Travis Brown
Functional programmer mostly.
Travis Brown
Jsonptr - Data structures and logic for resolving, assigning, and deleting by JSON Pointers

jsonptr - JSON Pointers for Rust Data structures and logic for resolving, assigning, and deleting by JSON Pointers (RFC 6901). Usage Resolve JSON Poin

Chance 38 Aug 28, 2022
Rust libraries and tools to help with interoperability and testing of serialization formats based on Serde.

The repository zefchain/serde-reflection is based on Facebook's repository novifinancial/serde-reflection. We are now maintaining the project here and

Zefchain Labs 29 Sep 25, 2022
JSON parser which picks up values directly without performing tokenization in Rust

Pikkr JSON parser which picks up values directly without performing tokenization in Rust Abstract Pikkr is a JSON parser which picks up values directl

Pikkr 609 Sep 16, 2022
Strongly typed JSON library for Rust

Serde JSON   Serde is a framework for serializing and deserializing Rust data structures efficiently and generically. [dependencies] serde_json = "1.0

null 3.3k Sep 29, 2022
JSON implementation in Rust

json-rust Parse and serialize JSON with ease. Changelog - Complete Documentation - Cargo - Repository Why? JSON is a very loose format where anything

Maciej Hirsz 489 Sep 25, 2022
Rust port of gjson,get JSON value by dotpath syntax

A-JSON Read JSON values quickly - Rust JSON Parser change name to AJSON, see issue Inspiration comes from gjson in golang Installation Add it to your

Chen Jiaju 86 Sep 4, 2022
rurl is like curl but with a json configuration file per request

rurl rurl is a curl-like cli tool made in rust, the difference is that it takes its params from a json file so you can have all different requests sav

Bruno Ribeiro da Silva 6 Sep 10, 2022
A rust script to convert a better bibtex json file from Zotero into nice organised notes in Obsidian

Zotero to Obsidian script This is a script that takes a better bibtex JSON file exported by Zotero and generates an organised collection of reference

Sashin Exists 2 Feb 12, 2022
A tool for outputs semantic difference of json

jsondiff A tool for outputs semantic difference of json. "semantic" means: sort object key before comparison sort array before comparison (optional, b

niboshi 3 Sep 22, 2021
Easily create dynamic css using json notation

jss! This crate provides an easy way to write dynamic css using json notation. This gives you more convenient than you think. Considering using a dyna

Jovansonlee Cesar 7 May 14, 2022