A multi-threaded Twitch chat archiving and downloading tool.

Overview

Twitch Chat Downloader πŸ—’οΈ

Build Status Release Status License:GPLv3 Rust:Nightly

tcd is a multi-threaded Twitch Chat Downloader built in Rust πŸ¦€ .

Usage: tcd [OPTIONS] <--channel <CHANNEL>|--video <VIDEO>>

Options:
  -c, --channel <CHANNEL>      The channel(s) to download
  -i, --client-id <CLIENT_ID>  The Twitch client ID to use in the request headers
  -f, --format <FORMAT>        Used with --output or --stdout [default: csv] [possible values: json, csv]
  -l, --limit <LIMIT>          Downloads the first n videos from each channel
  -e, --live                   If specified, polls for new videos every `poll` seconds
  -o, --output <OUTPUT>        If specified, pipes data to the file
  -p, --postgres [<POSTGRES>]  The PostgreSQL connection string [default: DATABASE_URL env]
  -q, --quiet                  Whether to print download progress
  -s, --stdout                 If specified, pipes data to stdout
  -t, --threads <THREADS>      The number of threads to use [default: 10]
  -v, --video <VIDEO>          The video ids to download the chat for
  -w, --wait <WAIT>            The number of minutes to wait between polls (`live` only) [default: 30]
  -h, --help                   Print help information
  -V, --version                Print version information

Pipe the chat messages of the first 5 videos of Atrioc, Linkus7 and Aspecticor to the file hitman.csv

tcd --channel atrioc --channel linkus7 --channel aspecticor --limit 5 --output hitman.csv

Save the chat from the videos with id 1649326959 and 1648474855 to the connected PostgreSQL database.

tcd --video 1649326959 --video 1648474855 --postgres

Building from source

# build the binary
cargo build --release

# execute the binary
target/release/tcd --help

Generating datasets

Some pre-made dataset scripts are located in the queries directory. You can run these with cargo run -p queries --example <name>.

Using pre-made datasets

Pre-made datasets can be downloaded from the Mediafire folder.

Piping data to a database

tcd supports saving data directly to a PostgreSQL database. First, apply the Prisma schema with the following commands:

# apply schema.prisma to the database
# note: this WILL wipe all database content
cargo prisma migrate dev --name init

# generate the Prisma client
cargo prisma generate

Or execute the migration.sql SQL statements against your database. Then, set the DATABASE_URL environment variable (a .env file works too), or supply the connection URL with --postgres <url>.

Output format

Data piped to a file or stdout will be in the following format:

--format csv

channel_id,video_id,comment_id,commenter_id,created_at,text
23211159,1642642569,3f445ae2-2f6e-4256-b367-df8132454786,157032028,"2022-11-03 21:25:22.754 +00:00","poggies"

--format json

[
  {
    "channelId": "i64",
    "videoId": "i64",
    "commentId": "string",
    "commenterId": "i64",
    "createdAt": "string",
    "text": "string"
  },
  {
    "channelId": 23211159,
    "videoId": 1642642569,
    "commentId": "3f445ae2-2f6e-4256-b367-df8132454786",
    "commenterId": 157032028,
    "createdAt": "2022-11-03 21:25:22.754 +00:00",
    "text": "poggies"
  }
]
You might also like...
LL Cool Twitch Tools

LL Cool Twitch Tools Hit me up on my twitch channel This project is a playground twitch API playground for me.

 (Pre-Release Software) Secure, Encrypted, P2P chat written atop Warp, IPFS, LibP2P, Dioxus and many more awesome projects and protocols.
(Pre-Release Software) Secure, Encrypted, P2P chat written atop Warp, IPFS, LibP2P, Dioxus and many more awesome projects and protocols.

Uplink Privacy First, Modular, P2P messaging client built atop Warp. Uplink is written in pure Rust with a UI in Dioxus (which is also written in Rust

A Rust synchronisation primitive for "Multiplexed Concurrent Single-Threaded Read" access

exit-left verb; 1. To exit or disappear in a quiet, non-dramatic fashion, making way for more interesting events. 2. (imperative) Leave the scene, and

Harmony chat client in the terminal!

ilo-toki Harmony chat client in the terminal! Usage Run it with cargo run once you've cloned the repo. If you'd like, you can install the program with

Tree-based TUI chat client
Tree-based TUI chat client

cove Cove is a TUI client for euphoria.io, a threaded real-time chat platform. It runs on Linux, Windows and macOS. Manual installation This section c

Concurrent and multi-stage data ingestion and data processing with Rust+Tokio

TokioSky Build concurrent and multi-stage data ingestion and data processing pipelines with Rust+Tokio. TokioSky allows developers to consume data eff

CarLI is a framework for creating single-command and multi-command CLI applications in Rust

CarLI is a framework for creating single-command and multi-command CLI applications in Rust. The framework provides error and IO types better suited for the command line environment, especially in cases where unit testing is needed.

Multi-Architecture Code Emission Library

macel Multi-Architecture Code Emission Library (macel) is a library which implements a low-level intermediate representation meant to expose on machin

Show unused code from multi-crate Rust projects

Warnalyzer Remove unused code from multi-crate Rust projects. The dead_code lint family of rustc is limited to one crate only and thus can't tell whet

Releases(v0.4.1)
Owner
Matthew Polak
Computer Science student at the University of Ottawa
Matthew Polak
Twitch chat in the terminal.

Twitch chat in the terminal. What it looks like: Keybinds: Normal mode Key Description c Go to the chat window chat. i Enter input mode for sending me

null 176 Jan 5, 2023
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

Ryan 59 Dec 10, 2022
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

null 59 Dec 10, 2022
apkeep - A command-line tool for downloading APK files from various sources

apkeep - A command-line tool for downloading APK files from various sources Installation Precompiled binaries for apkeep on various platforms can be d

Electronic Frontier Foundation 561 Dec 29, 2022
Desktop app for reading and downloading manga. With clean distraction-free design and no clutter

Tonbun Tonbun is a desktop app for reading and downloading manga. With clean distraction-free design and no clutter. Build with Rust, Tauri, Vue.js, a

null 23 Nov 30, 2022
Failed experiment in downloading random cat image, turning it into ascii and displaying it in Neovim.

cat.nvim Failed experiment in downloading random cat image, turning it into ascii and displaying it in Neovim. Failed as I realized far too late, that

James Vero 4 Aug 5, 2022
BoilerFiles is a CLI for downloading your boilerplate files from a public GitHub template repo.

BoilerFiles Available on crates.io. BoilerFiles is a CLI for downloading your boilerplate files from a public GitHub template repo. Features Are you t

Jan MΓΌller 4 Oct 1, 2022
Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive.

[WIP] REMOTE-ARCHIVE Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive. The idea b

null 4 Nov 7, 2022
Rust-advent - Learning Rust by solving advent of code challenges (Streaming live on Twitch every Monday)

Rust advent ?? ?? Learning Rust by implementing solutions for Advent of Code problems. ?? HEY, we are live-streaming our attempts to solve the exercis

Luciano Mammino 20 Nov 11, 2022
A Twitch OAuth token revoker written in Rust.

Revoker - A Twitch OAuth Token Revoker Details Revoker is a small CLI tool written in Rust, made so you can easily and urgently remove a Twitch OAuth

Mykola 4 May 20, 2022