A multi-threaded Twitch chat archiving and downloading tool.

Overview

Twitch Chat Downloader πŸ—’οΈ

Build Status Release Status License:GPLv3 Rust:Nightly

tcd is a multi-threaded Twitch Chat Downloader built in Rust πŸ¦€ .

Usage: tcd [OPTIONS] <--channel <CHANNEL>|--video <VIDEO>>

Options:
  -c, --channel <CHANNEL>      The channel(s) to download
  -i, --client-id <CLIENT_ID>  The Twitch client ID to use in the request headers
  -f, --format <FORMAT>        Used with --output or --stdout [default: csv] [possible values: json, csv]
  -l, --limit <LIMIT>          Downloads the first n videos from each channel
  -e, --live                   If specified, polls for new videos every `poll` seconds
  -o, --output <OUTPUT>        If specified, pipes data to the file
  -p, --postgres [<POSTGRES>]  The PostgreSQL connection string [default: DATABASE_URL env]
  -q, --quiet                  Whether to print download progress
  -s, --stdout                 If specified, pipes data to stdout
  -t, --threads <THREADS>      The number of threads to use [default: 10]
  -v, --video <VIDEO>          The video ids to download the chat for
  -w, --wait <WAIT>            The number of minutes to wait between polls (`live` only) [default: 30]
  -h, --help                   Print help information
  -V, --version                Print version information

Pipe the chat messages of the first 5 videos of Atrioc, Linkus7 and Aspecticor to the file hitman.csv

tcd --channel atrioc --channel linkus7 --channel aspecticor --limit 5 --output hitman.csv

Save the chat from the videos with id 1649326959 and 1648474855 to the connected PostgreSQL database.

tcd --video 1649326959 --video 1648474855 --postgres

Building from source

# build the binary
cargo build --release

# execute the binary
target/release/tcd --help

Generating datasets

Some pre-made dataset scripts are located in the queries directory. You can run these with cargo run -p queries --example <name>.

Using pre-made datasets

Pre-made datasets can be downloaded from the Mediafire folder.

Piping data to a database

tcd supports saving data directly to a PostgreSQL database. First, apply the Prisma schema with the following commands:

# apply schema.prisma to the database
# note: this WILL wipe all database content
cargo prisma migrate dev --name init

# generate the Prisma client
cargo prisma generate

Or execute the migration.sql SQL statements against your database. Then, set the DATABASE_URL environment variable (a .env file works too), or supply the connection URL with --postgres <url>.

Output format

Data piped to a file or stdout will be in the following format:

--format csv

channel_id,video_id,comment_id,commenter_id,created_at,text
23211159,1642642569,3f445ae2-2f6e-4256-b367-df8132454786,157032028,"2022-11-03 21:25:22.754 +00:00","poggies"

--format json

[
  {
    "channelId": "i64",
    "videoId": "i64",
    "commentId": "string",
    "commenterId": "i64",
    "createdAt": "string",
    "text": "string"
  },
  {
    "channelId": 23211159,
    "videoId": 1642642569,
    "commentId": "3f445ae2-2f6e-4256-b367-df8132454786",
    "commenterId": 157032028,
    "createdAt": "2022-11-03 21:25:22.754 +00:00",
    "text": "poggies"
  }
]
You might also like...
A Twitch OAuth token revoker written in Rust.

Revoker - A Twitch OAuth Token Revoker Details Revoker is a small CLI tool written in Rust, made so you can easily and urgently remove a Twitch OAuth

LL Cool Twitch Tools

LL Cool Twitch Tools Hit me up on my twitch channel This project is a playground twitch API playground for me.

Twitch Bot Project (Rust)

This is a Twitch bot written in Rust that connects to the Twitch IRC server and allows users to interact with the bot via chat messages. The bot can join multiple Twitch channels and respond to commands.

BoilerFiles is a CLI for downloading your boilerplate files from a public GitHub template repo.

BoilerFiles Available on crates.io. BoilerFiles is a CLI for downloading your boilerplate files from a public GitHub template repo. Features Are you t

Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive.

[WIP] REMOTE-ARCHIVE Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive. The idea b

Automatically replace Discord CDN links with local file links, by downloading the images to the public folder
Automatically replace Discord CDN links with local file links, by downloading the images to the public folder

Auto Undiscord Did you hear that Discord will be blocking external websites from using images hosted on their servers? Did you host every image on you

A Rust synchronisation primitive for "Multiplexed Concurrent Single-Threaded Read" access

exit-left verb; 1. To exit or disappear in a quiet, non-dramatic fashion, making way for more interesting events. 2. (imperative) Leave the scene, and

All in One AI Chat Tool( GPT-4 / GPT-3.5 /OpenAI API/Azure OpenAI)
All in One AI Chat Tool( GPT-4 / GPT-3.5 /OpenAI API/Azure OpenAI)

WeChatAI Create All in One personal AI chat assistant English | 中文介绍 This project is not for reward and profit, only used for personal Screenshot Supp

 (Pre-Release Software) Secure, Encrypted, P2P chat written atop Warp, IPFS, LibP2P, Dioxus and many more awesome projects and protocols.
(Pre-Release Software) Secure, Encrypted, P2P chat written atop Warp, IPFS, LibP2P, Dioxus and many more awesome projects and protocols.

Uplink Privacy First, Modular, P2P messaging client built atop Warp. Uplink is written in pure Rust with a UI in Dioxus (which is also written in Rust

Releases(v0.4.1)
Owner
Matthew Polak
Computer Science student at the University of Ottawa
Matthew Polak
Twitch chat in the terminal.

Twitch Chat IRC, in the terminal. What it looks like: Keybinds: Normal mode Key Description ? Have the keybinds window appear. i Enter insert mode for

null 172 Dec 30, 2022
Twitch chat in the terminal.

Twitch chat in the terminal. What it looks like: Keybinds: Normal mode Key Description c Go to the chat window chat. i Enter input mode for sending me

null 176 Jan 5, 2023
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

Ryan 59 Dec 10, 2022
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

null 59 Dec 10, 2022
An easy-to-use, incremental, multi-threaded garbage collector for Rust

Refuse An easy-to-use, incremental, multi-threaded garbage collector for Rust. //! A basic usage example demonstrating the garbage collector. use refu

Khonsu Labs 6 May 3, 2024
apkeep - A command-line tool for downloading APK files from various sources

apkeep - A command-line tool for downloading APK files from various sources Installation Precompiled binaries for apkeep on various platforms can be d

Electronic Frontier Foundation 561 Dec 29, 2022
Desktop app for reading and downloading manga. With clean distraction-free design and no clutter

Tonbun Tonbun is a desktop app for reading and downloading manga. With clean distraction-free design and no clutter. Build with Rust, Tauri, Vue.js, a

null 23 Nov 30, 2022
Failed experiment in downloading random cat image, turning it into ascii and displaying it in Neovim.

cat.nvim Failed experiment in downloading random cat image, turning it into ascii and displaying it in Neovim. Failed as I realized far too late, that

James Vero 4 Aug 5, 2022
KAIVM is a multiplatform Command Line Interface (CLI) designed to simplify the process of downloading, managing, configuring, and running different versions of Shinkai Node

KAIVM - Shinkai Version Manager KAIVM is a multiplatform Command Line Interface (CLI) designed to simplify the process of downloading, managing, confi

Alfredo Gallardo 7 May 1, 2024
Rust-advent - Learning Rust by solving advent of code challenges (Streaming live on Twitch every Monday)

Rust advent ?? ?? Learning Rust by implementing solutions for Advent of Code problems. ?? HEY, we are live-streaming our attempts to solve the exercis

Luciano Mammino 20 Nov 11, 2022