A multi-threaded Twitch chat archiving and downloading tool.

Last update: Dec 19, 2022

Overview

Twitch Chat Downloader 🗒️

tcd is a multi-threaded Twitch Chat Downloader built in Rust 🦀 .

Usage: tcd [OPTIONS] <--channel <CHANNEL>|--video <VIDEO>>

Options:
  -c, --channel <CHANNEL>      The channel(s) to download
  -i, --client-id <CLIENT_ID>  The Twitch client ID to use in the request headers
  -f, --format <FORMAT>        Used with --output or --stdout [default: csv] [possible values: json, csv]
  -l, --limit <LIMIT>          Downloads the first n videos from each channel
  -e, --live                   If specified, polls for new videos every `poll` seconds
  -o, --output <OUTPUT>        If specified, pipes data to the file
  -p, --postgres [<POSTGRES>]  The PostgreSQL connection string [default: DATABASE_URL env]
  -q, --quiet                  Whether to print download progress
  -s, --stdout                 If specified, pipes data to stdout
  -t, --threads <THREADS>      The number of threads to use [default: 10]
  -v, --video <VIDEO>          The video ids to download the chat for
  -w, --wait <WAIT>            The number of minutes to wait between polls (`live` only) [default: 30]
  -h, --help                   Print help information
  -V, --version                Print version information

Pipe the chat messages of the first 5 videos of Atrioc, Linkus7 and Aspecticor to the file hitman.csv

tcd --channel atrioc --channel linkus7 --channel aspecticor --limit 5 --output hitman.csv

Save the chat from the videos with id 1649326959 and 1648474855 to the connected PostgreSQL database.

tcd --video 1649326959 --video 1648474855 --postgres

Building from source

# build the binary
cargo build --release

# execute the binary
target/release/tcd --help

Generating datasets

Some pre-made dataset scripts are located in the queries directory. You can run these with cargo run -p queries --example <name>.

Using pre-made datasets

Pre-made datasets can be downloaded from the Mediafire folder.

Piping data to a database

tcd supports saving data directly to a PostgreSQL database. First, apply the Prisma schema with the following commands:

# apply schema.prisma to the database
# note: this WILL wipe all database content
cargo prisma migrate dev --name init

# generate the Prisma client
cargo prisma generate

Or execute the migration.sql SQL statements against your database. Then, set the DATABASE_URL environment variable (a .env file works too), or supply the connection URL with --postgres <url>.

Output format

Data piped to a file or stdout will be in the following format:

--format csv

channel_id,video_id,comment_id,commenter_id,created_at,text
23211159,1642642569,3f445ae2-2f6e-4256-b367-df8132454786,157032028,"2022-11-03 21:25:22.754 +00:00","poggies"

--format json

[
  {
    "channelId": "i64",
    "videoId": "i64",
    "commentId": "string",
    "commenterId": "i64",
    "createdAt": "string",
    "text": "string"
  },
  {
    "channelId": 23211159,
    "videoId": 1642642569,
    "commentId": "3f445ae2-2f6e-4256-b367-df8132454786",
    "commenterId": 157032028,
    "createdAt": "2022-11-03 21:25:22.754 +00:00",
    "text": "poggies"
  }
]

Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive.

[WIP] REMOTE-ARCHIVE Remote-Archive is a utility for exploring remote archive files without downloading the entire contents of the archive. The idea b

4 Nov 7, 2022

Twitch Bot Project (Rust)

This is a Twitch bot written in Rust that connects to the Twitch IRC server and allows users to interact with the bot via chat messages. The bot can join multiple Twitch channels and respond to commands.

4 Aug 17, 2023

Automatically replace Discord CDN links with local file links, by downloading the images to the public folder

Auto Undiscord Did you hear that Discord will be blocking external websites from using images hosted on their servers? Did you host every image on you

5 Nov 17, 2023

A Rust synchronisation primitive for "Multiplexed Concurrent Single-Threaded Read" access

exit-left verb; 1. To exit or disappear in a quiet, non-dramatic fashion, making way for more interesting events. 2. (imperative) Leave the scene, and

0 Dec 5, 2021

All in One AI Chat Tool( GPT-4 / GPT-3.5 /OpenAI API/Azure OpenAI)

WeChatAI Create All in One personal AI chat assistant English | 中文介绍 This project is not for reward and profit, only used for personal Screenshot Supp

38 Apr 25, 2023

(Pre-Release Software) Secure, Encrypted, P2P chat written atop Warp, IPFS, LibP2P, Dioxus and many more awesome projects and protocols.

Uplink Privacy First, Modular, P2P messaging client built atop Warp. Uplink is written in pure Rust with a UI in Dioxus (which is also written in Rust

13 Jan 25, 2023

xyz is a chat platform where people sign up, play a matching game, and say goodbye

xyz is an asynchronous chat and web service What you need Docker Desktop 🐳 Cargo (Rust package manager) 📦 Clone our project Follow the steps below t

12 Oct 11, 2023

Terminal UI to chat with large language models (LLM) using different model backends, and integrations with your favourite editors!

Oatmeal Terminal UI to chat with large language models (LLM) using different model backends, and integrations with your favourite editors! Overview In

88 Dec 4, 2023

Harmony chat client in the terminal!

ilo-toki Harmony chat client in the terminal! Usage Run it with cargo run once you've cloned the repo. If you'd like, you can install the program with