A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc.

Last update: Dec 21, 2022

Related tags

Command-line carbonbot

Overview

carbonbot

A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc.

Run

To quickly get started, copy conf/run_crawlers.sh to somewhere, fill in neccesary parameters and run it.

Trade

Crawl tick-by-tick trades:

docker run -d --name carbonbot-trade --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.trade.config.js

Level2 orderbook updates

Crawl realtime level2 orderbook incremental updates:

docker run -d --name carbonbot-l2_event --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_event.config.js

Level2 orderbook full snapshots from RESTful API

Crawl level2 orderbook full snapshots from RESTful API:

docker run -d --name carbonbot-l2_snapshot --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_snapshot.config.js

Level2 orderbook top-k snapshots

Crawl realtime level2 orderbook top-K snapshots:

docker run -d --name carbonbot-l2_topk --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_topk.config.js

Level3 orderbook updates

Crawl realtime level3 orderbook incremental updates:

docker run -d --name carbonbot-l3_event --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l3_event.config.js

BBO

Crawl realtime BBO:

docker run -d --name carbonbot-bbo --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.bbo.config.js

Ticker

Crawl 24hr rolling window tickers:

docker run -d --name carbonbot-ticker --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.ticker.config.js

Candlestick

Crawl candlesticks(i.e., OHLCV)

docker run -d --name carbonbot-candlestick --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.candlestick.config.js

Funding rate

Crawl funding rates

docker run -d --name carbonbot-funding_rate --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.funding_rate.config.js

Open interest

docker run -d --name carbonbot-open_interest --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.open_interest.config.js

Other

docker run -d --name carbonbot-other --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.other.config.js

Environment Variables:

Name	Required	Meaning
DATA_DIR	true	The local directory to store data
AWS_ACCESS_KEY_ID	true	AWS access key ID
AWS_SECRET_ACCESS_KEY	true	AWS secret access key
AWS_S3_DIR	true	AWS S3 destination path
REDIS_URL	false	If set to non-empty, data will be published to redis channels

The ghcr.io/crypto-crawler/carbonbot:latest container writes data to the local path temporarily, then moves data to AWS S3 every 15 minutes.

Build

docker pull rust:latest && docker pull node:bullseye-slim
docker build -t ghcr.io/crypto-crawler/carbonbot:latest .
docker push ghcr.io/crypto-crawler/carbonbot:latest

Download

File Name	MD5	Size	Magnet Link
parsed-trade-2021-07.zip	a4a4088c3c9ebccc70e4b10f77c044c3	84213736076	magnet:?xt=urn:btih:557afe1132dd5a67dada971009733ae6019fd84b&dn=parsed-trade-2021-07.zip&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=http%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce

Rust File Management CLI is a command-line tool written in Rust that provides essential file management functionalities. Whether you're working with files or directories, this tool simplifies common file operations with ease.

Rust FileOps Rust File Management CLI is a command-line tool written in Rust that provides essential file management functionalities. Whether you're w

5 May 2, 2024

Rust-based CLI library for OpenAI ChatGPT API

This is a Rust library that provides a CLI (command-line interface) for easy and convenient access to the OpenAI ChatGPT API. The library offers a simple and straightforward way to integrate the ChatGPT API into your Rust-based applications.

10 Mar 9, 2023

A rust library + CLI tool that tells you when swas will upload new video through complex calculations

A rust library + CLI tool that tells you when swas will upload new video through complex calculations. It also lets you search and play youtube videos of swas and other channels. Searching about youtube channels is also an option. Basically it's a youtube search cli tool written in rust.

4 Jun 10, 2022

Crunch is a command-line interface (CLI) to claim staking rewards every X hours for Substrate-based chains

Comments

The file is not in csv format

The file is not in csv format, but is stored with a csv suffix, and some data is omitted.

aws s3 sync --request-payer requester s3://carbonbot/monthly/parsed .
head -n2 binance.inverse_future.l2_event.BTC.USD.BTCUSD_211231.2021-07.csv

timestamp       snapshot        asks    bids    seq_id  prev_seq_id
1625097600016   false   [[36224.3,0.690144461,25000.0,250.0],[36228.1,0.104890955,3800.0,38.0],[36234.1,0.0,0.0,0.0]]  [[36205.3,0.005524053,200.0,2.0],[36211.2,0.0,0.0,0.0],[36211.3,0.0,0.0,0.0],[36212.6,0.0,0.0,0.0]]

it missing seq_id and prev_seq_id data.

What is the way to read these 'csv' in best practice?

opened by 0xZP 3

A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc.

Related tags

Overview

carbonbot

Run

Trade

Level2 orderbook updates

Level2 orderbook full snapshots from RESTful API

Level2 orderbook top-k snapshots

Level3 orderbook updates

BBO

Ticker

Candlestick

Funding rate

Open interest

Other

Build

Download

You might also like...

Rust File Management CLI is a command-line tool written in Rust that provides essential file management functionalities. Whether you're working with files or directories, this tool simplifies common file operations with ease.

Rust-based CLI library for OpenAI ChatGPT API

A rust library + CLI tool that tells you when swas will upload new video through complex calculations

Crunch is a command-line interface (CLI) to claim staking rewards every X hours for Substrate-based chains

A cli based pastebin in Rust, but very insecure

Text-based to-do management CLI & language server

Small and simple CLI app to generate .editorconfig based on a given settings.

Emulates an Edge hardware-based room sensor client purely as a CLI application

Scouty is a command-line interface (CLI) to keep an eye on substrate-based chains and hook things up

Comments

The file is not in csv format

Owner

A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)

A CLI tool you can pipe code and then ask for changes, add documentation, etc, using the OpenAI API.

🌊 ~ seaward is a crawler which searches for links or a specified word in a website.

A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.

A small, memory efficient crawler written in Rust.

A terminal ASCII media player. View images, gifs, videos, webcam, YouTube, etc.. directly in the terminal as ASCII art.

A Windows virtual display driver written in Rust (works with VR, etc)

A CLI tool to get help with CLI tools 🐙

CLI tool to find duplicate files based on their hashes.

turbocommit is a Rust-based CLI tool that generates high-quality git commit messages in accordance with the Conventional Commits specification, using OpenAI's