A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc.

Related tags

Command-line carbonbot
Overview

carbonbot

A CLI tool based on the crypto-crawler-rs library to crawl trade, level2, level3, ticker, funding rate, etc.

Run

To quickly get started, copy conf/run_crawlers.sh to somewhere, fill in neccesary parameters and run it.

Trade

Crawl tick-by-tick trades:

docker run -d --name carbonbot-trade --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.trade.config.js

Level2 orderbook updates

Crawl realtime level2 orderbook incremental updates:

docker run -d --name carbonbot-l2_event --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_event.config.js

Level2 orderbook full snapshots from RESTful API

Crawl level2 orderbook full snapshots from RESTful API:

docker run -d --name carbonbot-l2_snapshot --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_snapshot.config.js

Level2 orderbook top-k snapshots

Crawl realtime level2 orderbook top-K snapshots:

docker run -d --name carbonbot-l2_topk --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l2_topk.config.js

Level3 orderbook updates

Crawl realtime level3 orderbook incremental updates:

docker run -d --name carbonbot-l3_event --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.l3_event.config.js

BBO

Crawl realtime BBO:

docker run -d --name carbonbot-bbo --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.bbo.config.js

Ticker

Crawl 24hr rolling window tickers:

docker run -d --name carbonbot-ticker --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.ticker.config.js

Candlestick

Crawl candlesticks(i.e., OHLCV)

docker run -d --name carbonbot-candlestick --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.candlestick.config.js

Funding rate

Crawl funding rates

docker run -d --name carbonbot-funding_rate --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.funding_rate.config.js

Open interest

docker run -d --name carbonbot-open_interest --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.open_interest.config.js

Other

docker run -d --name carbonbot-other --restart always -v $YOUR_LOCAL_PATH:/data -e DATA_DIR=/data -e AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY" -e AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY" -e AWS_S3_DIR="s3://YOUR_BUCKET/path" -u "$(id -u):$(id -g)" ghcr.io/crypto-crawler/carbonbot:latest pm2-runtime start pm2.other.config.js

Environment Variables:

Name Required Meaning
DATA_DIR true The local directory to store data
AWS_ACCESS_KEY_ID true AWS access key ID
AWS_SECRET_ACCESS_KEY true AWS secret access key
AWS_S3_DIR true AWS S3 destination path
REDIS_URL false If set to non-empty, data will be published to redis channels

The ghcr.io/crypto-crawler/carbonbot:latest container writes data to the local path temporarily, then moves data to AWS S3 every 15 minutes.

Build

docker pull rust:latest && docker pull node:bullseye-slim
docker build -t ghcr.io/crypto-crawler/carbonbot:latest .
docker push ghcr.io/crypto-crawler/carbonbot:latest

Download

File Name MD5 Size Magnet Link
parsed-trade-2021-07.zip a4a4088c3c9ebccc70e4b10f77c044c3 84213736076 magnet:?xt=urn:btih:557afe1132dd5a67dada971009733ae6019fd84b&dn=parsed-trade-2021-07.zip&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=http%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce
You might also like...
Rust-based CLI library for OpenAI ChatGPT API
Rust-based CLI library for OpenAI ChatGPT API

This is a Rust library that provides a CLI (command-line interface) for easy and convenient access to the OpenAI ChatGPT API. The library offers a simple and straightforward way to integrate the ChatGPT API into your Rust-based applications.

bigduu 10 Mar 9, 2023
A rust library + CLI tool that tells you when swas will upload new video through complex calculations
A rust library + CLI tool that tells you when swas will upload new video through complex calculations

A rust library + CLI tool that tells you when swas will upload new video through complex calculations. It also lets you search and play youtube videos of swas and other channels. Searching about youtube channels is also an option. Basically it's a youtube search cli tool written in rust.

midnightFirefly 4 Jun 10, 2022
Crunch is a command-line interface (CLI) to claim staking rewards every X hours for Substrate-based chains
Crunch is a command-line interface (CLI) to claim staking rewards every X hours for Substrate-based chains

crunch · crunch is a command-line interface (CLI) to claim staking rewards every X hours for Substrate-based chains. Why use crunch To automate payout

null 39 Dec 8, 2022
A cli based pastebin in Rust, but very insecure

pasta A cli based pastebin in Rust, but very insecure Use nightly toolchain to build rustup override set nightly When this program is running, you can

Snehit Sah 3 Mar 25, 2022
Text-based to-do management CLI & language server
Text-based to-do management CLI & language server

☑️ Todome （日本語版はこちら） Todome is a notation developed for making and editing to-do lists. It is inspired by Todo.txt, and in fact some of the todome not

monaqa 16 Aug 17, 2022
Small and simple CLI app to generate .editorconfig based on a given settings.

add-editorconfig Small and simple CLI app to generate .editorconfig based on a given settings. Usage # Will create an .editorconfig in the current dir

Reinaldy Rafli 3 Jan 16, 2022
Emulates an Edge hardware-based room sensor client purely as a CLI application

ambi_mock_client Usage You must have Rust installed to build ambi_mock_client. You can find documentation on installing Rust here. Using cargo run c

Jim Hodapp 1 Jan 22, 2022
Scouty is a command-line interface (CLI) to keep an eye on substrate-based chains and hook things up
Scouty is a command-line interface (CLI) to keep an eye on substrate-based chains and hook things up

scouty is a command-line interface (CLI) to keep an eye on substrate-based chains and hook things up

TurboFlakes 15 Aug 6, 2022
A CLI-based pride flag generator written in Rust
A CLI-based pride flag generator written in Rust

🌈 prideful (in development) A CLI-based pride flag generator written in Rust. How to run Build the project using cargo. Install cargo by following th

Angelo-F 35 Sep 3, 2022
Comments
  • The file is not in csv format

    The file is not in csv format

    The file is not in csv format, but is stored with a csv suffix, and some data is omitted.

    aws s3 sync --request-payer requester s3://carbonbot/monthly/parsed .
head -n2 binance.inverse_future.l2_event.BTC.USD.BTCUSD_211231.2021-07.csv

timestamp       snapshot        asks    bids    seq_id  prev_seq_id
1625097600016   false   [[36224.3,0.690144461,25000.0,250.0],[36228.1,0.104890955,3800.0,38.0],[36234.1,0.0,0.0,0.0]]  [[36205.3,0.005524053,200.0,2.0],[36211.2,0.0,0.0,0.0],[36211.3,0.0,0.0,0.0],[36212.6,0.0,0.0,0.0]]

    it missing seq_id and prev_seq_id data.

    What is the way to read these 'csv' in best practice?

    opened by 0xZP 3
Owner
Rock-solid cryptocurrency crawlers
null
GitHub
A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)

FS Dir Cache A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.) When working on build s

Dawid Ciężarkiewicz 5 Aug 29, 2023
A CLI tool you can pipe code and then ask for changes, add documentation, etc, using the OpenAI API.

AiBro This is your own little coding bro, immersed in the world of AI, crypto, and all other types of over hyped tech trends. You can pipe it code and

Josh Bainbridge 5 Sep 5, 2023
🌊 ~ seaward is a crawler which searches for links or a specified word in a website.

?? seaward Installation cargo install seaward On NetBSD a pre-compiled binary is available from the official repositories. To install it, simply run:

null 3 Jul 16, 2023
A lightweight async Web crawler in Rust, optimized for concurrent scraping while respecting `robots.txt` rules.

??️ crawly A lightweight and efficient web crawler in Rust, optimized for concurrent scraping while respecting robots.txt rules. ?? Features Concurren

CrystalSoft 5 Aug 29, 2023
A small, memory efficient crawler written in Rust.

Atra - The smaller way to crawl !!This read me will we reworked in a few days. Currently I am working on a better version and a wiki for the config fi

Felix Engl 3 Mar 23, 2024
A terminal ASCII media player. View images, gifs, videos, webcam, YouTube, etc.. directly in the terminal as ASCII art.

Terminal Media Player View images, videos (files or YouTube links), webcam, etc directly in the terminal as ASCII. All images you see below are just m

Max Curzi 36 May 8, 2023
A Windows virtual display driver written in Rust (works with VR, etc)

Virtual Display Driver This is a Windows driver made in Rust which creates a virtual desktop. It has many uses, such as: A private virtual desktop for

Cherry 28 Sep 19, 2023
A CLI tool to get help with CLI tools 🐙

A CLI tool to get help with CLI tools ?? halp aims to help find the correct arguments for command-line tools by checking the predefined list of common

Orhun Parmaksız 566 Apr 16, 2023
CLI tool to find duplicate files based on their hashes.

Dupper Dupper is a CLI tool that helps you identify duplicate files based on their hashes (using the Seahash hashing algorithm). Installation You can

Rubén J.R. 4 Dec 27, 2022
turbocommit is a Rust-based CLI tool that generates high-quality git commit messages in accordance with the Conventional Commits specification, using OpenAI's

turbocommit is a Rust-based CLI tool that generates high-quality git commit messages in accordance with the Conventional Commits specification, using OpenAI's `gpt-3.5-turbo` language model. It is easy to use and a cost-effective way to keep git commit history at a higher quality, helping developers stay on track with their work.

Sett 16 Mar 26, 2023