Download your Stripe account to a SQLite database.

Table Dog

Last update: Oct 10, 2022

Related tags

Command-line tdog-cli

Overview

tdog

tdog is a CLI to download your Stripe account to a SQLite database.

Install

# macOS:
curl -L --output /usr/local/bin/tdog https://github.com/tabledog/tdog-cli/releases/latest/download/tdog-x86_64-apple-darwin

# Linux:
#curl -L --output /usr/local/bin/tdog https://github.com/tabledog/tdog-cli/releases/latest/download/tdog-x86_64-unknown-linux-gnu

chmod +x /usr/local/bin/tdog;

Releases
- macOS
- Linux
- Windows

Usage

tdog --stripe-key abc --target db.sqlite --watch

# Or use a JSON config:
tdog --json-file /path/config.json;
tdog --json "{}";

See the config JSON schema.

Building

git clone https://github.com/tabledog/tdog-cli;
cd tdog-cli;
cargo build --package tdog_cli

# Run
./target/debug/tdog_cli

Examples

Running once vs polling

When watch is true, the tdog process will continue to run and poll the Stripe /events endpoint, applying new writes to the database as they occur.

When watch is false, the tdog process will:

First run: Download all objects and then apply the events, exiting after done.
Second or greater runs: Apply any events that have occurred since the last run (as long as less that 30 days have passed, which is the limit of Stripes event history). Exit after done.

// Config path `cmd.args.options`:
{
    "options": {
        "watch": true
    }
}

SQLite

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_... download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "sqlite": {
                    "file": "/absolute/path/db.sqlite"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}

tdog --json-file config.json

MySQL

Start MySQL:

# Note: Version <= 5.6 not supported.

# Version 5.7
docker run --platform linux/x86_64 -e MYSQL_ROOT_PASSWORD=my-secret-pw -p 3306:3306 -d mysql:5.7

# Version 8
docker run --platform linux/x86_64 -e MYSQL_ROOT_PASSWORD=my-secret-pw -p 3306:3306 -d mysql:8

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_... download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "mysql": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 3306
                    },
                    "user": "root",
                    "pass": "my-secret-pw"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}

tdog --json-file config.json

Postgres

Start Postgres:

# Version 9.6.23
docker run --platform linux/x86_64 -e POSTGRES_PASSWORD=my-secret-pw -p 5432:5432 -d postgres:9.6.23


# Version 14
docker run --platform linux/x86_64 -e POSTGRES_PASSWORD=my-secret-pw -p 5432:5432 -d postgres:14

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_.. download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "postgres": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 5432
                    },
                    "user": "postgres",
                    "pass": "my-secret-pw"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}

tdog --json-file config.json

Postgres with TLS (SSL)

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_.. download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "postgres": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 5432
                    },
                    "user": "postgres",
                    "pass": "my-secret-pw",
                    "tls": {
                        // Certificate Authority certificate.
                        // Equal to `PGPASSWORD=x psql "port=5432 host=x user=postgres sslrootcert=/certs/ca.crt sslmode=verify-full"`
                        "ca_cert_file": "/certs/ca.crt"
                    }
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}

dlfile is a very simple, straightforward CLI to download a file

dlfile dlfile is a very simple, straightforward CLI to download a file, except it... Only downloads over HTTPS Requires TLS 1.2 or higher Executes in

3 Oct 21, 2022

A cli tool to download Imperial DoC past papers 🦀

Exams DoC Downloader A cli tool to download past papers from exams.doc.ic.ac.uk. An excercise in procrastination, written in Rust 🦀 . This currently

3 Apr 8, 2022

Download pdbs from symbol servers and cache locally, parse symbol paths from env vars

symsrv This crate lets you download and cache pdb files from symbol servers, according to the rules from the _NT_SYMBOL_PATH environment variable. It

6 Sep 15, 2022

Periodically download a youtube playlist, extract audio, convert to mp3, move to directory (possibly synced using syncthing).

auto-dl Periodically download a youtube playlist, extract audio, convert to mp3, move to directory (possibly synced using syncthing). drop https://git

10 Jan 12, 2023

Multi-protocol CLI download accelerator

dlrs dlrs is a multi-protocol download tool for the command line. Written in Rust, dlrs splits your downloads to speed them up where possible. Feature

7 Aug 7, 2023

Rustle: a powerful download manager and accelerator written in Rust

Rustle Rustle: a powerful download manager and accelerator written in Rust. Preview Features Resumable Downloads: Allow interrupted downloads to be re

4 Aug 19, 2023

A commandline tool to download video from bilibili.

Bilibili-dl 不断学习，持续更新中.... A commandline program to download video from bilibili. How to use bilibili-dl -h print help bilibili-dl download URL to

4 Oct 9, 2023

TTAutoRecord is a tool to automatically download TikTok Livestreams of users you follow.

TTAutoRecord: Automated TikTok Live Stream Recorder TTAutoRecord is a specialized utility designed to automate the recording of live streams on TikTok

4 Oct 30, 2023

A cli tool to automatically download and upload advent of code problems.

Advent of Code CLI CLI client for advent of code. The purpose of this tool is so you don't have to leave your editor while participating (if you are l

43 Dec 7, 2023

Comments

Rate limit fails, rate increases dramatically, eventually crashes process
So, I havent been able to complete a full download of my stripe data because something funky is going on with the rate limiter.

This is the context:

The command I'm using is tdog --json-file config.json.

I'm using a SQLite db, and I'm sure it's able to interact with the DB bc tdog successfully adds tables to the empty db at thestart of the run.

I have max_requests_per_second=20

Here's my config.json, click to expand

I redacted the key of course

{ "cmd": { "fn": "download", "args": { "from": { "stripe": { "secret_key": "rk_redacted", "max_requests_per_second": 20 } }, "to": { "sqlite": { "file": "main.db" } }, "options": { "watch": false, } } }, "log": "trace" }

What happens:

It runs for a while, seems like it's doing fine.

At some point, I get the "Database inserts are too slow" error message. On one run, the bad stuff (queue & rate limit blowing up) happened right after this warning. But on another run it kept going without issues for another 45min-1hr. So I think this is not correlated, but I'm not sure.

Then, at some point...

the queue grows

The req/sec increases dramatically, surpassing my rate limit

Eventually stripe starts returning 429s, and the program waits a bit, but then it keeps running way above stripes rate limit, immediately gets a 429 again... and eventually stripe just starts closing the connection.
opened by qbatten 2