Download your Stripe account to a SQLite database.

Overview

tdog

tdog is a CLI to download your Stripe account to a SQLite database.

Install

# macOS:
curl -L --output /usr/local/bin/tdog https://github.com/tabledog/tdog-cli/releases/latest/download/tdog-x86_64-apple-darwin

# Linux:
#curl -L --output /usr/local/bin/tdog https://github.com/tabledog/tdog-cli/releases/latest/download/tdog-x86_64-unknown-linux-gnu

chmod +x /usr/local/bin/tdog;

Usage

tdog --stripe-key abc --target db.sqlite --watch

# Or use a JSON config:
tdog --json-file /path/config.json;
tdog --json "{}";

See the config JSON schema.

Building

git clone https://github.com/tabledog/tdog-cli;
cd tdog-cli;
cargo build --package tdog_cli

# Run
./target/debug/tdog_cli

Examples

Running once vs polling

When watch is true, the tdog process will continue to run and poll the Stripe /events endpoint, applying new writes to the database as they occur.

When watch is false, the tdog process will:

  • First run: Download all objects and then apply the events, exiting after done.
  • Second or greater runs: Apply any events that have occurred since the last run (as long as less that 30 days have passed, which is the limit of Stripes event history). Exit after done.
// Config path `cmd.args.options`:
{
    "options": {
        "watch": true
    }
}

SQLite

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_... download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "sqlite": {
                    "file": "/absolute/path/db.sqlite"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}
tdog --json-file config.json

MySQL

Start MySQL:

# Note: Version <= 5.6 not supported.

# Version 5.7
docker run --platform linux/x86_64 -e MYSQL_ROOT_PASSWORD=my-secret-pw -p 3306:3306 -d mysql:5.7

# Version 8
docker run --platform linux/x86_64 -e MYSQL_ROOT_PASSWORD=my-secret-pw -p 3306:3306 -d mysql:8

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_... download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "mysql": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 3306
                    },
                    "user": "root",
                    "pass": "my-secret-pw"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}
tdog --json-file config.json

Postgres

Start Postgres:

# Version 9.6.23
docker run --platform linux/x86_64 -e POSTGRES_PASSWORD=my-secret-pw -p 5432:5432 -d postgres:9.6.23


# Version 14
docker run --platform linux/x86_64 -e POSTGRES_PASSWORD=my-secret-pw -p 5432:5432 -d postgres:14

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_.. download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "postgres": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 5432
                    },
                    "user": "postgres",
                    "pass": "my-secret-pw"
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}
tdog --json-file config.json

Postgres with TLS (SSL)

config.json

{
    "cmd": {
        "fn": "download",
        "args": {
            "from": {
                "stripe": {
                    "secret_key": "sk_test_.. download at https://dashboard.stripe.com/apikeys"
                }
            },
            "to": {
                "postgres": {
                    "addr": {
                        "ip": "127.0.0.1",
                        "port": 5432
                    },
                    "user": "postgres",
                    "pass": "my-secret-pw",
                    "tls": {
                        // Certificate Authority certificate.
                        // Equal to `PGPASSWORD=x psql "port=5432 host=x user=postgres sslrootcert=/certs/ca.crt sslmode=verify-full"`
                        "ca_cert_file": "/certs/ca.crt"
                    }
                }
            },
            "options": {
                "watch": true
            }
        }
    },
    "log": "info"
}
You might also like...
A cli tool to download Imperial DoC past papers 🦀

Exams DoC Downloader A cli tool to download past papers from exams.doc.ic.ac.uk. An excercise in procrastination, written in Rust 🦀 . This currently

Download pdbs from symbol servers and cache locally, parse symbol paths from env vars

symsrv This crate lets you download and cache pdb files from symbol servers, according to the rules from the _NT_SYMBOL_PATH environment variable. It

Save image from your clipboard 📋 as an image file directly from your command line! 🔥

Clpy 📋 Save copied image from clipboard as an image file directly from your command line! Note It works only on windows as of now. I'll be adding sup

FastSSH is a TUI that allows you to quickly connect to your services by navigating through your SSH config.
FastSSH is a TUI that allows you to quickly connect to your services by navigating through your SSH config.

Connect quickly to your services 🚀 FastSSH is a TUI that allows you to quickly connect to your services by navigating through your SSH config. Instal

Deadliner helps you keep track of the time left for your deadline by dynamically updating the wallpaper of your desktop with the time left.
Deadliner helps you keep track of the time left for your deadline by dynamically updating the wallpaper of your desktop with the time left.

Deadliner Watch the YouTube video What's Deadliner? Deadliner is a cross-platform desktop application for setting deadline for a project and keeping t

By mirroring traffic to and from your machine, mirrord surrounds your local service with a mirror image of its cloud environment.
By mirroring traffic to and from your machine, mirrord surrounds your local service with a mirror image of its cloud environment.

mirrord lets you easily mirror traffic from your Kubernetes cluster to your development environment. It comes as both Visual Studio Code extension and

A Yocto setup and management tool that helps you keep your environment up-to-date and in-sync with your team
A Yocto setup and management tool that helps you keep your environment up-to-date and in-sync with your team

yb (Yocto Buddy) yb is designed to make it easy to setup and (perhaps more importantly) keep Yocto environments up-to-date and in-sync with your team.

 create and test the style and formatting of text in your terminal applications
create and test the style and formatting of text in your terminal applications

description: create and test the style and formatting of text in your terminal applications docs: https://docs.rs/termstyle termstyle is a library tha

Currency exchange rates in your terminal

Usage example Rates is a scriptable CLI tool that brings currency exchange rates right into your terminal and supports 30+ fiat and 10K+ crypto currencies.

Comments
  • Rate limit fails, rate increases dramatically, eventually crashes process

    Rate limit fails, rate increases dramatically, eventually crashes process

    So, I havent been able to complete a full download of my stripe data because something funky is going on with the rate limiter.

    This is the context:

    • The command I'm using is tdog --json-file config.json.
    • I'm using a SQLite db, and I'm sure it's able to interact with the DB bc tdog successfully adds tables to the empty db at thestart of the run.
    • I have max_requests_per_second=20
    Here's my config.json, click to expand

    I redacted the key of course

    
    {
        "cmd": {
            "fn": "download",
            "args": {
                "from": {
                    "stripe": {
                        "secret_key": "rk_redacted",
                        "max_requests_per_second": 20
                    }
                },
                "to": {
                    "sqlite": {
                        "file": "main.db"
                    }
                },
                "options": {
                    "watch": false,
                }
            }
        },
        "log": "trace"
    }
    
    

    What happens:

    • It runs for a while, seems like it's doing fine.
    • At some point, I get the "Database inserts are too slow" error message. On one run, the bad stuff (queue & rate limit blowing up) happened right after this warning. But on another run it kept going without issues for another 45min-1hr. So I think this is not correlated, but I'm not sure.
    • Then, at some point...
      • the queue grows
      • The req/sec increases dramatically, surpassing my rate limit
    • Eventually stripe starts returning 429s, and the program waits a bit, but then it keeps running way above stripes rate limit, immediately gets a 429 again... and eventually stripe just starts closing the connection.
    opened by qbatten 2
Owner
Table Dog
Download your Stripe account to a SQL database.
Table Dog
Download binary for your OS from Github

github-bin-downloader Download binary for your OS from Github. Installation Install github-bin-downloader using cargo cargo install github-bin-downloa

353fc443 10 Dec 10, 2021
Nazuna - 🐦 Download Twitter videos using your terminal!

Nazuna ?? Download Twitter videos using your terminal! Installation Binary Download the desired file for your OS (Windows, Mac, Linux) from https://gi

Felipe Garcia 25 Sep 8, 2022
A cli tool to download specific GitHub directories or files

cloneit A cli tool to download specific GitHub directories or files. Installation From git git clone https://github.com/alok8bb/cloneit cd cloneit car

Alok 47 Sep 21, 2022
Cross-platform terminal program to download IEEE journals

IEEE Journal Downloader A cross-platform terminal program which tries to download every article in a specified journal and merge the documents into on

Fong Chien Yoong 18 Jul 23, 2022
CLI tool and crate to download FTDC data from mongodb clusters.

Download mongodb FTDC data Command line tool and lib crate to download FTDC data from mongodb clusters to investigate with e.g. keyhole. Crate For the

Mathias Oertel 0 Nov 7, 2021
A simple CLI to build VEXCode V5 Pro projects and download them to the V5 brain.

vexer A simple CLI to build VEXCode V5 Pro projects and download them to the V5 brain. (WIP) This currently is only tested on and only works on window

null 2 May 16, 2022
A simple, fast and interruptable download accelerator, written in Rust

snatch A simple, fast and interruptable download accelerator, written in Rust WARNING This project is no longer maintained by @k0pernicus and @jean-se

Dernier Cri ® 650 Aug 29, 2022
Use the osu!direct button on the osu! website to download beatmaps directly to osu

osum!direct-web Use the osu!direct button on the osu! website to download beatmaps directly to osu! - without osu! supporter. Badges Usage Run the app

oSumAtrIX 1 Jan 21, 2022
cli application to visualize crate download metrics.

cli application to visualize crate download metrics.

Orhan Balci 1 Jan 28, 2022
dlfile is a very simple, straightforward CLI to download a file

dlfile dlfile is a very simple, straightforward CLI to download a file, except it... Only downloads over HTTPS Requires TLS 1.2 or higher Executes in

Colin 2 Feb 7, 2022