Library for scripting analyses against crates.io's database dumps

David Tolnay

Last update: Dec 14, 2022

Related tags

Data processing db-dump

Overview

crates.io database dumps

Library for scripting analyses against crates.io's database dumps.

These database dumps contain all information exposed by the crates.io API packaged into a single download. An updated dump is published every 24 hours. The latest dump is available at https://static.crates.io/db-dump.tar.gz.

[dependencies]
db-dump = "0.2"

Examples

The examples/ directory of this repo contains several runnable example analyses.

total‑downloads	Computes time series of total downloads by day across all crates on crates.io
crate‑downloads	Computes time series of downloads of one specific crate
top‑crates	Computes the top few most directly depended upon crates
user‑dependencies	Computes the percentage of crates.io which depends directly on at least one crate by the specified user
user‑downloads	Computes time series of the fraction of crates.io downloads attributed to a single given user's crates

Each of these examples can be run using Cargo once you've downloaded a recent database dump:

$ wget https://static.crates.io/db-dump.tar.gz
$ cargo run --release --example total-downloads

Here is the implementation of the most basic example, total-downloads, and graph of the resulting table. It shows crates.io download rate doubling every 9 months, or equivalently 10× every 2.5 years!

use chrono::NaiveDate;
use std::collections::BTreeMap as Map;

fn main() -> db_dump::Result<()> {
    let mut downloads = Map::<NaiveDate, u64>::new();
    db_dump::Loader::new()
        .version_downloads(|row| {
            *downloads.entry(row.date).or_default() += row.downloads;
        })
        .load("./db-dump.tar.gz")?;

    for (date, count) in downloads {
        println!("{},{}", date, count);
    }

    Ok(())
}

Here is a graph from the user-downloads example:

License

^{Licensed under either of Apache License, Version 2.0 or MIT license at your option.}
_{Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.}

You might also like...

Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

Polars Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow Blazingly fast DataFrames in Rust, Python & Node.js Polars is

11.8k Jan 8, 2023

mach-dump can parse Mach-O core dumps taken with lldb from macOS and iOS devices.

mach-dump mach-dump can parse Mach-O core dumps taken with lldb from macOS and iOS devices. It has no external dependencies. Example use std::path::Pa

8 Sep 16, 2022

Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

294 Dec 23, 2022

Security advisory database for Rust crates published through crates.io

RustSec Advisory Database The RustSec Advisory Database is a repository of security advisories filed against Rust crates published via https://crates.

682 Jan 1, 2023

Soufflé is a variant of Datalog for tool designers crafting analyses in Horn clauses. Soufflé synthesizes a native parallel C++ program from a logic specification.

Welcome! This is the official repository for the Soufflé language project. The Soufflé language is similar to Datalog (but has terms known as records)

672 Dec 29, 2022

Comments

Expose timestamps as DateTime

The created_at and updated_at timestamps in the db dump CSV do not mention a time zone, but it's not correct to represent them as NaiveDateTime because they refer to instants in real time, just stated in UTC.

opened by dtolnay 0
Drop dependency on serde's derive feature

This allows everything downstream of serde in our dependency graph (bstr, csv, semver, serde_json) to begin building before serde_derive has finished building.

Before:

After:

opened by dtolnay 0
Serialize Id structs

It would be beneficial if other structs are also serializable (Rows).

E.g. I want to serialize crates and categories data to be used as json in a pipeline.

opened by elpiel 0
use Day

As per discussion in https://github.com/chronotope/chrono/issues/820 - I've added the Day<Tz> type to a branch in chrono and tested it out over here. If this is desired to be merged, I'll update the chrono dependency to point to a released version once it is released

opened by esheppa 0

Releases(0.4.1)

0.4.1(Dec 17, 2022)
Documentation improvements

Source code(tar.gz)
Source code(zip)
0.4.0(Oct 30, 2022)
Switch from NaiveDateTime to DateTime<Utc> and Date<Utc> to represent crates.io server time (#17, #18)

Source code(tar.gz)
Source code(zip)
0.3.11(Sep 25, 2022)
Add serde Serialize impl for various Id types (#13, thanks @elpiel)

Source code(tar.gz)
Source code(zip)
0.3.10(Sep 25, 2022)
Add Clone impls for row structs (#12, thanks @elpiel)

Source code(tar.gz)
Source code(zip)
0.3.9(Aug 24, 2022)
Parse new explicit_name column in dependencies.csv (#10, https://github.com/rust-lang/crates.io/pull/5091)

Parse new links column in versions.csv (#10, https://github.com/rust-lang/crates.io/pull/5111)

Source code(tar.gz)
Source code(zip)
0.3.8(Aug 18, 2022)
Parse new checksum column in versions.csv (#9, https://github.com/rust-lang/crates.io/pull/5077)

Source code(tar.gz)
Source code(zip)
0.3.7(Aug 3, 2022)
Add keywords to crates.io metadata

Source code(tar.gz)
Source code(zip)
0.3.6(Aug 2, 2022)
Add categories to crates.io metadata

Source code(tar.gz)
Source code(zip)
0.3.5(Jul 30, 2022)
Update progress bar to indicatif 0.17

Source code(tar.gz)
Source code(zip)
0.3.4(Feb 6, 2022)
Include csv file name in error messages

Fall back to BadgeType::Other on badge attributes parse error

Source code(tar.gz)
Source code(zip)
0.3.3(Dec 25, 2021)
Improve diagrams in readme

Source code(tar.gz)
Source code(zip)
0.3.2(Oct 8, 2021)
Update progress bar to indicatif 0.16

Source code(tar.gz)
Source code(zip)
0.3.1(Jun 19, 2021)
Support OwnerId as hashmap and btreemap key

Source code(tar.gz)
Source code(zip)
0.3.0(May 26, 2021)

Source code(tar.gz)
Source code(zip)
0.2.0(May 19, 2021)

https://github.com/rust-lang/crates.io/issues/3617#issuecomment-841315011
Source code(tar.gz)
Source code(zip)
0.1.1(Apr 24, 2021)

Source code(tar.gz)
Source code(zip)
0.1.0(Mar 16, 2021)

Source code(tar.gz)
Source code(zip)

Library for scripting analyses against crates.io's database dumps

Related tags

Overview

crates.io database dumps

Examples

License

You might also like...

Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

mach-dump can parse Mach-O core dumps taken with lldb from macOS and iOS devices.

Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Security advisory database for Rust crates published through crates.io

Soufflé is a variant of Datalog for tool designers crafting analyses in Horn clauses. Soufflé synthesizes a native parallel C++ program from a logic specification.

Bundle Cargo crates for use with macOS/iOS in Xcode

"Crates for Cheese" is a Rust collection library of those crates I consider a useful "extended standard".

A Distributed SQL Database - Building the Database in the Public to Learn Database Internals

A lightweight, embedded key-value database for mobile clients (i.e., iOS, Android), written in Rust.

A web service that generates images of dependency graphs for crates hosted on crates.io

crates is an extension aims to help people to manage their dependencies for rust (crates.io & TOML).

https://crates.io/crates/transistor

Crates - A collection of open source Rust crates from iqlusion

Download .crate files of all versions of all crates from crates.io

A gui tool written in Dioxus to make it easy to release a workspace of crates to crates.io

Crates Registry is a tool for serving and publishing crates and serving rustup installation in offline networks.

JSON-RPC endpoint proxy that dumps requests/responses for debugging

A Rust application that inserts Discogs data dumps into Postgres

Scans a given directory for software of unknown provinence (SOUP) and dumps them in a json-file

Comments

Expose timestamps as DateTime

Drop dependency on serde's derive feature

Serialize Id structs

use Day

Releases(0.4.1)

0.4.1(Dec 17, 2022)

0.4.0(Oct 30, 2022)

0.3.11(Sep 25, 2022)

0.3.10(Sep 25, 2022)

0.3.9(Aug 24, 2022)

0.3.8(Aug 18, 2022)

0.3.7(Aug 3, 2022)

0.3.6(Aug 2, 2022)

0.3.5(Jul 30, 2022)

0.3.4(Feb 6, 2022)

0.3.3(Dec 25, 2021)

0.3.2(Oct 8, 2021)

0.3.1(Jun 19, 2021)

0.3.0(May 26, 2021)

0.2.0(May 19, 2021)

0.1.1(Apr 24, 2021)

0.1.0(Mar 16, 2021)

Owner

David Tolnay

Yet Another Technical Analysis library [for Rust]

Rust DataFrame library

Rayon: A data parallelism library for Rust

sparse linear algebra library for rust

ConnectorX - Fastest library to load data from DB to DataFrames in Rust and Python

A cross-platform library to retrieve performance statistics data.

This library provides a data view for reading and writing data in a byte array.

Dataflow is a data processing library, primarily for machine learning

A bit vector with the Rust standard library's portable SIMD API

A rust library built to support building time-series based projection models