get-all-crates
Download .crate files of all versions of all crates from crates.io.
Useful for things like noisy-clippy which need to analyze the source code of all crates.
I wrote this tool to saturate a 1000 Mbps connection. From scratch, it can finish downloading in under 20 minutes. It can also check the checksums of a directory of downloaded crates in 40 seconds on 64 cores. For a more fully featured tool, which some of this code is based on, but which is much slower in my experience, check out https://git.shipyard.rs/jstrong/registry-backup.
Usage
$ cargo install get-all-crates
$ git clone https://github.com/rust-lang/crates.io-index /path/to/index
$ get-all-crates --index /path/to/index --out /path/to/crates
Warning: as of 2022 this writes out more than 100 GB of crates.
The output directory structure is similar to how Cargo's registry index is structured:
/path/to/crates
├── 1
│ └── m
│ └── m-0.1.1.crate
├── 2
│ └── nu
│ └── nu-0.73.0.crate
├── 3
│ └── s
│ └── syn
│ └── syn-1.0.107.crate
└── se
└── rd
├── serde
│ └── serde-1.0.151.crate
└── serde_json
└── serde_json-1.0.91.crate
1.3k Jan 1, 2023
837 Dec 30, 2022
50 Jan 1, 2023
0 Nov 7, 2021
2 Dec 13, 2022
54 Dec 20, 2022
13 Dec 30, 2022
20 Nov 9, 2020
164 Jan 4, 2023
28 May 28, 2022
0 Dec 23, 2021
682 Jan 1, 2023
335 Dec 26, 2022
13 Jan 18, 2023
5 Jul 6, 2023
338 Jan 1, 2023
20 Dec 28, 2022
8 Mar 12, 2022
9 Mar 4, 2022
19 Jan 4, 2023
3 Jul 30, 2022
3 Dec 15, 2022
3 Dec 10, 2022
31 Feb 15, 2023
4 Jun 9, 2023
1k Jan 5, 2023
30 Dec 26, 2022
10 Jun 7, 2023
2 Sep 12, 2021