Tool to `cargo vendor` with filtering

Overview

cargo vendor, but with filtering

The core cargo vendor tool is useful to save all dependencies. However, it doesn't offer any filtering; today cargo includes all platforms, but some projects only care about Linux for example.

More information: rust-lang/cargo#7058

Generating a vendor/ directory with filtering

Here's a basic example which filters out all crates that don't target Linux; for example this will drop out crates like winapi-x86_64-pc-windows-gnu and core-foundation that are Windows or MacOS only.

$ cargo vendor-filterer --platform=x86_64-unknown-linux-gnu

You can also declaratively specify the desired vendor configuration via the Cargo metadata key package.metadata.vendor-filter:

[package.metadata.vendor-filter]
platforms = ["x86_64-unknown-linux-gnu"]
all-features = true
exclude-crate-paths = [ { name = "curl-sys", exclude = "curl" },
                        { name = "libz-sys", exclude = "src/zlib" },
                        { name = "libz-sys", exclude = "src/zlib-ng" },
                      ]

Available options for for package.metadata.vendor-filter in Cargo.toml

  • platforms: List of rustc target triples; this is the same values accepted by e.g. cargo metadata --filter-platform. You can specify multiple values, however at the moment wildcards are not supported.
  • all-features: Enable all features of the current crate when vendoring.
  • exclude-crate-paths: Remove files and directories from target crates. A key use case for this is removing the vendored copy of C libraries embedded in crates like libz-sys, when you only want to support dynamically linking.

All of these options have corresponding CLI flags; see cargo vendor-filterer --help.

Generating reproducible vendor tarballs

You can also provide --format=tar.zstd to output a reproducible tar archive compressed via zstd; the default filename will be vendor.tar.zstd. Similarly there is --format=tar.gz for gzip, and --format=tar to output an uncompressed tar archive, which you can compress however you like.

This option requires:

  • An external GNU tar program
  • An external gzip or zstd program (for --format=tar.gz and --format=tar.zstd respectively)
  • SOURCE_DATE_EPOCH set in the environment, or an external git and the working directory must be a git repository

This uses the suggested code from https://reproducible-builds.org/docs/archives/ to output a reproducible archive; in other words, another process/tool can also perform a git clone of your project and regenerate the vendor tarball to verify it.

Comments
  • Add `--prefix=vendor`

    Add `--prefix=vendor`

    Our current tar default puts everything at the toplevel, which is in retrospect surprising given a more standard practice of including the vendor/ directory.

    For backwards compatibility, keep things as is, but add a --prefix= option which allows configuring this.

    Closes: https://github.com/coreos/cargo-vendor-filterer/issues/27

    opened by cgwalters 3
  • Add `platform-wildcards`

    Add `platform-wildcards`

    Right now the platforms is basically an API we need to commit to, but I think most users would want:

    platform-wildcards: ["*-linux-gnu", "wasm32-*"]

    for example. In particular I discovered over in https://github.com/coreos/coreos-installer/pull/894 that coreos-installer actually has s390x-specific dependencies.

    opened by cgwalters 2
  • Output tarball does not wrap crate directories in a containing directory

    Output tarball does not wrap crate directories in a containing directory

    cargo vendor produces an output directory (defaulting to vendor) which contains a subdirectory for each of the crates being vendored. It's been our practice in CoreOS projects to tar up that directory and ship the tarball as a release artifact, so that downstream distro packages can build the project without individually fetching dependencies.

    Tarballs produced by cargo-vendor-filterer don't serve as a drop-in replacement for this practice, since the crate directories are directly placed in the root of the tarball instead of in a vendor subdirectory. This causes problems for a couple reasons:

    • The downstream package needs to adjust its unpacking code to do mkdir vendor && tar xf vendor.tgz -C vendor instead of just tar xf vendor.tgz, which is a needless compatibility break.
    • It's generally an antipattern to distribute tarballs which, unless special attention is paid during unpacking, create many entries in the current working directory.

    There doesn't seem to be any way to configure this behavior, other than by avoiding --format tar* and packing the resulting tarball manually.

    bug 
    opened by bgilbert 1
  • main: Use `eprintln!` for logging

    main: Use `eprintln!` for logging

    We need to log to stderr for compatibility because users expect to be able to redirect the output of cargo vendor to a file as it generates a cargo config file.

    opened by cgwalters 0
  • Add `--format=tar` and `--format=tar.zstd`

    Add `--format=tar` and `--format=tar.zstd`

    For many of our projects we want to attach them to e.g. Github releases, which is best done as a single file archive. We spend a bit of effort to make the output of this reproducible.

    opened by cgwalters 0
  • Support excluding paths from crates

    Support excluding paths from crates

    In some use cases, one wants to e.g. drop vendored C sources and hence require dynamically linking to the target library. We do this in rpm-ostree with curl-sys.

    opened by cgwalters 0
  • Fixup case of being run as cargo subcommand

    Fixup case of being run as cargo subcommand

    Right now cargo passes us our own subcommand name as an extra argument, which overlaps with us accepting a path. Special case undoing this.

    It'd be better if cargo set an environment variable we could use to detect this.

    opened by cgwalters 0
  • Log to stderr, not stdout

    Log to stderr, not stdout

    At least in rpm-ostree we redirect the output of cargo vendor to a file to capture its suggested cargo.toml config, so we need to log to stderr the same way cargo vendor prints.

    opened by cgwalters 0
  • Inject dummy workspace to gather vendored dep metadata

    Inject dummy workspace to gather vendored dep metadata

    cargo wants to be nice and have things like cargo build in a subproject auto-discover the workspace.

    However we don't want this behavior when gathering vendored dep metadata. Force inject a [workspace] member; it'd be nice if there was an environment variable/CLI for this in cargo.

    opened by cgwalters 0
  • Skip non-crates-io packages

    Skip non-crates-io packages

    cargo vendor only vendors stuff from crates.io; we expect path dependencies are already included from e.g. a git checkout of the project.

    Closes: https://github.com/cgwalters/cargo-vendor-filterer/issues/1

    opened by cgwalters 0
  • Compress more tightly

    Compress more tightly

    The tar compression flags invoke gzip and zstd with their default compression levels, which are not especially tight. Since we're creating long-term archives, we should probably compress as tightly as reasonably possible. That does require that we invoke the compressor separately.

    opened by bgilbert 3
Owner
Colin Walters
@openshift & Fedora/RHEL @coreos engineer at @RedHatOfficial
Colin Walters
Cargo-eval - A cargo plugin to quickly evaluate some Rust source code.

cargo eval A cargo plugin to quickly evaluate some Rust source code. Installation $ cargo install --git https://github.com/timClicks/cargo-eval.git Us

Tim McNamara 9 Jan 1, 2022
Cargo-about - 📜 Cargo plugin to generate list of all licenses for a crate 🦀

?? cargo-about Cargo plugin for generating a license listing for all dependencies of a crate See the book ?? for in-depth documentation. Please Note:

Embark 259 Aug 22, 2022
A CLI tool that allow you to create a temporary new rust project using cargo with already installed dependencies

cargo-temp A CLI tool that allow you to create a new rust project in a temporary directory with already installed dependencies. Install Requires Rust

Yohan Boogaert 60 Sep 14, 2022
A simple, modern fuzzy finder tool to run examples in a Cargo project.

cargo-rx cargo-rx is a simple, modern Runner for Examples in a Cargo project. This crate provides a single executable: rx. Basically anywhere you woul

Ritvik Nag 12 May 3, 2022
A very simple third-party cargo subcommand to execute a custom command

cargo-x A very simple third-party cargo subcommand to execute a custom command Usage install cargo-x cargo install cargo-x or upgrade cargo install -

刘冲 8 May 5, 2022
a cargo subcommand for counting lines of code in Rust projects

cargo-count Linux: A cargo subcommand for displaying line counts of source code in projects, including a niave unsafe counter for Rust source files. T

Kevin K. 123 Sep 6, 2022
Cargo script subcommand

cargo-script cargo-script is a Cargo subcommand designed to let people quickly and easily run Rust "scripts" which can make use of Cargo's package eco

Daniel Keep 630 Sep 18, 2022
A cargo subcommand for checking and applying updates to installed executables

cargo-update A cargo subcommand for checking and applying updates to installed executables Documentation Manpage Installation Firstly, ensure you have

наб 752 Sep 19, 2022
Cargo subcommand `release`: everything about releasing a rust crate.

cargo release Features Ensure you are in a good state for release, including: Right branch Up-to-date with remote Clean tree Supports workspaces using

null 872 Sep 24, 2022
Watches over your Cargo project's source.

$ cargo watch Cargo Watch watches over your project's source for changes, and runs Cargo commands when they occur. If you've used nodemon, guard, or e

null 1.9k Sep 21, 2022
A utility for managing cargo dependencies from the command line.

cargo edit This tool extends Cargo to allow you to add, remove, and upgrade dependencies by modifying your Cargo.toml file from the command line. Curr

Pascal Hertleif 2.6k Sep 24, 2022
A cargo subcommand that displays ghidra function output through the use of the rizin rz-ghidra project.

cargo-rz-ghidra A cargo subcommand that displays ghidra function output through the use of the rizin rz-ghidra project. Install cargo install --git ht

wcampbell 3 May 10, 2022
Helps cargo build and run apps for iOS

cargo-xcodebuild Helps cargo build and run apps for iOS. ?? ⚙️ ?? Setup You need to install Xcode (NOT just Command Line Tools!), xcodegen, cargo-xcod

Igor Shaposhnik 25 Jul 4, 2022
Cargo subcommand to easily run targets/examples

cargo-select Cargo subcommand to easily run targets/examples/tests Fuzzy match against targets, examples or tests in current rust project. cargo-selec

null 12 Jul 18, 2022
CLI Tool for tagging and organizing files by tags.

wutag ?? ??️ CLI tool for tagging and organizing files by tags. Install If you use arch Linux and have AUR repositories set up you can use your favour

Wojciech Kępka 30 Sep 8, 2022
CLI tool to bake your fresh and hot MD files

At least once in your Rust dev lifetime you wanted to make sure all code examples in your markdown files are up-to-date, correct and code is formated, but you couldn't make that done with already existing tools - fear not!

Patryk Budzyński 39 May 8, 2021
qsv - Performant CLI tool to query CSVs through SQL

qsv Performant CLI tool to query CSVs through SQL Installation After cloning the repository, you can install a binary locally using cargo install --pa

Dermot Haughey 3 Oct 28, 2021
dua (-> Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory

dua (-> Disk Usage Analyzer) is a tool to conveniently learn about the usage of disk space of a given directory. It's parallel by default and will max

Sebastian Thiel 1.6k Sep 22, 2022
rip is a command-line deletion tool focused on safety, ergonomics, and performance

rip (Rm ImProved) rip is a command-line deletion tool focused on safety, ergonomics, and performance. It favors a simple interface, and does not imple

Kevin Liu 653 Sep 19, 2022