Taking a copy of the mozilla-central repository, and trying to run vendor-filterer on it yields the following:

$ cargo vendor-filterer
error: Executing cargo metadata (first run): Error during execution of `cargo metadata`: error: failed to parse manifest at `/tmp/gecko/Cargo.toml`

Caused by:
  missing field `name` for key `package`

cargo vendor-filterer fails to replace certain crates, like kernel32-sys or advapi32-sys, because cargo metadata fails on the vendored crates.

Steps to reproduce:

$ cargo new just-test
     Created binary (application) `just-test` package
$ cd just-test
$ cargo add local-encoding
    Updating crates.io index
      Adding local-encoding v0.2.0 to dependencies.
$ cargo vendor-filterer --platform x86_64-unknown-linux-gnu
Gathering metadata for platforms
   Vendoring fuchsia-cprng v0.1.1 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/fuchsia-cprng-0.1.1) to vendor/fuchsia-cprng
   Vendoring getopts v0.2.21 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/getopts-0.2.21) to vendor/getopts
   Vendoring kernel32-sys v0.2.2 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/kernel32-sys-0.2.2) to vendor/kernel32-sys
   Vendoring libc v0.2.135 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.135) to vendor/libc
   Vendoring local-encoding v0.2.0 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/local-encoding-0.2.0) to vendor/local-encoding
   Vendoring pulldown-cmark v0.0.3 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/pulldown-cmark-0.0.3) to vendor/pulldown-cmark
   Vendoring rand v0.4.6 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.4.6) to vendor/rand
   Vendoring rand_core v0.3.1 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.3.1) to vendor/rand_core-0.3.1
   Vendoring rand_core v0.4.2 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.4.2) to vendor/rand_core
   Vendoring rdrand v0.4.0 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/rdrand-0.4.0) to vendor/rdrand
   Vendoring remove_dir_all v0.5.3 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/remove_dir_all-0.5.3) to vendor/remove_dir_all
   Vendoring skeptic v0.4.0 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/skeptic-0.4.0) to vendor/skeptic
   Vendoring tempdir v0.3.7 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/tempdir-0.3.7) to vendor/tempdir
   Vendoring unicode-width v0.1.10 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.10) to vendor/unicode-width
   Vendoring winapi v0.2.8 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-0.2.8) to vendor/winapi-0.2.8
   Vendoring winapi v0.3.9 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-0.3.9) to vendor/winapi
   Vendoring winapi-build v0.1.1 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-build-0.1.1) to vendor/winapi-build
   Vendoring winapi-i686-pc-windows-gnu v0.4.0 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-i686-pc-windows-gnu-0.4.0) to vendor/winapi-i686-pc-windows-gnu
   Vendoring winapi-x86_64-pc-windows-gnu v0.4.0 (/crypt/iv/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-x86_64-pc-windows-gnu-0.4.0) to vendor/winapi-x86_64-pc-windows-gnu
To use vendored sources, add this to your .cargo/config.toml for this project:


[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"
Skipping just-test
Replacing unreferenced package with stub: winapi-x86_64-pc-windows-gnu
Replacing unreferenced package with stub: winapi-i686-pc-windows-gnu
Replacing unreferenced package with stub: winapi-build
Replacing unreferenced package with stub: winapi
error: Replacing with stub: winapi-0.2.8: Executing cargo metadata: Error during execution of `cargo metadata`: error: failed to load manifest for dependency `advapi32-sys`

Caused by:
  failed to read `/tmp/.private/iv/just-test/vendor/winapi-0.2.8/lib/advapi32/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Same error, but with kernel32-sys vendored crate, happens when trying to vendor dependencies for https://github.com/mozilla/sccache checked out to tag v0.3.0.

Our current tar default puts everything at the toplevel, which is in retrospect surprising given a more standard practice of including the vendor/ directory.

For backwards compatibility, keep things as is, but add a --prefix= option which allows configuring this.

Closes: https://github.com/coreos/cargo-vendor-filterer/issues/27

Rework stub crate generation

Closes: https://github.com/coreos/cargo-vendor-filterer/issues/32

Our approach of filtering down the Cargo.toml kept running into problems with trying to actually run cargo metadata on the result. Specifically some crates have path dependencies that can't actually be built at runtime anyways.

Creating stub crates is tricky, because we do need to keep things like the exposed feature set because Cargo will verify that, even if the target crate isn't being compiled.

But, we do need to remove other unwanted things such as build scripts.

Switch to filtering the Cargo.toml by parsing via TOML directly. We no longer run cargo metadata on the result.

ci: Validate vendor filtering for nushell

This will validate our vendoring works for a real-world package.

This is probably similar to #32

I tried running vendor-filterer on https://github.com/nushell/nushell/ with this config:

[package.metadata.vendor-filter]
platforms = ["x86_64-unknown-linux-gnu", "i686-unknown-linux-gnu"]
all-features = true
exclude-crate-paths = [
  { name = "libgit2-sys", exclude = "libgit2" },
  { name = "libsqlite3-sys", exclude = "sqlite3" },
  { name = "libsqlite3-sys", exclude = "sqlcipher" },
  { name = "libssh2-sys", exclude = "libssh2" },
  { name = "libz-sys", exclude = "src/zlib" },
  { name = "libz-sys", exclude = "src/zlib-ng" },
  { name = "lz4-sys", exclude = "liblz4" },
  { name = "openssl-src", exclude = "openssl" },
  { name = "zstd-sys", exclude = "zstd" },
]

and get this error:

Skipping nu-parser
Skipping nu_plugin_inc
Skipping nu-glob
Skipping nu_plugin_gstat
Skipping nu-engine
Skipping nu-path
Skipping nu_plugin_custom_values
Skipping nu-protocol
Skipping nu-test-support
Skipping nu-color-config
Skipping nu_plugin_query
Skipping nu-utils
Skipping nu-table
Skipping nu-term-grid
Skipping nu-system
Skipping nu-cli
Skipping nu-plugin
Skipping nu-json
Skipping nu_plugin_example
Skipping nu-command
Skipping nu
Skipping nu-pretty-hex
Replacing unreferenced package with stub: windows-sys
Replacing unreferenced package with stub: malloc_buf
Replacing unreferenced package with stub: ntapi
Replacing unreferenced package with stub: winres
Replacing unreferenced package with stub: riscv-target
Replacing unreferenced package with stub: windows-0.37.0
Replacing unreferenced package with stub: rdrand
Replacing unreferenced package with stub: crossterm_winapi
Replacing unreferenced package with stub: core-foundation
Replacing unreferenced package with stub: semver-parser-0.7.0
Replacing unreferenced package with stub: rustc-hash
Replacing unreferenced package with stub: objc
Replacing unreferenced package with stub: windows
Replacing unreferenced package with stub: shlex
Replacing unreferenced package with stub: wasm-bindgen
Replacing unreferenced package with stub: web-sys
Replacing unreferenced package with stub: wasi-0.9.0+wasi-snapshot-preview1
Replacing unreferenced package with stub: volatile-register
Replacing unreferenced package with stub: atomic-polyfill
Replacing unreferenced package with stub: android_system_properties
Replacing unreferenced package with stub: omnipath
Replacing unreferenced package with stub: windows_aarch64_msvc-0.36.1
Replacing unreferenced package with stub: fuchsia-zircon
Replacing unreferenced package with stub: windows_i686_gnu
Replacing unreferenced package with stub: wasm-bindgen-macro
Replacing unreferenced package with stub: winapi-util
Replacing unreferenced package with stub: fsevent-sys
Replacing unreferenced package with stub: peeking_take_while
Replacing unreferenced package with stub: rand_core-0.4.2
Replacing unreferenced package with stub: critical-section
Replacing unreferenced package with stub: winapi-build
Replacing unreferenced package with stub: cty
Replacing unreferenced package with stub: windows_i686_msvc-0.36.1
Replacing unreferenced package with stub: fuchsia-zircon-sys
Replacing unreferenced package with stub: cexpr
Replacing unreferenced package with stub: windows_x86_64_msvc-0.36.1
Replacing unreferenced package with stub: rand_core-0.3.1
Replacing unreferenced package with stub: bindgen
Replacing unreferenced package with stub: rand_hc
Replacing unreferenced package with stub: instant
Replacing unreferenced package with stub: vcell
Replacing unreferenced package with stub: wasm-bindgen-backend
Replacing unreferenced package with stub: windows_i686_gnu-0.36.1
Replacing unreferenced package with stub: windows_i686_gnu-0.37.0
Replacing unreferenced package with stub: embedded-hal
Replacing unreferenced package with stub: nb
Replacing unreferenced package with stub: schannel
Replacing unreferenced package with stub: windows_x86_64_gnullvm
error: Replacing with stub: ws2_32-sys: Executing cargo metadata: Error during execution of `cargo metadata`: error: failed to load manifest for dependency `winapi-build`

Caused by:
  failed to read `/mnt/progs/slack/builds/nushell/nushell-0.71.0/build/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

What's actually a bit annoying is that it stops after that error. It would be nice that, even if it encounters a lib that can't be replaced, that it just skips it and moves on with the rest.

Right now the platforms is basically an API we need to commit to, but I think most users would want:

platform-wildcards: ["*-linux-gnu", "wasm32-*"]

for example. In particular I discovered over in https://github.com/coreos/coreos-installer/pull/894 that coreos-installer actually has s390x-specific dependencies.

This is how my Cargo.toml looks like:

[package]
name = "my_package"
version = "0.1.0"
edition = "2021"

[dependencies]
thiserror = "1.0.30"
tracing = "0.1"
# Fix for https://github.com/Absolucy/tracing-oslog/pull/3
tracing-oslog = { git = "https://github.com/kgrech/tracing-oslog", branch = "main" }
tracing-core = "0.1"
tracing-subscriber = "0.3"

[dev-dependencies]
libc = "0.2"

You can see that tracing-oslog is not fetched from crates-io, but it is fetch from github. I would sill like to vendor it so that I can build my project without connection to internet.

However right now it is not vendored because of the .filter(|source| source.is_crates_io()) line.

I am making an assumption that the idea of this if is not to vendor local dependencies. However if it is a local dependency, then source for it is already None and there is no need to have a filter on top of it.

My assumption could be very well wrong. I can imagine the situation where some crates are pulled from local corporate git system and some are pulled from the internet. For such case, skipping .filter(|source| source.is_crates_io()) makes more sense, but it still looks far from perfect to me as it would fail as soon as you would like to make a fork of some github package on github.

If the intention was to support the second case, would not it be better idea to have allowlist and blocklist support for the source strings in form of regular expressions? Or faster alternative could to make filtering configurable by passing something like --skip-non-crates-io

cargo vendor produces an output directory (defaulting to vendor) which contains a subdirectory for each of the crates being vendored. It's been our practice in CoreOS projects to tar up that directory and ship the tarball as a release artifact, so that downstream distro packages can build the project without individually fetching dependencies.

Tarballs produced by cargo-vendor-filterer don't serve as a drop-in replacement for this practice, since the crate directories are directly placed in the root of the tarball instead of in a vendor subdirectory. This causes problems for a couple reasons:

• The downstream package needs to adjust its unpacking code to do mkdir vendor && tar xf vendor.tgz -C vendor instead of just tar xf vendor.tgz, which is a needless compatibility break.
• It's generally an antipattern to distribute tarballs which, unless special attention is paid during unpacking, create many entries in the current working directory.

There doesn't seem to be any way to configure this behavior, other than by avoiding --format tar* and packing the resulting tarball manually.

I'd missed that this exists earlier. This allows configuring vendoring for full workspaces too.

Closes: https://github.com/coreos/cargo-vendor-filterer/issues/41

It's really unfortunate that the stdlib doesn't have an ergonomic default API for this.

I noticed this when trying to output to a target with a nonexistent parent directory.

It is possible to add the following to the Cargo.toml file:

[dependencies]
tracing-oslog = { git = "https://github.com/kgrech/tracing-oslog", branch = "zero_byte_fix" }

In this case .cargo-checksum.json will look similar to this:

{
  "files": {
    ".editorconfig": "97449e38427d3efc434009cb1ae1942b56daf7b28a942484cb196d94f015b200",
    ".rustfmt.toml": "113bab55bd151637aa8aad2e0f6de79596d99a82d2e24449d198f6606e6b97ed",
    "Cargo.toml": "c74d0e88d99e379665046b4ba7b8a90218f7993280e6c30869f8e283a16f0684",
    "LICENSE.md": "b36ff589a7b586161394980ac056831e3795e7390270c9be98bb2fa85829646d",
    "README.md": "541e4e5e5c8ac8f5ee4982cc828ac6864bad3e6343e9d9ca324ae7824639df63",
    "build.rs": "3cb1a3c31663e69a5331d3a00ffed50b048c85d7fe8c5fb94dc2a87c9f82a125",
    "src/ffi.rs": "d9dde313263113f9883a1ef6eb1a3f71b8366199125284d9dd7489729fc0e861",
    "src/lib.rs": "af7f3c1dc4a7612f3519b812f8d6f9298f0f8af2e999ee3a23e6e9a5ddce5d75",
    "src/logger.rs": "e54218ee849b13046647f367a6064bbdce97fc22214105c7e39067cf7bc9be08",
    "src/stub.rs": "0fee8535f7f41a56207138af674e23747052191a64156aa97c1786c049f9f5f8",
    "src/visitor.rs": "dabe10d3299b69a0c7ecf35bd1a3fd0a375e5f9bf009f8b22b0b4516212dff85",
    "wrapper.c": "1b5f198a96432e182673ad51ed8cf70974f6a57477672a8afb3872523ae0286d",
    "wrapper.h": "347451ee2239759635a5ff8ccd4b738f37877261b5f9db76abd06b319593dc1b"
  },
  "package": null
}

Note that the package could be set to null.

It looks like we do not use it (yet), so just marked it optional and added a unit test

Is there way to specify multiple platforms from command line? When I try cargo vendor-filterer --platform='*'-unknown-linux-gnu (for sequoia-sop) here is error:

  error: Could not create LLVM TargetMachine for triple: m68k-unknown-linux-gnu: No available targets are compatible with triple "m68k-unknown-linux-gnu"

When I try to specify multiple --platform= options like:

     --platform=aarch64-unknown-linux-gnu \
     --platform=armv7-unknown-linux-gnueabihf \
     --platform=i686-unknown-linux-gnu \
     --platform=powerpc64le-unknown-linux-gnu \
     --platform=x86_64-unknown-linux-gnu

It does not compile on x86_64, perhaps something superfluous is filtered.

The tar compression flags invoke gzip and zstd with their default compression levels, which are not especially tight. Since we're creating long-term archives, we should probably compress as tightly as reasonably possible. That does require that we invoke the compressor separately.