Ran into this while updating the version of a crate that had never been audited before. I performed the version bump, certified the new version, and then had to manually remove the exemption from config.toml. I was told by @bholley that cargo vet should attempt to remove an exemption in this case and is likely being confused by the version bump.

STR:

mkdir test
cd test

cargo init --library
# add wast 43.0.0
vim Cargo.toml
cargo vet init
# note exemptions.wast exists in config.toml

vim Cargo.toml # bump wast to 44.0.0
cargo update
cargo vet # fails!

cargo vet certify wast 44.0.0
cargo vet # succeeds!

# note exemptions.wast 43.0.0 still exists in config.toml

STR: cd fresh-mozilla-central-clone cargo vet init cargo vet suggest

Resulting output: mio.txt

Note that 'mio' appears several times as a reverse-dependency, but doesn't show up as a recommendation for inspect. We should figure out why that is and resolve the inconsistency.

Notes fields generated with mach cargo vet are being unexpectedly deleted during mach vendor rust. I discussed with @bholley , who suggested that I file an issue here.

I'm able to reproduce the behavior with the Cargo.toml line from this patch:

+ cubeb-pulse = { git = "https://github.com/mozilla/cubeb-pulse-rs", rev="cf48897be5cbe147d051ebbbe1eaf5fd8fb6bbc9", optional = true, features=["pulse-dlopen"] }

Here's some command output:

# This first command doesn't seem to have an effect, but leaving it in just in case the output is helpful at all
$ ./mach cargo vet
ERROR 
  × 'cargo metadata' exited unsuccessfully
  ╰─▶ `cargo metadata` exited with an error:     Updating git repository `https://github.com/mozilla/cubeb-pulse-rs`
      error: the lock file /home/ashley/git/mozilla-unified/Cargo.lock needs to be updated but --locked was passed to prevent this
      If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

$ ./mach vendor rust
# ...
    Removing cubeb-pulse v0.4.0 (https://github.com/mozilla/cubeb-pulse-rs?rev=79096eb59b0bfcedb660716a67c0de34a25ea29e#79096eb5)                                                                                   
      Adding cubeb-pulse v0.4.1 (https://github.com/mozilla/cubeb-pulse-rs?rev=cf48897be5cbe147d051ebbbe1eaf5fd8fb6bbc9#cf48897b)                                                                                   
    Removing pulse v0.3.0 (https://github.com/mozilla/cubeb-pulse-rs?rev=79096eb59b0bfcedb660716a67c0de34a25ea29e#79096eb5)                                                                                         
      Adding pulse v0.3.0 (https://github.com/mozilla/cubeb-pulse-rs?rev=cf48897be5cbe147d051ebbbe1eaf5fd8fb6bbc9#cf48897b)                                                                                         
    Removing pulse-ffi v0.1.0 (https://github.com/mozilla/cubeb-pulse-rs?rev=79096eb59b0bfcedb660716a67c0de34a25ea29e#79096eb5)                                                                                     
      Adding pulse-ffi v0.1.0 (https://github.com/mozilla/cubeb-pulse-rs?rev=cf48897be5cbe147d051ebbbe1eaf5fd8fb6bbc9#cf48897b)
# ...

# Notes are removed by this point!
$ ./mach cargo vet
Vetting Succeeded (132 fully audited, 70 partially audited, 252 exempted)

# snip from diff output
diff --git a/supply-chain/audits.toml b/supply-chain/audits.toml
index 0ce3dbc2e61e2..efaf9ba795b75 100644
--- a/supply-chain/audits.toml
+++ b/supply-chain/audits.toml
@@ -266,9 +266,6 @@ delta = "0.10.1 -> 0.10.2"
 who = "Paul Adenot <[email protected]>"
 criteria = "safe-to-deploy"
 delta = "0.10.2 -> 0.10.3"
-note = """
-Mozilla-developed package.
-"""
 
 [[audits.cubeb-core]]
 who = "Matthew Gregan <[email protected]>"
@@ -287,9 +284,6 @@ delta = "0.10.1 -> 0.10.2"
 who = "Paul Adenot <[email protected]>"
 criteria = "safe-to-deploy"
 delta = "0.10.2 -> 0.10.3"
-note = """
-Mozilla-developed package.
-"""

Currently, on mozilla-central, if you run mach cargo vet, you get the following error:

ERROR 
  × There are some issues with your policy.audit-as-crates-io entries

Error: 
  × Some non-crates.io-fetched packages match published crates.io versions
  │   viaduct:0.1.0
  help: Add a `policy.*.audit-as-crates-io` entry for them

If you also have other errors, they don't appear. So when you run mach vendor rust, it doesn't find any vet error relevant to vendoring, and goes through, even when such errors do exist. Running mach cargo vet --locked under those conditions does prevent the policy.audit-as-crates-io from showing up and display the other errors (which phabricator CI catches because it uses --locked)

Consider two projects, A and B, who mutually trust eachother and import eachother's audits.

We are trying to validate Serde v10

A defines:

• 5: "reviewed"
• 8->9: "reviewed"
• B_criteria_map: "super_audited" -> "reviewed"

B defines:

• 5->8: "super_audited"
• 9->10: "audited"
• A_criteria_map: "reviewed" -> "audited"
• "super_audited" implies "audited"

Note how the two projects aren't agreeing on a bijective mapping -- B thinks "reviewed" is as good as audited, while A thinks you need to be super_audited to reach that bar.

When B goes to vet v10, it follows a chain of "audited" and concludes v10 is "audited" ✅

When A goes to vet v10, it makes it only to v9 on "reviewed" and then can't make progress ❌

This sucks, but ok sure they're supposed to agree to cooperate and this can be resolved. Now add in dependency_criteria -- unless there is a proper bijection, foreign audits with dependency_criteria are gibberish in your own terminology, so you need to either reject audits that can't be fully mapped into your terminology (and even then they might fail), or, I guess, try to track the audit resolution from the perspective of you and every foreign import your include at the same time and try to... somehow... make sense of that? This... gets messy very quickly.

And this is to say nothing of just, a third-party C trying to import A and/or B and make any sense of what's going on! Can they figure out v10 is "audited" but not "reviewed"? Are they actually expected to do the resolution from the simultaneous perspective of every file they include?

It seems like if your resolution chain for some crate involves a mix of local and foreign constraints, things get very wobbly and bad very quickly. It would be good to know:

• "how hard" we are expect to try to resolve criteria in these mixed situations
• under what conditions we're allowed to discard audits as "gibberish from my perspective"
• if we should try to signal/log when we do this
• some recommended "best practices" for orgs to follow to avoid things becoming impossible soup

There's kind of 4 tiers of "temp" in my books:

1. So temp that it evaporates when the process exits
2. In your system "temp location" with a randomly generated directory name, so nothing can "know" where to find it, not even ourselves on subsequent runs (security people will be happy if you do this)
3. In your system "temp location" with a deterministic name
4. Actually not temp at all, just in the current working directory

I currently implement 3 with name "cargo-vet-checkout" because it's helpful for debugging, but 2 and 4 are both reasonable (your example output implies 3). 1 is totally gibberish for our usecase but I am pedant so I had to list it.

No matter what default we pick, we should absolutely expose a CLI flag for configuring this a little, if only so the user can opt into some flavour of 4 if they want.

Worth noting that something like cargo vet suggest is actually a very heavyweight process, because we basically need to download two separate copies of your crategraph and then individually diff each copy! This suggests it might be desirable to be able to cache results between runs, to some extent?

My current implementation is very conservative and will just completely destroy and recreate cargo-vet-checkout whenever anything wants to write results to temp (suggest, fetch, diff).

This is mostly me setting up my "preferred" machinery for a CLI app (although this is my first time using clap's derive-based machinery, I am quite liking it here).

Some extra stuff like --verbose, --log-file, --output-file are just "standard" things I like a CLI app to have for debugging and making them more flexible to the user.

What actually is implemented:

• minimal vet init
• trivial vet (does not apply trusted/audits or deltas yet)

A lot of the machinery set up here should make many of the other commands relatively trivial (open toml, push value to a Vec, write toml back).

Right now when cargo-vet spits out an error (in this case some first-party packages match published crates.io versions), it doesn't format it as json even if --output-format=json is specified. This can cause opaque errors from infrastructure that's expecting json.

Here's another example, which stems from a toml parse error: https://treeherder.mozilla.org/jobs?repo=try&revision=8a3a904d1fc5eedc4fc349af5fcc18767b35144d

CC @glandium

Currently the cache is stored in the system's temporary directory. On platforms where XDG is relevant, we could consider using $XDG_CACHE_HOME instead.

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

$XDG_CACHE_HOME defines the base directory relative to which user-specific non-essential data files should be stored. If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used.

When running cargo vet certify, sometimes it picks a criteria on its own, sometimes not. Example of case where it doesn't: unicode-ident:1.0.0 -> 1.0.1 in m-c.

These were outlined in https://github.com/bholley/cargo-vet/issues/85#issuecomment-1133091909 and https://github.com/bholley/cargo-vet/issues/85#issuecomment-1133374812 , but not implemented in #140.

Bumps console from 0.15.0 to 0.15.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I experimented with setting up cargo vet on a couple repositories that have the following setup: some crates which we publish, and some other testing/development-only crates in the same workspace that depend on the published crates. If I set the criteria on testing crates to safe-to-run in the configuration file, then when I run cargo vet suggest, I only get suggestions for safe-to-run audits, and no suggestions for safe-to-deploy audits. If I explicitly set the criteria for all workspace crates in the configuration file, then cargo vet suggest gives me separate nonempty lists of suggested safe-to-run and safe-to-deploy audits, which is the behavior I want. It sounds like, via dependency edges, the policy changes on one workspace crate are affecting another workspace crate, and overriding the expected default behavior of safe-to-deploy for all workspace crates.

Here's a simplified example:

$ tree
.
├── Cargo.lock
├── Cargo.toml
├── crate-a
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── crate-b
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
└── supply-chain
    ├── audits.toml
    ├── config.toml
    └── imports.lock

5 directories, 9 files

$ cargo tree
crate-a v0.1.0 (/home/david/scratch/cargo-vet-workspace-crate-policies/crate-a)
└── ring v0.16.20
    ├── libc v0.2.138
    ├── once_cell v1.16.0
    ├── spin v0.5.2
    └── untrusted v0.7.1
    [build-dependencies]
    └── cc v1.0.78

crate-b v0.1.0 (/home/david/scratch/cargo-vet-workspace-crate-policies/crate-b)
├── crate-a v0.1.0 (/home/david/scratch/cargo-vet-workspace-crate-policies/crate-a) (*)
└── serde v1.0.151

No policy configurations, OK:

$ cargo vet suggest
recommended audits for safe-to-deploy:
    cargo vet inspect serde 1.0.151                       (used by crate-b)                                           (16352 lines)
    cargo vet inspect ring 0.16.20                        (used by crate-a)                                           (264901 lines)
    cargo vet inspect wasm-bindgen-shared 0.2.83          (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (519 lines)
    cargo vet inspect cfg-if 1.0.0                        (used by log, wasm-bindgen)                                 (588 lines)
    cargo vet inspect untrusted 0.7.1                     (used by ring)                                              (720 lines)
    cargo vet inspect winapi-i686-pc-windows-gnu 0.4.0    (used by winapi)                                            (1133 lines)
    cargo vet inspect winapi-x86_64-pc-windows-gnu 0.4.0  (used by winapi)                                            (1168 lines)
    cargo vet inspect wasm-bindgen-macro 0.2.83           (used by wasm-bindgen)                                      (1308 lines)
    cargo vet inspect spin 0.5.2                          (used by ring)                                              (1660 lines)
    cargo vet inspect wasm-bindgen-macro-support 0.2.83   (used by wasm-bindgen-macro)                                (2054 lines)
    cargo vet inspect unicode-ident 1.0.6                 (used by syn, proc-macro2)                                  (2772 lines)
    cargo vet inspect wasm-bindgen-backend 0.2.83         (used by wasm-bindgen-macro-support)                        (3014 lines)
    cargo vet inspect quote 1.0.23                        (used by syn, and 3 others)                                 (3873 lines)
    cargo vet inspect once_cell 1.16.0                    (used by ring, wasm-bindgen-backend)                        (4494 lines)
    cargo vet inspect log 0.4.17                          (used by wasm-bindgen-backend)                              (5695 lines)
    cargo vet inspect proc-macro2 1.0.49                  (used by syn, quote, and 2 others)                          (6078 lines)
    cargo vet inspect cc 1.0.78                           (used by ring)                                              (7019 lines)
    cargo vet inspect bumpalo 3.11.1                      (used by wasm-bindgen-backend)                              (10640 lines)
    cargo vet inspect js-sys 0.3.60                       (used by web-sys)                                           (12966 lines)
    cargo vet inspect wasm-bindgen 0.2.83                 (used by js-sys, web-sys)                                   (21672 lines)
    cargo vet inspect syn 1.0.107                         (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (57688 lines)
    cargo vet inspect libc 0.2.138                        (used by ring)                                              (99364 lines)
    cargo vet inspect winapi 0.3.9                        (used by ring)                                              (181329 lines)
    cargo vet inspect web-sys 0.3.60                      (used by ring)                                              (202213 lines)

estimated audit backlog: 909220 lines

Use |cargo vet certify| to record the audits.

After setting policy.crate-b.criteria = "safe-to-run", I get this unexpected behavior:

$ cargo vet suggest
recommended audits for safe-to-run:
    cargo vet inspect serde 1.0.151                       (used by crate-b)                                           (16352 lines)
    cargo vet inspect ring 0.16.20                        (used by crate-a)                                           (264901 lines)
    cargo vet inspect wasm-bindgen-shared 0.2.83          (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (519 lines)
    cargo vet inspect cfg-if 1.0.0                        (used by log, wasm-bindgen)                                 (588 lines)
    cargo vet inspect untrusted 0.7.1                     (used by ring)                                              (720 lines)
    cargo vet inspect winapi-i686-pc-windows-gnu 0.4.0    (used by winapi)                                            (1133 lines)
    cargo vet inspect winapi-x86_64-pc-windows-gnu 0.4.0  (used by winapi)                                            (1168 lines)
    cargo vet inspect wasm-bindgen-macro 0.2.83           (used by wasm-bindgen)                                      (1308 lines)
    cargo vet inspect spin 0.5.2                          (used by ring)                                              (1660 lines)
    cargo vet inspect wasm-bindgen-macro-support 0.2.83   (used by wasm-bindgen-macro)                                (2054 lines)
    cargo vet inspect unicode-ident 1.0.6                 (used by syn, proc-macro2)                                  (2772 lines)
    cargo vet inspect wasm-bindgen-backend 0.2.83         (used by wasm-bindgen-macro-support)                        (3014 lines)
    cargo vet inspect quote 1.0.23                        (used by syn, and 3 others)                                 (3873 lines)
    cargo vet inspect once_cell 1.16.0                    (used by ring, wasm-bindgen-backend)                        (4494 lines)
    cargo vet inspect log 0.4.17                          (used by wasm-bindgen-backend)                              (5695 lines)
    cargo vet inspect proc-macro2 1.0.49                  (used by syn, quote, and 2 others)                          (6078 lines)
    cargo vet inspect cc 1.0.78                           (used by ring)                                              (7019 lines)
    cargo vet inspect bumpalo 3.11.1                      (used by wasm-bindgen-backend)                              (10640 lines)
    cargo vet inspect js-sys 0.3.60                       (used by web-sys)                                           (12966 lines)
    cargo vet inspect wasm-bindgen 0.2.83                 (used by js-sys, web-sys)                                   (21672 lines)
    cargo vet inspect syn 1.0.107                         (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (57688 lines)
    cargo vet inspect libc 0.2.138                        (used by ring)                                              (99364 lines)
    cargo vet inspect winapi 0.3.9                        (used by ring)                                              (181329 lines)
    cargo vet inspect web-sys 0.3.60                      (used by ring)                                              (202213 lines)

estimated audit backlog: 909220 lines

Use |cargo vet certify| to record the audits.

$ cargo vet dump-graph
graph LR
    subgraph roots
        node4{crate-b:0.1.0}
    end
    subgraph workspace-members
        node3[/crate-a:0.1.0/]
    end
    subgraph first-party
    end
    subgraph third-party
    end
    node4 --> node3

Lastly, I set both policy.crate-a.criteria = "safe-to-deploy" and policy.crate-b.criteria = "safe-to-run", and get the behavior I want:

$ cargo vet suggest
recommended audits for safe-to-deploy:
    cargo vet inspect ring 0.16.20                        (used by crate-a)                                           (264901 lines)
    cargo vet inspect wasm-bindgen-shared 0.2.83          (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (519 lines)
    cargo vet inspect cfg-if 1.0.0                        (used by log, wasm-bindgen)                                 (588 lines)
    cargo vet inspect untrusted 0.7.1                     (used by ring)                                              (720 lines)
    cargo vet inspect winapi-i686-pc-windows-gnu 0.4.0    (used by winapi)                                            (1133 lines)
    cargo vet inspect winapi-x86_64-pc-windows-gnu 0.4.0  (used by winapi)                                            (1168 lines)
    cargo vet inspect wasm-bindgen-macro 0.2.83           (used by wasm-bindgen)                                      (1308 lines)
    cargo vet inspect spin 0.5.2                          (used by ring)                                              (1660 lines)
    cargo vet inspect wasm-bindgen-macro-support 0.2.83   (used by wasm-bindgen-macro)                                (2054 lines)
    cargo vet inspect unicode-ident 1.0.6                 (used by syn, proc-macro2)                                  (2772 lines)
    cargo vet inspect wasm-bindgen-backend 0.2.83         (used by wasm-bindgen-macro-support)                        (3014 lines)
    cargo vet inspect quote 1.0.23                        (used by syn, and 3 others)                                 (3873 lines)
    cargo vet inspect once_cell 1.16.0                    (used by ring, wasm-bindgen-backend)                        (4494 lines)
    cargo vet inspect log 0.4.17                          (used by wasm-bindgen-backend)                              (5695 lines)
    cargo vet inspect proc-macro2 1.0.49                  (used by syn, quote, and 2 others)                          (6078 lines)
    cargo vet inspect cc 1.0.78                           (used by ring)                                              (7019 lines)
    cargo vet inspect bumpalo 3.11.1                      (used by wasm-bindgen-backend)                              (10640 lines)
    cargo vet inspect js-sys 0.3.60                       (used by web-sys)                                           (12966 lines)
    cargo vet inspect wasm-bindgen 0.2.83                 (used by js-sys, web-sys)                                   (21672 lines)
    cargo vet inspect syn 1.0.107                         (used by wasm-bindgen-backend, wasm-bindgen-macro-support)  (57688 lines)
    cargo vet inspect libc 0.2.138                        (used by ring)                                              (99364 lines)
    cargo vet inspect winapi 0.3.9                        (used by ring)                                              (181329 lines)
    cargo vet inspect web-sys 0.3.60                      (used by ring)                                              (202213 lines)

recommended audits for safe-to-run:
    cargo vet inspect serde 1.0.151  (used by crate-b)  (16352 lines)

estimated audit backlog: 909220 lines

Use |cargo vet certify| to record the audits.

I installed cargo-vet from commit 61d140dabe996938cf10900b73e93b705270cc9f.

Introduction

Audits in cargo-vet are associated with audit criteria, which allows the system to track and enforce audits for multiple sets of properties. We further generalized this with the concept of dependency criteria, which allows audits for a crate to specify the criteria that should be enforced for that crate’s dependencies. Put differently: once we allow users to require arbitrary combinations of criteria for top-level third-party dependencies, we need to specify what gets enforced for transitive dependencies. We concluded that the requirements should be inherited down the crate graph by default, but that the default behavior could be overridden by specifying dependency-criteria on either an audit or a policy entry.

I’ve come to believe that dependency-criteria on audits was a mistake, and that we should remove the feature. It has never once been used in practice in any cargo-vet instance I’m aware of, which is a strong indicator that it is not successfully serving user needs. To understand why, let’s look at the original motivating use-cases.

Requirement Strengthening

The first and primary use-case was what we’ve sometimes referred to as “requirement strengthening”. The idea was that an auditor might conclude that the validity of the audit depends on certain properties of a dependent crate. For example, whether or not a crate is safe-to-deploy might depend on whether a dependent crate returns the right-sized parameters for an unsafely-allocated memory buffer, a properly-validated path for a filesystem operation, or the correct definition of a constant for an FFI call. Because semantic versioning generally allows the contents of the dependency to vary across deployments at the publisher’s discretion, it’s technically not enough even to verify these properties directly in the current version of the dependent crate, because they might change out from under you.

Unfortunately, I think it’s unrealistic to expect people to do this consistently and at-scale. A corner-cutting auditor might eyeball a block of unsafe code and conclude it’s fine without fully tracing through the information flow of its inputs. A judicious auditor might trace those inputs back through rest of the crate. A tenacious auditor might even trace them across crate boundaries into the current version of a dependent crate. But approximately nobody is going to do all that and then derive the full set of required properties, articulate those properties as coherent custom criteria, and then figure out how to encode all of that in cargo-vet.

Realistically most auditors will end up assuming that external crates behave as advertised, and so I don’t think it’s productive to try to set the bar higher than that. This does create some potential failure modes and attack vectors — i.e. the introduction of an exploitable defect via separate seemingly-innocuous changes in two different crates — but the concrete risk level is debatable and I don’t want to let perfect be the enemy of good.

Requirement Weakening

The second use-case is “requirement weakening”. The idea is that an auditor might discover while auditing crate X that its transitive dependency Y is only used in a very limited way (e.g. in development mode) that requires a weaker audit. Unfortunately, people aren’t thinking about this consideration when auditing X. They might think about it when auditing Y (“why do we have this dependency anyway? Oh I see, it’s used by X's tests”) — but at that point it’s generally too late, since the flight rules of the system generally don’t permit you to just go and amend the existing audit for X.

So in practice nobody is doing any requirement weakening, which means in practice it’s not saving us any work. Moreover, much of the value here is automatically captured by cargo-vet's default behavior of requiring only safe-to-run for dev-dependencies. We have found it occasionally useful to record dependency-criteria overrides on policy entries, and I propose we keep the concept there. These aren’t shareable across projects, but they’ve proven themselves to have actual utility whereas the ones on audit entries have not.

Simplifying the Implementation

Beyond simplifying the feature set and aligning it with real user needs, dropping dependency-criteria from audit entries would dramatically simplify the implementation. They are a large (perhaps the largest) driver of complexity in the algorithm, and I suspect removing them would make the code much easier to reason about.

Bumps serde from 1.0.137 to 1.0.151.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps thiserror from 1.0.31 to 1.0.38.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

The vast majority of third-party crates that we pull in are downloaded through crates.io, and audit-as-crates-io Does The Right Thing for much of the remainder. That said, the behavior specified on first-party code docs leaves the potential for audit gaps:

[dependencies]
# If `anyhow` at this revision were published onto crates.io, it would appear as 1.0.66.
# However, the sources may not equal 1.0.66 as published.
# anyhow = { git = "https://github.com/dtolnay/anyhow", rev = "60e88002dcf7315487cbb2c724c66d2297c7bf46" }
anyhow = "=1.0.66"

Currently, the audit for the git-based anyhow would apply to the published anyhow, and vice-versa. It'd be nice if there were a mechanism for cargo-vet to differentiate between these somehow. Especially since the documentation notes:

""" It's important to note that any audits for such derived crates still correspond to the crates.io version. This is what inspect and certify will display, and this is what you should review before certifying, since others in the ecosystem may rely on your audits when using the original crate without your particular modifications. """

Looking around, maybe #170 is relevant here?