Given this project's emphasis on performance, I thought it'd be nice to have some built-in benchmarking. This way, contributors can easily check if their changes improve or hinder performance against a relatively standardized test (of course, absolute values will vary machine-to-machine).

As a first attempt at this, I decided to use a Rust library Criterion. I hit some issues right away, though: Criterion makes it pretty difficult to benchmark functions that are not in src/lib.rs (see this Stack Overflow Q&A).

Because of this, I decided to take the rather aggressive step of renaming src/find_secrets.rs -> src/lib.rs. I think this has benefits beyond easy benchmarking; for example, if someone wanted to use ripsecrets as a Rust library, I think it makes sense to have a lib.rs file.

I also folded the p_random file into the matcher module, which I think works well, since it appears that p_random's public function is only used within the matcher module.

Currently, there's only one benchmark. It basically runs find_secrets on the ripsecrets directory itself (I think?). On my System76 Oryx Pro, mean bench time is about 56ms.

While secrets is basically "get secrets" and expect none to be returned, the name is, well, not very S.E.O. friendly. @sirwart, are you open to considering some other names while this tool is still in its infancy?

On each commit to main, run benchmarks and publish them to the gh-pages branch. That branch can be configured as the source of GitHub Pages, so it gets built and published on each change. See https://lafrenierejm.github.io/ripsecrets/dev/bench/ for an example of benchmarks published from my fork.

When trying to install ripsecrets via cargo I noticed that it now tries to download the entire sentry module because it's listed as a submodule, which takes a long time because it's a large repo. Either the configuration needs to change or removed as a submodule.

Attempt to rebase #25 onto the current HEAD of main, 6feb4f7dccc6834877fe61dcb94cbb547a55a176.

Copying the first part of @sts10's PR description:

Given this project's emphasis on performance, I thought it'd be nice to have some built-in benchmarking. This way, contributors can easily check if their changes improve or hinder performance against a relatively standardized test (of course, absolute values will vary machine-to-machine).

As a first attempt at this, I decided to use a Rust library Criterion. I hit some issues right away, though: Criterion makes it pretty difficult to benchmark functions that are not in src/lib.rs (see this Stack Overflow Q&A).

Because of this, I decided to take the rather aggressive step of renaming src/find_secrets.rs -> src/lib.rs. I think this has benefits beyond easy benchmarking; for example, if someone wanted to use ripsecrets as a Rust library, I think it makes sense to have a lib.rs file.

I write a simple code like

package main

func main() {
        clientSecretKey := "alkfjlaf^*flkajlfkay7782085ljafg"
        println(clientSecretKey)
}

and hope ripsecrets can tell me 'you hardcode the secret in source files', but there's nothing output

According to this PR: https://github.com/sirwart/ripsecrets/pull/2:

Once this is merged, someone (me?) could put a PR on https://github.com/pre-commit/pre-commit.com/blob/main/all-repos.yaml that adds https://github.com/sirwart/secrets to it.

Someone needs to add https://github.com/sirwart/ripsecrets to the pre-commit repository.

Add only-matching option to only output matching parts of the string (similar to how rg goes it).

I had to replace printer::Standard with printer::StandardBuilder in order to achieve that - hope it's not an issue.

Attempt 2 at implemented clap on this project. I found I had to make the rather controversial choice of enforcing use of subcommands, so users will need to run secrets check file1 file2 now. I don't love it, and if it's not expected I understand. But I do think integrating a argument parser will save some headaches as more command-line options are added.

The recent change (https://github.com/sirwart/ripsecrets/commit/c40ddbd63c7bf6f08c7e7fe11bfe2a751cf23368) made the separator between identifier and value optional, which results in more false positives. I think this change makes the behavior more reasonable. Alternatively, we can go back to requiring an assignment operator.

Uses FxHash rather than Rust's default hasher, in an attempt to improve performance.

My tests have been a bit inconclusive, so would appreciate it if others compared these changes to the main branch in whatever way they think best. FxHash might even be a touch slower in some cases? I'm not sure.

There are definitely other hashers that may be worth exploring, like ahash. Maybe alternatives to try can be discussed in the comments here.

Long secrets are not ignored.

See the example below. The first two secrets are ignored, while the last two are not.

.secretsignore

[secrets]
96etKOmnte-bpLDSIcwdhXYlC82gF8x-ERPqZ7oo1Ug
8AOiCMgwF1eg5yLDgw9D1eymTSOp21PJwr4zdQRQyYQ
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eODlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzc3LCJpYXQiOjE2NTExNDA0ODAsImp0aSI6ImQzAAJmYzBiNzI2NDRjMjY5ODI0NGFiMTQ2OTc1N2YyIiwidXNlcl9pZCI6MX0.87aml-57DmEUo4LrlZwnDw4iVfiWVNA90xxCi01M2h0
eyJ0eXAiOiJKV1QiCCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eWPlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzgxLCJpYXQiOjE2NTExNDEwODEsImp0aSI6Ijk1YjRjMDA2ODZjNTRkYTU4OTE1NWYzOTgzZjcxNmJiIiwidXNlcl9pZCI6MX0.zs-3zv1eCSu9JeRBJgFw6CBoZUA4B2R3z6gl2vNYwdA

Easily reproducible like this:

ripsecrets .secretsignore

Like this https://bind9.readthedocs.io/en/v9_16_5/advanced.html#loading-a-new-key See also discussion in https://github.com/sirwart/ripsecrets/pull/35

Hi!

Love the tool! The performance gains are awesome! Curious if you had any plans to make adding additional secret patterns to the tool more friendly (Maybe an external TOML file with user specified patterns?). If you could forsee this being part of the tool, would love to pull together a PR if we can agree on an implementation.

Otherwise - would be happy to PR some additional secret patterns!