There was a discussion in today's GitBOM meeting about the utility of separating generation of gitoids from the generation of a GitBOM DAG. @edwarnicke has implemented this split in Go (https://github.com/edwarnicke/gitoid) (WIP) and described it as being a valuable change. The idea is that by splitting this out, other uses of gitoids can be explored.

EDIT: Also, the gitoid crate name is available. :smile:

Created a const for the size of the hash byte array. If we add a new hash algorithm that has a hash size greater than 32 bytes, we can update the const to support the larger value.

Also, added async methods for generating a bunch of GitOids

Signed-off-by: David Pollak [email protected]

This is an initial effort at splitting the gitbom crate into two crates: gitbom, which provides the overall BOM structure, and gitoid, which only handles the construction of gitoids. This PR also includes some changes to the GitOid API as it existed in the singular crate to make the API more consistent between sync and async, including removal of the Source type and delegation of any mass construction of GitOids to users of the API, rather than baking the iteration into the API itself.

This also includes some cleanup / consistency improvements to the test code, as well as changing the methods and constants for HashAlgorithm and NUM_HASH_BYTES to be pub(crate) instead of pub.

The individual commit messages try to explain more concretely.

This commit adds error handling for all existing functions which may fail. We still don't handle panics, so that will need to be fixed, but otherwise we're all good.

It also fixes the test code to be easier to grow in the future and to have clearer output.

Signed-off-by: Andrew Lilley Brinker [email protected]

Hello all!

I have been experimenting with our async API methods over on gitbom-rs-cli.

Most recent experiment: https://github.com/nellshamrell/gitbom-rs-cli/pull/21 Previous experiment: https://github.com/nellshamrell/gitbom-rs-cli/pull/18

As noted in the comments on the previous experiment, I'm not seeing a performance gain when using the async methods (in fact, there appears to be a performance penalty, though it is of course possible I'm blocking on something unintentionally). GitBOM is heavily dependent on many file reads and file writes and, as the tokio documentation states:

"Although it seems like Tokio would be useful for projects that simply need to read a lot of files, Tokio provides no advantage here compared to an ordinary threadpool. This is because operating systems generally do not provide asynchronous file APIs."

It's possible we could make use of io_uring, there has been some experimental work done on this (including with tokio) but support for it is still not wide-spread. This would also only benefit us on Linux, not on Windows. It is possible there is some sort of library that would support async file operations on Windows, but then we would need to maintain two separate sets of functionality - one for Linux and one for Windows. This is not impossible, but it adds to complexity and increases the maintainership burden more than it needs to at this time (imo).

The work that was done to add the async methods was tremendously valuable - I know I learned a lot and I believe other maintainers did as well. However, I propose removing the async API calls at least for now, and leaving it the project which uses the library to decide how to implement performant use of it through threads, etc. We can always add this functionality back in should it become necessary.

This is intended to make the API clearer in separating the hash, which is part of the GitOid but not sufficient for representing the GitOid, from the overall GitOid structure. The GitOid can be represented as a URI, or you can extract the parts (hash algorithm, object type, hash) individually. If you have the Hash, now represented as a struct, you can either access the individual bytes through the Deref impl or as_bytes method, or you can get the hexadecimal encoding through the Display impl or the implied to_string method or explicit hex method.

Signed-off-by: Andrew Lilley Brinker [email protected]

These commits do a few things.

1. Bump the dependency versions to their latest.
2. Introduce an ObjectType enum and make it a parameter of all the GitOid constructors.
3. Rename the HashAlgorithm variants to Sha1 and Sha256, so the capitalization matches regular Rust style.
4. Introduce a GitOidBuilder type to make constructing many GitOids with the same configuration easier.
5. Modify the trait bound on GitOid::new_from_reader from BufReader<R>: Read to R: Read.

On the second item: for the gitbom crate I believe we only care about the ObjectType::Blob variant, but the gitoid crate might have other users who want to handle other types of Git objects.

On the builder, this is the simplest-possible builder at the moment; the configuration is only set in the constructor of the builder, and can't be changed after that. It may be worth adding setters and getters for the configuration, but that hasn't been done here.

This attempts to add in a c compatible gitoid::new_from_str function.

However, when I run cbindgen -v --config cbindgen.toml --crate gitoid --output gitoid.h, I'm still not seeing the function in gitoid.h.

Update: FIXED!

Signed-off-by: Nell Shamrell [email protected]

The Go API allows constructing a GitOid from a URL by parsing the URL and ensuring it fits the required format. This could be done in the Rust version by introducing a TryFrom<url::Url> implementation, likely along with a new_from_url constructor to provide uniformity with the other constructors.

It would be nice if the Tokio integration / async stuff were an optional (but default-on) feature. For folks who are sensitive about build times and/or size of the dependency tree and don't need the async features.

This commit does a few different things:

• Fixes clippy lints, mostly by actually fixing them, although a couple are "fixed" with #[allow(clippy::...)] annotations so that we don't see the warnings anymore.
• Adds or clarifies existing comments both to better document the parts of the API, and to explain the functioning of the code better in places.
• Renames some variables for clarity. In particular, any digest local variable is renamed digester, and Source::get_digest is now Source::get_digester. This is intended to make clearer that the function returns the type which performs the digest operations, and does not return the digest itself.
• Removes a couple of unnecessary allocations. Namely, 1) the logic for producing a bunch of futures for GitOid generation and then using join_all on them doesn't actually require allocating an intermediate Vec for the futures, join_all just takes an iterator; 2) GitBom::add doesn't need to make a Vec of one element to pass to add_many, it can use a regular array. This does raise a Minimum-Supported Rust Version question, as arrays implementing IntoIter was introduced a few versions ago. I can revert this change if we aren't comfortable with the compatibility issue.

EDIT: This also modifies the new_from_str function to match new_from_reader by taking a hash_algorithm parameter.

This commits adds a new Finder struct which matches different sources of GitOids against a known set of GitOids, returning both an identifier and the matching GitOid itself, along with reporting any errors that arise when resolving the new GitOids.

The docs for the Finder struct spell it out the clearest. This design is centered around the new IntoIdentifiedGitOid trait, which is implemented for a variety of existing types. The goal of this is to allow the Finder to be generic over things which produce GitOids, so that, for example, someone could add a mechanism to resolve GitOids from a Url without changing any existing code.

Signed-off-by: Andrew Lilley Brinker [email protected]

We'd like for the gitoid API (to start with) to be exposed over FFI so that other language implementations can build off of the existing Rust logic rather than building anew.

I've used cbindgen in the past and am inclined to try using it again, but I'm also open to other possibilities!

Right now the GitOid type can't be serialized directly, but has to first be converted into a string, and the reverse is true for deserialization. It would be better to directly implement serde::Serialize and serde::Deserialize on GitOid.

(Split off from #23)

The following is a list of capabilities which ought to be included in a minimal file-matching API.

• [ ] Match against the contents of a reader
• [ ] Find a first match in a set of files
• [ ] Find all matches in a set of files
• [ ] Find n matches in a set of files

Looking at the current API, I believe most of this crate could be made no-std compatible. The only parts used currently which require std are the std::io bits, so the question would be what API should/could be presented in no-std context.

It looks like moving the Read and Write traits into core is an eventual goal, currently blocked on the fact that std::io::Error involves boxing (and thus need alloc). The function on the Read and Write traits specify std::io::Error as the return type, so they can't move unless/until std::io::Error changes.

I'm doing some investigating on how people have dealt with this IO limitation in other contexts.