Gix is a command-line interface (CLI) to access git repositories

When trying to beat git verify-pack

Attempt 1

I remembered timings on a cold cache that indicated something around 5:50min for git to run a verify pack on the linux kernel pack. However, turns out that if the environment is a little more controlled, git is still considerably faster than us despite using an LRU cache and despite using multiple cores quite efficiently.

Observation

Git uses a streaming pack approach which is optimized to apply objects inversely. It works by

• decompressing all deltas
• applying all deltas that depend on a base, recursively (and thus avoiding to have to decompress deltas multiple times)

We work using a memory mapped file which is optimized for random access, but won't be very fast for this kind of workload.

How to fix

Wait until we have implemented a streaming pack as well and try again, having the same algorithmical benefits possibly faired with more efficient memory handling. Git for some reason limits the application to 3 threads, even though we do benefit from having more threads so could be faster just because of this. The streaming (indexing) phase of reading a pack can be parallelised in case we have a pack on disk, and it should be easy to implement if the index datastructure itself is threadsafe (but might not be worth the complexity or memory overhead, let's see).

Motivation

git2 is actually quite easy to use, with a relatively high-level API that hides a lot of the plumbing that actually has to happen under the hood. Currently gitoxide has no such interface which requires a lot of knowledge to use and can be cumbersome at times.

This PR aims to provide a draft of a new Easy mode that provides a similarly simple API while hiding all the details. A major goal is to keep the lower level accessible for those who need it to eek out a few more percent of performance while offering a higher level that doesn't sacrifice too much performance for convenience, striking just the right balance.

We will use cargo smart-release as a driver and validator for the new API as it touches references, objects and trees right now.

Note that this PR was created 'late' in the development process as it's the first to try out a new workflow to help receiving comments on major features or changes.

Tasks

• [x] be cargo-release compatible when tagging versions
• experiment with ways of handling threaded and non-threaded high-level APIs
  • [x] convenient sharing of mutable state in threaded and unthreaded contexts
  • [x] a way to detach extra state to create transfer objects to pass thread boundaries, and to reattach it
  • [x] See if non-panicking could be an option for this API. Soon both make no difference in terms of backtraces
  • [x] Write down and possibly practice how caches are supposed to be handled. What happens with new packs being created by a gc, or with changing config files?
• [x] streamline ergonomics of finding and peeling refs
• [x] make cargo smart-release use the git_easy API exclusively
• [x] proof of concept that GATs are working - YES, see 7f56dbd82db2abc18b8e6d228c8a5f54b3dbf32a
• [x] make clear what can and can't be stored in the cache - it looks like packed::Buffer doesn't really belong there as it may contain a system resource in the form of a memory map. This means it should rather be shared. Finish the packed-refs needs update story, essentially.
  • To assure consistency, it's now behind an Arc<RwLock> even within the State structure that aims to be more like thread-local.
• ~~finally pull in git-config to see where it should live.~~
• [x] light documentation to get the main ideas across

Note that it should be reasonable stable, so probably dogfood it for a while, see #164 .

Out of Scope - separate PR

• finally find a good solution for immutable vs mutable naming (which doesn't always work nicely. Ref suffix is a not candidate for those with borrows inside. Candidates for massive changes are git-object and git-ref and git-actor. borrow() would then rather be to_ref()
• rename oid to id like in git2
• rename find_existing() to find() and provide the optional version under try_find(), both for objects and refs in plumbing crates

It'd be nice if static pages were supported because we want to rewrite our thing in Rust and for security reasons we recommend not running complex scripts and programs on the HTTP server, which means we recommend configuring all git repos with the so-called "dumb" protocol.

Our current thing currently just uses git directly and clones/fetches everything into a single cache repo, which means we can't parallelize it, so we're looking into alternatives for that, whether by using something like gitoxide or having multiple temporary caches and joining them together after fetching. However, having static pages support is a must.

These seem to be different errors though, maybe the shell on MacOS has trouble?

MacOS

2022-04-14T09:48:59.9196440Z stderr: /Users/runner/work/gitoxide/gitoxide/git-repository/tests/fixtures/make_rebase_i_repo.sh: line 33: [: 2 35 1: integer expression expected
2022-04-14T09:48:59.9196950Z fatal: --preserve-merges was replaced by --rebase-merges
2022-04-14T09:48:59.9197450Z ', tests/tools/src/lib.rs:115:9
2022-04-14T09:48:59.9197760Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2022-04-14T09:48:59.9197950Z 
2022-04-14T09:48:59.9197960Z 
2022-04-14T09:48:59.9198020Z failures:

Windows

2022-04-14T09:40:43.0699609Z stderr: which: no gsed in (/mingw64/bin:/usr/bin:/c/Users/runneradmin/bin:/d/a/gitoxide/gitoxide/target/debug/build/curl-sys-7e85c957548a17f0/out/build:/d/a/gitoxide/gitoxide/target/debug/build/libgit2-sys-a405fc67d8aee456/out/build:/d/a/gitoxide/gitoxide/target/debug/build/libssh2-sys-0acf82cde4d4f0d1/out/build:/d/a/gitoxide/gitoxide/target/debug/build/libz-sys-a02af73321b6f02b/out/lib:/d/a/gitoxide/gitoxide/target/debug/build/wepoll-ffi-169df33785c3c0f8/out:/d/a/gitoxide/gitoxide/target/debug/deps:/d/a/gitoxide/gitoxide/target/debug:/c/Users/runneradmin/.rustup/toolchains/stable-x86_64-pc-windows-msvc/lib/rustlib/x86_64-pc-windows-msvc/lib:/c/Users/runneradmin/.cargo/bin:/c/Users/runneradmin/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin:/c/Program Files/MongoDB/Server/5.0/bin:/c/aliyun-cli:/c/vcpkg:/c/cf-cli:/c/Program Files (x86)/NSIS:/c/tools/zstd:/c/Program Files/Mercurial:/c/hostedtoolcache/windows/stack/2.7.5/x64:/c/cabal/bin:/c/ghcup/bin:/c/tools/ghc-9.2.2/bin:/c/Program Files/dotnet:/c/mysql/bin:/c/Program Files/R/R-4.1.3/bin/x64:/c/SeleniumWebDrivers/GeckoDriver:/c/Program Files (x86)/sbt/bin:/c/Program Files (x86)/GitHub CLI:/bin:/c/Program Files (x86)/pipx_bin:/c/hostedtoolcache/windows/go/1.17.8/x64/bin:/c/hostedtoolcache/windows/Python/3.7.9/x64/Scripts:/c/hostedtoolcache/windows/Python/3.7.9/x64:/c/hostedtoolcache/windows/Ruby/2.5.9/x64/bin:/c/tools/kotlinc/bin:/c/hostedtoolcache/windows/Java_Temurin-Hotspot_jdk/8.0.322-6/x64/bin:/c/npm/prefix:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/ProgramData/kind:/c/Program Files/Eclipse Foundation/jdk-8.0.302.8-hotspot/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/Chocolatey/bin:/c/Program Files/Docker:/c/Program Files/PowerShell/7:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/dotnet:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files (x86)/Microsoft SQL Server/110/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/120/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/130/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/140/DTS/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/nodejs:/c/Program Files/OpenSSL/bin:/c/Strawberry/c/bin:/c/Strawberry/perl/site/bin:/c/Strawberry/perl/bin:/c/ProgramData/chocolatey/lib/pulumi/tools/Pulumi/bin:/c/Program Files/TortoiseSVN/bin:/c/Program Files/CMake/bin:/c/ProgramData/chocolatey/lib/maven/apache-maven-3.8.5/bin:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/GitHub CLI:/c/tools/php:/c/Program Files (x86)/sbt/bin:/c/SeleniumWebDrivers/ChromeDriver:/c/SeleniumWebDrivers/EdgeDriver:/c/Program Files/Amazon/AWSCLIV2:/c/Program Files/Amazon/SessionManagerPlugin/bin:/c/Program Files/Amazon/AWSSAMCLI/bin:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files (x86)/Microsoft BizTalk Server:/c/Program Files/LLVM/bin:/c/Users/runneradmin/.dotnet/tools:/c/Users/runneradmin/.cargo/bin:/c/Users/runneradmin/AppData/Local/Microsoft/WindowsApps)
2022-04-14T09:40:43.0705473Z D:\a\gitoxide\gitoxide\git-repository\tests\fixtures\make_rebase_i_repo.sh: line 33: bc: command not found
2022-04-14T09:40:43.0706030Z fatal: --preserve-merges was replaced by --rebase-merges
2022-04-14T09:40:43.0706440Z ', tests\tools\src\lib.rs:115:9
2022-04-14T09:40:43.0706836Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Originally posted by @Byron in https://github.com/Byron/gitoxide/issues/382#issuecomment-1099008000

This is more of a heads-up:

git-protocol enables the symrefs capability for ls-refs by default. If the remote repo is actually a namespace, and contains refs which point to targets outside that namespace, upload-pack advertises symref-target:(null). For example:

Given:

refs/namespaces/A/refs/heads/main      # deafbeef
refs/namespaces/B/refs/heads/main-of-a # ref: refs/namespaces/A/refs/heads/main

ls-refs in the context of B gives:

upload-pack> deafbeef refs/heads/main-of-a symref-target:(null)

That makes sense, because namespaces are supposed to be opaque to clients, and so what refs exist outside the namespace should not be revealed.

However, when parsing into the Ref type, git-protocol takes the value of symref-target verbatim, so we get:

Ref::Symbolic {
    path: "refs/heads/main-of-a",
    target: "(null)",
    object: Sha1(deafbeef),
}

Within gitoxide, this is safe, because the target is validated when constructing a RefEdit (if we were to mirror the remote refs tree after a fetch). It may still be more saferer if target was of type Option<BString>, and "(null)" parsed as None, for example when a library user is handing that value out to other tools without validating it first.

git includeIf

TODO

• [ ] Both the symlink & realpath versions of paths will be matched outside of $GIT_DIR. E.g. if ~/git is a symlink to /mnt/storage/git, both gitdir:~/git and gitdir:/mnt/storage/git will match.

Tracked in #562

Tasks

• [x] roundtrip tests raw byte comparison is failing because of an order mismatch in the tree extension and that should be investigated / fixed for now though this can be can be worked around by
  • [x] comparing the in memory states
  • ~comparing only the relevant portions of bytes instead of all~ all bytes are relevant
  • ~creating the baseline index without tree extension~ not sure that is a possibility with git
• [x] check git-config docs for options that influence sparse index writing
  • [x] update gix progress to reflect those findings
• [x] ~sparse index can be written by providing the corresponding extension flag in the write::Options~
• [x] ~index decides automatically if it needs to be sparse or not based on what kind of options are being passed in~
• [x] writes sparse / regular index depending on if DIR entries exist in State or not
  • is_sparse flag tracks / caches this for us do we trust other functions to modify this flag correctly or do we need a way of verifying this, a.k.a. checking the DIR entries our selves

Notes Just for myself to remember and to hammer the point home: write_to just looks at the current State and writes whatever it finds to out. There are some optional extensions that can be configured directly, but everything else (like git-config stuff) it does not care about. This is handled entirely by different functions that mutate the State to the desired state before writing it out.

given a gitconfig with the [url xxxx] sections as below.

I need to iterate over all url sections and be able to retrieve the insteadOf config key (which works already) and the xxxx that is the subsection name (see the todo note)

example

    use git_config::file::GitConfig;
    use git_config::parser::{Key, Parser};
    use git_config::values::Value;
    use std::fs::read_to_string;

    let input = r#"
[url "ssh://[email protected]/"]
    insteadOf = https://github.com/
[url "ssh://[email protected]"]
    insteadOf = https://bitbucket.org/
"#;
    let config = GitConfig::try_from(input).unwrap();
    let url = config.se.sections_by_name("url");
    assert_eq!(url.len(), 2);

    for (i, u) in url.iter().enumerate() {
        let instead_of = u.value(&Key::from("insteadOf"));

        // todo(sub-section-name)
        // let xxxxx = u.retrieve_parent_sub_section_name();
        // assert_eq!(xxxxx, "ssh://[email protected]/")

        if i == 1 {
            assert_eq!(
                instead_of.unwrap().deref(),
                "https://github.com/".as_bytes()
            )
        } else {
            assert_eq!(
                instead_of.unwrap().as_ref(),
                "https://bitbucket.org/".as_bytes()
            )
        }
    }

Maybe there is also a better way of doing this, then I would be very pleased if you can give me a hint.

@avoidscorn @edward-shen What do you think about it? Is anything missing, should something be changed?

Please feel free to comment here or use the 'suggestion' feature of github reviews.

Apologies if the guide sounds a bit robotic or unfriendly but I feel pressed for time and rushed it a bit. However, I am happy to spend some time to make collaborating on the project more straightforward and would love changes to the guide that make it feel more approachable/polished :D.

Thanks

Hi! I'm a developer that's been tasked to implement GitOxide into a WASM stack in an effort to clone a project as fast as possible in the browser.

As part of that work, I'm looking to introduce a lot of new functionality into the project to get clone working. As an initial task, I'm hoping to get the git-config package able to read and write config files. I've got a (very) WIP port of config parsing logic from isomorphic-git over on my fork.

I won't bother asking for a code review right now, I'm brand new to Rust and have a colleague taking a look at the code right now to give me some pointers on how I can improve my work

That said, I am unsure of what direction we want to go in with regards to data organization. The current output is a pretty unmanageable tangled vector of Sections and Entries alike in a single struct

I've noticed that the current git-config file has both Section and Entry types that look to contain all of the relevant properties, so I'll be refactoring my current code to use that data structure.

Outside of that, I was hoping you could give me some kind of direction when it comes to the different mods in the git-config/lib.rs file:

https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L57-L72

https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L131-L217

I'm not entirely sure what borrowed or spawned is referring to (which I admit as a point-of-noobiness), and I'm not sure I track what Span is doing here.

I also would love some insight as to what a Token is referring to here:

https://github.com/Byron/gitoxide/blob/main/git-config/src/file.rs#L4-L23

As I'm not sure an instance that would have a Section and only a single Entry

Looking forward to learning more and being able to meaningfully contribute upstream!

Can Byron review the PR on video?

Please choose one - no choice means no recordings are made.

• [x] Record review and upload on YouTube publicly
• [ ] Record review and upload on YouTube, but keep video unlisted

If you ticked any of the above boxes, I will always share a link to the video in the PR.

Summary 💡

I'm trying to use various methods in git_repository. I usually use thiserror to build an Error type for my crate using #[from] to be able to losslessly wrap every error my crate encounters, this is a pretty common idiom so I'm confident you've encountered it. When another crate I use uses the same pattern, this is pretty easy, I just extend my Error enum with the 1 new case for that crate:

#[derive(thiserror::Error, Debug)]
enum Error {
    // ... existing variants
    NewCrateError(#[from] NewCrate::Error)
}

And then I can just stick ? on any fallible operation and the conversion into my crate's Error type magically happens. I'm running into 2 problems trying to use git-repository this way:

1. There is not 1 error type for the crate with N variants (1 variant per error), but instead N totally separate error types. This means I have to add a ton of variants to my enum.
2. Some of the error types in git-repository are generic, e.g. git_repository::git_odb::find::existing::Error<git_repository::git_odb::store::find::Error> so there's not even a finite number of them, further exploding the number of variants I need to add.

I wouldn't say git-repository is doing anything "wrong", which is why this is a feature request. I don't know if there's a good motivation for doing errors this way, it just makes it more difficult to apply the idiom that I've learned for lossless error propagation.

If I could write one impl that would let me box any error coming from git-repository that would also work nicely, but as is now I think the only trait they have in common is the std::error::Error trait, so AFAIK there is no good way to only do this for git-repository errors automatically.

Motivation 🔦

Lossless error handling, generally avoiding boxing where I can. I could switch to something like anyhow, but I'm writing a library crate so I prefer to keep user's options open.

Required features

What follows is the result of a real quick analysis

• [x] init repo
• [ ] git status
• [x] git fetch
• [ ] git add .
• [ ] manipulate the staging are (i.e. unstage)
• [ ] reset (soft) - needs higher level functionality
• [ ] git commit
• [x] git commit --amend (by hand)
• [ ] git checkout
• [x] list branches
• [x] get HEAD and name of unborn branch

Currently, only the split index with the link extension is read, but the shared index is not, which means the in-memory representation is different when comparing a split index with a regular index.

Where to best place the resolve_link_extension function?

• should this only happen when we try to write a link extension or just happen by default everytime we load an index?
• does this always require a git_index::File?

Tasks

• [x] test comparing a split index with an identical "regular" index (in memory)
• [x] read and merge the shared index into the in-memory representation and discard the link extension
• [ ] ~~I am removing entries but not the corresponding path names in the path_backing~~
• [ ] improve error handling - make sure to have distinct paths for the possible outcomes and dedicated errors with helpful messages (double check with git)
  • [x] bound check for split & shared index entries
  • [ ] entry marked for both replace & remove - not sure this applies to us at the moment
  • [x] split entry path is not empty
• [ ] test fixture with overlap between shared and base index (currently we only have a test thas has no overlap between the two files)

Notes

• git does not seem to delete / update the entries immediately, but rather uses the CE_REMOVE and CE_UPDATE_IN_BASE flags

Required Features

(among others that are already implemented)

• [x] a fast way to get the size and kind of an object by ID, by reading only headers, similar to read_header().
• [x] Something like NO_DOT_GIT option to prevent trying two directories if one is given. (#667)
• ~~a filter to only traverse packed objects if they match a given filter to allow blob-only traversal. (for max performance)~~
• [ ] Generalize git_odb::Store::verify_integrity() so that users can be called on each decoded object to perform their own computation.

The first integration PR in the repository and the follow-up.

What is a split index?

When using a split index the index is split up into two separate files:

• the split index at $GIT_DIR/index
• the shared index at $GIT_DIR/sharedindex.<SHA-1>

The shared index contains all entries while the split index contains and accumulates changes. These changes in the split index are occasionally written into the shared index, either "automatically" based on config settings, or by the git-update-index command. "Automatically" here means that this runs every time the index is being read or updated.

Link Extension

The link extension stores 2 bitmaps that record how the stored changes in the split index should be merged with the shared index.

The replace bitmap stores which entries in the shared index should be replaced by entries stored in the split index, i.e. [0, 1, 1, 0, 1] -> replace shared index entries at index 1, 2, 4 with split index entries at 0, 1, 2 Any additional entries in the split index should be added to the shared index.

The delete bitmap stores which entries in the shared index should be deleted, i.e. [1, 0, 0, 1] -> delete entries at index 0 and 3

Tasks

• [x] check git source code to learn more about how split index works and update this issue
• [ ] #655
• [ ] writing
  • [ ] write only the split index without ever merging / updating the shared index
    • [ ] if this turns out to be too much work just write a regular index and discard the split / shared index files
  • [ ] merge split and shared indexes (based on maxPercentChange) - source code reference
• [ ] update active shared index modification time every time the split index is being read / updated to prevent automatic deletion

Config Settings

• core.splitIndex enables the use of a split index
• splitIndex.maxPercentChange The percentage threshold of entries in the split index (compared to the shared index) that triggers a write to the shared index. It defaults to 20.
• splitIndex.sharedIndexExpire If the modification time of a shared index is older than this value it will be deleted. Takes values like "now", "never", "2.weeks.ago" (default). This needs to be parsed.

Notes

• split index and sparse index are incompatible in git
• git | date.c is used to parse dates

Questions

• Does the split index get emptied / recreated from scratch when writing its changes to the shared index?
• I assume the actively used shared index can and should never be deleted, but based only on the sharedIndexExpire setting it could be

References

• git-update-index | split index
• git-index | split index / link extension