Sha 256 support

[Byron]

Even though the foundation is set, it needs another push to actually make it work with different kinds of hashes.

Tasks

• remove hash-type specific methods from git-hash and replace them with parametric usage of git_hash::Kind
• all code assuming hashes of len 20 should receive this value as parameter instead. This is what git does for the old index and pack file formats.
• a way to pass --object-hash information to the gix CLI
• Remove default sha1 feature from gix-hash crate and deal with the fallout
  • Add forwarding features to every plumbing crate that depends on it directly or indirectly
  • adjust tests to set sha1 by default
  • configure docs.rs to enable sha1
  • CI should execute script that runs cargo check -p gix-<name> to show it's failing with an error message
  • adjust all other jobs
  • let gix chose SHA1 as default
• remove SHA1 mention from git-features feature toggles
• parameterize hash len when decoding non-blob objects (see this for an example)
• understand and implement pack idx V3. - see if git actually implements this, and maybe decide that gitoxide won't handle the transition period, is either one has or another.
• add new Sha256 enum variant, consider putting it behind a feature flag, and add a hasher for it as well.
• general tests for reading refs and objects of different len
• tests for writing and reading objects of different len, maybe even write a conversion program which transforms an entire repo and double-checks with git-fsck
• when cloning, check the uninmplemented!() invocation to configure the repo for expecting a different hash

Crates Completed

• gix-commitgraph - minor testing, run tests with multiple hash kinds

Tasks

(generated by Codex, needs individual review and adjustment)

Based on issue #281 and current repo state, here is an effort-ordered crate list.

• gix-hash (small, foundational): remove default sha1, keep explicit sha1/sha256 features, and keep no-default build/tests green; watch out for Cargo feature additivity and clear compile errors when no hash is selected.
• gix-refspec (small): add explicit SHA256 parse tests for object-hash-looking refspecs; watch out for logic that implicitly treats 40-hex as “an object hash”.
• gix-filter + gix-worktree-stream (small to medium): add explicit dual-hash tests for filter/stream paths; watch out for protocol/hash-kind encoding paths that currently default to SHA1 semantics.
• gix-blame + gix-diff (medium): remove hardcoded Kind::Sha1.null() sentinels where possible and add SHA256 fixtures; watch out for sentinel IDs that must stay “impossible/non-matching” rather than repo-real IDs.
• gix-object (medium): parameterize non-blob/tree decode by hash length and add SHA256 object decode/encode tests; watch out for the tree parser hotspot that still splits at 20 bytes (TODO(SHA256)).
• gix-ref (medium): add explicit refs/reflog read-write tests for both hash lengths; watch out for many 40-char and SHA1-null expectations in tests.
• gix-index (medium-high): extend index/extension roundtrip tests for SHA256; watch out for offset/checksum math tied to len_in_bytes() and trailing checksum region assumptions.
• gix-protocol + gix-transport (medium-high): add object-format=sha256 negotiation tests end-to-end; watch out for code paths that currently accept/fallback only to SHA1.
• gix-odb (high): add explicit loose/packed read-write and prefix lookup tests under SHA256; watch out for SHA1-shaped helpers (e.g., 20-byte helpers and 40-hex prefix assumptions).
• gix-pack (very high): full SHA256 coverage for pack data/index/multi-index read-write and verification; watch out for SHA1-only assertions and format/version assumptions in pack/index code.
• gix (very high): implement and test clone/fetch repo reconfiguration when remote hash kind differs; watch out for the current clone unimplemented!() on object-hash mismatch.
• double-check for TODO(SHA256) or similar, and for gix_hash::Kind::Sha1::null() and similar

Implementation ideas

• make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

• One way to do that with approximately zero overhead would be to such functions generic on the object ID, using a trait that has a method to get the type. Then object IDs with a known type return a constant from that method, and object IDs with a runtime dispatched type return the value of that enum.

  • Josh Triplett (@joshtriplett) - taken verbatim as I'd barely be able to improve on it when paraphrasing. In short, have a trait for oid or allow efficient conversions to oid (it's just a slice, so that should work for specifically sized types as well especially if these were provided by git-hash.

Notes

• find ways to use the existing highly-parallel pack traversal (along with integration of loose-objects) to build an inverse-ref table to quickly traverse objects bottom-up to change the hash used along with all references, while being fast. This ties into being able to build new packs quickly, ideally even with delta-compression (the latter then has to be re-created as most objects actually change) - re-using deltas for blobs is the only way.
  • The existing traversal can mutate data in the tree, which is enough to decode the object and keep direct references for later.

Related

• Make sha1 optional in gix-hash #2332
• feat: add sha256 feature to gix-object #2359
• Add GIX_TEST_FIXTURE_HASH to gix-testtools #2377

[shea256]

+1 to this feature. Excited to see this implemented.

[msrd0]

+1. Given that git no longer considers sha256 support experimental and forges such as forgejo added support, this has became more relevant for new git repositories.

[Sturdy7435]

+1. Without sha256 support Starship prompt cannot detect repositories using that algorithm.

[jokeyrhyme]

Sebastian Thiel (@Byron)

make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

should the old SHA1 support also be put behind a feature toggle (enabled by default), so that builds can choose to only support SHA256 ?

[Byron]

+1. Without sha256 support Starship prompt cannot detect repositories using that algorithm.

Starship would definitely be a great motivation to finally go through with this.

gix-hash has been created with that in mind:

gitoxide/gix-hash/src/object_id.rs

Lines 13 to 14 in ec69c88

	/// A SHA 1 hash digest
	Sha1([u8; SIZE_OF_SHA1_DIGEST]),

Right now it's an enum and one could add another variant to it, behind a feature toggle, as the size of these hash digests definitely affects the resource requirements of operations like clone. As the size of these hashes would increase, this would affect the memory consumption even when cloning SHA1 repositories (unfortunately).
Maybe there are better ways to represent this - I think Git simply allocates for it which would be one solution here as well.

In any case, if this were to move forward, I think it would be good to review the hash representation first, as it's very relevant for performance.

@Byron

make sure once Sha256 is added as ObjectId variant, that it's behind a feature toggle to allow builds that opt-out of SHA256 support to not unnecessarily use more memory than needed. Maybe there are alternatives to this, too.

should the old SHA1 support also be put behind a feature toggle (enabled by default), so that builds can choose to only support SHA256 ?

I see the point and gitoxide should allow to be tailored to what application developers need. However, it by default it will try to be compatible to Git which means SHA1 and SHA256 must be supported, which is certainly what this would start out as.
Once there is explicit demand, I think a feature toggle for turning off SHA1 support could be contributed. This will be some work though, as the test-suite directly refers to it in plenty of places.