Skip to contentSkip to conversation

Documentation/user-manual.txt: try to clarify on object hashes

PATCH Superseded Dirk Gouders 2 participants 10 messages Feb 29, 2024 – Mar 12, 2024 Documentation

lore
Share & tools
lore
#0 lore raw

cover Documentation/user-manual.txt: try to clarify on object hashes

I'm not sure if such patches are welcome -- they could cause more work than it's worth it. On the other hand, the contradiction irritaded me and I had to dig a bit and check it by example and writing this down could perhaps help others who also stumble over this.

If someone knows the exact reasoning why the hashing changed from _after_ compressed to _before_ compressed, we could perhaps add that, too?

Dirk Gouders (1):
Documentation/user-manual.txt: example for generating object hashes

 Documentation/user-manual.txt | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/1 Documentation/user-manual.txt: example for generating object hashes

If someone spends the time to work through the documentation, the subject "hashes" can lead to contradictions:

The README of the initial commit states hashes are generated from compressed data (which changed very soon), whereas Documentation/user-manual.txt says they are generated from original data.

Don't give doubts a chance: clarify this and present a simple example on how object hashes can be generated manually.

---
 Documentation/user-manual.txt | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
Documentation/user-manual.txt +33 -0
index 6433903491..8dfb81e045 100644
@@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data
 plus this header, so `sha1sum` 'file' does not match the object name
 for 'file'.
 
+Starting with the initial commit, hashing was done on the compressed
+data and the file README of that commit explicitely states this:
+
+"The SHA1 hash is always the hash of the _compressed_ object, not the
+original one."
+
+This changed soon after that with commit
+d98b46f8d9a3 (Do SHA1 hash _before_ compression.).  Unfortunately, the
+commit message doesn't provide the detailed reasoning.
+
+The following is a short example that demonstrates how hashes can be
+generated manually:
+
+Let's asume a small text file with the content "Hello git.\n"
+-------------------------------------------------
+$ cat > hello.txt <<EOF
+Hello git.
+EOF
+-------------------------------------------------
+
+We can now manually generate the hash `git` would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is
+  11 bytes.
+
+- Prepend the object header to the file content and feed this to
+  sha1sum(1):
+
+-------------------------------------------------
+$ printf "blob 11\0" | cat - hello.txt | sha1sum
+7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 .
+-------------------------------------------------
+
 As a result, the general consistency of an object can always be tested
 independently of the contents or the type of the object: all objects can
 be validated by verifying that (a) their hashes match the content of the
Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

Quoted 10 lines · Dirk Gouders

If someone spends the time to work through the documentation, the subject "hashes" can lead to contradictions:

The README of the initial commit states hashes are generated from compressed data (which changed very soon), whereas Documentation/user-manual.txt says they are generated from original data.

Don't give doubts a chance: clarify this and present a simple example on how object hashes can be generated manually.

I'd rather not to waste readers' attention to historical wart.

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4096

Quoted 3 lines
Documentation/user-manual.txt +0 -0
@@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data
 plus this header, so `sha1sum` 'file' does not match the object name
 for 'file'.

The paragraph above (part of it is hidden before the hunk) clearly states what the naming rules are. We hash the original and then compress. If I use an implementation of Git that drives the zlib at compression level 1, and if you clone from my repository with another implementation of Git whose zlib is driven at compression level 9, our .git/objects/01/2345...90 files may not be identical, but when uncompressed they should store the same contents, so "hash then compress" is the only sensible choice that is not affected by the compression to give stable names to objects.

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4106

Quoted 9 lines
Documentation/user-manual.txt +9 -0
+Starting with the initial commit, hashing was done on the compressed
+data and the file README of that commit explicitely states this:
+
+"The SHA1 hash is always the hash of the _compressed_ object, not the
+original one."
+
+This changed soon after that with commit
+d98b46f8d9a3 (Do SHA1 hash _before_ compression.).  Unfortunately, the
+commit message doesn't provide the detailed reasoning.

These three are about Git development history, which by itself may be of interest for some people, but the main target audience of the user-manual is probably different from them. They may be interested to learn how Git works, but it is only to feel that they understand how the "magic" things Git does, like "a cryptographic hash of contents is enough to uniquely identify the contents being tracked", works well to trust their precious contents [*].

Side note:
https://lore.kernel.org/git/[email protected]/
explains the reason behind the change to those who did not find it obvious.

FYI, another "breaking" change we did earlier in the history of the project was to update the sort order of paths in tree objects. We do not need to confuse readers by talking about the original and updated sort order. The only thing they need, when they want to get the feeling that they understand how things work, is the description of how things work in the version of Git they have ready access to. Historical mistakes we made, corrections we made and why, are certainly of interest but not for the target audience of this document.

On the other hand, ...

Quoted 13 lines
Documentation/user-manual.txt +13 -0
+The following is a short example that demonstrates how hashes can be
+generated manually:
+
+Let's asume a small text file with the content "Hello git.\n"
+-------------------------------------------------
+$ cat > hello.txt <<EOF
+Hello git.
+EOF
+-------------------------------------------------
+
+We can now manually generate the hash `git` would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4130

Quoted 10 lines
Documentation/user-manual.txt +10 -0
+  11 bytes.
+
+- Prepend the object header to the file content and feed this to
+  sha1sum(1):
+
+-------------------------------------------------
+$ printf "blob 11\0" | cat - hello.txt | sha1sum
+7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 .
+-------------------------------------------------
+

... something like the above (modulo coding style) would be a useful addition to help those who want to convince themselves they understand how (some parts of) Git works under the hood, and I think it would be a welcome addition to some subset of such readers (the rest of the world may feel it is way too much detail, though).

I would draw the line between this one and a similar description and demonstration of historical mistakes, which is not as relevant as how things work in the current system. In other words, to me, it is OK to dig a bit deep to show how the current scheme works but it is way too much to do the same for versions of the system that do not exist anymore.

But others may draw the line differently and consider even the above a bit too much detail, which is a position I would also accept.

Thanks.

#3 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

Quoted 10 lines · Junio C Hamano
Quoted 8 lines · Dirk Gouders

If someone spends the time to work through the documentation, the subject "hashes" can lead to contradictions:

The README of the initial commit states hashes are generated from compressed data (which changed very soon), whereas Documentation/user-manual.txt says they are generated from original data.

Quoted 4 lines
Quoted 2 lines

Don't give doubts a chance: clarify this and present a simple example on how object hashes can be generated manually.

I'd rather not to waste readers' attention to historical wart.

Yes, but -- I should have mentioned it -- the document itself suggests to read the initial commit.

But I don't mean to argue about that, perhaps I digged to deep into details.

Quoted 5 lines

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4096

Quoted 3 lines
Documentation/user-manual.txt +0 -0
@@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data
 plus this header, so `sha1sum` 'file' does not match the object name
 for 'file'.

The paragraph above (part of it is hidden before the hunk) clearly

Quoted 8 lines

states what the naming rules are. We hash the original and then compress. If I use an implementation of Git that drives the zlib at compression level 1, and if you clone from my repository with another implementation of Git whose zlib is driven at compression level 9, our .git/objects/01/2345...90 files may not be identical, but when uncompressed they should store the same contents, so "hash then compress" is the only sensible choice that is not affected by the compression to give stable names to objects.

Thank your for that detail.

Quoted 23 lines

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4106

Quoted 9 lines
Documentation/user-manual.txt +9 -0
+Starting with the initial commit, hashing was done on the compressed
+data and the file README of that commit explicitely states this:
+
+"The SHA1 hash is always the hash of the _compressed_ object, not the
+original one."
+
+This changed soon after that with commit
+d98b46f8d9a3 (Do SHA1 hash _before_ compression.).  Unfortunately, the
+commit message doesn't provide the detailed reasoning.

These three are about Git development history, which by itself may be of interest for some people, but the main target audience of the user-manual is probably different from them. They may be interested to learn how Git works, but it is only to feel that they understand how the "magic" things Git does, like "a cryptographic hash of contents is enough to uniquely identify the contents being tracked", works well to trust their precious contents [*].

Side note:
https://lore.kernel.org/git/[email protected]/
explains the reason behind the change to those who did not find it obvious.

Quoted 9 lines

FYI, another "breaking" change we did earlier in the history of the project was to update the sort order of paths in tree objects. We do not need to confuse readers by talking about the original and updated sort order. The only thing they need, when they want to get the feeling that they understand how things work, is the description of how things work in the version of Git they have ready access to. Historical mistakes we made, corrections we made and why, are certainly of interest but not for the target audience of this document.

Again thank you, very interesting reading.

Quoted 43 lines

On the other hand, ...

Quoted 13 lines
Documentation/user-manual.txt +13 -0
+The following is a short example that demonstrates how hashes can be
+generated manually:
+
+Let's asume a small text file with the content "Hello git.\n"
+-------------------------------------------------
+$ cat > hello.txt <<EOF
+Hello git.
+EOF
+-------------------------------------------------
+
+We can now manually generate the hash `git` would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is

Documentation/user-manual.txt that is used to name the object is the hash of the original data · line 4130

Quoted 10 lines
Documentation/user-manual.txt +10 -0
+  11 bytes.
+
+- Prepend the object header to the file content and feed this to
+  sha1sum(1):
+
+-------------------------------------------------
+$ printf "blob 11\0" | cat - hello.txt | sha1sum
+7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 .
+-------------------------------------------------
+

... something like the above (modulo coding style) would be a useful addition to help those who want to convince themselves they understand how (some parts of) Git works under the hood, and I think it would be a welcome addition to some subset of such readers (the rest of the world may feel it is way too much detail, though).

I would draw the line between this one and a similar description and demonstration of historical mistakes, which is not as relevant as how things work in the current system. In other words, to me, it is OK to dig a bit deep to show how the current scheme works but it is way too much to do the same for versions of the system that do not exist anymore.

But others may draw the line differently and consider even the above a bit too much detail, which is a position I would also accept.

Thanks.

#4 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

Quoted 4 lines · Dirk Gouders
Quoted 1 line

I'd rather not to waste readers' attention to historical wart.

Yes, but -- I should have mentioned it -- the document itself suggests to read the initial commit.

Ahh, yes, we'd need to hedge that part. Good thinking.

I am still not sure if the first hunk below is a good idea or it is too much detail. The second hunk may be worth doing.

Thanks.

 Documentation/user-manual.txt | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Documentation/user-manual.txt +4 -2
index 6433903491..1027055784 100644
@@ -4093,7 +4093,8 @@ that not only specifies their type, but also provides size information
 about the data in the object.  It's worth noting that the SHA-1 hash
 that is used to name the object is the hash of the original data
 plus this header, so `sha1sum` 'file' does not match the object name
-for 'file'.
+for 'file' (the earliest versions of Git hashed slightly differently
+but the conclusion is still the same).
 
 As a result, the general consistency of an object can always be tested
 independently of the contents or the type of the object: all objects can
@@ -4123,7 +4124,8 @@ $ git switch --detach e83c5163
 ----------------------------------------------------
 
 The initial revision lays the foundation for almost everything Git has
-today, but is small enough to read in one sitting.
+today (even though details may differ in a few places), but is small
+enough to read in one sitting.
 
 Note that terminology has changed since that revision.  For example, the
 README in that revision uses the word "changeset" to describe what we
#5 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

In reply to Junio C Hamano
Quoted 23 lines · Junio C Hamano
Quoted 21 lines · Dirk Gouders
Documentation/user-manual.txt +21 -0
+The following is a short example that demonstrates how hashes can be
+generated manually:
+
+Let's asume a small text file with the content "Hello git.\n"
+-------------------------------------------------
+$ cat > hello.txt <<EOF
+Hello git.
+EOF
+-------------------------------------------------
+
+We can now manually generate the hash `git` would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is
+  11 bytes.
+
+- Prepend the object header to the file content and feed this to
+  sha1sum(1):
+
+-------------------------------------------------
+$ printf "blob 11\0" | cat - hello.txt | sha1sum
+7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 .
Quoted 8 lines
Quoted 2 lines

+-------------------------------------------------
+

... something like the above (modulo coding style) would be a useful addition to help those who want to convince themselves they understand how (some parts of) Git works under the hood, and I think it would be a welcome addition to some subset of such readers (the rest of the world may feel it is way too much detail, though).

May I ask what you meant by "modulo coding style", e.g. where I should look at to make the code of similar style?

I would also add that git-hash-object(1) could be used to verify the result if you think that is OK.

In addition to a suggestion in another mail, the commit would consist of substantial content you suggested and perhaps, you could tell me how to express this; would a Helped-By be correct?

Dirk

#6 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

Quoted 2 lines · Dirk Gouders

May I ask what you meant by "modulo coding style", e.g. where I should look at to make the code of similar style?

Documentation/CodingGuidelines would be a good start, but

* A here-doc for a single liner is probably an overkill. Why not

echo "Hello, world" >file

In either way, in our codebase a redirection operator '>' (or '<') has one whitespace before it, and no whitespace after it before the file.

* printf piped to "cat - file" whose output feeds another pipe looked unusual. More usual way novices write may be

{ printf ... ; cat file; } | sha1sum

were the two things I noticed.

Quoted 2 lines

I would also add that git-hash-object(1) could be used to verify the result if you think that is OK.

git hash-object can be used to replace that whole thing ;-)

Quoted 3 lines

In addition to a suggestion in another mail, the commit would consist of substantial content you suggested and perhaps, you could tell me how to express this; would a Helped-By be correct?

I think many may prefer to downcase B in By, but if it is "substantial", probably. I do not think I gave much in this discussion to become a substantial addition to the original, though.

#7 lore raw

Re: [PATCH 1/1] Documentation/user-manual.txt: example for generating object hashes

Quoted 27 lines · Junio C Hamano
Quoted 2 lines · Dirk Gouders

May I ask what you meant by "modulo coding style", e.g. where I should look at to make the code of similar style?

Documentation/CodingGuidelines would be a good start, but

* A here-doc for a single liner is probably an overkill. Why not

echo "Hello, world" >file

In either way, in our codebase a redirection operator '>' (or '<') has one whitespace before it, and no whitespace after it before the file.

* printf piped to "cat - file" whose output feeds another pipe looked unusual. More usual way novices write may be

{ printf ... ; cat file; } | sha1sum

were the two things I noticed.

Quoted 2 lines

I would also add that git-hash-object(1) could be used to verify the result if you think that is OK.

git hash-object can be used to replace that whole thing ;-)

Quoted 7 lines
Quoted 3 lines

In addition to a suggestion in another mail, the commit would consist of substantial content you suggested and perhaps, you could tell me how to express this; would a Helped-By be correct?

I think many may prefer to downcase B in By, but if it is "substantial", probably. I do not think I gave much in this discussion to become a substantial addition to the original, though.

Thank you for the explanation (some of which I should have found by myself).

I will send the prepared v2 when I solved my struggling with range-diffs; that concept is new to me and I have a slow brain -- if one at all.

Dirk

#8 lore raw

v2 cover Documentation/user-manual.txt: try to clarify on object hashes

In reply to Dirk Gouders

This is the second round of adding a hashing example to user-manual.txt.
---
Changes in v2:
- Do not go into detail about hashing in the history.
- Change code according to coding guidelines.
- Fix a typo (s/asume/assume/) and change the wording of that sentence.
- Write Git instead of git.
- To fit the whole document, change sample content to "Hello world", lentgh 12.
- Add verification of hash using git hash-object.
- Provide for empty lines around code blocks.
---
Dirk Gouders (1):
Documentation/user-manual.txt: example for generating object hashes

 Documentation/user-manual.txt | 36 +++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

Range-diff against v1:
1: 6995f866e7 ! 1: 568c59d69f Documentation/user-manual.txt: example for generating object hashes
@@ Metadata
## Commit message ##
Documentation/user-manual.txt: example for generating object hashes

- If someone spends the time to work through the documentation, the
- subject "hashes" can lead to contradictions:
+ Add a simple example on how object hashes can be generated manually.

- The README of the initial commit states hashes are generated from
- compressed data (which changed very soon), whereas
- Documentation/user-manual.txt says they are generated from original
- data.
-
- Don't give doubts a chance: clarify this and present a simple example
- on how object hashes can be generated manually.
+ Further, because the document suggests to have a look at the initial
+ commit, clarify that some details changed since that time.

## Documentation/user-manual.txt ##
-@@ Documentation/user-manual.txt: that is used to name the object is the hash of the original data +@@ Documentation/user-manual.txt: that not only specifies their type, but also provides size information
+ about the data in the object. It's worth noting that the SHA-1 hash
+ that is used to name the object is the hash of the original data
plus this header, so sha1sum 'file' does not match the object name
- for 'file'.
-
-+Starting with the initial commit, hashing was done on the compressed
-+data and the file README of that commit explicitely states this:
-+
-+"The SHA1 hash is always the hash of the _compressed_ object, not the
-+original one."
+-for 'file'.
++for 'file' (the earliest versions of Git hashed slightly differently
++but the conclusion is still the same).
+
-+This changed soon after that with commit
-+d98b46f8d9a3 (Do SHA1 hash _before_ compression.). Unfortunately, the
-+commit message doesn't provide the detailed reasoning.
++The following is a short example that demonstrates how these hashes
++can be generated manually:
+
-+The following is a short example that demonstrates how hashes can be
-+generated manually:
++Let's assume a small text file with some simple content:
+
-+Let's asume a small text file with the content "Hello git.\n"
+-------------------------------------------------
-+$ cat > hello.txt <<EOF
-+Hello git.
-+EOF
++$ echo "Hello world" >hello.txt
+-------------------------------------------------
+
-+We can now manually generate the hash git would use for this file:
++We can now manually generate the hash Git would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is
-+ 11 bytes.
++ 12 bytes.
+
+- Prepend the object header to the file content and feed this to
-+ sha1sum(1):
++ sha1sum:
+
+-------------------------------------------------
-+$ printf "blob 11\0" | cat - hello.txt | sha1sum -+7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 .
++$ { printf "blob 12\0"; cat hello.txt; } | sha1sum ++802992c4220de19a90767f3000a79a31b98d0df7 -
+-------------------------------------------------
+
++This manually constructed hash can be verified using git hash-object
++which of course hides the addition of the header:
++
++-------------------------------------------------
++$ git hash-object hello.txt
++802992c4220de19a90767f3000a79a31b98d0df7
++-------------------------------------------------
+
As a result, the general consistency of an object can always be tested
independently of the contents or the type of the object: all objects can
- be validated by verifying that (a) their hashes match the content of the
+@@ Documentation/user-manual.txt: $ git switch --detach e83c5163
+ ----------------------------------------------------
+
+ The initial revision lays the foundation for almost everything Git has
+-today, but is small enough to read in one sitting.
++today (even though details may differ in a few places), but is small
++enough to read in one sitting.
+
+ Note that terminology has changed since that revision. For example, the
+ README in that revision uses the word "changeset" to describe what we

Signature · 1 line
2.43.0
#9 lore raw

PATCH v2 1/1 Documentation/user-manual.txt: example for generating object hashes

Add a simple example on how object hashes can be generated manually.

Further, because the document suggests to have a look at the initial commit, clarify that some details changed since that time.

---
 Documentation/user-manual.txt | 36 +++++++++++++++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)
Documentation/user-manual.txt +34 -2
index 6433903491..90a4189358 100644
@@ -4093,7 +4093,38 @@ that not only specifies their type, but also provides size information
 about the data in the object.  It's worth noting that the SHA-1 hash
 that is used to name the object is the hash of the original data
 plus this header, so `sha1sum` 'file' does not match the object name
-for 'file'.
+for 'file' (the earliest versions of Git hashed slightly differently
+but the conclusion is still the same).
+
+The following is a short example that demonstrates how these hashes
+can be generated manually:
+
+Let's assume a small text file with some simple content:
+
+-------------------------------------------------
+$ echo "Hello world" >hello.txt
+-------------------------------------------------
+
+We can now manually generate the hash Git would use for this file:
+
+- The object we want the hash for is of type "blob" and its size is
+  12 bytes.
+
+- Prepend the object header to the file content and feed this to
+  `sha1sum`:
+
+-------------------------------------------------
+$ { printf "blob 12\0"; cat hello.txt; } | sha1sum
+802992c4220de19a90767f3000a79a31b98d0df7  -
+-------------------------------------------------
+
+This manually constructed hash can be verified using `git hash-object`
+which of course hides the addition of the header:
+
+-------------------------------------------------
+$ git hash-object hello.txt
+802992c4220de19a90767f3000a79a31b98d0df7
+-------------------------------------------------
 
 As a result, the general consistency of an object can always be tested
 independently of the contents or the type of the object: all objects can
@@ -4123,7 +4154,8 @@ $ git switch --detach e83c5163
 ----------------------------------------------------
 
 The initial revision lays the foundation for almost everything Git has
-today, but is small enough to read in one sitting.
+today (even though details may differ in a few places), but is small
+enough to read in one sitting.
 
 Note that terminology has changed since that revision.  For example, the
 README in that revision uses the word "changeset" to describe what we
Signature · 1 line
2.43.0