Skip to contentSkip to conversation

scripts/setlocalversion: make git describe output more reliable

PATCH Superseded Rasmus Villemoes 6 participants 17 messages Sep 10, 2020 – Sep 24, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts/setlocalversion: make git describe output more reliable

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.

---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.

 scripts/setlocalversion | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/setlocalversion +2 -2
index 20f2efd57b11..c5262f0d953d 100755
@@ -45,7 +45,7 @@ scm_version()
 
 	# Check for git and a git repo.
 	if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-	   head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+	   head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then
 
 		# If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
 		# it, because this version is defined in the top level Makefile.
@@ -59,7 +59,7 @@ scm_version()
 			fi
 			# If we are past a tagged commit (like
 			# "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-			if atag="$(git describe 2>/dev/null)"; then
+			if atag="$(git describe --abbrev=15 2>/dev/null)"; then
 				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
 
 			# If we don't have a tag at all we print -g{commitish}.
Signature · 1 line
2.23.0
Acked-by on v1 1/1
#1 lore raw
Quoted 41 lines · Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.

Quoted 1 line

Makes sense to me.

Quoted 33 lines

---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.

 scripts/setlocalversion | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/setlocalversion +2 -2
index 20f2efd57b11..c5262f0d953d 100755
@@ -45,7 +45,7 @@ scm_version()
 
 	# Check for git and a git repo.
 	if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-	   head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+	   head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then
 
 		# If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
 		# it, because this version is defined in the top level Makefile.
@@ -59,7 +59,7 @@ scm_version()
 			fi
 			# If we are past a tagged commit (like
 			# "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-			if atag="$(git describe 2>/dev/null)"; then
+			if atag="$(git describe --abbrev=15 2>/dev/null)"; then
 				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
 
 			# If we don't have a tag at all we print -g{commitish}.
#2 lore raw
In reply to Rasmus Villemoes
Quoted 34 lines · Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 7 lines

Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

The kernelrelease is formed by
[kernel version] + [some digits of git hash].

For example, "git describe" shows as follows:

v5.9.0-rc4-00034-g7fe10096c150

Linus gives a new tag every week (or every two week).

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

We have 14000 - 15000 commits in each release,
not 100M.

Quoted 37 lines

---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.

 scripts/setlocalversion | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/setlocalversion +2 -3
index 20f2efd57b11..c5262f0d953d 100755
@@ -45,7 +45,7 @@ scm_version()
 
        # Check for git and a git repo.
        if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-          head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+          head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then
 
                # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
                # it, because this version is defined in the top level Makefile.
@@ -59,7 +59,7 @@ scm_version()
                        fi
                        # If we are past a tagged commit (like
                        # "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-                       if atag="$(git describe 2>/dev/null)"; then
+                       if atag="$(git describe --abbrev=15 2>/dev/null)"; then
                                echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
 
                        # If we don't have a tag at all we print -g{commitish}.
--

2.23.0

Signature · 2 lines
Best Regards
Masahiro Yamada
Reviewed-by
#3 lore raw
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

For the patch:

Quoted 3 lines

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

...

Quoted 3 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

For the rev-parse usage ("unlikely case where we have no signed tag"), the total number of objects is definitely relevant, and the man-page says:
"unique prefix with at least length characters"
i.e., it might be longer, if needed for uniqueness.

For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name" I'm not quite sure whether the uniqueness is including anything about the tag-relative prefix, but if not, then we have the same problem.

At least, that's my reading of the situation.

Brian

#4 lore raw
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 11 lines

For the patch:

Quoted 3 lines

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

...

Quoted 3 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

No.

Quoted 2 lines

For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"

Yes, the abbreviated hash that git describe produces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.

As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.

But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of uname -r etc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.

Rasmus

Side note 1: Note that there really isn't any such thing as "last tag/previous tag" in a DAG; I think what git does is a best-effort search for the eligible tag that minimizes #({objects reachable from commit-to-be-described} - {objects reachable from tag}), where eligible tag means at least reachable from commit-to-be-described (so that set difference is a proper one), but there can be additional constraints (e.g. --match=...).

Side note 2: Linus or Greg releases are actually not interesting here (see the logic in setlocalversion that stops early if we're exactly at an annotated tag) - the whole raison d'etre for setlocalversion is that people do maintain and build non-mainline kernels, and it's extremely useful for uname -a to accurately tell just which commit is running on the target.

#5 lore raw
Quoted 63 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 11 lines

For the patch:

Quoted 3 lines

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

...

Quoted 3 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

No.

Quoted 2 lines

For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"

Yes, the abbreviated hash that git describe produces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.

As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.

But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of uname -r etc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.

Rasmus

Side note 1: Note that there really isn't any such thing as "last tag/previous tag" in a DAG; I think what git does is a best-effort search for the eligible tag that minimizes #({objects reachable from commit-to-be-described} - {objects reachable from tag}), where eligible tag means at least reachable from commit-to-be-described (so that set difference is a proper one), but there can be additional constraints (e.g. --match=...).

Quoted 6 lines

Side note 2: Linus or Greg releases are actually not interesting here (see the logic in setlocalversion that stops early if we're exactly at an annotated tag) - the whole raison d'etre for setlocalversion is that people do maintain and build non-mainline kernels, and it's extremely useful for uname -a to accurately tell just which commit is running on the target.

This is because you use the output from git as-is.

Why are you still trying to rely on such obscure behavior of git?

It is pretty easy to get the fixed number of digits reliably.

For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7

It always returns a 7-digits hash,
and two different people will get the same result for sure.

Your solution, --short=15, means "at least 15", which still contains ambiguity.

As I already noted, the kernelrelease string is constructed in this format:

<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>

For example, if I enable CONFIG_LOCALVERSION_AUTO=y in today's Linus tree, I got this:

5.9.0-rc5-00005-gfc4f28bb3daf

What if the number of digits were 7?

5.9.0-rc5-00005-gfc4f28b

I see no problem here.

Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.

Why do we care about the uniqueness of the abbreviated hash in the whole git history?

Note:
We prepend $(KERNELVERSION) to the result
of setlocalversion. [1]

[1] https://github.com/torvalds/linux/blob/v5.9-rc4/Makefile#L1175

Signature · 2 lines
Best Regards
Masahiro Yamada
#6 lore raw
Quoted 85 lines · Masahiro Yamada
Quoted 39 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 11 lines

For the patch:

Quoted 3 lines

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

...

Quoted 3 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

No.

Quoted 2 lines

For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"

Yes, the abbreviated hash that git describe produces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.

As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.

Quoted 14 lines

But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of uname -r etc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.

This is because you use the output from git as-is.

Why are you still trying to rely on such obscure behavior of git?

It is pretty easy to get the fixed number of digits reliably.

For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7

It always returns a 7-digits hash,
and two different people will get the same result for sure.

Your solution, --short=15, means "at least 15", which still contains ambiguity.

As I already noted, the kernelrelease string is constructed in this format:

<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>

For example, if I enable CONFIG_LOCALVERSION_AUTO=y in today's Linus tree, I got this:

Quoted 8 lines

5.9.0-rc5-00005-gfc4f28bb3daf

What if the number of digits were 7?

5.9.0-rc5-00005-gfc4f28b

I see no problem here.

The problem is that currently, the build relies on things which cannot easily be controlled or reproduced between different developers: Apart from git version (which is reasonable to mandate is the same, just like one must use same compiler, binutils etc. to get binary reproducible output), it depends on whether ~/.gitconfig has a core.abbrev setting - and even worse, it depends on the _total number of objects in the source git repo_, i.e. something that has nothing to do with what is currently in the work tree at all.

And that leads to real bugs when one builds external modules that end up in one directory in the rootfs, while uname -r tells modprobe to look in some other directory (differing in the length of the abbreviated hash).

Quoted 6 lines

Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.

Why do we care about the uniqueness of the abbreviated hash in the whole git history?

Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.

But what I mostly care about is getting a consistent result. And yes, you're right that the porcelain command 'git describe' could end up using something more than 15 digits (though that's extremely unlikely). So if you prefer, I can try to rewrite the logic purely in terms of plumbing commands. But that's a much more invasive patch, and one would obviously lose the guarantee of the abbreviation being unique among current git objects.

Alternatively, would you consider a Kconfig knob, LOCALVERSION_ABBREV?

Rasmus

#7 lore raw
Quoted 116 lines · Rasmus Villemoes
Quoted 85 lines · Masahiro Yamada
Quoted 39 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 11 lines

For the patch:

Quoted 3 lines

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

...

Quoted 3 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

No.

Quoted 2 lines

For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"

Yes, the abbreviated hash that git describe produces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.

As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.

Quoted 14 lines

But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of uname -r etc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.

This is because you use the output from git as-is.

Why are you still trying to rely on such obscure behavior of git?

It is pretty easy to get the fixed number of digits reliably.

For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7

It always returns a 7-digits hash,
and two different people will get the same result for sure.

Your solution, --short=15, means "at least 15", which still contains ambiguity.

As I already noted, the kernelrelease string is constructed in this format:

<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>

For example, if I enable CONFIG_LOCALVERSION_AUTO=y in today's Linus tree, I got this:

Quoted 8 lines

5.9.0-rc5-00005-gfc4f28bb3daf

What if the number of digits were 7?

5.9.0-rc5-00005-gfc4f28b

I see no problem here.

The problem is that currently, the build relies on things which cannot easily be controlled or reproduced between different developers: Apart from git version (which is reasonable to mandate is the same, just like one must use same compiler, binutils etc. to get binary reproducible output), it depends on whether ~/.gitconfig has a core.abbrev setting - and even worse, it depends on the _total number of objects in the source git repo_, i.e. something that has nothing to do with what is currently in the work tree at all.

And that leads to real bugs when one builds external modules that end up in one directory in the rootfs, while uname -r tells modprobe to look in some other directory (differing in the length of the abbreviated hash).

Quoted 6 lines

Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.

Why do we care about the uniqueness of the abbreviated hash in the whole git history?

Quoted 7 lines

Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.

I like:

   git rev-parse --verify HEAD 2>/dev/null | cut -c1-15

better than:

   git rev-parse --verify --short=15 HEAD 2>/dev/null

The former produces the deterministic kernelrelease string.

But, let's reconsider if we need as long as 15-digits.

There are a couple of kinds of objects in git: commit, tree, blob.

I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.

But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.

In the context of the kernel version, we can exclude tree and blob objects.

In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.

We have ~15000 commits per release. So, the difference is just 4x factor, though...

BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.

For example, "git show 06a0d" fails, but
"git log 06a0d" works.

masahiro@oscar:~/ref/linux$ git  show   06a0d
error: short SHA1 06a0d is ambiguous
hint: The candidates are:
hint:   06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued
'softback_lines' cursor() argument
hint:   06a0d81911b3 tree
hint:   06a0dc5a84d2 tree
hint:   06a0d1947c77 blob
hint:   06a0df434249 blob
fatal: ambiguous argument '06a0d': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
masahiro@oscar:~/ref/linux$ git  log --oneline  -1   06a0d
06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argument

What is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.

masahiro@oscar:~/ref/linux$ git  show   v5.9-rc5-00002-g06a0d  | head -1
commit 06a0df4d1b8b13b551668e47b11fd7629033b7df

I guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.

Keeping those above in mind, I believe 15-digits is too long.

So, I propose two options.

[1] 7 digits

The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.

As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.

[2] 12 digits

This matches to the Fixes: tag policy specified in Documentation/process/submitting-patches.rst

The abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)

I wait for some comments.

Quoted 9 lines

But what I mostly care about is getting a consistent result. And yes, you're right that the porcelain command 'git describe' could end up using something more than 15 digits (though that's extremely unlikely). So if you prefer, I can try to rewrite the logic purely in terms of plumbing commands. But that's a much more invasive patch, and one would obviously lose the guarantee of the abbreviation being unique among current git objects.

Alternatively, would you consider a Kconfig knob, LOCALVERSION_ABBREV?

No, I do not think LOCALVERSION_ABBREV is a good idea.

We should eliminate this problem
irrespective of the kernel configuration.

--
Best Regards
Masahiro Yamada

#8 lore raw
Quoted 87 lines · Masahiro Yamada
Quoted 9 lines · Rasmus Villemoes
Quoted 6 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes

Why do we care about the uniqueness of the abbreviated hash in the whole git history?

Quoted 7 lines

Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.

I like:

   git rev-parse --verify HEAD 2>/dev/null | cut -c1-15

better than:

   git rev-parse --verify --short=15 HEAD 2>/dev/null

The former produces the deterministic kernelrelease string.

But, let's reconsider if we need as long as 15-digits.

There are a couple of kinds of objects in git: commit, tree, blob.

I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.

But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.

In the context of the kernel version, we can exclude tree and blob objects.

In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.

We have ~15000 commits per release. So, the difference is just 4x factor, though...

BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.

For example, "git show 06a0d" fails, but
"git log 06a0d" works.

masahiro@oscar:~/ref/linux$ git  show   06a0d
error: short SHA1 06a0d is ambiguous
hint: The candidates are:
hint:   06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued
'softback_lines' cursor() argument
hint:   06a0d81911b3 tree
hint:   06a0dc5a84d2 tree
hint:   06a0d1947c77 blob
hint:   06a0df434249 blob
fatal: ambiguous argument '06a0d': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
masahiro@oscar:~/ref/linux$ git  log --oneline  -1   06a0d
06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argument

What is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.

Quoted 8 lines
masahiro@oscar:~/ref/linux$ git  show   v5.9-rc5-00002-g06a0d  | head -1
commit 06a0df4d1b8b13b551668e47b11fd7629033b7df

I guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.

Well, yes, but unfortunately only so far as applying the same "the user means a commit object" logic; git doesn't do anything to actually use the prefix to disambiguate. In fact, looking at a semi-random commit in 4.19-stable v4.19.114-7-g236c445eb352:

$ git show 236c44
error: short SHA1 236c44 is ambiguous
hint: The candidates are:
hint:   236c445eb352 commit 2020-03-13 - drm/bochs: downgrade
pci_request_region failure from error to warning
hint:   236c448cb6e7 commit 2011-09-08 - usbmon vs. tcpdump: fix dropped
packet count
hint:   236c445b1930 blob

Now you're right that we get

$ git show v4.19.114-7-g236c445 | head -n1
commit 236c445eb3529aa7c976f8812513c3cb26d77e27

so it knows we're not looking at that blob. But "git show v4.19.114-7-g236c44" still fails because there are two commits. Adding to the fun is that "git show v4.19.114-7-g236c448" actually shows the usbmon commit, which is old v3.2 stuff and certainly not a descendant of v4.19.114.

I didn't actually know that git would even accept those prefixed strings as possible refspecs, and for a moment you had me hoping that git would actually do the disambiguation using that prefix, which would certainly make 7 hex chars enough; unfortunately that's not the case.

Quoted 1 line

Keeping those above in mind, I believe 15-digits is too long.

Well, as you indicated, commits are probably ~1/4 of all objects, i.e. half a hexchar worth of bits. So the commit/object distinction shouldn't matter very much for the choice of suitable fixed length.

Quoted 15 lines

So, I propose two options.

[1] 7 digits

The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.

As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.

Quoted 8 lines

[2] 12 digits

This matches to the Fixes: tag policy specified in Documentation/process/submitting-patches.rst

The abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)

I can live with 12. It would be extremely rare that I would have to do some extra git yoga to figure out which commit they actually mean. I think we can still use git describe to do the bulk of the work (the 'git rev-parse | cut ... is easy, but it's somewhat tedious to find the closest tag, then compute the #commits-on-top part), then just have the awk script used for pretty-printing (the %05d of the #commits-on-top) can probably also be used to chop the abbreviated sha1 at 12 chars, should git have needed to make it longer. Please see below for an updated patch+commit log.

Quoted 4 lines
Quoted 1 line

Alternatively, would you consider a Kconfig knob, LOCALVERSION_ABBREV?

No, I do not think LOCALVERSION_ABBREV is a good idea.

Neither do I; I considered it before sending the patch but decided that yes:

Quoted 2 lines

We should eliminate this problem
irrespective of the kernel configuration.

Rasmus

====

something like this, then?

scripts/setlocalversion: make git describe output more reliable

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.

scripts/setlocalversion +12 -5
index 20f2efd57b11..b51072167df1 100755
@@ -45,7 +45,7 @@ scm_version()
 
 	# Check for git and a git repo.
 	if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-	   head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+	   head=$(git rev-parse --verify HEAD 2>/dev/null); then
 
 		# If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
 		# it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version()
 			fi
 			# If we are past a tagged commit (like
 			# "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-			if atag="$(git describe 2>/dev/null)"; then
-				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
-
-			# If we don't have a tag at all we print -g{commitish}.
+                        #
+                        # Ensure the abbreviated sha1 has exactly 12
+                        # hex characters, to make the output
+                        # independent of git version, local
+                        # core.abbrev settings and/or total number of
+                        # objects in the current repository - passing
+                        # --abbrev=12 ensures a minimum of 12, and the
+                        # awk substr() then picks the 'g' and first 12
+                        # hex chars.
+			if atag="$(git describe --abbrev=12 2>/dev/null)"; then
+				echo "$atag" | awk -F- '{printf("-%05d-%s",

$(NF-1),substr($(NF),0,13))}'
+
+ # If we don't have a tag at all we print -g{commitish},
+ # again using exactly 12 hex chars.
else
+ head="$(echo $head | cut -c1-12)"
printf '%s%s' -g $head
fi
fi

#9 lore raw
Quoted 114 lines · Rasmus Villemoes
Quoted 87 lines · Masahiro Yamada
Quoted 9 lines · Rasmus Villemoes
Quoted 6 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes

Why do we care about the uniqueness of the abbreviated hash in the whole git history?

Quoted 7 lines

Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.

I like:

   git rev-parse --verify HEAD 2>/dev/null | cut -c1-15

better than:

   git rev-parse --verify --short=15 HEAD 2>/dev/null

The former produces the deterministic kernelrelease string.

But, let's reconsider if we need as long as 15-digits.

There are a couple of kinds of objects in git: commit, tree, blob.

I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.

But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.

In the context of the kernel version, we can exclude tree and blob objects.

In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.

We have ~15000 commits per release. So, the difference is just 4x factor, though...

BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.

For example, "git show 06a0d" fails, but
"git log 06a0d" works.

masahiro@oscar:~/ref/linux$ git  show   06a0d
error: short SHA1 06a0d is ambiguous
hint: The candidates are:
hint:   06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued
'softback_lines' cursor() argument
hint:   06a0d81911b3 tree
hint:   06a0dc5a84d2 tree
hint:   06a0d1947c77 blob
hint:   06a0df434249 blob
fatal: ambiguous argument '06a0d': unknown revision or path not in the
working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
masahiro@oscar:~/ref/linux$ git  log --oneline  -1   06a0d
06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argument

What is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.

Quoted 8 lines
masahiro@oscar:~/ref/linux$ git  show   v5.9-rc5-00002-g06a0d  | head -1
commit 06a0df4d1b8b13b551668e47b11fd7629033b7df

I guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.

Well, yes, but unfortunately only so far as applying the same "the user means a commit object" logic; git doesn't do anything to actually use the prefix to disambiguate. In fact, looking at a semi-random commit in 4.19-stable v4.19.114-7-g236c445eb352:

$ git show 236c44
error: short SHA1 236c44 is ambiguous
hint: The candidates are:
hint:   236c445eb352 commit 2020-03-13 - drm/bochs: downgrade
pci_request_region failure from error to warning
hint:   236c448cb6e7 commit 2011-09-08 - usbmon vs. tcpdump: fix dropped
packet count
hint:   236c445b1930 blob

Now you're right that we get

Quoted 8 lines
$ git show v4.19.114-7-g236c445 | head -n1
commit 236c445eb3529aa7c976f8812513c3cb26d77e27

so it knows we're not looking at that blob. But "git show v4.19.114-7-g236c44" still fails because there are two commits. Adding to the fun is that "git show v4.19.114-7-g236c448" actually shows the usbmon commit, which is old v3.2 stuff and certainly not a descendant of v4.19.114.

Oh, I did not notice that.
Maybe git can be improved to check the validity of the 'git describe' form, but that is another story.

Quoted 35 lines

I didn't actually know that git would even accept those prefixed strings as possible refspecs, and for a moment you had me hoping that git would actually do the disambiguation using that prefix, which would certainly make 7 hex chars enough; unfortunately that's not the case.

Quoted 1 line

Keeping those above in mind, I believe 15-digits is too long.

Well, as you indicated, commits are probably ~1/4 of all objects, i.e. half a hexchar worth of bits. So the commit/object distinction shouldn't matter very much for the choice of suitable fixed length.

Quoted 15 lines

So, I propose two options.

[1] 7 digits

The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.

As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.

Quoted 8 lines

[2] 12 digits

This matches to the Fixes: tag policy specified in Documentation/process/submitting-patches.rst

The abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)

Quoted 9 lines

I can live with 12. It would be extremely rare that I would have to do some extra git yoga to figure out which commit they actually mean. I think we can still use git describe to do the bulk of the work (the 'git rev-parse | cut ... is easy, but it's somewhat tedious to find the closest tag, then compute the #commits-on-top part), then just have the awk script used for pretty-printing (the %05d of the #commits-on-top) can probably also be used to chop the abbreviated sha1 at 12 chars, should git have needed to make it longer. Please see below for an updated patch+commit log.

LGTM.

Could you send it to ML as a patch?

Quoted 95 lines
Quoted 4 lines
Quoted 1 line

Alternatively, would you consider a Kconfig knob, LOCALVERSION_ABBREV?

No, I do not think LOCALVERSION_ABBREV is a good idea.

Neither do I; I considered it before sending the patch but decided that yes:

Quoted 2 lines

We should eliminate this problem
irrespective of the kernel configuration.

Rasmus

====

something like this, then?

scripts/setlocalversion: make git describe output more reliable

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.

scripts/setlocalversion +12 -5
index 20f2efd57b11..b51072167df1 100755
@@ -45,7 +45,7 @@ scm_version()
 
        # Check for git and a git repo.
        if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-          head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+          head=$(git rev-parse --verify HEAD 2>/dev/null); then
 
                # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
                # it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version()
                        fi
                        # If we are past a tagged commit (like
                        # "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-                       if atag="$(git describe 2>/dev/null)"; then
-                               echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
-
-                       # If we don't have a tag at all we print -g{commitish}.
+                        #
+                        # Ensure the abbreviated sha1 has exactly 12
+                        # hex characters, to make the output
+                        # independent of git version, local
+                        # core.abbrev settings and/or total number of
+                        # objects in the current repository - passing
+                        # --abbrev=12 ensures a minimum of 12, and the
+                        # awk substr() then picks the 'g' and first 12
+                        # hex chars.
+                       if atag="$(git describe --abbrev=12 2>/dev/null)"; then
+                               echo "$atag" | awk -F- '{printf("-%05d-%s",

$(NF-1),substr($(NF),0,13))}'

scripts/setlocalversion +4 -0
+
+                       # If we don't have a tag at all we print -g{commitish},
+                       # again using exactly 12 hex chars.
                        else
+                               head="$(echo $head | cut -c1-12)"
                                printf '%s%s' -g $head
                        fi
                fi
Signature · 2 lines
Best Regards
Masahiro Yamada
#10 lore raw
In reply to Masahiro Yamada
Quoted 62 lines · Masahiro Yamada
Quoted 34 lines · Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Quoted 7 lines

Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.

I agree that any randomness should be avoided.

My question is, do we need 15-digits?

The kernelrelease is formed by
[kernel version] + [some digits of git hash].

For example, "git describe" shows as follows:

v5.9.0-rc4-00034-g7fe10096c150

Linus gives a new tag every week (or every two week).

Quoted 8 lines

So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?

We have 14000 - 15000 commits in each release,
not 100M.

I kinda agree with this...we need to chopped down the excess bits from the information .

Indeed , a 15 digits is too long to keep up.

~Bhaskar

Quoted 44 lines
Quoted 39 lines

---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.

 scripts/setlocalversion | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/setlocalversion +2 -3
index 20f2efd57b11..c5262f0d953d 100755
@@ -45,7 +45,7 @@ scm_version()
 
        # Check for git and a git repo.
        if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-          head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+          head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then
 
                # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
                # it, because this version is defined in the top level Makefile.
@@ -59,7 +59,7 @@ scm_version()
                        fi
                        # If we are past a tagged commit (like
                        # "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-                       if atag="$(git describe 2>/dev/null)"; then
+                       if atag="$(git describe --abbrev=15 2>/dev/null)"; then
                                echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
 
                        # If we don't have a tag at all we print -g{commitish}.
--

2.23.0

Signature · 2 lines
Best Regards
Masahiro Yamada
#11 lore raw
In reply to Masahiro Yamada
Quoted 3 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes

...

Quoted 7 lines
Quoted 7 lines

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).

Hi Masahiro

Can I get you to carry this through the kbuild tree? Unless of course your questions/concerns haven't been addressed.

Thanks,
Rasmus

#12 lore raw

PATCH v2 1/1 scripts/setlocalversion: make git describe output more reliable

In reply to Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.

---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.

 scripts/setlocalversion | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
scripts/setlocalversion +16 -5
index 20f2efd57b11..bb709eda96cd 100755
@@ -45,7 +45,7 @@ scm_version()
 
 	# Check for git and a git repo.
 	if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-	   head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+	   head=$(git rev-parse --verify HEAD 2>/dev/null); then
 
 		# If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
 		# it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version()
 			fi
 			# If we are past a tagged commit (like
 			# "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-			if atag="$(git describe 2>/dev/null)"; then
-				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
-
-			# If we don't have a tag at all we print -g{commitish}.
+			#
+			# Ensure the abbreviated sha1 has exactly 12
+			# hex characters, to make the output
+			# independent of git version, local
+			# core.abbrev settings and/or total number of
+			# objects in the current repository - passing
+			# --abbrev=12 ensures a minimum of 12, and the
+			# awk substr() then picks the 'g' and first 12
+			# hex chars.
+			if atag="$(git describe --abbrev=12 2>/dev/null)"; then
+				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),substr($(NF),0,13))}'
+
+			# If we don't have a tag at all we print -g{commitish},
+			# again using exactly 12 hex chars.
 			else
+				head="$(echo $head | cut -c1-12)"
 				printf '%s%s' -g $head
 			fi
 		fi
Signature · 1 line
2.23.0
#13 lore raw

Thanks for the patch Rasmus. Overall it looks good to me, be aligned to the stable patch submission rules makes sense. A tiny thing though:

I did not calculate the exact collision probability with 12 characters and it does not make sense to even discuss this, if this is a current rule for stable patches. However we have a couple of 12's scattered in the code. And if the submission rule changes in the future, we should have a single location to update it.

So I suggest you introduce something on the line of:

...
num_chars=12
...
--abbrev=$num_chars
...

I guess you get the picture.

Greetings from the sunny mountains,

Nico

Quoted 73 lines · Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.

---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.

 scripts/setlocalversion | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
scripts/setlocalversion +10 -5
index 20f2efd57b11..bb709eda96cd 100755
@@ -45,7 +45,7 @@ scm_version()
 
 	# Check for git and a git repo.
 	if test -z "$(git rev-parse --show-cdup 2>/dev/null)" &&
-	   head=$(git rev-parse --verify --short HEAD 2>/dev/null); then
+	   head=$(git rev-parse --verify HEAD 2>/dev/null); then
 
 		# If we are at a tagged commit (like "v2.6.30-rc6"), we ignore
 		# it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version()
 			fi
 			# If we are past a tagged commit (like
 			# "v2.6.30-rc5-302-g72357d5"), we pretty print it.
-			if atag="$(git describe 2>/dev/null)"; then
-				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}'
-
-			# If we don't have a tag at all we print -g{commitish}.
+			#
+			# Ensure the abbreviated sha1 has exactly 12
+			# hex characters, to make the output
+			# independent of git version, local
+			# core.abbrev settings and/or total number of
+			# objects in the current repository - passing
+			# --abbrev=12 ensures a minimum of 12, and the
+			# awk substr() then picks the 'g' and first 12
+			# hex chars.

scripts/setlocalversion scm_version() · line 80

Quoted 10 lines
scripts/setlocalversion +6 -0
+			if atag="$(git describe --abbrev=12 2>/dev/null)"; then
+				echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),substr($(NF),0,13))}'
+
+			# If we don't have a tag at all we print -g{commitish},
+			# again using exactly 12 hex chars.
 			else
+				head="$(echo $head | cut -c1-12)"
 				printf '%s%s' -g $head
 			fi
 		fi

--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

#14 lore raw
Quoted 5 lines · Nico Schottelius

Thanks for the patch Rasmus. Overall it looks good to me, be aligned to the stable patch submission rules makes sense. A tiny thing though:

I did not calculate the exact collision probability with 12 characters

For reference, the math is something like this: Consider a repo with N+1 objects. We look at one specific object (for setlocalversion its the head commit being built, for the stable rules its whatever particular commit one is interested in for backporting), and want to know the probability that its sha1 collides with some other object in the first b bits (here b=48). Assuming the sha1s are independent and uniformly distributed, the probability of not colliding with one specific other commit is x=1-1/2^b, and the probability of not colliding with any of the other N commits is x^N, making the probability of a collision 1-x^N = (1-x)(1+x+x^2+...+x^{N-1}). Now the N terms in the second factor are very-close-to-but-slightly-smaller-than 1, so an upper bound for this probability is (1-x)N = N/2^b, which is also what one would naively expect. [This estimate is always valid, but it becomes a void statement of "the probability is less then 1" when N is >= 2^b].

So, assuming some vendor kernel repo that has all of Greg's stable.git (around 10M objects I think) and another 10M objects because random vendor, that works out to 20e6/2^48 = 7.1e-8, 71 ppb.

Quoted 6 lines

So I suggest you introduce something on the line of:

...
num_chars=12
...
--abbrev=$num_chars

I considered that, but it becomes quite ugly since it needs to get into the awk script (as a 13, though perhaps we could get awk to do the +1, I don't really speak awk), where we'd then need to use " instead of ' and then escape the $ that are to be interpreted by awk and not the shell. So I think it's more readable with hardcoding and comments explaining why they are there; should anyone ever want to change 12.

Rasmus

#15 lore raw

Hey Rasmus,

Quoted 11 lines · Rasmus Villemoes
Quoted 6 lines

So I suggest you introduce something on the line of:

...
num_chars=12
...
--abbrev=$num_chars

I considered that, but it becomes quite ugly since it needs to get into the awk script (as a 13, though perhaps we could get awk to do the +1, I don't really speak awk), where we'd then need to use " instead of ' and then escape the $ that are to be interpreted by awk and not the shell.

No need for that, awk can read (and use) environment variables...

Quoted 2 lines

So I think it's more readable with hardcoding and comments explaining why they are there; should anyone ever want to change 12.

... so that in practice you only need to change 13 to var+1 and 12 to $var.

Cheers,

Nico

--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

#16 lore raw
In reply to Rasmus Villemoes
Quoted 36 lines · Rasmus Villemoes

When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.

The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by

(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig

So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.

Quoted 7 lines

---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.

 scripts/setlocalversion | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

Applied to linux-kbuild.
Thanks.

Signature · 2 lines
Best Regards
Masahiro Yamada