scripts/setlocalversion: make git describe output more reliable
PATCH 1/1 scripts/setlocalversion: make git describe output more reliable
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.
The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by
(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig
So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).
Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.
---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.
scripts/setlocalversion | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/setlocalversion +2 -2
@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.
@@ -59,7 +59,7 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then + if atag="$(git describe --abbrev=15 2>/dev/null)"; then echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' # If we don't have a tag at all we print -g{commitish}.
Signature · 1 line
2.23.0
Quoted 41 lines · Rasmus Villemoes
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.
Quoted 1 line
Signed-off-by Rasmus Villemoes <[email protected]>
Makes sense to me.
Quoted 33 lines
---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.scripts/setlocalversion | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/setlocalversion +2
-2@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.@@ -59,7 +59,7 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then + if atag="$(git describe --abbrev=15 2>/dev/null)"; then echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' # If we don't have a tag at all we print -g{commitish}.
Quoted 34 lines · Rasmus Villemoes
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).
Quoted 7 lines
Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
The kernelrelease is formed by
[kernel version] + [some digits of git hash].
For example, "git describe" shows as follows:
v5.9.0-rc4-00034-g7fe10096c150
Linus gives a new tag every week (or every two week).
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?
We have 14000 - 15000 commits in each release,
not 100M.
Quoted 37 lines
Signed-off-by Rasmus Villemoes <[email protected]>---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.scripts/setlocalversion | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/setlocalversion +2
-3@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.@@ -59,7 +59,7 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then + if atag="$(git describe --abbrev=15 2>/dev/null)"; then echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' # If we don't have a tag at all we print -g{commitish}. --2.23.0
Signature · 2 lines
Best Regards Masahiro Yamada
Brian Norris
Reviewed-byQuoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).
For the patch:
Quoted 3 lines
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
...
Quoted 3 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?
For the rev-parse usage ("unlikely case where we have no signed tag"), the total number of objects is definitely relevant, and the man-page says:
"unique prefix with at least length characters"
i.e., it might be longer, if needed for uniqueness.
For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name" I'm not quite sure whether the uniqueness is including anything about the tag-relative prefix, but if not, then we have the same problem.
At least, that's my reading of the situation.
Brian
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).
Quoted 11 lines
For the patch:
Reviewed-by Brian Norris <[email protected]>Quoted 3 lines
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
...
Quoted 3 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?
No.
Quoted 2 lines
For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"
Yes, the abbreviated hash that git describe produces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.
As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.
But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of uname -r etc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.
Rasmus
Side note 1: Note that there really isn't any such thing as "last tag/previous tag" in a DAG; I think what git does is a best-effort search for the eligible tag that minimizes #({objects reachable from commit-to-be-described} - {objects reachable from tag}), where eligible tag means at least reachable from commit-to-be-described (so that set difference is a proper one), but there can be additional constraints (e.g. --match=...).
Side note 2: Linus or Greg releases are actually not interesting here (see the logic in setlocalversion that stops early if we're exactly at an annotated tag) - the whole raison d'etre for setlocalversion is that people do maintain and build non-mainline kernels, and it's extremely useful for uname -a to accurately tell just which commit is running on the target.
Quoted 63 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).Quoted 11 lines
For the patch:
Reviewed-by Brian Norris <[email protected]>Quoted 3 lines
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
...
Quoted 3 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?No.
Quoted 2 lines
For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"Yes, the abbreviated hash that
git describeproduces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.
But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of
uname -retc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.Rasmus
Side note 1: Note that there really isn't any such thing as "last tag/previous tag" in a DAG; I think what git does is a best-effort search for the eligible tag that minimizes #({objects reachable from commit-to-be-described} - {objects reachable from tag}), where eligible tag means at least reachable from commit-to-be-described (so that set difference is a proper one), but there can be additional constraints (e.g. --match=...).
Quoted 6 lines
Side note 2: Linus or Greg releases are actually not interesting here (see the logic in setlocalversion that stops early if we're exactly at an annotated tag) - the whole raison d'etre for setlocalversion is that people do maintain and build non-mainline kernels, and it's extremely useful for
uname -ato accurately tell just which commit is running on the target.
This is because you use the output from git as-is.
Why are you still trying to rely on such obscure behavior of git?
It is pretty easy to get the fixed number of digits reliably.
For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7
It always returns a 7-digits hash,
and two different people will get the same result for sure.
Your solution, --short=15, means "at least 15", which still contains ambiguity.
As I already noted, the kernelrelease string is constructed in this format:
<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>
For example, if I enable CONFIG_LOCALVERSION_AUTO=y in today's Linus tree, I got this:
5.9.0-rc5-00005-gfc4f28bb3daf
What if the number of digits were 7?
5.9.0-rc5-00005-gfc4f28b
I see no problem here.
Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.
Why do we care about the uniqueness of the abbreviated hash in the whole git history?
Note:
We prepend $(KERNELVERSION) to the result
of setlocalversion. [1]
[1] https://github.com/torvalds/linux/blob/v5.9-rc4/Makefile#L1175
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 85 lines · Masahiro Yamada
Quoted 39 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).Quoted 11 lines
For the patch:
Reviewed-by Brian Norris <[email protected]>Quoted 3 lines
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
...
Quoted 3 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?No.
Quoted 2 lines
For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"Yes, the abbreviated hash that
git describeproduces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.
Quoted 14 lines
But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of
uname -retc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.This is because you use the output from git as-is.
Why are you still trying to rely on such obscure behavior of git?
It is pretty easy to get the fixed number of digits reliably.
For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7It always returns a 7-digits hash,
and two different people will get the same result for sure.Your solution, --short=15, means "at least 15", which still contains ambiguity.
As I already noted, the kernelrelease string is constructed in this format:
<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>
For example, if I enable
CONFIG_LOCALVERSION_AUTO=yin today's Linus tree, I got this:
Quoted 8 lines
5.9.0-rc5-00005-gfc4f28bb3daf
What if the number of digits were 7?
5.9.0-rc5-00005-gfc4f28b
I see no problem here.
The problem is that currently, the build relies on things which cannot easily be controlled or reproduced between different developers: Apart from git version (which is reasonable to mandate is the same, just like one must use same compiler, binutils etc. to get binary reproducible output), it depends on whether ~/.gitconfig has a core.abbrev setting - and even worse, it depends on the _total number of objects in the source git repo_, i.e. something that has nothing to do with what is currently in the work tree at all.
And that leads to real bugs when one builds external modules that end up in one directory in the rootfs, while uname -r tells modprobe to look in some other directory (differing in the length of the abbreviated hash).
Quoted 6 lines
Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.
Why do we care about the uniqueness of the abbreviated hash in the whole git history?
Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.
But what I mostly care about is getting a consistent result. And yes, you're right that the porcelain command 'git describe' could end up using something more than 15 digits (though that's extremely unlikely). So if you prefer, I can try to rewrite the logic purely in terms of plumbing commands. But that's a much more invasive patch, and one would obviously lose the guarantee of the abbreviation being unique among current git objects.
Alternatively, would you consider a Kconfig knob, LOCALVERSION_ABBREV?
Rasmus
Quoted 116 lines · Rasmus Villemoes
Quoted 85 lines · Masahiro Yamada
Quoted 39 lines · Rasmus Villemoes
Quoted 9 lines · Brian Norris
Quoted 7 lines · Masahiro Yamada
Quoted 5 lines · Rasmus Villemoes
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).Quoted 11 lines
For the patch:
Reviewed-by Brian Norris <[email protected]>Quoted 3 lines
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
...
Quoted 3 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?No.
Quoted 2 lines
For git-describe (the case where we have a tag to base off):
"use <n> digits, or as many digits as needed to form a unique object name"Yes, the abbreviated hash that
git describeproduces is unique among all objects (and objects are more than just commits) in the current repo, so what matters for probability-of-collision is the total number of objects - linus.git itself probably grows ~60000 objects per release.As for "do we need 15 digits", well, in theory the next time I merge the next rt-stable tag into our kernel I could end up with a commit that matches some existing object in the first 33 hex chars at which point I should have chosen 34 - but of course that's so unlikely that it's irrelevant.
Quoted 14 lines
But the upshot of that is that there really is no objective answer to "how many digits do we need", so it becomes a tradeoff between "low enough probability that anyone anywhere in the next few years would have needed more than X when building their own kernel" and readability of
uname -retc. So I decided somewhat arbitrarily that each time one rolls a new release, there should be less than 1e-9 probability that HEAD collides with some other object when abbreviated to X hexchars. X=12 doesn't pass that criteria even when the repo has only 10M objects (and, current linus.git already has objects that need 12 to be unique, so such collisions do happen in practice, though of course very rarely). 13 and 14 are just weird numbers, so I ended with 15, which also allows many many more objects in the repo before the probability crosses that 1e-9 threshold.This is because you use the output from git as-is.
Why are you still trying to rely on such obscure behavior of git?
It is pretty easy to get the fixed number of digits reliably.
For example,
git rev-parse --verify HEAD 2>/dev/null | cut -c1-7It always returns a 7-digits hash,
and two different people will get the same result for sure.Your solution, --short=15, means "at least 15", which still contains ambiguity.
As I already noted, the kernelrelease string is constructed in this format:
<kernel-version>-<number-of-commits-on-top>-<abbreviated-commit-hash>
For example, if I enable
CONFIG_LOCALVERSION_AUTO=yin today's Linus tree, I got this:Quoted 8 lines
5.9.0-rc5-00005-gfc4f28bb3daf
What if the number of digits were 7?
5.9.0-rc5-00005-gfc4f28b
I see no problem here.
The problem is that currently, the build relies on things which cannot easily be controlled or reproduced between different developers: Apart from git version (which is reasonable to mandate is the same, just like one must use same compiler, binutils etc. to get binary reproducible output), it depends on whether ~/.gitconfig has a core.abbrev setting - and even worse, it depends on the
_totalnumber of objects in the source gitrepo_, i.e. something that has nothing to do with what is currently in the work tree at all.And that leads to real bugs when one builds external modules that end up in one directory in the rootfs, while
uname -rtells modprobe to look in some other directory (differing in the length of the abbreviated hash).Quoted 6 lines
Even if we have another object that starts with "fc4f28b", the kernelrelease string is still unique thanks to the <kernel-version>-<number-of-commits-on-top> prefix.
Why do we care about the uniqueness of the abbreviated hash in the whole git history?
Quoted 7 lines
Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.
I like:
git rev-parse --verify HEAD 2>/dev/null | cut -c1-15
better than:
git rev-parse --verify --short=15 HEAD 2>/dev/null
The former produces the deterministic kernelrelease string.
But, let's reconsider if we need as long as 15-digits.
There are a couple of kinds of objects in git: commit, tree, blob.
I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.
But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.
In the context of the kernel version, we can exclude tree and blob objects.
In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.
We have ~15000 commits per release. So, the difference is just 4x factor, though...
BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.
For example, "git show 06a0d" fails, but
"git log 06a0d" works.
masahiro@oscar:~/ref/linux$ git show 06a0d error: short SHA1 06a0d is ambiguous hint: The candidates are: hint: 06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued 'softback_lines' cursor() argument hint: 06a0d81911b3 tree hint: 06a0dc5a84d2 tree hint: 06a0d1947c77 blob hint: 06a0df434249 blob fatal: ambiguous argument '06a0d': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' masahiro@oscar:~/ref/linux$ git log --oneline -1 06a0d 06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argument
What is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.
masahiro@oscar:~/ref/linux$ git show v5.9-rc5-00002-g06a0d | head -1 commit 06a0df4d1b8b13b551668e47b11fd7629033b7df
I guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.
Keeping those above in mind, I believe 15-digits is too long.
So, I propose two options.
[1] 7 digits
The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.
As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.
[2] 12 digits
This matches to the Fixes: tag policy specified in Documentation/process/submitting-patches.rst
The abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)
I wait for some comments.
Quoted 9 lines
But what I mostly care about is getting a consistent result. And yes, you're right that the porcelain command 'git describe' could end up using something more than 15 digits (though that's extremely unlikely). So if you prefer, I can try to rewrite the logic purely in terms of plumbing commands. But that's a much more invasive patch, and one would obviously lose the guarantee of the abbreviation being unique among current git objects.
Alternatively, would you consider a
Kconfigknob,LOCALVERSION_ABBREV?
No, I do not think LOCALVERSION_ABBREV is a good idea.
We should eliminate this problem
irrespective of the kernel configuration.
--
Best Regards
Masahiro Yamada
Quoted 87 lines · Masahiro Yamada
Quoted 9 lines · Rasmus Villemoes
Quoted 6 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes
Why do we care about the uniqueness of the abbreviated hash in the whole git history?
Quoted 7 lines
Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.
I like:
git rev-parse --verify HEAD 2>/dev/null | cut -c1-15better than:
git rev-parse --verify --short=15 HEAD 2>/dev/nullThe former produces the deterministic kernelrelease string.
But, let's reconsider if we need as long as 15-digits.
There are a couple of kinds of objects in git: commit, tree, blob.
I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.
But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.
In the context of the kernel version, we can exclude tree and blob objects.
In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.
We have ~15000 commits per release. So, the difference is just 4x factor, though...
BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.For example, "git show 06a0d" fails, but
"git log 06a0d" works.masahiro@oscar:~/ref/linux$ git show 06a0d error: short SHA1 06a0d is ambiguous hint: The candidates are: hint: 06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued 'softback_lines' cursor() argument hint: 06a0d81911b3 tree hint: 06a0dc5a84d2 tree hint: 06a0d1947c77 blob hint: 06a0df434249 blob fatal: ambiguous argument '06a0d': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' masahiro@oscar:~/ref/linux$ git log --oneline -1 06a0d 06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argumentWhat is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.
Quoted 8 lines
masahiro@oscar:~/ref/linux$ git show v5.9-rc5-00002-g06a0d | head -1 commit 06a0df4d1b8b13b551668e47b11fd7629033b7dfI guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.
Well, yes, but unfortunately only so far as applying the same "the user means a commit object" logic; git doesn't do anything to actually use the prefix to disambiguate. In fact, looking at a semi-random commit in 4.19-stable v4.19.114-7-g236c445eb352:
$ git show 236c44 error: short SHA1 236c44 is ambiguous hint: The candidates are: hint: 236c445eb352 commit 2020-03-13 - drm/bochs: downgrade pci_request_region failure from error to warning hint: 236c448cb6e7 commit 2011-09-08 - usbmon vs. tcpdump: fix dropped packet count hint: 236c445b1930 blob
Now you're right that we get
$ git show v4.19.114-7-g236c445 | head -n1 commit 236c445eb3529aa7c976f8812513c3cb26d77e27
so it knows we're not looking at that blob. But "git show v4.19.114-7-g236c44" still fails because there are two commits. Adding to the fun is that "git show v4.19.114-7-g236c448" actually shows the usbmon commit, which is old v3.2 stuff and certainly not a descendant of v4.19.114.
I didn't actually know that git would even accept those prefixed strings as possible refspecs, and for a moment you had me hoping that git would actually do the disambiguation using that prefix, which would certainly make 7 hex chars enough; unfortunately that's not the case.
Quoted 1 line
Keeping those above in mind, I believe 15-digits is too long.
Well, as you indicated, commits are probably ~1/4 of all objects, i.e. half a hexchar worth of bits. So the commit/object distinction shouldn't matter very much for the choice of suitable fixed length.
Quoted 15 lines
So, I propose two options.
[1] 7 digits
The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.
As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.
Quoted 8 lines
[2] 12 digits
This matches to the Fixes: tag policy specified in
Documentation/process/submitting-patches.rstThe abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)
I can live with 12. It would be extremely rare that I would have to do some extra git yoga to figure out which commit they actually mean. I think we can still use git describe to do the bulk of the work (the 'git rev-parse | cut ... is easy, but it's somewhat tedious to find the closest tag, then compute the #commits-on-top part), then just have the awk script used for pretty-printing (the %05d of the #commits-on-top) can probably also be used to chop the abbreviated sha1 at 12 chars, should git have needed to make it longer. Please see below for an updated patch+commit log.
Quoted 4 lines
Quoted 1 line
Alternatively, would you consider a
Kconfigknob,LOCALVERSION_ABBREV?No, I do not think
LOCALVERSION_ABBREVis a good idea.
Neither do I; I considered it before sending the patch but decided that yes:
Quoted 2 lines
We should eliminate this problem
irrespective of the kernel configuration.
Rasmus
====
something like this, then?
scripts/setlocalversion: make git describe output more reliable
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.
The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by
(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig
So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.
scripts/setlocalversion +12 -5
@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then - echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' - - # If we don't have a tag at all we print -g{commitish}. + # + # Ensure the abbreviated sha1 has exactly 12 + # hex characters, to make the output + # independent of git version, local + # core.abbrev settings and/or total number of + # objects in the current repository - passing + # --abbrev=12 ensures a minimum of 12, and the + # awk substr() then picks the 'g' and first 12 + # hex chars. + if atag="$(git describe --abbrev=12 2>/dev/null)"; then + echo "$atag" | awk -F- '{printf("-%05d-%s",
$(NF-1),substr($(NF),0,13))}'
+
+ # If we don't have a tag at all we print -g{commitish},
+ # again using exactly 12 hex chars.
else
+ head="$(echo $head | cut -c1-12)"
printf '%s%s' -g $head
fi
fi
Quoted 114 lines · Rasmus Villemoes
Quoted 87 lines · Masahiro Yamada
Quoted 9 lines · Rasmus Villemoes
Quoted 6 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes
Why do we care about the uniqueness of the abbreviated hash in the whole git history?
Quoted 7 lines
Because when I ask a customer "what kernel are you running", and they tell me "4.19.45-rt67-00567-123abc8", I prefer that 123abc8 uniquely identifies the right commit, instead of me having to dig around each of the commits starting with that prefix and see which one of them matches "is exactly 567 commits from 4.19.45-rt67". 7 hexchars is nowhere near enough for that today, which is why Linus himself did that "auto-tune abbrev based on repo size" patch for git.git.
I like:
git rev-parse --verify HEAD 2>/dev/null | cut -c1-15better than:
git rev-parse --verify --short=15 HEAD 2>/dev/nullThe former produces the deterministic kernelrelease string.
But, let's reconsider if we need as long as 15-digits.
There are a couple of kinds of objects in git: commit, tree, blob.
I think you came up with 15-digits to ensure the uniqueness in _all_ kinds of objects.
But, when your customer says "4.19.45-rt67-00567-123abc8", 123abc8 is apparently a commit instead of a tree or a blob.
In the context of the kernel version, we can exclude tree and blob objects.
In other words, I think "grows ~60000 objects per release" is somewhat over-estimating.
We have ~15000 commits per release. So, the difference is just 4x factor, though...
BTW, I did some experiments, and I noticed
"git log" accepts a shorter hash
than "git show" does presumably because
"git log" only takes a commit.For example, "git show 06a0d" fails, but
"git log 06a0d" works.masahiro@oscar:~/ref/linux$ git show 06a0d error: short SHA1 06a0d is ambiguous hint: The candidates are: hint: 06a0df4d1b8b commit 2020-09-08 - fbcon: remove now unusued 'softback_lines' cursor() argument hint: 06a0d81911b3 tree hint: 06a0dc5a84d2 tree hint: 06a0d1947c77 blob hint: 06a0df434249 blob fatal: ambiguous argument '06a0d': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' masahiro@oscar:~/ref/linux$ git log --oneline -1 06a0d 06a0df4d1b8b fbcon: remove now unusued 'softback_lines' cursor() argumentWhat is interesting is, if you prepend <tag>-<number-of-commits>- to the abbreviated hash, "git show" accepts as short as a commit "git log" does.
Quoted 8 lines
masahiro@oscar:~/ref/linux$ git show v5.9-rc5-00002-g06a0d | head -1 commit 06a0df4d1b8b13b551668e47b11fd7629033b7dfI guess git cleverly understands it refers to a commit object since "v5.9-rc5-00002-" prefix only makes sense
in the commit context.Well, yes, but unfortunately only so far as applying the same "the user means a commit object" logic; git doesn't do anything to actually use the prefix to disambiguate. In fact, looking at a semi-random commit in 4.19-stable v4.19.114-7-g236c445eb352:
$ git show 236c44 error: short SHA1 236c44 is ambiguous hint: The candidates are: hint: 236c445eb352 commit 2020-03-13 - drm/bochs: downgrade pci_request_region failure from error to warning hint: 236c448cb6e7 commit 2011-09-08 - usbmon vs. tcpdump: fix dropped packet count hint: 236c445b1930 blobNow you're right that we get
Quoted 8 lines
$ git show v4.19.114-7-g236c445 | head -n1 commit 236c445eb3529aa7c976f8812513c3cb26d77e27so it knows we're not looking at that blob. But "git show v4.19.114-7-g236c44" still fails because there are two commits. Adding to the fun is that "git show v4.19.114-7-g236c448" actually shows the usbmon commit, which is old v3.2 stuff and certainly not a descendant of v4.19.114.
Oh, I did not notice that.
Maybe git can be improved to check the validity of the 'git describe' form, but that is another story.
Quoted 35 lines
I didn't actually know that git would even accept those prefixed strings as possible refspecs, and for a moment you had me hoping that git would actually do the disambiguation using that prefix, which would certainly make 7 hex chars enough; unfortunately that's not the case.
Quoted 1 line
Keeping those above in mind, I believe 15-digits is too long.
Well, as you indicated, commits are probably ~1/4 of all objects, i.e. half a hexchar worth of bits. So the commit/object distinction shouldn't matter very much for the choice of suitable fixed length.
Quoted 15 lines
So, I propose two options.
[1] 7 digits
The abbreviated hash part may not uniquely identify the commit. In that case, you need some extra git operations to find out which one is it.
As for the kernel build, <kernel-version>-<number-of-commits>-<7-digits> is enough to provide the unique kernelrelease string.
Quoted 8 lines
[2] 12 digits
This matches to the Fixes: tag policy specified in
Documentation/process/submitting-patches.rstThe abbreviated hash part is very likely unique in the commit object space.
(Of course, it is impossible to guarantee the uniqueness)
Quoted 9 lines
I can live with 12. It would be extremely rare that I would have to do some extra git yoga to figure out which commit they actually mean. I think we can still use git describe to do the bulk of the work (the 'git rev-parse | cut ... is easy, but it's somewhat tedious to find the closest tag, then compute the #commits-on-top part), then just have the awk script used for pretty-printing (the %05d of the #commits-on-top) can probably also be used to chop the abbreviated sha1 at 12 chars, should git have needed to make it longer. Please see below for an updated patch+commit log.
LGTM.
Could you send it to ML as a patch?
Quoted 95 lines
Quoted 4 lines
Quoted 1 line
Alternatively, would you consider a
Kconfigknob,LOCALVERSION_ABBREV?No, I do not think
LOCALVERSION_ABBREVis a good idea.Neither do I; I considered it before sending the patch but decided that yes:
Quoted 2 lines
We should eliminate this problem
irrespective of the kernel configuration.Rasmus
====
something like this, then?
scripts/setlocalversion: make git describe output more reliableWhen building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.scripts/setlocalversion +12
-5@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.@@ -59,11 +59,22 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then - echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' - - # If we don't have a tag at all we print -g{commitish}. + # + # Ensure the abbreviated sha1 has exactly 12 + # hex characters, to make the output + # independent of git version, local + # core.abbrev settings and/or total number of + # objects in the current repository - passing + # --abbrev=12 ensures a minimum of 12, and the + # awk substr() then picks the 'g' and first 12 + # hex chars. + if atag="$(git describe --abbrev=12 2>/dev/null)"; then + echo "$atag" | awk -F- '{printf("-%05d-%s",$(NF-1),substr($(NF),0,13))}'
scripts/setlocalversion +4
-0+ + # If we don't have a tag at all we print -g{commitish}, + # again using exactly 12 hex chars. else + head="$(echo $head | cut -c1-12)" printf '%s%s' -g $head fi fi
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 62 lines · Masahiro Yamada
Quoted 34 lines · Rasmus Villemoes
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).Quoted 7 lines
Now, why is 60 bits enough for everyone? It's not mathematically guaranteed that git won't have to use 16 in some git repo, but it is beyond unlikely: Even in a repo with 100M objects, the probability that any given commit (i.e. the one being described) clashes with some other object in the first 15 hex chars is less than 1e-10, and currently a git repo tracking Linus', -stable and -rt only has around 10M objects.
I agree that any randomness should be avoided.
My question is, do we need 15-digits?
The kernelrelease is formed by
[kernel version] + [some digits of git hash].For example, "git describe" shows as follows:
v5.9.0-rc4-00034-g7fe10096c150
Linus gives a new tag every week (or every two week).
Quoted 8 lines
So, I think the conflict happens
only when we have two commits that start with the same 7-digits in the _same_ release. Is this correct?We have 14000 - 15000 commits in each release,
not 100M.
I kinda agree with this...we need to chopped down the excess bits from the information .
Indeed , a 15 digits is too long to keep up.
~Bhaskar
Quoted 44 lines
Quoted 39 lines
Signed-off-by Rasmus Villemoes <[email protected]>---
I could probably fix things by adding a 'git config --local core.abbrev 15' step to the Yocto build process after the repo to build from has been cloned but before building has started. But in the interest of binary reproducibility outside of just Yocto builds, I think it's better if this lives in the kernel.scripts/setlocalversion | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/setlocalversion +2
-3@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify --short=15 HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.@@ -59,7 +59,7 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then + if atag="$(git describe --abbrev=15 2>/dev/null)"; then echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' # If we don't have a tag at all we print -g{commitish}. --2.23.0
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 3 lines · Masahiro Yamada
Quoted 1 line · Rasmus Villemoes
...
Quoted 7 lines
Quoted 7 lines
So in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, use an explicit --abbrev=15 option (and for consistency, also use rev-parse --short=15 for the unlikely case of no signed tags being usable).
Hi Masahiro
Can I get you to carry this through the kbuild tree? Unless of course your questions/concerns haven't been addressed.
Thanks,
Rasmus
PATCH v2 1/1 scripts/setlocalversion: make git describe output more reliable
In reply to Rasmus VillemoesWhen building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under /lib/modules/ where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.
The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this git describe: The output of that can be affected by
(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfig
So in order to avoid uname -a output relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.
---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.
scripts/setlocalversion | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
scripts/setlocalversion +16 -5
@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.
@@ -59,11 +59,22 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then - echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' - - # If we don't have a tag at all we print -g{commitish}. + # + # Ensure the abbreviated sha1 has exactly 12 + # hex characters, to make the output + # independent of git version, local + # core.abbrev settings and/or total number of + # objects in the current repository - passing + # --abbrev=12 ensures a minimum of 12, and the + # awk substr() then picks the 'g' and first 12 + # hex chars. + if atag="$(git describe --abbrev=12 2>/dev/null)"; then + echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),substr($(NF),0,13))}' + + # If we don't have a tag at all we print -g{commitish}, + # again using exactly 12 hex chars. else + head="$(echo $head | cut -c1-12)" printf '%s%s' -g $head fi fi
Signature · 1 line
2.23.0
Thanks for the patch Rasmus. Overall it looks good to me, be aligned to the stable patch submission rules makes sense. A tiny thing though:
I did not calculate the exact collision probability with 12 characters and it does not make sense to even discuss this, if this is a current rule for stable patches. However we have a couple of 12's scattered in the code. And if the submission rule changes in the future, we should have a single location to update it.
So I suggest you introduce something on the line of:
...
num_chars=12
...
--abbrev=$num_chars
...
I guess you get the picture.
Greetings from the sunny mountains,
Nico
Quoted 73 lines · Rasmus Villemoes
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.Signed-off-by Rasmus Villemoes <[email protected]>---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.scripts/setlocalversion | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)scripts/setlocalversion +10
-5@@ -45,7 +45,7 @@ scm_version() # Check for git and a git repo. if test -z "$(git rev-parse --show-cdup 2>/dev/null)" && - head=$(git rev-parse --verify --short HEAD 2>/dev/null); then + head=$(git rev-parse --verify HEAD 2>/dev/null); then # If we are at a tagged commit (like "v2.6.30-rc6"), we ignore # it, because this version is defined in the top level Makefile.@@ -59,11 +59,22 @@ scm_version() fi # If we are past a tagged commit (like # "v2.6.30-rc5-302-g72357d5"), we pretty print it. - if atag="$(git describe 2>/dev/null)"; then - echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),$(NF))}' - - # If we don't have a tag at all we print -g{commitish}. + # + # Ensure the abbreviated sha1 has exactly 12 + # hex characters, to make the output + # independent of git version, local + # core.abbrev settings and/or total number of + # objects in the current repository - passing + # --abbrev=12 ensures a minimum of 12, and the + # awk substr() then picks the 'g' and first 12 + # hex chars.
scripts/setlocalversion scm_version() · line 80
Quoted 10 lines
scripts/setlocalversion +6
-0+ if atag="$(git describe --abbrev=12 2>/dev/null)"; then + echo "$atag" | awk -F- '{printf("-%05d-%s", $(NF-1),substr($(NF),0,13))}' + + # If we don't have a tag at all we print -g{commitish}, + # again using exactly 12 hex chars. else + head="$(echo $head | cut -c1-12)" printf '%s%s' -g $head fi fi
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Quoted 5 lines · Nico Schottelius
Thanks for the patch Rasmus. Overall it looks good to me, be aligned to the stable patch submission rules makes sense. A tiny thing though:
I did not calculate the exact collision probability with 12 characters
For reference, the math is something like this: Consider a repo with N+1 objects. We look at one specific object (for setlocalversion its the head commit being built, for the stable rules its whatever particular commit one is interested in for backporting), and want to know the probability that its sha1 collides with some other object in the first b bits (here b=48). Assuming the sha1s are independent and uniformly distributed, the probability of not colliding with one specific other commit is x=1-1/2^b, and the probability of not colliding with any of the other N commits is x^N, making the probability of a collision 1-x^N = (1-x)(1+x+x^2+...+x^{N-1}). Now the N terms in the second factor are very-close-to-but-slightly-smaller-than 1, so an upper bound for this probability is (1-x)N = N/2^b, which is also what one would naively expect. [This estimate is always valid, but it becomes a void statement of "the probability is less then 1" when N is >= 2^b].
So, assuming some vendor kernel repo that has all of Greg's stable.git (around 10M objects I think) and another 10M objects because random vendor, that works out to 20e6/2^48 = 7.1e-8, 71 ppb.
Quoted 6 lines
So I suggest you introduce something on the line of:
...
num_chars=12
...
--abbrev=$num_chars
I considered that, but it becomes quite ugly since it needs to get into the awk script (as a 13, though perhaps we could get awk to do the +1, I don't really speak awk), where we'd then need to use " instead of ' and then escape the $ that are to be interpreted by awk and not the shell. So I think it's more readable with hardcoding and comments explaining why they are there; should anyone ever want to change 12.
Rasmus
Hey Rasmus,
Quoted 11 lines · Rasmus Villemoes
Quoted 6 lines
So I suggest you introduce something on the line of:
...
num_chars=12
...
--abbrev=$num_charsI considered that, but it becomes quite ugly since it needs to get into the awk script (as a 13, though perhaps we could get awk to do the +1, I don't really speak awk), where we'd then need to use " instead of ' and then escape the $ that are to be interpreted by awk and not the shell.
No need for that, awk can read (and use) environment variables...
Quoted 2 lines
So I think it's more readable with hardcoding and comments explaining why they are there; should anyone ever want to change 12.
... so that in practice you only need to change 13 to var+1 and 12 to $var.
Cheers,
Nico
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
Quoted 36 lines · Rasmus Villemoes
When building for an embedded target using Yocto, we're sometimes observing that the version string that gets built into vmlinux (and thus what uname -a reports) differs from the path under
/lib/modules/where modules get installed in the rootfs, but only in the length of the -gabc123def suffix. Hence modprobe always fails.The problem is that Yocto has the concept of "sstate" (shared state), which allows different developers/buildbots/etc. to share build artifacts, based on a hash of all the metadata that went into building that artifact - and that metadata includes all dependencies (e.g. the compiler used etc.). That normally works quite well; usually a clean build (without using any sstate cache) done by one developer ends up being binary identical to a build done on another host. However, one thing that can cause two developers to end up with different builds [and thus make one's vmlinux package incompatible with the other's kernel-dev package], which is not captured by the metadata hashing, is this
git describe: The output of that can be affected by(1) git version: before 2.11 git defaulted to a minimum of 7, since 2.11 (git.git commit e6c587) the default is dynamic based on the number of objects in the repo
(2) hence even if both run the same git version, the output can differ based on how many remotes are being tracked (or just lots of local development branches or plain old garbage)
(3) and of course somebody could have a core.abbrev config setting in ~/.gitconfigSo in order to avoid
uname -aoutput relying on such random details of the build environment which are rather hard to ensure are consistent between developers and buildbots, make sure the abbreviated sha1 always consists of exactly 12 hex characters. That is consistent with the current rule for -stable patches, and is almost always enough to identify the head commit unambigously - in the few cases where it does not, the v5.4.3-00021- prefix would certainly nail it down.
Quoted 7 lines
Signed-off-by Rasmus Villemoes <[email protected]>---
v2: use 12 instead of 15, and ensure that the result does have exactly 12 hex chars.scripts/setlocalversion | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
Applied to linux-kbuild.
Thanks.
Signature · 2 lines
Best Regards Masahiro Yamada