Skip to contentSkip to conversation

suport/download: fix git wrapper with submodules on older git versions

PATCH New Yann E. MORIN 3 participants 6 messages May 24, 2020 – May 30, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

Older versions of git store the absolute path of the submodules' repository as stored in the super-prject, e.g.:

    $ cat some-submodule/.git
    gitdir: /path/to/super-project/.git/modules/some-submodule

Obviously, this is not very reproducible.

More recent versions of git, however, store relative paths, shich de-facto makes it reproducible.

Fix older versions by replacing the absolute paths with relative ones.

---
 support/download/git | 13 +++++++++++++
 1 file changed, 13 insertions(+)
support/download/git +13 -0
index 075f665bbf..15d8c66e05 100755
@@ -176,6 +176,19 @@ date="$( _git log -1 --pretty=format:%cD )"
 # There might be submodules, so fetch them.
 if [ ${recurse} -eq 1 ]; then
     _git submodule update --init --recursive
+
+    # Older versions of git will store the absolute path of the git tree
+    # in the .git of submodules, while newer versions just use relative
+    # paths. Detect and fix the older variants to use relative paths, so
+    # that the archives are reproducible across a wider range of git
+    # versions. However, we can't do that if git is too old and uses
+    # full repositories for submodules.
+    cmd='printf "%s\n" "${path}/"'
+    for module_dir in $( _git submodule --quiet foreach "'${cmd}'" ); do
+        [ -f "${module_dir}/.git" ] || continue
+        relative_dir="$( sed -r -e 's,/+,/,g; s,[^/]+/,../,g' <<<"${module_dir}" )"
+        sed -r -i -e "s:^gitdir\: $(pwd)/:gitdir\: "${relative_dir}":" "${module_dir}/.git"
+    done
 fi
 
 # Generate the archive, sort with the C locale so that it is reproducible.
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

Yann,

Quoted 32 lines · Yann E. MORIN

Older versions of git store the absolute path of the submodules' repository as stored in the super-prject, e.g.:

     $ cat some-submodule/.git
     gitdir: /path/to/super-project/.git/modules/some-submodule

Obviously, this is not very reproducible.

More recent versions of git, however, store relative paths, shich de-facto makes it reproducible.

Fix older versions by replacing the absolute paths with relative ones.

---
  support/download/git | 13 +++++++++++++
  1 file changed, 13 insertions(+)
support/download/git +6 -0
index 075f665bbf..15d8c66e05 100755
@@ -176,6 +176,19 @@ date="$( _git log -1 --pretty=format:%cD )"
  # There might be submodules, so fetch them.
  if [ ${recurse} -eq 1 ]; then
      _git submodule update --init --recursive
+
+    # Older versions of git will store the absolute path of the git tree
+    # in the .git of submodules, while newer versions just use relative
+    # paths. Detect and fix the older variants to use relative paths, so
+    # that the archives are reproducible across a wider range of git
+    # versions. However, we can't do that if git is too old and uses

support/download/git date="$( _git log -1 --pretty=format:%cD )" · line 194

Quoted 10 lines
support/download/git +7 -0
+    # full repositories for submodules.
+    cmd='printf "%s\n" "${path}/"'
+    for module_dir in $( _git submodule --quiet foreach "'${cmd}'" ); do
+        [ -f "${module_dir}/.git" ] || continue
+        relative_dir="$( sed -r -e 's,/+,/,g; s,[^/]+/,../,g' <<<"${module_dir}" )"
+        sed -r -i -e "s:^gitdir\: $(pwd)/:gitdir\: "${relative_dir}":" "${module_dir}/.git"
+    done
  fi
 
  # Generate the archive, sort with the C locale so that it is reproducible.

Should we expand the find to ignore files named '.git' so that these don't get added to the tarball at all?

find . -not -type d \
?????? -and -not -path "./.git/*" -and -not -name ".git" >"${output}.list"

Seems like it'd be in-line with our current exclusion of the .git/ subfolder because that relative git reference wouldn't be valid after the tarball got unpacked anyway.

Signature · 8 lines
Vincent Fazio
Embedded Software Engineer - Linux
Extreme Engineering Solutions, Inc
http://www.xes-inc.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200524/478bbddc/attachment.html>
#2 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

Vincent, All,

On 2020-05-24 20:55 -0500, Vincent Fazio spake thusly:

Quoted 26 lines · Yann E. MORIN

Older versions of git store the absolute path of the submodules' repository as stored in the super-prject, e.g.:

     $ cat some-submodule/.git    gitdir: /path/to/super-project/.git/modules/some-submodule
 Obviously, this is not very reproducible.
 More recent versions of git, however, store relative paths, shich
 de-facto makes it reproducible.Fix older versions by replacing the absolute paths with relative ones.
 Signed-off-by: Yann E. MORIN
 [1]<[email protected]>--- support/download/git | 13 +++++++++++++
  1 file changed, 13 insertions(+)diff --git a/support/download/git b/support/download/git
 index 075f665bbf..15d8c66e05 100755--- a/support/download/git
 +++ b/support/download/git@@ -176,6 +176,19 @@ date="$( _git log -1 --pretty=format:%cD )"
  # There might be submodules, so fetch them.
  if [ ${recurse} -eq 1 ]; then     _git submodule update --init --recursive
 ++    # Older versions of git will store the absolute path of the git tree
 +    # in the .git of submodules, while newer versions just use relative
 +    # paths. Detect and fix the older variants to use relative paths, so
 +    # that the archives are reproducible across a wider range of git
 +    # versions. However, we can't do that if git is too old and uses
 +    # full repositories for submodules.+    cmd='printf "%s\n" "${path}/"'
 +    for module_dir in $( _git submodule --quiet foreach "'${cmd}'" ); do
 +        [ -f "${module_dir}/.git" ] || continue
 +        relative_dir="$( sed -r -e 's,/+,/,g; s,[^/]+/,../,g' <<<"${module_dir}" )"
 +        sed -r -i -e "s:^gitdir\: $(pwd)/:gitdir\: "${relative_dir}":" "${module_dir}/.git"
 +    done fi  # Generate the archive, sort with the C locale so that it is reproducible.
Quoted 4 lines

Should we expand the find to ignore files named '.git' so that these don't get added to the tarball at all?

find . -not -type d \
?????? -and -not -path "./.git/*" -and -not -name ".git" >"${output}.list"

We do not want to do tht, because we want to reproduce the existign tarballs. And those existign tarballs already contain the .git files.

Note however that, for people wo have prehistoric git versions, git submodulkes will be entire repositories of their own, i.e. the .git of submodules is a directory with an actual repository, instead of a plain file with a gitdir indirection. For those people, tarballs from git archives are not reproducible but we don't care.

But for the case that concerns us, we don't want to drop the .git files.

Yeah, that might be an oversight from back when we introduced support for submodules, but it's now too late...

Regards,
Yann E. MORIN.

Quoted 10 lines

Seems like it'd be in-line with our current exclusion of the .git/ subfolder because that relative git reference wouldn't be valid after the tarball got unpacked anyway.

-- Vincent FazioEmbedded Software Engineer - Linux Extreme Engineering Solutions, Inc [2]http://www.xes-inc.com

Links:
1. mailto:yann.morin.1998 at free.fr/
2. http://www.xes-inc.com/

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

Yann,

Quoted 29 lines · Yann E. MORIN

Vincent, All,

On 2020-05-24 20:55 -0500, Vincent Fazio spake thusly:

Quoted 26 lines · Yann E. MORIN

Older versions of git store the absolute path of the submodules' repository as stored in the super-prject, e.g.:

      $ cat some-submodule/.git    gitdir: /path/to/super-project/.git/modules/some-submodule
  Obviously, this is not very reproducible.
  More recent versions of git, however, store relative paths, shich
  de-facto makes it reproducible.Fix older versions by replacing the absolute paths with relative ones.
  Signed-off-by: Yann E. MORIN
  [1]<[email protected]>--- support/download/git | 13 +++++++++++++
   1 file changed, 13 insertions(+)diff --git a/support/download/git b/support/download/git
  index 075f665bbf..15d8c66e05 100755--- a/support/download/git
  +++ b/support/download/git@@ -176,6 +176,19 @@ date="$( _git log -1 --pretty=format:%cD )"
   # There might be submodules, so fetch them.
   if [ ${recurse} -eq 1 ]; then     _git submodule update --init --recursive
  ++    # Older versions of git will store the absolute path of the git tree
  +    # in the .git of submodules, while newer versions just use relative
  +    # paths. Detect and fix the older variants to use relative paths, so
  +    # that the archives are reproducible across a wider range of git
  +    # versions. However, we can't do that if git is too old and uses
  +    # full repositories for submodules.+    cmd='printf "%s\n" "${path}/"'
  +    for module_dir in $( _git submodule --quiet foreach "'${cmd}'" ); do
  +        [ -f "${module_dir}/.git" ] || continue
  +        relative_dir="$( sed -r -e 's,/+,/,g; s,[^/]+/,../,g' <<<"${module_dir}" )"
  +        sed -r -i -e "s:^gitdir\: $(pwd)/:gitdir\: "${relative_dir}":" "${module_dir}/.git"
  +    done fi  # Generate the archive, sort with the C locale so that it is reproducible.
Quoted 8 lines
Quoted 4 lines

Should we expand the find to ignore files named '.git' so that these don't get added to the tarball at all?

find . -not -type d \
?????? -and -not -path "./.git/*" -and -not -name ".git" >"${output}.list"

We do not want to do tht, because we want to reproduce the existign tarballs. And those existign tarballs already contain the .git files.

Gotcha. In the case that this is a one-off patch and may be ported, I completely agree. I was thinking of this in terms of the PAX patch series we've been discussing via IRC

Quoted 11 lines

Note however that, for people wo have prehistoric git versions, git submodulkes will be entire repositories of their own, i.e. the .git of submodules is a directory with an actual repository, instead of a plain file with a gitdir indirection. For those people, tarballs from git archives are not reproducible but we don't care.

But for the case that concerns us, we don't want to drop the .git files.

Yeah, that might be an oversight from back when we introduced support for submodules, but it's now too late...

However, when we introduce the new PAX tarball, i think we should consider dropping all directories and files named '.git'...

Sure it was an oversight before with the GNU tarballs, but we know it's a problem and i don't think we need to carry it forward. Or is there a detail I'm missing here as well?

Quoted 14 lines

Regards,
Yann E. MORIN.

Quoted 10 lines

Seems like it'd be in-line with our current exclusion of the .git/ subfolder because that relative git reference wouldn't be valid after the tarball got unpacked anyway.

-- Vincent FazioEmbedded Software Engineer - Linux Extreme Engineering Solutions, Inc [2]http://www.xes-inc.com

Links:
1. mailto:yann.morin.1998 at free.fr/
2. http://www.xes-inc.com/

Signature · 4 lines
Vincent Fazio
Embedded Software Engineer - Linux
Extreme Engineering Solutions, Inc
http://www.xes-inc.com
#4 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

In reply to Yann E. MORIN

support/download/git date="$( _git log -1 --pretty=format:%cD )" · line 185

Quoted 6 lines · On Sun, 24 May 2020 13:47:18 +0200 "Yann E. MORIN" <[email protected]>…
support/download/git +6 -0
+    # Older versions of git will store the absolute path of the git tree
+    # in the .git of submodules, while newer versions just use relative
+    # paths. Detect and fix the older variants to use relative paths, so
+    # that the archives are reproducible across a wider range of git
+    # versions. However, we can't do that if git is too old and uses
+    # full repositories for submodules.

If I understand correctly, there are three "eras":

- Really old Git versions, where full repositories are used for submodules, where we can't do anything.

- Old Git versions, that stored absolute paths.

- Recent Git versions, that store relative paths.

Would it be possible to identify which versions we're talking about here? I'm sure you've done that research, and I think it makes sense to capture that, as we will certainly wonder what we mean by "older versions", "old version", "new version.

What is new, old, or older today, will feel quite different 5 years from now.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/1 suport/download: fix git wrapper with submodules on older git versions

Thomas, All,

On 2020-05-29 23:41 +0200, Thomas Petazzoni spake thusly:

Quoted 9 lines

support/download/git date="$( _git log -1 --pretty=format:%cD )" · line 185

Quoted 6 lines · On Sun, 24 May 2020 13:47:18 +0200 "Yann E. MORIN" <[email protected]>…
support/download/git +6 -0
+    # Older versions of git will store the absolute path of the git tree
+    # in the .git of submodules, while newer versions just use relative
+    # paths. Detect and fix the older variants to use relative paths, so
+    # that the archives are reproducible across a wider range of git
+    # versions. However, we can't do that if git is too old and uses
+    # full repositories for submodules.
Quoted 8 lines

If I understand correctly, there are three "eras":

- Really old Git versions, where full repositories are used for submodules, where we can't do anything.

- Old Git versions, that stored absolute paths.

- Recent Git versions, that store relative paths.

Spot-on.

Quoted 7 lines

Would it be possible to identify which versions we're talking about here? I'm sure you've done that research, and I think it makes sense to capture that, as we will certainly wonder what we mean by "older versions", "old version", "new version.

What is new, old, or older today, will feel quite different 5 years from now.

Sorry, you presumed too much: I haven't dug the nitty-gritty details on when git transitioned from one area to another...

I just stumbled on this issue while working on the conversion of the archives generated from a git tree, which got me scratch my head for quite some time... I should have noted the conditions back then, true, but I forgot...

Regards,
Yann E. MORIN.

Quoted 5 lines

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'