Skip to contentSkip to conversation

support/download/svn: generate reproducible svn archives

PATCH New Heiko Thiery 5 participants 12 messages Dec 22, 2019 – May 22, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

To generate a reproducible archive from a svn repository mainly the same aproach is done like for the archives from a git repository.

---
 support/download/svn | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
support/download/svn +16 -1
index 542b25c0a2..505cdd58b1 100755
@@ -38,4 +38,19 @@ _svn() {
 
 _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
 
-tar czf "${output}" "${basename}"
+# Generate the archive, sort with the C locale so that it is reproducible.
+# We do not want the .svn dir; we keep other .svn files, in case they are the
+# only files in their directory.
+find "${basename}" -not -type d \
+       -and -not -path "./.svn/*" >"${output}.list"
+LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
+
+# Create GNU-format tarballs, since that's the format of the tarballs on
+# sources.buildroot.org and used in the *.hash files
+tar cf - --transform="s#^\./#${basename}/#" \
+         --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+         -T "${output}.list.sorted" >"${output}.tar"
+gzip -6 -n <"${output}.tar" >"${output}"
+
+rm -f "${output}.list"
+rm -f "${output}.list.sorted"
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Quoted 4 lines · On Sun, 22 Dec 2019 22:31:49 +0100 Heiko Thiery <[email protected]> wrote:

To generate a reproducible archive from a svn repository mainly the same aproach is done like for the archives from a git repository.

Thanks. Could you check whether the tarballs that are now produced by this are identical to the tarballs we already have on sources.buildroot.org ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Hi,

Am So., 22. Dez. 2019 um 22:36 Uhr schrieb Thomas Petazzoni <[email protected]>:

Quoted 4 lines

On Sun, 22 Dec 2019 22:31:49 +0100
Heiko Thiery <[email protected]> wrote:

Quoted 8 lines
Quoted 4 lines

To generate a reproducible archive from a svn repository mainly the same aproach is done like for the archives from a git repository.

Thanks. Could you check whether the tarballs that are now produced by this are identical to the tarballs we already have on sources.buildroot.org ?

I checked this for the fis package and unfortunately this is not the case. Do you expect to have the same hashes?

Quoted 5 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#3 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Hi,

Quoted 6 lines
Quoted 3 lines

Thanks. Could you check whether the tarballs that are now produced by this are identical to the tarballs we already have on sources.buildroot.org ?

I checked this for the fis package and unfortunately this is not the case. Do you expect to have the same hashes?

this is how the archive from sources.buildroot.org looks:

tar tvf fis-2892.tar.gz
drwxr-xr-x peko/peko 0 2014-02-09 12:26 fis-2892/
-rw-r--r-- peko/peko 1699 2007-01-18 17:42 fis-2892/endswap.cc
-rwxr-xr-x peko/peko 4229 2006-12-13 22:30 fis-2892/disklog

-rwxr-xr-x peko/peko       532 2004-07-13 22:56 fis-2892/cvstotbz2
-rwxr-xr-x peko/peko       451 2004-04-19 14:04 fis-2892/overwrite
-rwxr-xr-x peko/peko       196 2007-03-17 01:24 fis-2892/write_dvd

-rw-r--r-- peko/peko 15659 2007-05-02 22:42 fis-2892/fis.c
-rw-r--r-- peko/peko 6507 2007-01-28 14:55 fis-2892/apexctl.cc
-rw-r--r-- peko/peko 348 2007-03-17 01:24 fis-2892/Makefile
-rw-r--r-- peko/peko 1858 2007-01-23 18:42 fis-2892/sercommhdr.cc
-rwxr-xr-x peko/peko 534 2004-07-13 22:56 fis-2892/cvstotgz
-rwxr-xr-x peko/peko 1021 2006-12-13 22:30 fis-2892/report_disklog
-rw-r--r-- peko/peko 9956 2007-03-25 01:14 fis-2892/fis.cc
-rwxr-xr-x peko/peko 372 2004-01-25 16:04 fis-2892/cvsmv
-rwxr-xr-x peko/peko 402 2005-10-31 19:17 fis-2892/svntotbz2

and this is the generated on;

tar tvf fis-2892.tar.gz
-rw-r--r-- 0/0 348 2019-12-22 00:00 fis-2892/Makefile
-rw-r--r-- 0/0 6507 2019-12-22 00:00 fis-2892/apexctl.cc

-rwxr-xr-x 0/0             372 2019-12-22 00:00 fis-2892/cvsmv
-rwxr-xr-x 0/0             532 2019-12-22 00:00 fis-2892/cvstotbz2
-rwxr-xr-x 0/0             534 2019-12-22 00:00 fis-2892/cvstotgz

-rwxr-xr-x 0/0 4229 2019-12-22 00:00 fis-2892/disklog
-rw-r--r-- 0/0 1699 2019-12-22 00:00 fis-2892/endswap.cc
-rw-r--r-- 0/0 15659 2019-12-22 00:00 fis-2892/fis.c
-rw-r--r-- 0/0 9956 2019-12-22 00:00 fis-2892/fis.cc
-rwxr-xr-x 0/0 451 2019-12-22 00:00 fis-2892/overwrite
-rwxr-xr-x 0/0 1021 2019-12-22 00:00 fis-2892/report_disklog
-rw-r--r-- 0/0 1858 2019-12-22 00:00 fis-2892/sercommhdr.cc
-rwxr-xr-x 0/0 402 2019-12-22 00:00 fis-2892/svntotbz2
-rwxr-xr-x 0/0 196 2019-12-22 00:00 fis-2892/write_dvd

The diffrerences I see are:
- the s.b.o archive has an entry for the directory
- the s.b.o archive has the user/group owner peko while the generated has 0/0
- the file ordering is different

--
Heiko

#4 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

In reply to Heiko Thiery
Quoted 6 lines · On Sun, 22 Dec 2019 22:40:43 +0100 Heiko Thiery <[email protected]> wrote:
Quoted 3 lines

Thanks. Could you check whether the tarballs that are now produced by this are identical to the tarballs we already have on sources.buildroot.org ?

I checked this for the fis package and unfortunately this is not the case. Do you expect to have the same hashes?

Probably not, because with your change we now generate the tarballs differently.

Normally, it is annoying because it means the hash has changed, breaking the build for older Buildroot users, if we update the tarballs on sources.buildroot.org.

However, in this case, current Buildroot does not have any hash (as far as I can see) for Subversion-fetched packages. So we could introduce your change and update the tarballs on sources.buildroot.org at the same time, and then introduce hashes in those packages.

Peter, Yann, what do you think ?

We have only very few Subversion-fetched packages, I think we should keep it simple.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Thomas, Heiko, All,

On 2019-12-22 22:57 +0100, Thomas Petazzoni spake thusly:

Quoted 17 lines
Quoted 6 lines · On Sun, 22 Dec 2019 22:40:43 +0100 Heiko Thiery <[email protected]> wrote:
Quoted 3 lines

Thanks. Could you check whether the tarballs that are now produced by this are identical to the tarballs we already have on sources.buildroot.org ?

I checked this for the fis package and unfortunately this is not the case. Do you expect to have the same hashes?

Probably not, because with your change we now generate the tarballs differently.

Normally, it is annoying because it means the hash has changed, breaking the build for older Buildroot users, if we update the tarballs on sources.buildroot.org.

Quoted 6 lines

However, in this case, current Buildroot does not have any hash (as far as I can see) for Subversion-fetched packages. So we could introduce your change and update the tarballs on sources.buildroot.org at the same time, and then introduce hashes in those packages.

Peter, Yann, what do you think ?

I am 100% on-line with regenerating the tarballs so they are reproducible, even if that means updating s.b.o.

Quoted 2 lines

We have only very few Subversion-fetched packages, I think we should keep it simple.

In practice, we have only two: fis and open2300. The other packages that may use svn are those where the user would set the version, so they would anyway be excluded from the hash check.

Regards,
Yann E. MORIN.

Quoted 5 lines

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Hi,

Quoted 9 lines
Quoted 9 lines

Normally, it is annoying because it means the hash has changed, breaking the build for older Buildroot users, if we update the tarballs on sources.buildroot.org.

However, in this case, current Buildroot does not have any hash (as far as I can see) for Subversion-fetched packages. So we could introduce your change and update the tarballs on sources.buildroot.org at the same time, and then introduce hashes in those packages.

Quoted 4 lines
Quoted 1 line

Peter, Yann, what do you think ?

I am 100% on-line with regenerating the tarballs so they are reproducible, even if that means updating s.b.o.

As long as it doesn't break anything for existing users, regenerating the tarballs are also fine by me.

Quoted 6 lines
Quoted 2 lines

We have only very few Subversion-fetched packages, I think we should keep it simple.

In practice, we have only two: fis and open2300. The other packages that may use svn are those where the user would set the version, so they would anyway be excluded from the hash check.

Talking about those two packages, weren't we going to remove them? They are both very old and haven't been updated since they were added.

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Hello,

+Alex in Cc. Alex, there's an open2300 question for you below.

Quoted 6 lines · On Mon, 23 Dec 2019 23:05:34 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 3 lines

In practice, we have only two: fis and open2300. The other packages that may use svn are those where the user would set the version, so they would anyway be excluded from the hash check.

Talking about those two packages, weren't we going to remove them? They are both very old and haven't been updated since they were added.

That's indeed another way. For the "fis" package, I think this is reasonable, as even the use-case is very obsolete (RedBoot stuff). For open2300, I'm not sure.

Alex: is open2300 still relevant today? You introduced this package in Buildroot many years ago.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hello,
+Alex in Cc. Alex, there's an open2300 question for you below.

As far as I can see, you didn't actually put Alex in CC.

Quoted 16 lines
Quoted 6 lines · On Mon, 23 Dec 2019 23:05:34 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 3 lines

In practice, we have only two: fis and open2300. The other packages that may use svn are those where the user would set the version, so they would anyway be excluded from the hash check.

Talking about those two packages, weren't we going to remove them? They are both very old and haven't been updated since they were added.

That's indeed another way. For the "fis" package, I think this is reasonable, as even the use-case is very obsolete (RedBoot stuff). For open2300, I'm not sure.

Alex: is open2300 still relevant today? You introduced this package in Buildroot many years ago.

Signature · 1 line
Bye, Peter Korsgaard
#9 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

Hi,

Quoted 3 lines · Peter Korsgaard
Quoted 2 lines

Alex: is open2300 still relevant today? You introduced this package in Buildroot many years ago.

I'm not personally using it at the moment as I now have an x86 box connected to the weather station and I compiled open2300 on it directly. I'm fine with the removal.

Signature · 3 lines
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

In reply to Heiko Thiery

Heiko, All,

On 2019-12-22 22:31 +0100, Heiko Thiery spake thusly:

Quoted 18 lines

To generate a reproducible archive from a svn repository mainly the same aproach is done like for the archives from a git repository.

---
 support/download/svn | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
support/download/svn +1 -1
index 542b25c0a2..505cdd58b1 100755
@@ -38,4 +38,19 @@ _svn() {
 
 _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
 
-tar czf "${output}" "${basename}"
+# Generate the archive, sort with the C locale so that it is reproducible.

support/download/svn _svn() { · line 51

Quoted 10 lines
support/download/svn +10 -0
+# We do not want the .svn dir; we keep other .svn files, in case they are the
+# only files in their directory.
+find "${basename}" -not -type d \
+       -and -not -path "./.svn/*" >"${output}.list"
+LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
+
+# Create GNU-format tarballs, since that's the format of the tarballs on
+# sources.buildroot.org and used in the *.hash files
+tar cf - --transform="s#^\./#${basename}/#" \
+         --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \

Where does "${date}" comes from? Nothing is setting it...

So I've added some code to that effect, and pushed to master, thanks.

Regards,
Yann E. MORIN.

Quoted 8 lines
support/download/svn +5 -0
+         -T "${output}.list.sorted" >"${output}.tar"
+gzip -6 -n <"${output}.tar" >"${output}"
+
+rm -f "${output}.list"
+rm -f "${output}.list.sorted"
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#11 lore raw

PATCH 1/1 support/download/svn: generate reproducible svn archives

In reply to Heiko Thiery

Heiko, All,

On 2019-12-22 22:31 +0100, Heiko Thiery spake thusly:

Quoted 4 lines

To generate a reproducible archive from a svn repository mainly the same aproach is done like for the archives from a git repository.

That patch has been long applied now, but I just noticed that we forgot to enable hash checks for svn tarbals:

https://git.buildroot.org/buildroot/tree/package/pkg-generic.mk#n580

I'll take it on me to fix that, as I'm already working in this area...

Regards,
Yann E. MORIN.

Quoted 32 lines
---
 support/download/svn | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
support/download/svn +16 -1
index 542b25c0a2..505cdd58b1 100755
@@ -38,4 +38,19 @@ _svn() {
 
 _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
 
-tar czf "${output}" "${basename}"
+# Generate the archive, sort with the C locale so that it is reproducible.
+# We do not want the .svn dir; we keep other .svn files, in case they are the
+# only files in their directory.
+find "${basename}" -not -type d \
+       -and -not -path "./.svn/*" >"${output}.list"
+LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
+
+# Create GNU-format tarballs, since that's the format of the tarballs on
+# sources.buildroot.org and used in the *.hash files
+tar cf - --transform="s#^\./#${basename}/#" \
+         --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+         -T "${output}.list.sorted" >"${output}.tar"
+gzip -6 -n <"${output}.tar" >"${output}"
+
+rm -f "${output}.list"
+rm -f "${output}.list.sorted"
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'