download/cvs: add a 10 minute timeout

PATCH New Arnout Vandecappelle (Essensium/Mind) 4 participants 9 messages May 31, 2018 – Jun 17, 2018

Share & tools

Arnout Vandecappelle (Essensium/Mind)

author

2018-05-31 07:05 UTC #0 lore raw

PATCH 1/2 download/cvs: add a 10 minute timeout

Apparently, CVS servers can be deadlocked and in that case clients will retry connecting to them indefinitely. Cfr.
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78/build-end.log
Apparently, the sf.net CVS server got in such a deadlock on 2018-05-18, and almost 2 weeks later it is still not fixed.

Instead of just hanging, we should fall back on BR2_SECONDARY_SITE. To achieve this, it's sufficient to add a timeout to the CVS command.

The timeout value is of course arbitrary. However, we can assume that nobody will be putting large projects under CVS any more. So if the download takes more than 5 minutes, it's probably broken. Let's put the timeout at 10 minutes then.

Fixes:
http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105
http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78
http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
 support/download/cvs | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

support/download/cvs +6 -2

index 3f77b849e4..9d0dc3cb3a 100755
@@ -33,9 +33,13 @@ done
 shift $((OPTIND-1)) # Get rid of our options
 
 # Caller needs to single-quote its arguments to prevent them from
-# being expanded a second time (in case there are spaces in them)
+# being expanded a second time (in case there are spaces in them).
+# If the CVS server is deadlocked, the client will never return (cfr.
+# http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78/build-end.log
+# ). Since nobody sane will put large code bases in CVS, a timeout of
+# 10 minutes should do the trick.
 _cvs() {
-    eval ${CVS} "${@}"
+    eval timeout 10m ${CVS} "${@}"
 }
 
 if [[ ${rev} =~ ^[0-9] ]]; then

Signature · 1 line

2.17.1

Arnout Vandecappelle (Essensium/Mind)

author

2018-05-31 07:05 UTC #1 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

More than 3 years after the 5.45.3 release, a tarball for that release was uploaded to sourceforge.net. The differences between this tarball and the CVS checkout are minimal:

- There are no CVS directories, of course, but we don't need them.
- File timestamps are different.
- expect.tests is missing, but we don't execute tests anyway.
- configure script is different, but we AUTORECONF anyway.
- 'fixcat' script is missing, but it is not called anywhere.

Since sourceforge.net has broken CVS downloads, now is a good time to switch away from it.

While we're at it, add a hash file including license.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Cc Fabio Porcedda <[email protected]>

---
Either this one or the previous one is sufficient to fix the issue, of course. Still, I think both are relevant for master.

Note that there is also a 5.45.4 release from a few months ago. But I don't know if anyone still uses expect :-).

Note that this was the last internal package to use CVS. So we no longer have autobuild coverage of the cvs download method.

---
 package/expect/expect.hash | 4 ++++
 package/expect/expect.mk   | 7 ++++---
 2 files changed, 8 insertions(+), 3 deletions(-)
 create mode 100644 package/expect/expect.hash

package/expect/expect.hash new file +4 -0

new file mode 100644
index 0000000000..c89f323906
@@ -0,0 +1,4 @@
+# From https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz.SHA256
+sha256  c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9  expect5.45.3.tar.gz
+# Locally calculated
+sha256  b2415b17dc8d9a287f4509047ef5ac3436baef7ba7c50faef5222dcdf61a2bab  README

package/expect/expect.mk +4 -3

index d78ff6c208..ccde33a71b 100644
@@ -4,12 +4,13 @@
 #
 ################################################################################
 
-# Version 5.45.3
-EXPECT_VERSION = 2014-05-02
-EXPECT_SITE = cvs://expect.cvs.sourceforge.net:/cvsroot/expect
+EXPECT_VERSION = 5.45.3
+EXPECT_SITE = https://sourceforge.net/projects/expect/files/Expect/$(EXPECT_VERSION)
+EXPECT_SOURCE = expect$(EXPECT_VERSION).tar.gz
 EXPECT_LICENSE = Public domain
 EXPECT_LICENSE_FILES = README
 
+# 0001-enable-cross-compilation.patch
 EXPECT_AUTORECONF = YES
 EXPECT_DEPENDENCIES = tcl
 EXPECT_CONF_OPTS = --with-tcl=$(BUILD_DIR)/tcl-$(TCL_VERSION)/unix

Signature · 1 line

2.17.1

Peter Korsgaard

2018-05-31 17:38 UTC #2 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

Quoted 29 lines

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle (Essensium/Mind) <[email protected]> writes:
More than 3 years after the 5.45.3 release, a tarball for that release was uploaded to sourceforge.net. The differences between this tarball and the CVS checkout are minimal:
- There are no CVS directories, of course, but we don't need them.
- File timestamps are different.
- expect.tests is missing, but we don't execute tests anyway.
- configure script is different, but we AUTORECONF anyway.
- 'fixcat' script is missing, but it is not called anywhere.
Since sourceforge.net has broken CVS downloads, now is a good time to switch away from it.
While we're at it, add a hash file including license.
Fixes:
http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105
http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78
http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Fabio Porcedda <[email protected]>
---
Either this one or the previous one is sufficient to fix the issue, of course. Still, I think both are relevant for master.

Quoted 5 lines

Note that there is also a 5.45.4 release from a few months ago. But I don't know if anyone still uses expect :-).
Note that this was the last internal package to use CVS. So we no longer have autobuild coverage of the cvs download method.

True, but with so few upstream projects using CVS that was bound to happen sooner or later.

Committed, thanks.

Signature · 1 line

Bye, Peter Korsgaard

Thomas Petazzoni

2018-06-01 09:09 UTC #3 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

In reply to Arnout Vandecappelle (Essensium/Mind)

Hello,

Quoted 16 lines · Arnout Vandecappelle (Essensium/Mind)

package/expect/expect.hash new file +4 -0

new file mode 100644
index 0000000000..c89f323906
@@ -0,0 +1,4 @@
+# From https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz.SHA256
+sha256  c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9  expect5.45.3.tar.gz
+# Locally calculated
+sha256  b2415b17dc8d9a287f4509047ef5ac3436baef7ba7c50faef5222dcdf61a2bab  README

package/expect/expect.mk +0 -0

index d78ff6c208..ccde33a71b 100644
@@ -4,12 +4,13 @@
 #

package/expect/expect.mk line 11

Quoted 10 lines

package/expect/expect.mk +3 -3

 ################################################################################
 
-# Version 5.45.3
-EXPECT_VERSION = 2014-05-02
-EXPECT_SITE = cvs://expect.cvs.sourceforge.net:/cvsroot/expect
+EXPECT_VERSION = 5.45.3
+EXPECT_SITE = https://sourceforge.net/projects/expect/files/Expect/$(EXPECT_VERSION)
+EXPECT_SOURCE = expect$(EXPECT_VERSION).tar.gz
 EXPECT_LICENSE = Public domain
 EXPECT_LICENSE_FILES = README

Romain's autobuiler is not happy with this:

http://autobuild.buildroot.net/results/dd2/dd29953f28577674070717cbb5feba8e12aad8e3/build-end.log

it downloads some html file. But the same URL downloaded here provides the correct tarball.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Romain Naour

2018-06-01 10:14 UTC #4 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

Hi Thomas,

Le 01/06/2018 ? 11:09, Thomas Petazzoni a ?crit?:

Quoted 34 lines

Hello,

Quoted 16 lines · Arnout Vandecappelle (Essensium/Mind)

package/expect/expect.hash new file +4 -0

new file mode 100644
index 0000000000..c89f323906
@@ -0,0 +1,4 @@
+# From https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz.SHA256
+sha256  c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9  expect5.45.3.tar.gz
+# Locally calculated
+sha256  b2415b17dc8d9a287f4509047ef5ac3436baef7ba7c50faef5222dcdf61a2bab  README

package/expect/expect.mk +0 -0

index d78ff6c208..ccde33a71b 100644
@@ -4,12 +4,13 @@
 #

package/expect/expect.mk line 11

Quoted 10 lines

package/expect/expect.mk +3 -3

 ################################################################################
 
-# Version 5.45.3
-EXPECT_VERSION = 2014-05-02
-EXPECT_SITE = cvs://expect.cvs.sourceforge.net:/cvsroot/expect
+EXPECT_VERSION = 5.45.3
+EXPECT_SITE = https://sourceforge.net/projects/expect/files/Expect/$(EXPECT_VERSION)
+EXPECT_SOURCE = expect$(EXPECT_VERSION).tar.gz
 EXPECT_LICENSE = Public domain
 EXPECT_LICENSE_FILES = README

Romain's autobuiler is not happy with this:

Quoted 4 lines

http://autobuild.buildroot.net/results/dd2/dd29953f28577674070717cbb5feba8e12aad8e3/build-end.log
it downloads some html file. But the same URL downloaded here provides the correct tarball.

$ sha256sum instance-?/dl/expect-*
2b1833202a2371812e9faf67eb0127871b9bd53720e52460552e165534fbde88
instance-1/dl/expect-2014-05-02.tar.gz
48c7a5d2a2081384fb42aaac5eedbec6b5511aa02398611b4d4f6754053990e9
instance-2/dl/expect-2014-05-02.tar.gz

$ ls -lt instance-?/dl/expect-2014-05-02.tar.gz
-rw-r--r--. 2 naourr naourr 635868 25 mai   01:12
instance-1/dl/expect-2014-05-02.tar.gz
-rw-r--r--. 1 naourr naourr 636242 25 avril 11:44
instance-2/dl/expect-2014-05-02.tar.gz

So, it seems Buildroot generated an archive named expect-2014-05-02.tar.gz after downloading from cvs. Unfortunately, it's the same as the new upstream archive...

Note: tar --version
tar (GNU tar) 1.30

I'll remove the wrong (old) archive.

Best regards,
Romain

Quoted 3 lines

Thomas

Romain Naour

2018-06-01 10:53 UTC #5 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

Le 01/06/2018 ? 12:14, Romain Naour a ?crit?:

Quoted 48 lines

Hi Thomas,
Le 01/06/2018 ? 11:09, Thomas Petazzoni a ?crit?:
Quoted 34 lines
Hello,
Quoted 16 lines · Arnout Vandecappelle (Essensium/Mind)
package/expect/expect.hash new file +4 -0
new file mode 100644
index 0000000000..c89f323906
@@ -0,0 +1,4 @@
+# From https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz.SHA256
+sha256  c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9  expect5.45.3.tar.gz
+# Locally calculated
+sha256  b2415b17dc8d9a287f4509047ef5ac3436baef7ba7c50faef5222dcdf61a2bab  README
package/expect/expect.mk +0 -0
index d78ff6c208..ccde33a71b 100644
@@ -4,12 +4,13 @@
 #
package/expect/expect.mk line 11
Quoted 10 lines
package/expect/expect.mk +3 -3
 ################################################################################
 
-# Version 5.45.3
-EXPECT_VERSION = 2014-05-02
-EXPECT_SITE = cvs://expect.cvs.sourceforge.net:/cvsroot/expect
+EXPECT_VERSION = 5.45.3
+EXPECT_SITE = https://sourceforge.net/projects/expect/files/Expect/$(EXPECT_VERSION)
+EXPECT_SOURCE = expect$(EXPECT_VERSION).tar.gz
 EXPECT_LICENSE = Public domain
 EXPECT_LICENSE_FILES = README
Romain's autobuiler is not happy with this:
Quoted 4 lines
http://autobuild.buildroot.net/results/dd2/dd29953f28577674070717cbb5feba8e12aad8e3/build-end.log
it downloads some html file. But the same URL downloaded here provides the correct tarball.
$ sha256sum instance-?/dl/expect-*
2b1833202a2371812e9faf67eb0127871b9bd53720e52460552e165534fbde88
instance-1/dl/expect-2014-05-02.tar.gz
48c7a5d2a2081384fb42aaac5eedbec6b5511aa02398611b4d4f6754053990e9
instance-2/dl/expect-2014-05-02.tar.gz

Quoted 8 lines

$ ls -lt instance-?/dl/expect-2014-05-02.tar.gz
-rw-r--r--. 2 naourr naourr 635868 25 mai   01:12
instance-1/dl/expect-2014-05-02.tar.gz
-rw-r--r--. 1 naourr naourr 636242 25 avril 11:44
instance-2/dl/expect-2014-05-02.tar.gz
So, it seems Buildroot generated an archive named expect-2014-05-02.tar.gz after downloading from cvs. Unfortunately, it's the same as the new upstream archive...

Sorry, the new archive is expect5.45.3.tar.gz

I downloaded this archive using a make source on this machine without any issue.

sha256sum dl/expect/expect5.45.3.tar.gz c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9 dl/expect/expect5.45.3.tar.gz

It's maybe a sourceforge issue (again).

Best regards,
Romain

Quoted 13 lines

Note: tar --version
tar (GNU tar) 1.30
I'll remove the wrong (old) archive.
Best regards,
Romain
Quoted 3 lines
Thomas

Arnout Vandecappelle

author

2018-06-01 11:08 UTC #6 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

In reply to Thomas Petazzoni

Quoted 34 lines · Thomas Petazzoni

Hello,

Quoted 16 lines · Arnout Vandecappelle (Essensium/Mind)

package/expect/expect.hash new file +4 -0

new file mode 100644
index 0000000000..c89f323906
@@ -0,0 +1,4 @@
+# From https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz.SHA256
+sha256  c520717b7195944a69ce1492ec82ca0ac3f3baf060804e6c5ee6d505ea512be9  expect5.45.3.tar.gz
+# Locally calculated
+sha256  b2415b17dc8d9a287f4509047ef5ac3436baef7ba7c50faef5222dcdf61a2bab  README

package/expect/expect.mk +0 -0

index d78ff6c208..ccde33a71b 100644
@@ -4,12 +4,13 @@
 #

package/expect/expect.mk line 11

Quoted 10 lines

package/expect/expect.mk +3 -3

 ################################################################################
 
-# Version 5.45.3
-EXPECT_VERSION = 2014-05-02
-EXPECT_SITE = cvs://expect.cvs.sourceforge.net:/cvsroot/expect
+EXPECT_VERSION = 5.45.3
+EXPECT_SITE = https://sourceforge.net/projects/expect/files/Expect/$(EXPECT_VERSION)
+EXPECT_SOURCE = expect$(EXPECT_VERSION).tar.gz
 EXPECT_LICENSE = Public domain
 EXPECT_LICENSE_FILES = README

Romain's autobuiler is not happy with this:

Quoted 4 lines

http://autobuild.buildroot.net/results/dd2/dd29953f28577674070717cbb5feba8e12aad8e3/build-end.log
it downloads some html file. But the same URL downloaded here provides the correct tarball.

Yeah, in Romain's autobuilder it is redirected to

https://sourceforge.net/#!/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz

while locally I get

https://sourceforge.net/projects/expect/files/Expect/5.45.3/expect5.45.3.tar.gz/download

which is the right one...

Regards,
Arnout

Quoted 3 lines

Thomas

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Peter Korsgaard

2018-06-17 14:59 UTC #7 lore raw

PATCH 2/2 package/expect: download tarball instead of cvs

In reply to Arnout Vandecappelle (Essensium/Mind)

Quoted 29 lines

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle (Essensium/Mind) <[email protected]> writes:
More than 3 years after the 5.45.3 release, a tarball for that release was uploaded to sourceforge.net. The differences between this tarball and the CVS checkout are minimal:
- There are no CVS directories, of course, but we don't need them.
- File timestamps are different.
- expect.tests is missing, but we don't execute tests anyway.
- configure script is different, but we AUTORECONF anyway.
- 'fixcat' script is missing, but it is not called anywhere.
Since sourceforge.net has broken CVS downloads, now is a good time to switch away from it.
While we're at it, add a hash file including license.
Fixes:
http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105
http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78
http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Fabio Porcedda <[email protected]>
---
Either this one or the previous one is sufficient to fix the issue, of course. Still, I think both are relevant for master.

Quoted 5 lines

Note that there is also a 5.45.4 release from a few months ago. But I don't know if anyone still uses expect :-).
Note that this was the last internal package to use CVS. So we no longer have autobuild coverage of the cvs download method.

Committed to 2018.02.x, thanks.

Signature · 1 line

Bye, Peter Korsgaard

Peter Korsgaard

2018-05-31 17:35 UTC #8 lore raw

PATCH 1/2 download/cvs: add a 10 minute timeout

In reply to Arnout Vandecappelle (Essensium/Mind)

Quoted 8 lines

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle (Essensium/Mind) <[email protected]> writes:
Apparently, CVS servers can be deadlocked and in that case clients will retry connecting to them indefinitely. Cfr.
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78/build-end.log
Apparently, the sf.net CVS server got in such a deadlock on 2018-05-18, and almost 2 weeks later it is still not fixed.

Quoted 5 lines

Instead of just hanging, we should fall back on BR2_SECONDARY_SITE. To achieve this, it's sufficient to add a timeout to the CVS command.
The timeout value is of course arbitrary. However, we can assume that nobody will be putting large projects under CVS any more.

Tell that to the BSD people ;)

Quoted 8 lines

So if the download takes more than 5 minutes, it's probably broken. Let's put the timeout at 10 minutes then.
Fixes:
http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105
http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c
http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78
http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4

This is somewhat ugly - But lets indeed keep pragmatic, almost no projects use CVS anymore.

Committed, thanks.

Signature · 1 line

Bye, Peter Korsgaard