Skip to contentSkip to conversation

scripts/autobuild-run: make it Python 3.x compatible

PATCH New Thomas Petazzoni 3 participants 5 messages Dec 3, 2019 – Dec 4, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible

With Python 3.7, the autobuild-run did not work due to the following issues:

- The urlparse module no longer exists, it's not urllib.parse

- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x

- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as urlopen_closing() returns a stream of bytes and not strings. So we need to call decode_bytes() on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.

---
 scripts/autobuild-run | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
scripts/autobuild-run +6 -3
index 5921edd..e475ea8 100755
@@ -145,14 +145,15 @@ from distutils.version import StrictVersion
 import platform
 from threading import Thread, Event
 import datetime
-import urlparse
 
 if sys.hexversion >= 0x3000000:
     import configparser
     import urllib.request as _urllib
+    import urllib.parse as urlparse
 else:
     import ConfigParser as configparser
     import urllib2 as _urllib
+    import urlparse
 
 urlopen = _urllib.urlopen
 urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
@@ -307,8 +308,10 @@ class Builder:
         list. This way, branches with a higher weight are more likely to
         be selected.
         """
+        csv_branches = []
         with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r:
-            csv_branches = r.readlines()
+            for l in r.readlines():
+                csv_branches.append(decode_bytes(l))
         branches = []
         for branch in csv.reader(csv_branches):
             branches += [branch[0]] * int(branch[1])
@@ -830,7 +833,7 @@ def main():
     # Enforce the sanest umask here, to avoid buildroot doing it on its
     # own and causing a double-make call, thus adding extraneous lines
     # in case of failures.
-    os.umask(0022)
+    os.umask(0o022)
 
     def sigterm_handler(signum, frame):
         """Kill all children"""
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible

Hi Thomas,

Quoted 4 lines · Thomas Petazzoni

With Python 3.7, the autobuild-run did not work due to the following issues:

- The urlparse module no longer exists, it's not urllib.parse

s/not/now/

Quoted 21 lines

- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x

- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as urlopen_closing() returns a stream of bytes and not strings. So we need to call decode_bytes() on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.

---
 scripts/autobuild-run | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
scripts/autobuild-run +0 -0
index 5921edd..e475ea8 100755

scripts/autobuild-run from distutils.version import StrictVersion · line 152

Quoted 9 lines
scripts/autobuild-run +1 -1
@@ -145,14 +145,15 @@ from distutils.version import StrictVersion
 import platform
 from threading import Thread, Event
 import datetime
-import urlparse
 
 if sys.hexversion >= 0x3000000:
     import configparser
     import urllib.request as _urllib
+    import urllib.parse as urlparse

Would that work with any Python 3.x?

Quoted 28 lines
scripts/autobuild-run +5 -2
 else:
     import ConfigParser as configparser
     import urllib2 as _urllib
+    import urlparse
 
 urlopen = _urllib.urlopen
 urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
@@ -307,8 +308,10 @@ class Builder:
         list. This way, branches with a higher weight are more likely to
         be selected.
         """
+        csv_branches = []
         with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r:
-            csv_branches = r.readlines()
+            for l in r.readlines():
+                csv_branches.append(decode_bytes(l))
         branches = []
         for branch in csv.reader(csv_branches):
             branches += [branch[0]] * int(branch[1])
@@ -830,7 +833,7 @@ def main():
     # Enforce the sanest umask here, to avoid buildroot doing it on its
     # own and causing a double-make call, thus adding extraneous lines
     # in case of failures.
-    os.umask(0022)
+    os.umask(0o022)
 
     def sigterm_handler(signum, frame):
         """Kill all children"""
Signature · 3 lines
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
#2 lore raw

PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible

Hello,

Quoted 3 lines · On Tue, 3 Dec 2019 19:11:37 +0200 Baruch Siach <[email protected]> wrote:
Quoted 1 line

- The urlparse module no longer exists, it's not urllib.parse

s/not/now/

Indeed.

Quoted 6 lines
Quoted 4 lines

if sys.hexversion >= 0x3000000:
import configparser
import urllib.request as _urllib
+ import urllib.parse as urlparse

Would that work with any Python 3.x?

https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation of urllib.parse.urlparse() that say:

"""
Changed in version 3.2: Added IPv6 URL parsing capabilities. """

So it means that urllib.parse.urlparse() was there at least before Python 3.2, which means it was most likely there since Python 3.x.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible

Quoted 22 lines · Thomas Petazzoni

Hello,

Quoted 3 lines · On Tue, 3 Dec 2019 19:11:37 +0200 Baruch Siach <[email protected]> wrote:
Quoted 1 line

- The urlparse module no longer exists, it's not urllib.parse

s/not/now/

Indeed.

Quoted 6 lines
Quoted 4 lines

if sys.hexversion >= 0x3000000:
import configparser
import urllib.request as _urllib
+ import urllib.parse as urlparse

Would that work with any Python 3.x?

https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation of urllib.parse.urlparse() that say:

Quoted 6 lines

"""
Changed in version 3.2: Added IPv6 URL parsing capabilities. """

So it means that urllib.parse.urlparse() was there at least before Python 3.2, which means it was most likely there since Python 3.x.

And anyway, I think we generally should assume Python >= 3.4. Debian
oldoldstable (jessie) has 3.4 already. Trying to support older versions will make our life unnecessarily difficult.

Regards,
Arnout

#4 lore raw

PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible

In reply to Thomas Petazzoni
Quoted 15 lines · On Tue, 3 Dec 2019 17:54:57 +0100 Thomas Petazzoni…

With Python 3.7, the autobuild-run did not work due to the following issues:

- The urlparse module no longer exists, it's not urllib.parse

- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x

- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as urlopen_closing() returns a stream of bytes and not strings. So we need to call decode_bytes() on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.

Quoted 4 lines
---
 scripts/autobuild-run | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

Applied to buildroot-test, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com