scripts/autobuild-run: make it Python 3.x compatible
PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible
With Python 3.7, the autobuild-run did not work due to the following issues:
- The urlparse module no longer exists, it's not urllib.parse
- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x
- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as urlopen_closing() returns a stream of bytes and not strings. So we need to call decode_bytes() on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.
--- scripts/autobuild-run | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
scripts/autobuild-run +6 -3
@@ -145,14 +145,15 @@ from distutils.version import StrictVersion import platform from threading import Thread, Event import datetime -import urlparse if sys.hexversion >= 0x3000000: import configparser import urllib.request as _urllib + import urllib.parse as urlparse else: import ConfigParser as configparser import urllib2 as _urllib + import urlparse urlopen = _urllib.urlopen urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
@@ -307,8 +308,10 @@ class Builder: list. This way, branches with a higher weight are more likely to be selected. """ + csv_branches = [] with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r: - csv_branches = r.readlines() + for l in r.readlines(): + csv_branches.append(decode_bytes(l)) branches = [] for branch in csv.reader(csv_branches): branches += [branch[0]] * int(branch[1])
@@ -830,7 +833,7 @@ def main(): # Enforce the sanest umask here, to avoid buildroot doing it on its # own and causing a double-make call, thus adding extraneous lines # in case of failures. - os.umask(0022) + os.umask(0o022) def sigterm_handler(signum, frame): """Kill all children"""
Signature · 1 line
2.23.0
PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible
Hi Thomas,
Quoted 4 lines · Thomas Petazzoni
With Python 3.7, the autobuild-run did not work due to the following issues:
- The urlparse module no longer exists, it's not urllib.parse
s/not/now/
Quoted 21 lines
- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x
- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as
urlopen_closing()returns a stream of bytes and not strings. So we need to calldecode_bytes()on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.Signed-off-by Thomas Petazzoni <[email protected]>--- scripts/autobuild-run | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)scripts/autobuild-run +0
-0
scripts/autobuild-run from distutils.version import StrictVersion · line 152
Quoted 9 lines
scripts/autobuild-run +1
-1@@ -145,14 +145,15 @@ from distutils.version import StrictVersion import platform from threading import Thread, Event import datetime -import urlparse if sys.hexversion >= 0x3000000: import configparser import urllib.request as _urllib + import urllib.parse as urlparse
Would that work with any Python 3.x?
Quoted 28 lines
scripts/autobuild-run +5
-2else: import ConfigParser as configparser import urllib2 as _urllib + import urlparse urlopen = _urllib.urlopen urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))@@ -307,8 +308,10 @@ class Builder: list. This way, branches with a higher weight are more likely to be selected. """ + csv_branches = [] with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r: - csv_branches = r.readlines() + for l in r.readlines(): + csv_branches.append(decode_bytes(l)) branches = [] for branch in csv.reader(csv_branches): branches += [branch[0]] * int(branch[1])@@ -830,7 +833,7 @@ def main(): # Enforce the sanest umask here, to avoid buildroot doing it on its # own and causing a double-make call, thus adding extraneous lines # in case of failures. - os.umask(0022) + os.umask(0o022) def sigterm_handler(signum, frame): """Kill all children"""
Signature · 3 lines
http://baruch.siach.name/blog/ ~. .~ Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
- baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible
Hello,
Quoted 3 lines · On Tue, 3 Dec 2019 19:11:37 +0200 Baruch Siach <[email protected]> wrote:
Quoted 1 line
- The urlparse module no longer exists, it's not urllib.parse
s/not/now/
Indeed.
Quoted 6 lines
Quoted 4 lines
if sys.hexversion >=
0x3000000:
import configparser
import urllib.request as_urllib
+ import urllib.parse as urlparseWould that work with any Python 3.x?
https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation of urllib.parse.urlparse() that say:
"""
Changed in version 3.2: Added IPv6 URL parsing capabilities. """
So it means that urllib.parse.urlparse() was there at least before Python 3.2, which means it was most likely there since Python 3.x.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible
Quoted 22 lines · Thomas Petazzoni
Hello,
Quoted 3 lines · On Tue, 3 Dec 2019 19:11:37 +0200 Baruch Siach <[email protected]> wrote:
Quoted 1 line
- The urlparse module no longer exists, it's not urllib.parse
s/not/now/
Indeed.
Quoted 6 lines
Quoted 4 lines
if sys.hexversion >=
0x3000000:
import configparser
import urllib.request as_urllib
+ import urllib.parse as urlparseWould that work with any Python 3.x?
https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation ofurllib.parse.urlparse()that say:
Quoted 6 lines
"""
Changed in version 3.2: Added IPv6 URL parsing capabilities. """So it means that
urllib.parse.urlparse()was there at least before Python 3.2, which means it was most likely there since Python 3.x.
And anyway, I think we generally should assume Python >= 3.4. Debian
oldoldstable (jessie) has 3.4 already. Trying to support older versions will make our life unnecessarily difficult.
Regards,
Arnout
PATCH 1/1 scripts/autobuild-run: make it Python 3.x compatible
In reply to Thomas PetazzoniQuoted 15 lines · On Tue, 3 Dec 2019 17:54:57 +0100 Thomas Petazzoni…
With Python 3.7, the autobuild-run did not work due to the following issues:
- The urlparse module no longer exists, it's not urllib.parse
- 0022 is no longer recognized as an octal value, we must use 0o022, which also works in Python 2.x
- reading the CSV file with the list of branches through the CSV parser failed due to the lack of decoding, as
urlopen_closing()returns a stream of bytes and not strings. So we need to calldecode_bytes()on each element of the CSV array. Since the CSV file is typically 3 or 4 lines long, we don't really need to optimize things.
Quoted 4 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- scripts/autobuild-run | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
Applied to buildroot-test, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com