autobuild-run: python3 compat and misc improvements
v2 cover autobuild-run: python3 compat and misc improvements
Changes v1->v2:
* catch up with git HEAD: further cleanup/python3-compat changes (patches 1-6)
* "move check_requirements() to SystemInfo" is mostly identical to v1;
changed: use "upload" var, remove entry from TODO list
* "encapsulate subprocess calls" had to be reworked
It now wraps subprocess.Popen() instead of subprocess.call(), and implements run_cmd(), run_cmd_write_to() and (new:) run_cmd_get_stdout() on top of it. This covers all uses cases of the subprocess module and, in the case of run_cmd_get_stdout(), handles the bytes->str conversion.
In v1, the "set-locale" patch added a subprocess.check_output() wrapper, which has been dropped.
* "set subprocess env" is a separate commit now,
used to be part of the "set-locale" patch
Andr? Erdmann (10):
autobuild-run: fixup imports
autobuild-run, python3: dict.iteritems()->items() autobuild-run, python3: decode subprocess output autobuild-run, python3: encode/decode mmap data
autobuild-run: avoid use of builtin as varname
autobuild-run: reorganize imports
autobuild-run: move check_requirements() to SystemInfo
autobuild-run: encapsulate subprocess calls
autobuild-run: control subprocess env
autobuild-run: set locale to en_US or C
scripts/autobuild-run | 280 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 183 insertions(+), 97 deletions(-)
Signature · 1 line
2.3.2
PATCH v2 1/10 autobuild-run: fixup imports
* drop "import ConfigParser", module gets imported as "configparser" See commit 16fcf29310bbb2533a98eb2247fc1415cb27e514
* drop "import argparse", currently unused
--- scripts/autobuild-run | 2 -- 1 file changed, 2 deletions(-)
scripts/autobuild-run +0 -2
@@ -121,8 +121,6 @@ import shutil from time import localtime, strftime import sys import hashlib -import argparse -import ConfigParser import docopt if sys.hexversion >= 0x3000000:
Signature · 1 line
2.3.2
PATCH v2 2/10 autobuild-run, python3: dict.iteritems()->items()
In reply to André ErdmannSome dict methods, like iteritems(), have been dropped in Python 3, while the semantics of others, e.g. items(), has changed.
The Py3k variant of items() returns a "view" object, whereas the Py2k variant returns a list, but both methods essentially do the same thing here.
--- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/autobuild-run +1 -1
@@ -105,7 +105,7 @@ Default values for the arguments are: %s """ % '\n '.join( - ['%s = %s' % (key, val) for (key, val) in defaults.iteritems()]) + ['%s = %s' % (key, val) for (key, val) in defaults.items()]) __doc__ = doc
Signature · 1 line
2.3.2
PATCH v2 3/10 autobuild-run, python3: decode subprocess output
In reply to André Erdmannsubprocess.check_output() and its py 2.6 compat variant, subprocess.Popen().communicate()[0], return a bytes object in Python 3, but regexp.search() in get_failure_reason() wants a str:
Quoted 1 line
Quoted 1 line
Quoted 1 line
re.search(r'.', b'a')
TypeError: can't use a string pattern on a bytes-like object
--- scripts/autobuild-run | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
scripts/autobuild-run +3 -2
@@ -576,8 +576,9 @@ def send_results(result, **kwargs): def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = subprocess.Popen(["tail", "-n", "3", - os.path.join(outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0].splitlines() + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() import re regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/')
Signature · 1 line
2.3.2
PATCH v2 4/10 autobuild-run, python3: encode/decode mmap data
In reply to André Erdmannmmap objects return bytes in Python 3,
which leads to errors when used in str context, for example:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
fh =
open("file", "r")fh_mmap=mmap.mmap(fh.fileno(), 0, access=ACCESS_READ)fh_mmap.find("word")
TypeError: 'str' does not support the buffer interface
Adds a new compat function, encode_str(), which is a no-op in Python 2.
--- scripts/autobuild-run | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/autobuild-run +6 -2
@@ -139,12 +139,16 @@ if sys.hexversion >= 0x3000000: def decode_byte_list(bl): return [b.decode() for b in bl] + + def encode_str(s): + return s.encode() else: def _identity(e): return e decode_bytes = _identity decode_byte_list = _identity + encode_str = _identity MAX_DURATION = 60 * 60 * 4 VERSION = 1
@@ -607,10 +611,10 @@ def send_results(result, **kwargs): mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mf.seek(0) # Search for first action on the failed package - offset = mf.find('>>> %s' % ' '.join(reason)) + offset = mf.find(encode_str('>>> %s' % ' '.join(reason))) if offset != -1: with open(resultfile, "w") as endlog: - endlog.write(mf[offset:]) + endlog.write(decode_bytes(mf[offset:])) else: # not found, use last 500 lines as fallback extract_last_500_lines()
Signature · 1 line
2.3.2
PATCH v2 5/10 autobuild-run: avoid use of builtin as varname
In reply to André Erdmann"file" is a type name in Python 2.
--- scripts/autobuild-run | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/autobuild-run +3 -3
@@ -637,10 +637,10 @@ def send_results(result, **kwargs): for root, dirs, files in os.walk(srcroot): dest = os.path.join(destroot, os.path.relpath(root, srcroot)) - for file in files: - if file == 'config.log': + for fname in files: + if fname == 'config.log': os.makedirs(dest) - shutil.copy(os.path.join(root, file), os.path.join(dest, file)) + shutil.copy(os.path.join(root, fname), os.path.join(dest, fname)) copy_config_log_files()
Signature · 1 line
2.3.2
PATCH v2 6/10 autobuild-run: reorganize imports
In reply to André Erdmann* move "re", "mmap" to the top of the file
* sort imports alphabetically
--- scripts/autobuild-run | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
scripts/autobuild-run +9 -9
@@ -111,17 +111,19 @@ __doc__ = doc import contextlib import csv -from random import randint -import subprocess +import docopt +import errno +import hashlib +import mmap import multiprocessing -import signal import os -import errno +from random import randint +import re import shutil -from time import localtime, strftime +import signal +import subprocess import sys -import hashlib -import docopt +from time import localtime, strftime if sys.hexversion >= 0x3000000: import configparser
@@ -584,7 +586,6 @@ def send_results(result, **kwargs): ["tail", "-n", "3", os.path.join(outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0]).splitlines() - import re regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') for line in lastlines: m = regexp.search(line)
@@ -606,7 +607,6 @@ def send_results(result, **kwargs): if not reason: extract_last_500_lines() else: - import mmap f = open(os.path.join(outputdir, "logfile"), 'r') mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mf.seek(0)
Signature · 1 line
2.3.2
PATCH v2 7/10 autobuild-run: move check_requirements() to SystemInfo
In reply to André Erdmann* build up the list of (additional) essential programs in main()
* can check for optional progs in fixup_config() without having to list them in sysinfo
* it'd be quite easy to dump the prog list to the logfile (for name, have_prog in sysinfo.progs.items(): ...)
--- scripts/autobuild-run | 123 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 73 insertions(+), 50 deletions(-)
scripts/autobuild-run +73 -50
@@ -47,8 +47,6 @@ # - Instead of excluding all configurations that have # BR2_PACKAGE_CLASSPATH=y, improve the script to detect whether the # necessary host machine requirements are there to build classpath. -# - Integrate method check-requirements with the SysInfo class, distinghuishing -# between required and optional dependencies. # - Extend required dependencies to subversion, mercurial, cpio, wget, python, # etc. # - Detect selection of multiple virtual package providers and don't consider it
@@ -166,51 +164,72 @@ def check_version(): print("ERROR: script version too old, please upgrade.") sys.exit(1) -def has_prog(name, flags=os.X_OK, env=os.environ): - if not name or name[0] == os.sep: raise ValueError(name) - - prog_path = env.get("PATH", None) - # for windows compatibility, we'd need to take PATHEXT into account - - if prog_path: - for prog_dir in filter(None, prog_path.split(os.pathsep)): - # os.join() not necessary: non-empty prog_dir and name[0] != os.sep - prog = prog_dir + os.sep + name - if os.access(prog, flags): - return True - # -- - return False - -def check_requirements(upload=False): - needed_progs = ["make", "git", "gcc", "timeout"] - missing_requirements = False - - if upload: - needed_progs.append("curl") - - for prog in needed_progs: - if not has_prog(prog): - print("ERROR: your system lacks the '%s' program" % prog) - missing_requirements = True - - if missing_requirements: - sys.exit(1) - class SystemInfo: - def __init__(self): - devnull = open(os.devnull, "w") - # _grep_gcj :: str -> int - _grep_gcj = lambda prog: \ - subprocess.call("%s -version | grep gcj" % prog, shell=True, - stdout=devnull, stderr=devnull) - - self.has_bzr = has_prog("bzr") - - self.has_java = has_prog("java") and _grep_gcj("java") == 1 + DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] + DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] - self.has_javac = has_prog("javac") and _grep_gcj("javac") == 1 + def __init__(self): + self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) + self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS) + self.progs = {} + + def find_prog(self, name, flags=os.X_OK, env=os.environ): + if not name or name[0] == os.sep: raise ValueError(name) + + prog_path = env.get("PATH", None) + # for windows compatibility, we'd need to take PATHEXT into account + + if prog_path: + for prog_dir in filter(None, prog_path.split(os.pathsep)): + # os.join() not necessary: non-empty prog_dir + # and name[0] != os.sep + prog = prog_dir + os.sep + name + if os.access(prog, flags): + return prog + # -- + return None - self.has_jar = has_prog("jar") + def has(self, prog): + """Checks whether a program is available. + Lazily evaluates missing entries. + + Returns: None if prog not found, else path to the program [evaluates to True] + """ + try: + return self.progs[prog] + except KeyError: + pass + + have_it = self.find_prog(prog) + # java[c] needs special care + if have_it and prog in ('java', 'javac'): + with open(os.devnull, "w") as devnull: + if subprocess.call("%s -version | grep gcj" % prog, shell=True, + stdout=devnull, stderr=devnull) != 1: + have_it = False + # -- + self.progs[prog] = have_it + return have_it + + def check_requirements(self): + """Checks program dependencies. + + Returns: True if all mandatory programs are present, else False. + """ + do_check_has_prog = self.has + + missing_requirements = False + for prog in self.needed_progs: + if not do_check_has_prog(prog): + print("ERROR: your system lacks the '%s' program" % prog) + missing_requirements = True + + # check optional programs here, + # else they'd get checked by each worker instance + for prog in self.optional_progs: + do_check_has_prog(prog) + + return not missing_requirements def get_toolchain_configs(): """Fetch and return the possible toolchain configurations
@@ -338,14 +357,14 @@ def fixup_config(**kwargs): if "BR2_PACKAGE_MROUTED=y\n" in configlines and \ "BR2_TOOLCHAIN_USES_UCLIBC=y\n" in configlines: configlines.remove("BR2_PACKAGE_MROUTED=y\n") - if "BR2_NEEDS_HOST_JAVA=y\n" in configlines and not sysinfo.has_java: + if "BR2_NEEDS_HOST_JAVA=y\n" in configlines and not sysinfo.has("java"): return False - if "BR2_NEEDS_HOST_JAVAC=y\n" in configlines and not sysinfo.has_javac: + if "BR2_NEEDS_HOST_JAVAC=y\n" in configlines and not sysinfo.has("javac"): return False - if "BR2_NEEDS_HOST_JAR=y\n" in configlines and not sysinfo.has_jar: + if "BR2_NEEDS_HOST_JAR=y\n" in configlines and not sysinfo.has("jar"): return False # python-nfc needs bzr - if 'BR2_PACKAGE_PYTHON_NFC=y\n' in configlines and not sysinfo.has_bzr: + if 'BR2_PACKAGE_PYTHON_NFC=y\n' in configlines and not sysinfo.has("bzr"): return False # The ctng toolchain is affected by PR58854 if 'BR2_PACKAGE_LTTNG_TOOLS=y\n' in configlines and \
@@ -771,11 +790,15 @@ def main(): # explicitly on None. upload = (args['--http-login'] is not None) \ and (args['--http-password'] is not None) - check_requirements(upload) - if not upload: + if upload: + sysinfo.needed_progs.append("curl") + else: print("WARN: due to the lack of http login/password details, results will not be submitted") print("WARN: tarballs of results will be kept locally only") + if not sysinfo.check_requirements(): + sys.exit(1) + def sigterm_handler(signum, frame): """Kill all children"""
Signature · 1 line
2.3.2
PATCH v2 8/10 autobuild-run: encapsulate subprocess calls
In reply to André ErdmannPreparation step for passing LANG to worker (sub-)processes, allows to redirect stdin/stdout/stderr, which all default to devnull now unless specified otherwise.
This makes the "yes"-pipe in "make oldconfig" redundant.
--- scripts/autobuild-run | 92 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 55 insertions(+), 37 deletions(-)
scripts/autobuild-run +55 -37
@@ -172,6 +172,7 @@ class SystemInfo: self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS) self.progs = {} + self.devnull = open(os.devnull, "w") def find_prog(self, name, flags=os.X_OK, env=os.environ): if not name or name[0] == os.sep: raise ValueError(name)
@@ -203,10 +204,8 @@ class SystemInfo: have_it = self.find_prog(prog) # java[c] needs special care if have_it and prog in ('java', 'javac'): - with open(os.devnull, "w") as devnull: - if subprocess.call("%s -version | grep gcj" % prog, shell=True, - stdout=devnull, stderr=devnull) != 1: - have_it = False + if self.run_cmd("%s -version | grep gcj" % prog, shell=True) != 1: + have_it = False # -- self.progs[prog] = have_it return have_it
@@ -231,6 +230,27 @@ class SystemInfo: return not missing_requirements + def popen(self, cmdv, **kwargs): + kwargs.setdefault('stdin', self.devnull) + kwargs.setdefault('stdout', self.devnull) + kwargs.setdefault('stderr', self.devnull) + return subprocess.Popen(cmdv, **kwargs) + + def run_cmd(self, cmdv, **kwargs): + proc = self.popen(cmdv, **kwargs) + proc.communicate() + return proc.poll() + + def run_cmd_write_to(self, outstream, cmdv, **kwargs): + kwargs.update(stdout=outstream, stderr=outstream) + return self.run_cmd(cmdv, **kwargs) + + def run_cmd_get_stdout(self, cmdv, **kwargs): + proc = self.popen(cmdv, stdout=subprocess.PIPE, **kwargs) + stdout_data, _ = proc.communicate() + return (proc.poll(), decode_bytes(stdout_data) if stdout_data else None) + + def get_toolchain_configs(): """Fetch and return the possible toolchain configurations
@@ -274,6 +294,7 @@ def prepare_build(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] log_write(log, "INFO: preparing a new build")
@@ -298,15 +319,15 @@ def prepare_build(**kwargs): # didn't exist already. srcdir = os.path.join(idir, "buildroot") if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", "git://git.busybox.net/buildroot", srcdir], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["git", "clone", "git://git.busybox.net/buildroot", srcdir]) if ret != 0: log_write(log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "pull"], cwd=abssrcdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to(log, ["git", "pull"], cwd=abssrcdir) if ret != 0: log_write(log, "ERROR: could not pull Buildroot sources") return -1
@@ -315,7 +336,7 @@ def prepare_build(**kwargs): outputdir = os.path.join(idir, "output") if os.path.exists(outputdir): # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) + sysinfo.run_cmd(["rm", "-rf", outputdir]) os.mkdir(outputdir) return 0
@@ -459,6 +480,7 @@ def gen_config(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
@@ -493,10 +515,7 @@ def gen_config(**kwargs): with open(os.path.join(outputdir, ".config"), "w+") as configf: configf.writelines(configlines) - devnull = open(os.devnull, "w") - - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"]) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1
@@ -504,23 +523,20 @@ def gen_config(**kwargs): # Now, generate the random selection of packages, and fixup # things if needed. while True: - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, - "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, + "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"]) if ret != 0: log_write(log, "ERROR: cannot generate random configuration") return -1 if fixup_config(**kwargs): break - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"]) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1 - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"]) if ret != 0: log_write(log, "ERROR: cannot savedefconfig") return -1
@@ -532,6 +548,7 @@ def do_build(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
@@ -547,7 +564,7 @@ def do_build(**kwargs): "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % kwargs['njobs']] \ + kwargs['make_opts'].split() - sub = subprocess.Popen(cmd, stdout=f, stderr=f) + sub = sysinfo.popen(cmd, stdout=f, stderr=f) kwargs['buildpid'][kwargs['instance']] = sub.pid ret = sub.wait() kwargs['buildpid'][kwargs['instance']] = 0
@@ -560,7 +577,7 @@ def do_build(**kwargs): if ret != 0: log_write(log, "INFO: build failed") return -1 - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir], stdout=f, stderr=f) + ret = sysinfo.run_cmd_write_to(f, ["make", "O=%s" % outputdir, "-C", srcdir]) if ret != 0: log_write(log, "INFO: build failed during legal-info") return -1
@@ -577,6 +594,7 @@ def send_results(result, **kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] outputdir = os.path.abspath(os.path.join(idir, "output")) srcdir = os.path.join(idir, "buildroot")
@@ -595,15 +613,14 @@ def send_results(result, **kwargs): shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"), os.path.join(resultdir, "licenses-manifest.csv")) - subprocess.call(["git log master -n 1 --pretty=format:%%H > %s" % \ + sysinfo.run_cmd(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=srcdir, stderr=None) def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() + lastlines = sysinfo.run_cmd_get_stdout( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")])[1].splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') for line in lastlines:
@@ -618,9 +635,9 @@ def send_results(result, **kwargs): """Save the last part of the build log, starting from the failed package""" def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ + sysinfo.run_cmd(["tail -500 %s > %s" % \ (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + shell=True, stderr=None) reason = get_failure_reason() if not reason:
@@ -677,8 +694,8 @@ def send_results(result, **kwargs): # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["tar", "cjf", "results.tar.bz2", "results"], cwd=outputdir) if ret != 0: log_write(log, "ERROR: could not make results tarball") sys.exit(1)
@@ -687,13 +704,14 @@ def send_results(result, **kwargs): # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - "http://autobuild.buildroot.org/submit/"], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, + ["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + "http://autobuild.buildroot.org/submit/"]) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) else:
Signature · 1 line
2.3.2
PATCH v2 8/10 autobuild-run: encapsulate subprocess calls
Hi Andr?, Thomas, all,
Quoted 6 lines · Andr? Erdmann
Preparation step for passing LANG to worker (sub-)processes, allows to redirect stdin/stdout/stderr, which all default to devnull now unless specified otherwise.
This makes the "yes"-pipe in "make oldconfig" redundant.Signed-off-by Andr? Erdmann <[email protected]>
[...]
Quoted 11 lines
return 0
scripts/autobuild-run +1
-0@@ -459,6 +480,7 @@ def gen_config(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
scripts/autobuild-run def gen_config(**kwargs): · line 518
Quoted 8 lines
scripts/autobuild-run +1
-4@@ -493,10 +515,7 @@ def gen_config(**kwargs): with open(os.path.join(outputdir, ".config"), "w+") as configf: configf.writelines(configlines) - devnull = open(os.devnull, "w") - - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"])
Any reason to drop the yes pipe in the command?
Quoted 11 lines
scripts/autobuild-run +1
-3if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1@@ -504,23 +523,20 @@ def gen_config(**kwargs): # Now, generate the random selection of packages, and fixup # things if needed. while True: - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, - "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir,
scripts/autobuild-run def gen_config(**kwargs): · line 534
Quoted 10 lines
scripts/autobuild-run +2
-2+ "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"]) if ret != 0: log_write(log, "ERROR: cannot generate random configuration") return -1 if fixup_config(**kwargs): break - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"])
ditto
Quoted 10 lines
scripts/autobuild-run +1
-2if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1 - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"]) if ret != 0: log_write(log, "ERROR: cannot savedefconfig") return -1
[...]
scripts/autobuild-run def send_results(result, **kwargs): · line 618
Quoted 9 lines
scripts/autobuild-run +2
-2@@ -595,15 +613,14 @@ def send_results(result, **kwargs): shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"), os.path.join(resultdir, "licenses-manifest.csv")) - subprocess.call(["git log master -n 1 --pretty=format:%%H > %s" % \ + sysinfo.run_cmd(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=srcdir, stderr=None)
IIRC, stderr=None dumps the error output in the process error output, so only available for the autobuilder owner.
So, how about log stderr in the log file using sysinfo.run_cmd_get_output (stdout would be anyway redirected to gitid, and stderr would be part of the log)?
Quoted 12 lines
scripts/autobuild-run +2
-3def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() + lastlines = sysinfo.run_cmd_get_stdout( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")])[1].splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') for line in lastlines:
scripts/autobuild-run def send_results(result, **kwargs): · line 640
Quoted 8 lines
scripts/autobuild-run +2
-2@@ -618,9 +635,9 @@ def send_results(result, **kwargs): """Save the last part of the build log, starting from the failed package""" def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ + sysinfo.run_cmd(["tail -500 %s > %s" % \ (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + shell=True, stderr=None)
ditto
Quoted 39 lines
scripts/autobuild-run +10
-10reason = get_failure_reason() if not reason:@@ -677,8 +694,8 @@ def send_results(result, **kwargs): # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["tar", "cjf", "results.tar.bz2", "results"], cwd=outputdir) if ret != 0: log_write(log, "ERROR: could not make results tarball") sys.exit(1)@@ -687,13 +704,14 @@ def send_results(result, **kwargs): # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - "http://autobuild.buildroot.org/submit/"], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, + ["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + "http://autobuild.buildroot.org/submit/"]) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) else: --2.3.2
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Regards,
Signature · 1 line
Samuel
PATCH v2 8/10 autobuild-run: encapsulate subprocess calls
Hi,
2015-04-12 9:51 GMT+02:00 Samuel Martin <[email protected]>:
Quoted 24 lines
Hi Andr?, Thomas, all,
Quoted 6 lines · Andr? Erdmann
Preparation step for passing LANG to worker (sub-)processes, allows to redirect stdin/stdout/stderr, which all default to devnull now unless specified otherwise.
This makes the "yes"-pipe in "make oldconfig" redundant.Signed-off-by Andr? Erdmann <[email protected]>[...]
Quoted 14 lines
return 0
scripts/autobuild-run +1
-0@@ -459,6 +480,7 @@ def gen_config(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the@@ -493,10 +515,7 @@ def gen_config(**kwargs): with open(os.path.join(outputdir, ".config"), "w+") as configf: configf.writelines(configlines)
Quoted 7 lines
scripts/autobuild-run def gen_config(**kwargs): · line 518
Quoted 5 lines
scripts/autobuild-run +1
-4- devnull = open(os.devnull, "w") - - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"])Any reason to drop the yes pipe in the command?
It's not needed anymore: SystemInfo::run_cmd() redirects stdin to /dev/null, so this command reads as "make ... oldconfig < /dev/null", which behaves identical to "yes '' | make ... oldconfig".
Quoted 38 lines
Quoted 11 lines
scripts/autobuild-run +1
-3if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1@@ -504,23 +523,20 @@ def gen_config(**kwargs): # Now, generate the random selection of packages, and fixup # things if needed. while True: - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, - "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir,scripts/autobuild-run def gen_config(**kwargs): · line 534
Quoted 10 lines
scripts/autobuild-run +2
-2+ "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"]) if ret != 0: log_write(log, "ERROR: cannot generate random configuration") return -1 if fixup_config(**kwargs): break - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"])ditto
Quoted 10 lines
scripts/autobuild-run +1
-2if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1 - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"]) if ret != 0: log_write(log, "ERROR: cannot savedefconfig") return -1[...]
Quoted 4 lines
scripts/autobuild-run +0
-0@@ -595,15 +613,14 @@ def send_results(result, **kwargs): shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"), os.path.join(resultdir, "licenses-manifest.csv"))
Quoted 11 lines
scripts/autobuild-run def send_results(result, **kwargs): · line 618
Quoted 5 lines
scripts/autobuild-run +2
-2- subprocess.call(["git log master -n 1 --pretty=format:%%H > %s" % \ + sysinfo.run_cmd(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=srcdir, stderr=None)IIRC, stderr=None dumps the error output in the process error output, so only available for the autobuilder owner.
So, how about log stderr in the log file using sysinfo.run_cmd_get_output (stdout would be anyway redirected to gitid, and stderr would be part of the log)?
Sure! But I'd rather do that in a follow-up patch since this one tries to keep the functional changes at a minimum, except for the stdin-/dev/null redirection. (The "git log" command used to write to console before this series, same applies to the "tar" command below.)
Quoted 61 lines
Quoted 12 lines
scripts/autobuild-run +2
-3def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() + lastlines = sysinfo.run_cmd_get_stdout( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")])[1].splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') for line in lastlines:scripts/autobuild-run def send_results(result, **kwargs): · line 640
Quoted 8 lines
scripts/autobuild-run +2
-2@@ -618,9 +635,9 @@ def send_results(result, **kwargs): """Save the last part of the build log, starting from the failed package""" def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ + sysinfo.run_cmd(["tail -500 %s > %s" % \ (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + shell=True, stderr=None)ditto
Quoted 39 lines
scripts/autobuild-run +10
-10reason = get_failure_reason() if not reason:@@ -677,8 +694,8 @@ def send_results(result, **kwargs): # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["tar", "cjf", "results.tar.bz2", "results"], cwd=outputdir) if ret != 0: log_write(log, "ERROR: could not make results tarball") sys.exit(1)@@ -687,13 +704,14 @@ def send_results(result, **kwargs): # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - "http://autobuild.buildroot.org/submit/"], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, + ["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + "http://autobuild.buildroot.org/submit/"]) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) else: --2.3.2
Signature · 1 line
Andr?
PATCH v2 9/10 autobuild-run: control subprocess env
In reply to André ErdmannPreparation step for passing LANG to worker (sub-)processes, allows to control execution environment of subprocesses without affecting the "autobuild-run" process itself.
--- scripts/autobuild-run | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/autobuild-run +4 -2
@@ -173,11 +173,12 @@ class SystemInfo: self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS) self.progs = {} self.devnull = open(os.devnull, "w") + self.env = os.environ.copy() - def find_prog(self, name, flags=os.X_OK, env=os.environ): + def find_prog(self, name, flags=os.X_OK): if not name or name[0] == os.sep: raise ValueError(name) - prog_path = env.get("PATH", None) + prog_path = self.env.get("PATH", None) # for windows compatibility, we'd need to take PATHEXT into account if prog_path:
@@ -234,6 +235,7 @@ class SystemInfo: kwargs.setdefault('stdin', self.devnull) kwargs.setdefault('stdout', self.devnull) kwargs.setdefault('stderr', self.devnull) + kwargs['env'] = self.env return subprocess.Popen(cmdv, **kwargs) def run_cmd(self, cmdv, **kwargs):
Signature · 1 line
2.3.2
PATCH v2 9/10 autobuild-run: control subprocess env
Hi Andr?, Thomas, all,
Quoted 5 lines · Andr? Erdmann
Preparation step for passing LANG to worker (sub-)processes, allows to control execution environment of subprocesses without affecting the "autobuild-run" process itself.
Signed-off-by Andr? Erdmann <[email protected]>
Regards,
Signature · 1 line
Samuel
PATCH v2 10/10 autobuild-run: set locale to en_US or C
In reply to André Erdmannsome python scripts break if the locale is set to C, try en_US{.UTF-8,} first
Additionally, drop all locale env vars (LC_*, LANG[GUAGE]) when setting the new locale.
---
Just for reference, a small python example that breaks if LANG is set to C:
$ LANG=C python -c "print(u'Andr\xe9')" UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)
--- scripts/autobuild-run | 48 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-)
scripts/autobuild-run +44 -4
@@ -168,6 +168,12 @@ class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] + # list of default locales (in lowercase, without "-", descending order) + # some python scripts break if the locale is set to C, try en_US first + TRY_LOCALES = ['en_us.utf8', 'en_us', 'c'] + # list of locale environment variables that should be (re-)set by set_locale() + LOCALE_KEYS = ['LANG'] + def __init__(self): self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS)
@@ -231,6 +237,41 @@ class SystemInfo: return not missing_requirements + def set_locale(self): + def is_locale_env_varname(w): + # w[:4] == 'LANG' and (not w[4:] or w[4:] == 'UAGE') ... + return w[:3] == 'LC_' or w == 'LANG' or w == 'LANGUAGE' + + ret, locales_str = self.run_cmd_get_stdout(["locale", "-a"]) + if ret != os.EX_OK: + return False + + # create a dict + # <locale identifier> (as listed in TRY_LOCALES) => <locale env name> + locales = dict(( + (k.lower().replace("-", ""), k) for k in locales_str.split(None) + )) + + for loc_key in filter(lambda x: x in locales, self.TRY_LOCALES): + # cannot modify self.env while iterating over it, + # create intermediate list + env_old_locale_keys = [ + k for k in self.env.keys() if is_locale_env_varname(k) + ] + for k in env_old_locale_keys: + del self.env[k] + + # set new locale once + for vname in self.LOCALE_KEYS: + self.env[vname] = locales[loc_key] + return True + # -- end for + # practically impossible to reach this return if 'c' is in TRY_LOCALES: + return None + + def sanitize_env(self): + self.set_locale() + def popen(self, cmdv, **kwargs): kwargs.setdefault('stdin', self.devnull) kwargs.setdefault('stdout', self.devnull)
@@ -789,12 +830,11 @@ def merge(dict_1, dict_2): def main(): - # Avoid locale settings of autobuilder machine leaking in, for example - # showing error messages in another language. - os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() + # Avoid locale settings of autobuilder machine leaking in, for example + # showing error messages in another language. + sysinfo.sanitize_env() args = docopt.docopt(doc, version=VERSION)
Signature · 1 line
2.3.2
PATCH v2 10/10 autobuild-run: set locale to en_US or C
Hi Andr?, Thomas, all,
Quoted 1 line · Andr? Erdmann
some python scripts break if the locale is set to C, try en_US{.UTF-8,} first
Could you give an example of such a script?
Quoted 34 lines
Additionally, drop all locale env vars (LC_*, LANG[GUAGE]) when setting the new locale.
Signed-off-by Andr? Erdmann <[email protected]>---
Just for reference, a small python example that breaks if LANG is set to C:
$ LANG=C python -c "print(u'Andr\xe9')" UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)--- scripts/autobuild-run | 48 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-)scripts/autobuild-run +6
-0@@ -168,6 +168,12 @@ class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] + # list of default locales (in lowercase, without "-", descending order) + # some python scripts break if the locale is set to C, try en_US first + TRY_LOCALES = ['en_us.utf8', 'en_us', 'c'] + # list of locale environment variables that should be (re-)set by set_locale() + LOCALE_KEYS = ['LANG'] + def __init__(self): self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS)
scripts/autobuild-run class SystemInfo: · line 242
Quoted 6 lines
scripts/autobuild-run +3
-0@@ -231,6 +237,41 @@ class SystemInfo: return not missing_requirements + def set_locale(self): + def is_locale_env_varname(w): + # w[:4] == 'LANG' and (not w[4:] or w[4:] == 'UAGE') ...
Hmm... I must say the comment is rather confusing!
scripts/autobuild-run class SystemInfo: · line 243
Quoted 1 line
scripts/autobuild-run +1
-0+ return w[:3] == 'LC_' or w == 'LANG' or w == 'LANGUAGE'
or: return w.startswith('LC_') or w in ['LANG', 'LANGUAGE']
scripts/autobuild-run class SystemInfo: · line 252
Quoted 9 lines
scripts/autobuild-run +9
-0+ + ret, locales_str = self.run_cmd_get_stdout(["locale", "-a"])[ + if ret != os.EX_OK: + return False + + # create a dict + # <locale identifier> (as listed in TRY_LOCALES) => <locale env name> + locales = dict(( + (k.lower().replace("-", ""), k) for k in locales_str.split(None)
s/None//
scripts/autobuild-run class SystemInfo: · line 255
Quoted 3 lines
scripts/autobuild-run +3
-0+ )) + + for loc_key in filter(lambda x: x in locales, self.TRY_LOCALES):
Is a for loop really needed here?
You could just do:
try:[
loc_key = filter(lambda x: x in locales, self.TRY_LOCALES)[0]
except (TypeError, IndexError):
# set "C" as locale
loc_key = self.TRY_LOCALES[-1]scripts/autobuild-run class SystemInfo: · line 262
Quoted 7 lines
scripts/autobuild-run +7
-0+ # cannot modify self.env while iterating over it, + # create intermediate list + env_old_locale_keys = [ + k for k in self.env.keys() if is_locale_env_varname(k) + ] + for k in env_old_locale_keys: + del self.env[k]
Setting the new value will automatically override the old value, so deleting it before looks a bit overkill...
Quoted 34 lines
scripts/autobuild-run +15
-5+ + # set new locale once + for vname in self.LOCALE_KEYS: + self.env[vname] = locales[loc_key] + return True + # -- end for + # practically impossible to reach this return if 'c' is in TRY_LOCALES: + return None + + def sanitize_env(self): + self.set_locale() + def popen(self, cmdv, **kwargs): kwargs.setdefault('stdin', self.devnull) kwargs.setdefault('stdout', self.devnull)@@ -789,12 +830,11 @@ def merge(dict_1, dict_2): def main(): - # Avoid locale settings of autobuilder machine leaking in, for example - # showing error messages in another language. - os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() + # Avoid locale settings of autobuilder machine leaking in, for example + # showing error messages in another language. + sysinfo.sanitize_env() args = docopt.docopt(doc, version=VERSION) --2.3.2
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Regards,
Signature · 1 line
Samuel
PATCH v2 10/10 autobuild-run: set locale to en_US or C
Hi,
2015-04-12 10:31 GMT+02:00 Samuel Martin <[email protected]>:
Quoted 6 lines
Hi Andr?, Thomas, all,
Quoted 1 line · Andr? Erdmann
some python scripts break if the locale is set to C, try en_US{.UTF-8,} first
Could you give an example of such a script?
Apart from the small snippet I've included below the commit message:
no, I don't have any real-world script at hand that exposes this issue and is sufficiently small. In general, it occurs when a script relies on the default system encoding derived from the LANG/LC_ and has to deal with non-ascii strings, e.g. when using open() and not io.open()/codecs.open() to read a file.
Quoted 35 lines
Quoted 35 lines
Additionally, drop all locale env vars (LC_*, LANG[GUAGE]) when setting the new locale.
Signed-off-by Andr? Erdmann <[email protected]>---
Just for reference, a small python example that breaks if LANG is set to C:
$ LANG=C python -c "print(u'Andr\xe9')" UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)--- scripts/autobuild-run | 48 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-)scripts/autobuild-run +6
-0@@ -168,6 +168,12 @@ class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] + # list of default locales (in lowercase, without "-", descending order) + # some python scripts break if the locale is set to C, try en_US first + TRY_LOCALES = ['en_us.utf8', 'en_us', 'c'] + # list of locale environment variables that should be (re-)set by set_locale() + LOCALE_KEYS = ['LANG'] + def __init__(self): self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS)@@ -231,6 +237,41 @@ class SystemInfo:
Quoted 7 lines
scripts/autobuild-run class SystemInfo: · line 242
Quoted 5 lines
scripts/autobuild-run +3
-0return not missing_requirements + def set_locale(self): + def is_locale_env_varname(w): + # w[:4] == 'LANG' and (not w[4:] or w[4:] == 'UAGE') ...Hmm... I must say the comment is rather confusing!
I'll drop it.
I usually take some "this could be optimized" notes while coding, this one aims at reducing the str parts that need to be matched more than once, because the word w matches "LANG" or "LANGUAGE" if the first 4 chars match "LANG" and the remainder is either empty or "UAGE".
Quoted 17 lines
scripts/autobuild-run class SystemInfo: · line 243
Quoted 1 line
scripts/autobuild-run +1
-0+ return w[:3] == 'LC_' or w == 'LANG' or w == 'LANGUAGE'or: return
w.startswith('LC_')or w in ['LANG', 'LANGUAGE']scripts/autobuild-run class SystemInfo: · line 252
Quoted 9 lines
scripts/autobuild-run +9
-0+ + ret, locales_str = self.run_cmd_get_stdout(["locale", "-a"])[ + if ret != os.EX_OK: + return False + + # create a dict + # <locale identifier> (as listed in TRY_LOCALES) => <locale env name> + locales = dict(( + (k.lower().replace("-", ""), k) for k in locales_str.split(None)s/None//
scripts/autobuild-run class SystemInfo: · line 255
Quoted 3 lines
scripts/autobuild-run +3
-0+ )) + + for loc_key in filter(lambda x: x in locales, self.TRY_LOCALES):
Quoted 8 lines
Is a for loop really needed here?
You could just do:
try:[
loc_key=filter(lambda x: x in locales, self.TRY_LOCALES)[0]
except (TypeError, IndexError):# set "C" as locale loc_key = self.TRY_LOCALES[-1]
I've chosen a for-loop here because it works with both python 2 and python 3 equally well:
* in python 2: filter() returns a list => "filter(...)[0]" enclosed by a "try/except IndexError" block
* in python 3: filter() returns a generator object => "next(filter(...))" enclosed by a "try/except StopIteration" block
None of these variants are compatible with each other, because you can't access generator items by index (filter(...)[0] raises a TypeError in py3), and lists are not iterators (next(filter(...)) raises a TypeError in py2).
What would work for both py2/py3 is "next(iter(filter(...)))" enclosed by a "try/except StopIteration" block, but the for-loop is more readable.
Quoted 10 lines
scripts/autobuild-run class SystemInfo: · line 262
Quoted 7 lines
scripts/autobuild-run +7
-0+ # cannot modify self.env while iterating over it, + # create intermediate list + env_old_locale_keys = [ + k for k in self.env.keys() if is_locale_env_varname(k) + ] + for k in env_old_locale_keys: + del self.env[k]Setting the new value will automatically override the old value, so deleting it before looks a bit overkill...
Deleting/Inserting is asymmetrical:
* drop any env var matching LANG, LANGUAGE or LC_* (LC_MESSAGES et al)
* set LANG to the new locale
We could also simply set LC_ALL instead of deleting LC_*, but that opens up the possibility to leak LC_ env vars when LC_ALL gets unset/emptied:
$ env LC_MESSAGES=en_US.UTF-8 LC_ALL=de_DE.UTF-8 sh -c 'locale | grep LC_MES; export LC_ALL=; locale | grep LC_MES;'
LC_MESSAGES="de_DE.UTF-8"
LC_MESSAGES=en_US.UTF-8
Quoted 34 lines
Quoted 34 lines
scripts/autobuild-run +15
-5+ + # set new locale once + for vname in self.LOCALE_KEYS: + self.env[vname] = locales[loc_key] + return True + # -- end for + # practically impossible to reach this return if 'c' is in TRY_LOCALES: + return None + + def sanitize_env(self): + self.set_locale() + def popen(self, cmdv, **kwargs): kwargs.setdefault('stdin', self.devnull) kwargs.setdefault('stdout', self.devnull)@@ -789,12 +830,11 @@ def merge(dict_1, dict_2): def main(): - # Avoid locale settings of autobuilder machine leaking in, for example - # showing error messages in another language. - os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() + # Avoid locale settings of autobuilder machine leaking in, for example + # showing error messages in another language. + sysinfo.sanitize_env() args = docopt.docopt(doc, version=VERSION) --2.3.2
Signature · 1 line
Andr?
v2 cover autobuild-run: python3 compat and misc improvements
In reply to André ErdmannDear Andr? Erdmann,
Quoted 8 lines · Andr? Erdmann
Andr? Erdmann (10):
autobuild-run: fixup imports
autobuild-run, python3: dict.iteritems()->items() autobuild-run, python3: decode subprocess output autobuild-run, python3: encode/decode mmap data
autobuild-run: avoid use of builtin as varname
autobuild-run: reorganize imports
autobuild-run: movecheck_requirements()to SystemInfo
Those 7 first patches applied.
Quoted 3 lines
autobuild-run: encapsulate subprocess calls
autobuild-run: control subprocess env
autobuild-run: set locale toen_USor C
I'll need more time (or help from someone else) to review these changes.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com