autobuild-run: control subprocess stdio and env
v3 cover autobuild-run: control subprocess stdio and env
Changes v2->v3:
* add __doc__ comments to SystemInfo popen(), run_cmd*()
* incorporate Samuel Martin's suggestions:
* s/split(None)/split()
* drop bogus comment in is_locale_env_varname()
and use a more compact word matching variant
* redirect stderr to the log file when running "git log", "tail"
But did not re-add the "yes |" pipe to "make ...config" commands and kept set_locale() as-is. I don't think that changes are necessary here, see my replies in the v2 series, but it's open to discussion, of course.
Changes v1->v2:
* catch up with git HEAD: further cleanup/python3-compat changes (patches 1-6)
* "move check_requirements() to SystemInfo" is mostly identical to v1;
changed: use "upload" var, remove entry from TODO list
* "encapsulate subprocess calls" had to be reworked
It now wraps subprocess.Popen() instead of subprocess.call(), and implements run_cmd(), run_cmd_write_to() and (new:) run_cmd_get_stdout() on top of it. This covers all uses cases of the subprocess module and, in the case of run_cmd_get_stdout(), handles the bytes->str conversion.
In v1, the "set-locale" patch added a subprocess.check_output() wrapper, which has been dropped.
* "set subprocess env" is a separate commit now,
used to be part of the "set-locale" patch
Andr? Erdmann (4):
autobuild-run: encapsulate subprocess calls
autobuild-run: log stderr of "git log" / "tail"
autobuild-run: control subprocess env
autobuild-run: set locale to en_US or C
scripts/autobuild-run | 183 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 140 insertions(+), 43 deletions(-)
Signature · 1 line
2.3.6
PATCH v3 1/4 autobuild-run: encapsulate subprocess calls
Preparation step for passing LANG to worker (sub-)processes, allows to redirect stdin/stdout/stderr, which all default to devnull now unless specified otherwise.
This makes the "yes"-pipe in "make oldconfig" redundant.
--- scripts/autobuild-run | 130 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 93 insertions(+), 37 deletions(-)
scripts/autobuild-run +93 -37
@@ -172,6 +172,7 @@ class SystemInfo: self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS) self.progs = {} + self.devnull = open(os.devnull, "w") def find_prog(self, name, flags=os.X_OK, env=os.environ): if not name or name[0] == os.sep: raise ValueError(name)
@@ -203,10 +204,8 @@ class SystemInfo: have_it = self.find_prog(prog) # java[c] needs special care if have_it and prog in ('java', 'javac'): - with open(os.devnull, "w") as devnull: - if subprocess.call("%s -version | grep gcj" % prog, shell=True, - stdout=devnull, stderr=devnull) != 1: - have_it = False + if self.run_cmd("%s -version | grep gcj" % prog, shell=True) != 1: + have_it = False # -- self.progs[prog] = have_it return have_it
@@ -231,6 +230,65 @@ class SystemInfo: return not missing_requirements + def popen(self, cmdv, **kwargs): + """subprocess.Popen() wrapper that redirects stdin, stdout and + stderr to /dev/null unless specified otherwise (via kwargs). + + Returns: subprocess.Popen object + + arguments: + * cmdv -- command (usually a list, tuple or str) + * **kwargs -- additional keyword arguments that will be passed to + subprocess.Popen() + """ + kwargs.setdefault('stdin', self.devnull) + kwargs.setdefault('stdout', self.devnull) + kwargs.setdefault('stderr', self.devnull) + return subprocess.Popen(cmdv, **kwargs) + + def run_cmd(self, cmdv, **kwargs): + """Runs a command and blocks until it terminates. + stdin, stdout and stderr get redirected to /dev/null by default, + see SystemInfo.popen() for details. + + Returns: exit code of the command + + arguments: see SystemInfo.popen() + """ + proc = self.popen(cmdv, **kwargs) + proc.communicate() + return proc.poll() + + def run_cmd_write_to(self, outstream, cmdv, **kwargs): + """Similar to SystemInfo.run_cmd(), + but redirects stdout and stderr to the given output stream. + + Returns: exit code of the command + + Note: + run_cmd_write_to(log, cmd) and run_cmd(cmd, stdout=log, stderr=log) + are interchangeable. + + arguments: + * outstream -- default output stream for stdout/stderr + * cmdv -- command, see SystemInfo.popen() + * **kwargs -- see SystemInfo.popen() + """ + kwargs.update(stdout=outstream, stderr=outstream) + return self.run_cmd(cmdv, **kwargs) + + def run_cmd_get_stdout(self, cmdv, **kwargs): + """Similar to SystemInfo.run_cmd(), but captures stdout. + + Returns: 2-tuple (exit code of the command, decoded stdout or None) + + arguments: see SystemInfo.popen() + """ + proc = self.popen(cmdv, stdout=subprocess.PIPE, **kwargs) + stdout_data, _ = proc.communicate() + return (proc.poll(), decode_bytes(stdout_data) if stdout_data else None) + + def get_toolchain_configs(): """Fetch and return the possible toolchain configurations
@@ -274,6 +332,7 @@ def prepare_build(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] log_write(log, "INFO: preparing a new build")
@@ -298,15 +357,15 @@ def prepare_build(**kwargs): # didn't exist already. srcdir = os.path.join(idir, "buildroot") if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", "git://git.busybox.net/buildroot", srcdir], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["git", "clone", "git://git.busybox.net/buildroot", srcdir]) if ret != 0: log_write(log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "pull"], cwd=abssrcdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to(log, ["git", "pull"], cwd=abssrcdir) if ret != 0: log_write(log, "ERROR: could not pull Buildroot sources") return -1
@@ -315,7 +374,7 @@ def prepare_build(**kwargs): outputdir = os.path.join(idir, "output") if os.path.exists(outputdir): # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) + sysinfo.run_cmd(["rm", "-rf", outputdir]) os.mkdir(outputdir) return 0
@@ -459,6 +518,7 @@ def gen_config(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
@@ -493,10 +553,7 @@ def gen_config(**kwargs): with open(os.path.join(outputdir, ".config"), "w+") as configf: configf.writelines(configlines) - devnull = open(os.devnull, "w") - - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"]) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1
@@ -504,23 +561,20 @@ def gen_config(**kwargs): # Now, generate the random selection of packages, and fixup # things if needed. while True: - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, - "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, + "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"]) if ret != 0: log_write(log, "ERROR: cannot generate random configuration") return -1 if fixup_config(**kwargs): break - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "oldconfig"]) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1 - ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"], - stdout=devnull, stderr=devnull) + ret = sysinfo.run_cmd(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"]) if ret != 0: log_write(log, "ERROR: cannot savedefconfig") return -1
@@ -532,6 +586,7 @@ def do_build(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
@@ -547,7 +602,7 @@ def do_build(**kwargs): "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % kwargs['njobs']] \ + kwargs['make_opts'].split() - sub = subprocess.Popen(cmd, stdout=f, stderr=f) + sub = sysinfo.popen(cmd, stdout=f, stderr=f) kwargs['buildpid'][kwargs['instance']] = sub.pid ret = sub.wait() kwargs['buildpid'][kwargs['instance']] = 0
@@ -563,7 +618,7 @@ def do_build(**kwargs): cmd = ["make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + kwargs['make_opts'].split() - ret = subprocess.call(cmd, stdout=f, stderr=f) + ret = sysinfo.run_cmd_write_to(f, cmd) if ret != 0: log_write(log, "INFO: build failed during legal-info") return -1
@@ -580,6 +635,7 @@ def send_results(result, **kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + sysinfo = kwargs['sysinfo'] outputdir = os.path.abspath(os.path.join(idir, "output")) srcdir = os.path.join(idir, "buildroot")
@@ -598,15 +654,14 @@ def send_results(result, **kwargs): shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"), os.path.join(resultdir, "licenses-manifest.csv")) - subprocess.call(["git log master -n 1 --pretty=format:%%H > %s" % \ + sysinfo.run_cmd(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=srcdir, stderr=None) def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() + lastlines = sysinfo.run_cmd_get_stdout( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")])[1].splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') for line in lastlines:
@@ -621,9 +676,9 @@ def send_results(result, **kwargs): """Save the last part of the build log, starting from the failed package""" def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ + sysinfo.run_cmd(["tail -500 %s > %s" % \ (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + shell=True, stderr=None) reason = get_failure_reason() if not reason:
@@ -680,8 +735,8 @@ def send_results(result, **kwargs): # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, ["tar", "cjf", "results.tar.bz2", "results"], cwd=outputdir) if ret != 0: log_write(log, "ERROR: could not make results tarball") sys.exit(1)
@@ -690,13 +745,14 @@ def send_results(result, **kwargs): # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - "http://autobuild.buildroot.org/submit/"], - stdout=log, stderr=log) + ret = sysinfo.run_cmd_write_to( + log, + ["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + "http://autobuild.buildroot.org/submit/"]) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) else:
Signature · 1 line
2.3.6
PATCH v3 1/4 autobuild-run: encapsulate subprocess calls
Dear Andr? Erdmann,
Quoted 6 lines · Andr? Erdmann
Preparation step for passing LANG to worker (sub-)processes, allows to redirect stdin/stdout/stderr, which all default to devnull now unless specified otherwise.
This makes the "yes"-pipe in "make oldconfig" redundant.Signed-off-by Andr? Erdmann <[email protected]>
I'm fine with the idea, but I have a doubt about the implementation.
scripts/autobuild-run class SystemInfo:
Quoted 7 lines
scripts/autobuild-run +4
-0+ def popen(self, cmdv, **kwargs): + def run_cmd(self, cmdv, **kwargs): + def run_cmd_write_to(self, outstream, cmdv, **kwargs): + def run_cmd_get_stdout(self, cmdv, **kwargs):
Why are these methods part of the SysInfo class? What is the relation between running commands, and verifying which tools are available on the host machine?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 2/4 autobuild-run: log stderr of "git log" / "tail"
In reply to André ErdmannRather than printing error messages from "git log" and "tail" to the autobuilder's console, redirect them to the instance's logfile.
--- scripts/autobuild-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/autobuild-run +2 -2
@@ -656,7 +656,7 @@ def send_results(result, **kwargs): sysinfo.run_cmd(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir, stderr=None) + shell=True, cwd=srcdir, stderr=log) def get_failure_reason(): # Output is a tuple (package, version), or None.
@@ -678,7 +678,7 @@ def send_results(result, **kwargs): def extract_last_500_lines(): sysinfo.run_cmd(["tail -500 %s > %s" % \ (os.path.join(outputdir, "logfile"), resultfile)], - shell=True, stderr=None) + shell=True, stderr=log) reason = get_failure_reason() if not reason:
Signature · 1 line
2.3.6
PATCH v3 3/4 autobuild-run: control subprocess env
In reply to André ErdmannPreparation step for passing LANG to worker (sub-)processes, allows to control execution environment of subprocesses without affecting the "autobuild-run" process itself.
--- scripts/autobuild-run | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/autobuild-run +4 -2
@@ -173,11 +173,12 @@ class SystemInfo: self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS) self.progs = {} self.devnull = open(os.devnull, "w") + self.env = os.environ.copy() - def find_prog(self, name, flags=os.X_OK, env=os.environ): + def find_prog(self, name, flags=os.X_OK): if not name or name[0] == os.sep: raise ValueError(name) - prog_path = env.get("PATH", None) + prog_path = self.env.get("PATH", None) # for windows compatibility, we'd need to take PATHEXT into account if prog_path:
@@ -244,6 +245,7 @@ class SystemInfo: kwargs.setdefault('stdin', self.devnull) kwargs.setdefault('stdout', self.devnull) kwargs.setdefault('stderr', self.devnull) + kwargs['env'] = self.env return subprocess.Popen(cmdv, **kwargs) def run_cmd(self, cmdv, **kwargs):
Signature · 1 line
2.3.6
PATCH v3 4/4 autobuild-run: set locale to en_US or C
In reply to André Erdmannsome python scripts break if the locale is set to C, try en_US first
Additionally, drop all locale env vars (LC_*, LANG[GUAGE]) when setting the new locale.
--- scripts/autobuild-run | 47 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-)
scripts/autobuild-run +43 -4
@@ -168,6 +168,12 @@ class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] + # list of default locales (in lowercase, without "-", descending order) + # some python scripts break if the locale is set to C, try en_US first + TRY_LOCALES = ['en_us.utf8', 'en_us', 'c'] + # list of locale environment variables that should be (re-)set by set_locale() + LOCALE_KEYS = ['LANG'] + def __init__(self): self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS) self.optional_progs = list(self.__class__.DEFAULT_OPTIONAL_PROGS)
@@ -231,6 +237,40 @@ class SystemInfo: return not missing_requirements + def set_locale(self): + def is_locale_env_varname(w): + return w.startswith('LC_') or w in ['LANG', 'LANGUAGE'] + + ret, locales_str = self.run_cmd_get_stdout(["locale", "-a"]) + if ret != os.EX_OK: + return False + + # create a dict + # <locale identifier> (as listed in TRY_LOCALES) => <locale env name> + locales = dict(( + (k.lower().replace("-", ""), k) for k in locales_str.split() + )) + + for loc_key in filter(lambda x: x in locales, self.TRY_LOCALES): + # cannot modify self.env while iterating over it, + # create intermediate list + env_old_locale_keys = [ + k for k in self.env.keys() if is_locale_env_varname(k) + ] + for k in env_old_locale_keys: + del self.env[k] + + # set new locale once + for vname in self.LOCALE_KEYS: + self.env[vname] = locales[loc_key] + return True + # -- end for + # practically impossible to reach this return if 'c' is in TRY_LOCALES: + return None + + def sanitize_env(self): + self.set_locale() + def popen(self, cmdv, **kwargs): """subprocess.Popen() wrapper that redirects stdin, stdout and stderr to /dev/null unless specified otherwise (via kwargs).
@@ -830,12 +870,11 @@ def merge(dict_1, dict_2): def main(): - # Avoid locale settings of autobuilder machine leaking in, for example - # showing error messages in another language. - os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() + # Avoid locale settings of autobuilder machine leaking in, for example + # showing error messages in another language. + sysinfo.sanitize_env() args = docopt.docopt(doc, version=VERSION)
Signature · 1 line
2.3.6
PATCH v3 4/4 autobuild-run: set locale to en_US or C
Dear Andr? Erdmann,
Quoted 9 lines · Andr? Erdmann
some python scripts break if the locale is set to C, try
en_USfirstAdditionally, drop all locale env vars (LC_*, LANG[GUAGE]) when setting the new locale.
Signed-off-by Andr? Erdmann <[email protected]>--- scripts/autobuild-run | 47 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-)
This commit looks very, very complicated. What is the original problem exactly? You say "some python scripts break", but that isn't a very precise description.
Quoted 20 lines
scripts/autobuild-run +20
-0+ def set_locale(self): + def is_locale_env_varname(w): + return w.startswith('LC_') or w in ['LANG', 'LANGUAGE'] + + ret, locales_str = self.run_cmd_get_stdout(["locale", "-a"]) + if ret != os.EX_OK: + return False + + # create a dict + # <locale identifier> (as listed in TRY_LOCALES) => <locale env name> + locales = dict(( + (k.lower().replace("-", ""), k) for k in locales_str.split() + )) + + for loc_key in filter(lambda x: x in locales, self.TRY_LOCALES): + # cannot modify self.env while iterating over it, + # create intermediate list + env_old_locale_keys = [ + k for k in self.env.keys() if is_locale_env_varname(k) + ]
scripts/autobuild-run class SystemInfo: · line 269
Quoted 10 lines
scripts/autobuild-run +10
-0+ for k in env_old_locale_keys: + del self.env[k] + + # set new locale once + for vname in self.LOCALE_KEYS: + self.env[vname] = locales[loc_key] + return True + # -- end for + # practically impossible to reach this return if 'c' is in TRY_LOCALES: + return None
There's really nothing simpler than this mess?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
v3 cover autobuild-run: control subprocess stdio and env
In reply to André ErdmannAndr?,
Quoted 5 lines · Andr? Erdmann
Andr? Erdmann (4):
autobuild-run: encapsulate subprocess calls
autobuild-run: log stderr of "git log" / "tail"
autobuild-run: control subprocess env
autobuild-run: set locale toen_USor C
There were some questions on patches 1 and 4, but you never gave any feedback. Can you take into account the comments, and resend an updated version?
In the mean time, I've marked this version of the patches as Changes Requested in patchwork.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com