autobuild-run: fix typo in help text
PATCH v2 1/7 autobuild-run: fix typo in help text
--- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/autobuild-run +1 -1
@@ -86,7 +86,7 @@ Options: -s, --submitter SUBMITTER name/machine of submitter Defaults to %(--submitter)s. --http-url URL URL of resource to submit your results. - Defaults to $(--http-url)s. + Defaults to %(--http-url)s. --http-login LOGIN username to send results with Not set by default. --http-password PASSWORD password to send results with (for security
Signature · 1 line
2.11.0
PATCH v2 2/7 autobuild-run: add '-d, --debug' option
While changing and debugging the autobuild-run script, it is convenient to be able to see the log directly on stdout rather than opening the log file. So provide an option for this.
The option is called '--debug' with the idea that other debugging features added later can reuse the same option.
--- scripts/autobuild-run | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
scripts/autobuild-run +7 -2
@@ -101,6 +101,7 @@ Options: Defaults to %(--pid-file)s. -c, --config CONFIG path to configuration file Not set by default. + -d, --debug Send log output to stdout instead of log file --toolchains-url URL URL of toolchain configuration file Format of the configuration file:
@@ -800,7 +801,10 @@ def run_instance(**kwargs): if not os.path.exists(idir): os.mkdir(idir) - kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") + if kwargs['debug']: + kwargs['log'] = sys.stdout + else: + kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") log_write(kwargs['log'], "INFO: instance started") while True:
@@ -926,7 +930,8 @@ def main(): nice = (args['--nice'] or 0), toolchains_url = args['--toolchains-url'], upload = upload, - buildpid = buildpid + buildpid = buildpid, + debug = args['--debug'] )) p.start() processes.append(p)
Signature · 1 line
2.11.0
PATCH v2 3/7 autobuild-run: use 'olddefconfig' instead of 'oldconfig'
In reply to Arnout Vandecappelle (Essensium/Mind)Back in the days that autobuild-run was created, we didn't have an olddefconfig target so we had to pipe yes into oldconfig instead. Nowadays, however, we can use the olddefconfig target.
---
v2: Also use 'olddefconfig' for the second call to 'oldconfig'
--- scripts/autobuild-run | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/autobuild-run +4 -4
@@ -558,8 +558,8 @@ def gen_config(**kwargs): devnull = open(os.devnull, "w") - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "olddefconfig"], + stdout=devnull, stderr=devnull) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1
@@ -586,8 +586,8 @@ def gen_config(**kwargs): if fixup_config(**kwargs): break - ret = subprocess.call(["yes '' 2>/dev/null| make O=%s -C %s oldconfig" % \ - (outputdir, srcdir)], shell=True, stdout=devnull, stderr=devnull) + ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "olddefconfig"], + stdout=devnull, stderr=devnull) if ret != 0: log_write(log, "ERROR: cannot oldconfig") return -1
Signature · 1 line
2.11.0
PATCH v2 4/7 autobuild-run: check that toolchain config lines are still present
In reply to Arnout Vandecappelle (Essensium/Mind)Some lines from the toolchain config may be removed due to dependency issues. Currently this is covered by explicit conditions in the autobuild-run script, e.g. checking that libc is not glibc before enabling BR2_STATIC_LIBS. However, that binds this script pretty tightly to the logic in Buildroot itself.
So instead, just check that the toolchain configuration is still valid after running 'olddefconfig', and discard the configuration if it isn't. This is similar to how it's done in the test-pkg script.
---
v2: don't report the missing lines in the log file (Thomas)
--- scripts/autobuild-run | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
scripts/autobuild-run +6 -1
@@ -366,6 +366,11 @@ def is_toolchain_usable(**kwargs): with open(os.path.join(outputdir, ".config")) as configf: configlines = configf.readlines() + # Check that the toolchain configuration is still present + for toolchainline in kwargs['config']: + if toolchainline not in configlines: + return False + # The latest Linaro toolchains on x86-64 hosts requires glibc # 2.14+ on the host. if platform.machine() == 'x86_64':
@@ -564,7 +569,7 @@ def gen_config(**kwargs): log_write(log, "ERROR: cannot oldconfig") return -1 - if not is_toolchain_usable(**kwargs): + if not is_toolchain_usable(config=config["contents"], **kwargs): return -1 # Now, generate the random selection of packages, and fixup
Signature · 1 line
2.11.0
PATCH v2 5/7 autobuild-run: remove check for glibc
In reply to Arnout Vandecappelle (Essensium/Mind)Now that we check the validity of the toolchain config after running 'make olddefconfig', it is no longer necessary to have an explicit check of glibc before enabling BR2_STATIC_LIBS.
This will be necessary to be able to use the in-tree toolchain configs.
--- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/autobuild-run +1 -1
@@ -554,7 +554,7 @@ def gen_config(**kwargs): configlines.append("BR2_INIT_SYSTEMD=y\n") elif randint(0, 20) == 0: configlines.append("BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y\n") - if config["libc"] != "glibc" and randint(0, 20) == 0: + if randint(0, 20) == 0: configlines.append("BR2_STATIC_LIBS=y\n") # Write out the configuration file
Signature · 1 line
2.11.0
PATCH v2 6/7 autobuild-run: simplify, don't make a dict of config
In reply to Arnout Vandecappelle (Essensium/Mind)A toolchain config was defined as a dict with keys url, hostarch, libc and contents. Of these, only contents and libc were actually used outside of get_toolchain_configs(). And the use of libc has been removed in the previous patch.
So there is no reason anymore to make config a dict. Just replace it with its contents.
--- scripts/autobuild-run | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-)
scripts/autobuild-run +11 -15
@@ -252,11 +252,8 @@ class SystemInfo: def get_toolchain_configs(**kwargs): """Fetch and return the possible toolchain configurations - This function returns an array of dictionaries, with for each toolchain: - - url: the URL of the toolchain defconfig - - libc: the C library used by the toolchain - - hostarch: the host architecture for which the toolchain is built - - contents: an array of lines of the defconfig + This function returns an array of toolchain configurations. Each + toolchain configuration is itself an array of lines of the defconfig. """ toolchains_url = kwargs['toolchains_url']
@@ -271,31 +268,30 @@ def get_toolchain_configs(**kwargs): for row in csv.reader(l): config = {} - config["url"] = row[0] - config["hostarch"] = row[1] + url = row[0] + config_hostarch = row[1] keep = False # Keep all toolchain configs that work regardless of the host # architecture - if config['hostarch'] == "any": + if config_hostarch == "any": keep = True # Keep all toolchain configs that can work on the current host # architecture - if hostarch == config["hostarch"]: + if hostarch == config_hostarch: keep = True # Assume that x86 32 bits toolchains work on x86_64 build # machines - if hostarch == 'x86_64' and config["hostarch"] == "x86": + if hostarch == 'x86_64' and config_hostarch == "x86": keep = True if not keep: continue - config["libc"] = row[2] - with urlopen_closing(config["url"]) as r: - config["contents"] = decode_byte_list(r.readlines()) + with urlopen_closing(url) as r: + config = decode_byte_list(r.readlines()) configs.append(config) return configs
@@ -542,7 +538,7 @@ def gen_config(**kwargs): i = randint(0, len(configs) - 1) config = configs[i] - configlines = config["contents"] + configlines = config # Amend the configuration with a few things. configlines.append("BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y\n")
@@ -569,7 +565,7 @@ def gen_config(**kwargs): log_write(log, "ERROR: cannot oldconfig") return -1 - if not is_toolchain_usable(config=config["contents"], **kwargs): + if not is_toolchain_usable(config=config, **kwargs): return -1 # Now, generate the random selection of packages, and fixup
Signature · 1 line
2.11.0
PATCH v2 6/7 autobuild-run: simplify, don't make a dict of config
Hello,
scripts/autobuild-run def gen_config(**kwargs): · line 541
Quoted 2 lines · Arnout Vandecappelle (Essensium/Mind)
scripts/autobuild-run +1
-1- configlines = config["contents"] + configlines = config
So, using directly the variable named "config" in the remainder of this function would be more logical, no?
(Note: as I said, I have already applied the patch, so further improvements need to be done as follow-up patches)
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH v2 6/7 autobuild-run: simplify, don't make a dict of config
Quoted 10 lines · Thomas Petazzoni
Hello,
scripts/autobuild-run def gen_config(**kwargs): · line 541
Quoted 2 lines · Arnout Vandecappelle (Essensium/Mind)
scripts/autobuild-run +1
-1- configlines = config["contents"] + configlines = configSo, using directly the variable named "config" in the remainder of this function would be more logical, no?
Indeed, it looks kind of ridiculous now...
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v2 7/7 autobuild-run: use in-tree toolchain configs
In reply to Arnout Vandecappelle (Essensium/Mind)Instead of having the list of toolchain configs as a CSV file, they are now maintained within the buildroot tree itself. Therefore, autobuild-run must get the toolchain configs from there.
The toolchains-url option is replaced with a toolchains-path option. This path is relative to the buildroot directory and defaults to support/config-fragments/autobuild. It is possible to override this with a relative or absolute path to the directory containing the toolchain configs.
The toolchain configs no longer contain the hostarch information. For the predefined external toolchains, this is handled by checking after the olddefconfig: because of "depends on HOSTARCH=...", the toolchain config will have been removed. For the custom external toolchains, we instead rely on the URL: it ends with -HOSTARCH.tar.gz.
Since we don't fetch from a URL anymore, the exception handling introduced in 5f1016ab0 is no longer needed.
---
Clearly, at the moment, the custom external toolchain tarball names *don't* end with -HOSTARCH, so the hostarch check doesn't actually work. However, I think this is the most elegent solution. Other alternatives are:
- Adding a dummy config option BR2_HOSTARCH=... to the toolchain config fragments. This works pretty nicely, because then it will be covered automatically by the check that all the toolchain config lines are still there. However, this requires manual handling when generating the toolchain configs (they are no longer the result of "make savedefconfig"). Also, it is not enough for the x86 case (i.e. the ctng toolchains): these also work on x86_64 autobuilders, so special handling for that case would be needed.
- As it happens, all br-* toolchains are x86_64, and all ctng toolchains are x86, so we could just do string matching on those to derive the hostarch. However, that would be too much of a hack for my comfort.
Any other ideas are welcome!
If this solution is chosen, I propose to replace
http://patchwork.ozlabs.org/patch/748187/ with the renamed tarballs immediately while applying.
--- scripts/autobuild-run | 69 ++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 37 deletions(-)
scripts/autobuild-run +32 -37
@@ -67,7 +67,7 @@ defaults = { '--nice': 0, '--pid-file': '/tmp/buildroot-autobuild.pid', '--http-url': 'http://autobuild.buildroot.org/submit/', - '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv', + '--toolchains-path': 'support/config-fragments/autobuild', } doc = """autobuild-run - run Buildroot autobuilder
@@ -102,7 +102,8 @@ Options: -c, --config CONFIG path to configuration file Not set by default. -d, --debug Send log output to stdout instead of log file - --toolchains-url URL URL of toolchain configuration file + --toolchains-path URL Path to the toolchain configuration files + (relative to buildroot dir). Format of the configuration file:
@@ -169,7 +170,7 @@ else: encode_str = _identity MAX_DURATION = 60 * 60 * 8 -VERSION = 1 +VERSION = 2 def log_write(logf, msg): logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg))
@@ -255,10 +256,14 @@ def get_toolchain_configs(**kwargs): This function returns an array of toolchain configurations. Each toolchain configuration is itself an array of lines of the defconfig. """ - toolchains_url = kwargs['toolchains_url'] + toolchains_path = kwargs['toolchains_path'] + + if os.path.isabs(toolchains_path): + toolchains_dir = toolchains_path + else: + idir = "instance-%d" % kwargs['instance'] + toolchains_dir = os.path.join(idir, "buildroot", toolchains_path) - with urlopen_closing(toolchains_url) as r: - l = decode_byte_list(r.readlines()) configs = [] (_, _, _, _, hostarch) = os.uname()
@@ -266,33 +271,26 @@ def get_toolchain_configs(**kwargs): if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86': hostarch = 'x86' - for row in csv.reader(l): - config = {} - url = row[0] - config_hostarch = row[1] - keep = False - - # Keep all toolchain configs that work regardless of the host - # architecture - if config_hostarch == "any": + for configfile in os.listdir(toolchains_dir): + if configfile.endswith('.config'): keep = True - - # Keep all toolchain configs that can work on the current host - # architecture - if hostarch == config_hostarch: - keep = True - - # Assume that x86 32 bits toolchains work on x86_64 build - # machines - if hostarch == 'x86_64' and config_hostarch == "x86": - keep = True - - if not keep: - continue - - with urlopen_closing(url) as r: - config = decode_byte_list(r.readlines()) - configs.append(config) + with open(os.path.join(toolchains_dir, configfile)) as r: + config = decode_byte_list(r.readlines()) + for configline in config: + # If the toolchain is for a specific arch, check it. The prebuilt toolchains + # are only for x86 or x86_64. If the URL doesn't specify an arch, assume it + # is OK. + m = re.match(r'^BR2_TOOLCHAIN_EXTERNAL_URL=".*(x86|x86_64)\.tar', configline) + if m: + configarch = m.group(1) + if configarch != hostarch: + keep = False + # Assume that x86 32 bits toolchains work on x86_64 + # build machines + if configarch == 'x86' and hostarch == 'x86_64': + keep = True + if keep: + configs.append(config) return configs def prepare_build(**kwargs):
@@ -530,10 +528,7 @@ def gen_config(**kwargs): log_write(log, "INFO: generate the configuration") # Select a random toolchain configuration - try: - configs = get_toolchain_configs(**kwargs) - except: - return -1 + configs = get_toolchain_configs(**kwargs) i = randint(0, len(configs) - 1) config = configs[i]
@@ -929,7 +924,7 @@ def main(): submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), - toolchains_url = args['--toolchains-url'], + toolchains_path = args['--toolchains-path'], upload = upload, buildpid = buildpid, debug = args['--debug']
Signature · 1 line
2.11.0
Thomas Petazzoni
AppliedPATCH v2 1/7 autobuild-run: fix typo in help text
In reply to Arnout Vandecappelle (Essensium/Mind)Hello,
Quoted 4 lines · Arnout Vandecappelle (Essensium/Mind)
--- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
I've applied patches 1 to 6. For patch 7, it needs to wait until I apply the in-tree autobuild defconfigs.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com