autobuild: misc fixes and enhancements (branch yem/limits)
v3 cover autobuild: misc fixes and enhancements (branch yem/limits)
Hello All!
This series, against the autobuilder tree, adds two new configuration options, a sample runtime config file, and a SysV init startup script. It also fixes an infinite loop when the host glibc is too old and the Linaro toolchains are used.
The two new rutime options are:
--nice N specify the niceness of the 'make' command
--pid-file file where to save the main process PID
The niceness allows limitting the impact of the autobuilder on the rest of the job, when on a shared machine. A better solution would be to implement a container with cgroups, but a systemd unit file would be so much convenient! ;-)
The PID file allows to simplify the startup script, and really ensure that we get the autobuild process' PID.
A SysV init startup script, as well as a sample defaults config file are provided, to ease running the autobuild script as a daemon.
A sample runtime configuration file is provided as well.
As for the infinite loop, the fix consist in two changes:
- separating the check that the configuration is actually fixable from the actual fixup,
- adding a safeguard that limits the loop to at most 100 iterationd, and ignores that configuration if the limit is reached.
Note that nothing in there is really bullet-proof, but has been running smoothly on a private instance for about a few hours now, and the failure/succes rate is on par with what we currently have in the autobuilders (two patches to fix build issues due to the host configuration have already been caught, and patches submitted)
Changes v2 -> v3:
- add the infinite loop fixes
Changes v1 -> v2;
- add PID file handling
- add startup script and sample configuration files
- typo in --nice help text
Regards,
Yann E. MORIN.
The following changes since commit 611b515aa40c635d1af24883a9eab9e1e363a33c:
Add mips32r6 and mips64r6 toolchains (2015-11-22 15:31:07 +0100)
are available in the git repository at:
git://git.busybox.net/~ymorin/git/buildroot-test yem/limits
for you to fetch changes up to a89f4ae40bd1f7d5600226eb0ac20f27098413fb:
autobuild: check for Linaro toolchains earlier (2015-11-27 16:00:21 +0100)
----------------------------------------------------------------
Yann E. MORIN (7):
autobuild: add config option to set niceness
autobuild: store PID to PID file
conf: add a sample run-time configuration file
etc: add SysV-init startup script and sample config file
autobuild: avoid infinite loop when sanitising the configuration
autobuild: add a function to check if the configuration is fixable
autobuild: check for Linaro toolchains earlier
conf/buildroot-autobuild.sample | 5 ++ etc/default/buildroot-autobuild | 23 +++++++ etc/init.d/buildroot-autobuild | 148 ++++++++++++++++++++++++++++++++++++++++ scripts/autobuild-run | 66 +++++++++++++++--- 4 files changed, 231 insertions(+), 11 deletions(-) create mode 100644 conf/buildroot-autobuild.sample create mode 100644 etc/default/buildroot-autobuild create mode 100755 etc/init.d/buildroot-autobuild
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 1/7 autobuild: add config option to set niceness
When the machine running the autobuilder is shared, it is important that the autobuilder instances do not hog all of the CPU.
Add an option to the configuration, so that the user can set the niceness to run the instances with. By default, the niceness is 0, which means the current niceness.
Properly doing so would require setting up a container with proper CPU cgroup, but it's a bit complicated.
--- scripts/autobuild-run | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/autobuild-run +9 -1
@@ -64,6 +64,7 @@ defaults = { '--njobs': '1', '--submitter': 'N/A', '--make-opts': '', + '--nice': 0, } doc = """autobuild-run - run Buildroot autobuilder
@@ -77,6 +78,8 @@ Options: Defaults to %(--ninstances)s. -j, --njobs NJOBS number of parallel jobs Defaults to %(--njobs)s. + --nice N Niceness, positive number + Defaults to %(--nice)s. -s, --submitter SUBMITTER name/machine of submitter Defaults to %(--submitter)s. --http-login LOGIN username to send results with
@@ -101,6 +104,7 @@ Format of the configuration file: [main] ninstances = <value> njobs = <value> + nice = <value> http-login = <value> http-password = <value> submitter = <value>
@@ -540,6 +544,7 @@ def do_build(**kwargs): idir = "instance-%d" % kwargs['instance'] log = kwargs['log'] + nice = kwargs['nice'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the
@@ -551,7 +556,9 @@ def do_build(**kwargs): f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") - cmd = ["timeout", str(MAX_DURATION), "make", "O=%s" % outputdir, + cmd = ["timeout", str(MAX_DURATION), + "nice", "-n", nice, + "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % kwargs['njobs']] \ + kwargs['make_opts'].split()
@@ -851,6 +858,7 @@ def main(): http_password = args['--http-password'], submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), + nice = (args['--nice'] or 0), upload = upload, buildpid = buildpid ))
Signature · 1 line
1.9.1
PATCH v3 1/7 autobuild: add config option to set niceness
Yann,
Quoted 7 lines · Yann E. MORIN
When the machine running the autobuilder is shared, it is important that the autobuilder instances do not hog all of the CPU.
Add an option to the configuration, so that the user can set the niceness to run the instances with. By default, the niceness is 0, which means the current niceness.
Quoted 7 lines
Properly doing so would require setting up a container with proper CPU cgroup, but it's a bit complicated.
Signed-off-by "Yann E. MORIN" <[email protected]>--- scripts/autobuild-run | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
Applied to buildroot-test, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 2/7 autobuild: store PID to PID file
In reply to Yann E. MORINLet the autobuild script store its own PID file, rather than retrieve it from the init script.
--- scripts/autobuild-run | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/autobuild-run +7 -0
@@ -65,6 +65,7 @@ defaults = { '--submitter': 'N/A', '--make-opts': '', '--nice': 0, + '--pid-file': '/tmp/buildroot-autobuild.pid', } doc = """autobuild-run - run Buildroot autobuilder
@@ -92,6 +93,8 @@ Options: --make-opts OPTSTRING string of extra options to pass to Buildroot make, such as specific command wrappers Empty by default. + --pid-file PATH path to a file where to store the PID + Defaults to %(--pid-file)s. -c, --config CONFIG path to configuration file Not set by default.
@@ -807,6 +810,10 @@ def main(): # load in defaults at lowest priority args = merge(args, defaults) + # Save our PID very early, so we can be stopped + with open(args['--pid-file'], "w+") as pidf: + pidf.write("%d" % os.getpid()) + # http_login/password could theoretically be allowed as empty, so check # explicitly on None. upload = (args['--http-login'] is not None) \
Signature · 1 line
1.9.1
PATCH v3 2/7 autobuild: store PID to PID file
Yann,
Quoted 7 lines · Yann E. MORIN
Let the autobuild script store its own PID file, rather than retrieve it from the init script.
Signed-off-by "Yann E. MORIN" <[email protected]>--- scripts/autobuild-run | 7 +++++++ 1 file changed, 7 insertions(+)
Applied to buildroot-test, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 3/7 conf: add a sample run-time configuration file
In reply to Yann E. MORIN--- conf/buildroot-autobuild.sample | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 conf/buildroot-autobuild.sample
conf/buildroot-autobuild.sample new file +5 -0
@@ -0,0 +1,5 @@ +[main] +ninstances = 4 +njobs = 4 +nice = 20 +submitter = John Doe
Signature · 1 line
1.9.1
PATCH v3 3/7 conf: add a sample run-time configuration file
Yann,
Quoted 12 lines · Yann E. MORIN
Signed-off-by "Yann E. MORIN" <[email protected]>--- conf/buildroot-autobuild.sample | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 conf/buildroot-autobuild.sampleconf/buildroot-autobuild.sample new file +0
-0
conf/buildroot-autobuild.sample line 4
Quoted 4 lines
conf/buildroot-autobuild.sample new file +4
-0@@ -0,0 +1,5 @@ +[main] +ninstances = 4 +njobs = 4 +nice = 20
From man nice(1):
Niceness values range from -20 (most favorable to the process) to 19 (least favorable to the process).
So I've changed your example nice value from 20 to 19, and then applied to buildroot-test.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 4/7 etc: add SysV-init startup script and sample config file
In reply to Yann E. MORIN--- etc/default/buildroot-autobuild | 23 +++++++ etc/init.d/buildroot-autobuild | 148 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) create mode 100644 etc/default/buildroot-autobuild create mode 100755 etc/init.d/buildroot-autobuild
etc/default/buildroot-autobuild new file +23 -0
@@ -0,0 +1,23 @@ +# The absolute path, outside the chroot, where the +# buildroot-test tree is cloned, optional +AUTOBUILD_DIR="/home/johndoe/buildroot-test" + +# The absolute path, outside the chroot, where to +# share downloads (not yet implemented) +#AUTOBUILD_DL_DIR="/home/johndoe/dl" + +# The absolute path to chroot into to rn the autobuild script +AUTOBUILD_CHROOT="/var/chroot-autobuild" + +# User (in the chroot) that runs the autobuild script) +AUTOBUILD_USER="buildroot" + +# The absolute path, in the chroot, that is the base of the +# autobuild work dir. In there, you'll have: +# - AUTOBUILD_CHROOT_DIR/buildroot-test +# the buildroot-test tree (bind-mounted from AUTOBUILD_DIR if set) +# - AUTOBUILD_CHROOT_DIR/buildroot-autobuild.conf +# the run-time configuration of the autobuild script +# - AUTOBUILD_CHROOT_DIR/run/ +# the parent directory for all build instances +AUTOBUILD_CHROOT_DIR="/home/buildroot/autobuild/"
etc/init.d/buildroot-autobuild new file +148 -0
148 lines not on this page · raw message
Signature · 1 line
1.9.1
PATCH v3 4/7 etc: add SysV-init startup script and sample config file
Yann,
etc/default/buildroot-autobuild line 3
Quoted 9 lines · Yann E. MORIN
etc/default/buildroot-autobuild new file +3
-0@@ -0,0 +1,23 @@ +# The absolute path, outside the chroot, where the +# buildroot-test tree is cloned, optional +AUTOBUILD_DIR="/home/johndoe/buildroot-test"
What if my buildroot-test tree is just inside the chroot ?
etc/default/buildroot-autobuild line 6
Quoted 2 lines
etc/default/buildroot-autobuild new file +2
-0+# The absolute path, outside the chroot, where to +# share downloads (not yet implemented)
"not yet implemented" -> why do we have some code about it in the script below ?
etc/default/buildroot-autobuild line 9
Quoted 3 lines
etc/default/buildroot-autobuild new file +3
-0+#AUTOBUILD_DL_DIR="/home/johndoe/dl" + +# The absolute path to chroot into to rn the autobuild script
rn -> run
etc/default/buildroot-autobuild line 12
Quoted 3 lines
etc/default/buildroot-autobuild new file +3
-0+AUTOBUILD_CHROOT="/var/chroot-autobuild" + +# User (in the chroot) that runs the autobuild script)
last closing parenthesis unneeded
etc/default/buildroot-autobuild line 16
Quoted 4 lines
etc/default/buildroot-autobuild new file +4
-0+AUTOBUILD_USER="buildroot" + +# The absolute path, in the chroot, that is the base of the +# autobuild work dir. In there, you'll have:
What does it mean "you'll have" ? Does it mean that the init script will set up everything this way, or that it expect things to be already set up that way ?
Quoted 66 lines
etc/default/buildroot-autobuild new file +7
-0+# - AUTOBUILD_CHROOT_DIR/buildroot-test +# the buildroot-test tree (bind-mounted from AUTOBUILD_DIR if set) +# - AUTOBUILD_CHROOT_DIR/buildroot-autobuild.conf +# the run-time configuration of the autobuild script +# - AUTOBUILD_CHROOT_DIR/run/ +# the parent directory for all build instances +AUTOBUILD_CHROOT_DIR="/home/buildroot/autobuild/"etc/init.d/buildroot-autobuild new file +53
-0@@ -0,0 +1,148 @@ +#!/bin/sh +# vim: ft=sh + +### BEGIN INIT INFO +# Provides: buildroot-autobuild +# Required-Start: $network +# Required-Stop: $network +# Default-Start: 2 3 4 5 +# Default-Stop: 1 +# Short-Description: Buildroot autobuilds +### END INIT INFO + +# Expected configuration in the configuration file; +# AUTOBUILD_DIR the absolute path to the shared the autobuild +# git tree; optional +# AUTOBUILD_DL_DIR the absolute path to the shared DL dir; optional +# AUTOBUILD_CHROOT the absolute path to the chroot to run in +# AUTOBUILD_USER the username to run as +# +# The following variable is to be interpreted inside the chroot: +# AUTOBUILD_CHROOT_DIR the absolute path to the directory builds run in + +CFG_FILE="/etc/default/buildroot-autobuild" + +if [ ! -e "${CFG_FILE}" ]; then + printf "ERROR: no autobuilder configuration file\n" >&2 + exit 1 +fi +. "${CFG_FILE}" +if [ -z "${AUTOBUILD_USER}" ]; then + printf "ERROR: no autobuild user\n" >&2 + exit 1 +fi +if [ -z "${AUTOBUILD_DIR}" ]; then + printf "ERROR: no autobuild dir\n" >&2 + exit 1 +fi +if [ -z "${AUTOBUILD_CHROOT}" ]; then + printf "ERROR: no autobuild chroot\n" >&2 + exit 1 +fi +if [ -z "${AUTOBUILD_CHROOT_DIR}" ]; then + printf "ERROR: no autobuild chroot dir\n" >&2 + exit 1 +fi + +# Derived configuration: +# AUTOBUILD_RUN_DIR instances will be created in there +# AUTOBUILD_CMD the autobuild script to run +# AUTOBUILD_CFG the autobuild runtime configuration +AUTOBUILD_RUN_DIR="${AUTOBUILD_CHROOT_DIR}/run" +AUTOBUILD_CMD="${AUTOBUILD_CHROOT_DIR}/buildroot-test/scripts/autobuild-run" +AUTOBUILD_CFG="${AUTOBUILD_CHROOT_DIR}/buildroot-autobuild.conf"
etc/init.d/buildroot-autobuild line 63
Quoted 10 lines
etc/init.d/buildroot-autobuild new file +10
-0+AUTOBUILD_CHROOT_PID_FILE="${AUTOBUILD_CHROOT_DIR}/buildroot-autobuild.pid" +AUTOBUILD_PID_FILE="${AUTOBUILD_CHROOT}/${AUTOBUILD_CHROOT_DIR}/buildroot-autobuild.pid" + +autobuild_start() { + echo "Starting buildroot-autobuild" + + CMD="cd '${AUTOBUILD_RUN_DIR}'" + CMD="${CMD}; '${AUTOBUILD_CMD}' -c '${AUTOBUILD_CFG}' --pid-file '${AUTOBUILD_CHROOT_PID_FILE}' &" + + do_chroot "rm -rf '${AUTOBUILD_RUN_DIR}'"
Why ? Not only this is unnecessary because the autobuild-run script automatically removes instance-X/output. But it is actively harmful because it completely removes the download cache of each instance.
etc/init.d/buildroot-autobuild line 73
Quoted 10 lines
etc/init.d/buildroot-autobuild new file +10
-0+ do_chroot "mkdir -p '${AUTOBUILD_RUN_DIR}'" + do_chroot "${CMD}" +} + +autobuild_stop() { + echo "Stopping buildroot-autobuild" + if [ -f "${AUTOBUILD_PID_FILE}" ]; then + kill $(cat "${AUTOBUILD_PID_FILE}") + fi + rm -f "${AUTOBUILD_PID_FILE}"
The rm -f could possibly be within the if, no ?
Other than that, it generally looks good, but maybe
the /etc/default/buildroot-autobuild file needs a few more comments to explain the expected directory layout and preparations to be done by the user in order to be able to use this script.
Also, maybe you should make the use of the chroot optional: both of us are running autobuild-run in a chroot, but not everyone is doing this.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 4/7 etc: add SysV-init startup script and sample config file
Thomas, All,
On 2015-11-28 15:36 +0100, Thomas Petazzoni spake thusly:
Quoted 13 lines · Yann E. MORIN
etc/default/buildroot-autobuild line 3
Quoted 10 lines
etc/default/buildroot-autobuild new file +3
-0@@ -0,0 +1,23 @@ +# The absolute path, outside the chroot, where the +# buildroot-test tree is cloned, optional +AUTOBUILD_DIR="/home/johndoe/buildroot-test"What if my buildroot-test tree is just inside the chroot ?
That setting is "optional". See below for how it is used.
Quoted 5 lines
etc/default/buildroot-autobuild line 6
Quoted 2 lines
etc/default/buildroot-autobuild new file +2
-0+# The absolute path, outside the chroot, where to +# share downloads (not yet implemented)"not yet implemented" -> why do we have some code about it in the script below ?
Damn, I forgot to remove it...
Quoted 8 lines
etc/default/buildroot-autobuild line 16
Quoted 4 lines
etc/default/buildroot-autobuild new file +4
-0+AUTOBUILD_USER="buildroot" + +# The absolute path, in the chroot, that is the base of the +# autobuild work dir. In there, you'll have:What does it mean "you'll have" ? Does it mean that the init script will set up everything this way, or that it expect things to be already set up that way ?
Ok, I should have expanded on it. Here it is:
- the buildroot-autobuild.conf has to be provided
- the run/ directory is created
- the buildroot-test/ directory is either:
- bind-mounted if AUTOBUILD_DIR is set, or
- must be already present otherwise
Quoted 10 lines
etc/default/buildroot-autobuild line 16
Quoted 7 lines
etc/default/buildroot-autobuild new file +7
-0+autobuild_start() { + echo "Starting buildroot-autobuild" + + CMD="cd '${AUTOBUILD_RUN_DIR}'" + CMD="${CMD}; '${AUTOBUILD_CMD}' -c '${AUTOBUILD_CFG}' --pid-file '${AUTOBUILD_CHROOT_PID_FILE}' &" + + do_chroot "rm -rf '${AUTOBUILD_RUN_DIR}'"Why ? Not only this is unnecessary because the autobuild-run script automatically removes instance-X/output.
Ah, damn. It's in prepare_build(). I missed it.
Quoted 8 lines
etc/default/buildroot-autobuild line 22
Quoted 6 lines
etc/default/buildroot-autobuild new file +6
-0+autobuild_stop() { + echo "Stopping buildroot-autobuild" + if [ -f "${AUTOBUILD_PID_FILE}" ]; then + kill $(cat "${AUTOBUILD_PID_FILE}") + fi + rm -f "${AUTOBUILD_PID_FILE}"The rm -f could possibly be within the if, no ?
Yes.
Quoted 4 lines
Other than that, it generally looks good, but maybe
the/etc/default/buildroot-autobuildfile needs a few more comments to explain the expected directory layout and preparations to be done by the user in order to be able to use this script.
Will update.
Quoted 2 lines
Also, maybe you should make the use of the chroot optional: both of us are running autobuild-run in a chroot, but not everyone is doing this.
Yes. It will anyway be needed when we implement the multi-chroot setup.
Thanks!
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 5/7 autobuild: avoid infinite loop when sanitising the configuration
In reply to Yann E. MORINThere are (hard-to-debug) cases where we can't quickly converge to a valid configuration. In which case, the sanitising loop is just a plain infinite loop.
Limit the loop to at most 100 (arbitrary) iterations, so we're not forever stuck.
--- scripts/autobuild-run | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/autobuild-run +7 -0
@@ -518,7 +518,14 @@ def gen_config(**kwargs): # Now, generate the random selection of packages, and fixup # things if needed. + # Safe-guard, in case we can not quickly come to a valid + # configuration: allow at most 100 (arbitrary) iterations. + bounded_loop = 100 while True: + if bounded_loop == 0: + log_write(log, "ERROR: cannot generate random configuration after 100 iterations") + return -1 + bounded_loop -= 1 ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "KCONFIG_PROBABILITY=%d" % randint(1,30), "randpackageconfig"], stdout=devnull, stderr=devnull)
Signature · 1 line
1.9.1
PATCH v3 5/7 autobuild: avoid infinite loop when sanitising the configuration
Yann,
Quoted 11 lines · Yann E. MORIN
There are (hard-to-debug) cases where we can't quickly converge to a valid configuration. In which case, the sanitising loop is just a plain infinite loop.
Limit the loop to at most 100 (arbitrary) iterations, so we're not forever stuck.
Signed-off-by "Yann E. MORIN" <[email protected]>--- scripts/autobuild-run | 7 +++++++ 1 file changed, 7 insertions(+)
Applied to buildroot-test, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 6/7 autobuild: add a function to check if the configuration is fixable
In reply to Yann E. MORINCurrently, the fixup_config() function is responsible for two things:
- checking whether the configuration can be fixed,
- actually fixing it.
When we have an initial basic configuration, we currently loop over-and-over again until the configuration is actually fixed (limted to 100 iterations).
However, there are cases where the configuration can *not* be fixed, because it contains options incompatible with the host system [0], and wich are not randomised (i.e. looping will never fix the configuration).
Introduce a new function, is_config_fixable() which is responsible for checking that the initial configuration does not contain any option that is incompatible with the host system.
That function currently does nothing ands always consider the configuration to be fixable; actual tests will be added in followup commits.
[0] like the latest Linaro ARM toolchains which requires a glibc >= 2.14.
--- scripts/autobuild-run | 13 +++++++++++++ 1 file changed, 13 insertions(+)
scripts/autobuild-run +13 -0
@@ -332,6 +332,15 @@ def prepare_build(**kwargs): return 0 +def is_config_fixable(**kwargs): + """Check if the configuration is actually fixable + + This functions checks that the configuration does not contain + any unfixable options. + """ + + return True + def fixup_config(**kwargs): """Finalize the configuration and reject any problematic combinations
@@ -516,6 +525,10 @@ def gen_config(**kwargs): log_write(log, "ERROR: cannot oldconfig") return -1 + if not is_config_fixable(**kwargs): + log_write(log, "ERROR: configuration is not fixable") + return -1 + # Now, generate the random selection of packages, and fixup # things if needed. # Safe-guard, in case we can not quickly come to a valid
Signature · 1 line
1.9.1
PATCH v3 6/7 autobuild: add a function to check if the configuration is fixable
Yann,
Quoted 16 lines · Yann E. MORIN
Currently, the
fixup_config()function is responsible for two things:
- checking whether the configuration can be fixed,
- actually fixing it.When we have an initial basic configuration, we currently loop over-and-over again until the configuration is actually fixed (limted to 100 iterations).
However, there are cases where the configuration can *not* be fixed, because it contains options incompatible with the host system [0], and wich are not randomised (i.e. looping will never fix the configuration).
Introduce a new function,
is_config_fixable()which is responsible for checking that the initial configuration does not contain any option that is incompatible with the host system.
Quoted 7 lines
That function currently does nothing ands always consider the configuration to be fixable; actual tests will be added in followup commits.
[0] like the latest Linaro ARM toolchains which requires a glibc >= 2.14.
Signed-off-by "Yann E. MORIN" <[email protected]>
I've applied, but after doing some fairly significant changes, both to the code and to the commit log. I didn't like the name "is_config_fixable", because really it's not about the configuration being "fixable" or anything like that. It is just about the toolchain being usable or not. So "is_toolchain_usable" was IMO a much better choice here.
Also, showing an error in the logs is not appropriate: it is an expected situation to sometimes have a toolchain that cannot be used. Frightening the user with an error in this case is not appropriate.
See the final commit at
http://git.buildroot.net/buildroot-test/commit/?id=f762f3630c270bbde27fec0585bb597b472b53f1.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH v3 7/7 autobuild: check for Linaro toolchains earlier
In reply to Yann E. MORINWhen we get a configuration with those toolchains, we can not fix it. So we need to abort that configuration.
Fix the glibc version we check against (2.14, not 2.24).
--- scripts/autobuild-run | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
scripts/autobuild-run +19 -10
@@ -339,6 +339,25 @@ def is_config_fixable(**kwargs): any unfixable options. """ + idir = "instance-%d" % kwargs['instance'] + sysinfo = kwargs['sysinfo'] + + outputdir = os.path.join(idir, "output") + with open(os.path.join(outputdir, ".config")) as configf: + configlines = configf.readlines() + + # The latest Linaro toolchains on x86-64 hosts requires glibc + # 2.14+ on the host. + if platform.machine() == 'x86_64': + if 'BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y\n' in configlines or \ + 'BR2_TOOLCHAIN_EXTERNAL_LINARO_AARCH64=y\n' in configlines or \ + 'BR2_TOOLCHAIN_EXTERNAL_LINARO_ARMEB=y\n' in configlines: + ldd_version_output = subprocess.Popen(['ldd', '--version'], stdout=subprocess.PIPE).communicate()[0] + glibc_version = ldd_version_output.splitlines()[0].split()[-1] + if StrictVersion('2.14') > StrictVersion(glibc_version): + log_write(log, "WARN: ignoring the Linaro ARM toolchains becausee too old host glibc") + return False + return True def fixup_config(**kwargs):
@@ -457,16 +476,6 @@ def fixup_config(**kwargs): if 'BR2_PACKAGE_WESTON=y\n' in configlines and \ 'BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/mipsel-ctng-linux-uclibc.tar.xz"\n' in configlines: return False - # The latest Linaro toolchains on x86-64 hosts requires glibc - # 2.24+ on the host. - if platform.machine() == 'x86_64': - if 'BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM=y\n' in configlines or \ - 'BR2_TOOLCHAIN_EXTERNAL_LINARO_AARCH64=y\n' in configlines or \ - 'BR2_TOOLCHAIN_EXTERNAL_LINARO_ARMEB=y\n' in configlines: - ldd_version_output = subprocess.Popen(['ldd', '--version'], stdout=subprocess.PIPE).communicate()[0] - glibc_version = ldd_version_output.splitlines()[0].split()[-1] - if StrictVersion('2.24') > StrictVersion(glibc_version): - return False with open(os.path.join(outputdir, ".config"), "w+") as configf: configf.writelines(configlines)
Signature · 1 line
1.9.1
PATCH v3 7/7 autobuild: check for Linaro toolchains earlier
Yann,
Quoted 9 lines · Yann E. MORIN
When we get a configuration with those toolchains, we can not fix it. So we need to abort that configuration.
Fix the glibc version we check against (2.14, not 2.24).
Signed-off-by "Yann E. MORIN" <[email protected]>--- scripts/autobuild-run | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-)
Applied to buildroot-test, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com