upgrade-helper.py: remove gcc-runtime pre-build step from upgrade loop
PATCH 1/3 upgrade-helper.py: remove gcc-runtime pre-build step from upgrade loop
From: Chen Qi <[email protected]>
Building gcc-runtime for each machine before upgrading recipes only slows things down with no net benefit.
1) It will not error out early.
2) Building recipes will natually build gcc-runtime and things will be more parallelized.
3) Yocto project is in a good status and will very unlikely to have gcc-runtime build error.
--- upgrade-helper.py | 15 --------------- 1 file changed, 15 deletions(-)
upgrade-helper.py +0 -15
@@ -468,21 +468,6 @@ class Updater(object): pkggroups_ctx.append({"name":",".join([pkg_ctx['PN'] for pkg_ctx in pkgs_ctx]),"pkgs":pkgs_ctx,"error":None, 'base_dir':self.uh_recipes_all_dir}) I(" ############################################################") - if pkggroups_ctx and not self.args.skip_compilation: - I(" Building gcc runtimes ...") - for machine in self.opts['machines']: - I(" building gcc runtime for %s" % machine) - try: - self.bb.complete("gcc-runtime", machine) - except Exception as e: - E(" Can't build gcc-runtime for %s." % machine) - - if isinstance(e, Error): - E(e.stdout + e.stderr) - else: - import traceback - traceback.print_exc(file=sys.stdout) - succeeded_pkggroups_ctx = [] failed_pkggroups_ctx = [] attempted_pkggroups = 0
Signature · 1 line
2.49.0
PATCH 2/3 modules/steps.py: refactor compile() to use parallel per-machine builds
From: Chen Qi <[email protected]>
Reasons:
1. Building recipes one by one consumes little build resource. In practice, when building one recipe machine by machine in a serialized way, the machine load is very very low. This means AUH is not making use of the machines' computing resources.
2. Yocto project has good and mature technique to control the build to not exhaust machines' resource, e.g., BB_NUMBER_THREADS, BB_PRESSURE_MAX_XXX. So we really don't need to worry about AUH exhausting server resources.
Technical Details:
1. Each machine gets its own build directory ($BUILDDIR-{machine}) with BUILDDIR and BBPATH env vars set accordingly, and conf/ copied from the original build directory. All machines compile in parallel via multiprocessing.Pool, with errors collected and reported after all workers complete.
2. Use original build directory for the first machine in parallel build. Skip creating a separate build directory for the first machine and build directly in the original BUILDDIR. Only the remaining machines get their own $BUILDDIR-{machine} copies. The purpose is that if testimage is enabled, then the build can just the previous results.
3. Always refresh per-machine conf/ directory before compilation. This ensure anything new in the main build directory will get into new build directory.
4. Add --serial-build option to use sequential per-machine compilation. When --serial-build is passed, compile each machine serially using the original build logic instead of the parallel multiprocessing approach.
Note:
After this change, AUH will use parallel build by default. The '--serial-build' is there if users really need to use serial build for some reason.
--- modules/steps.py | 96 +++++++++++++++++++++++++++++++++++++---------- upgrade-helper.py | 3 ++ 2 files changed, 80 insertions(+), 19 deletions(-)
modules/steps.py +77 -19
@@ -22,10 +22,12 @@ import os import shutil import re +import multiprocessing from logging import debug as D from logging import info as I from logging import warning as W +from logging import error as E from errors import Error, DevtoolError, CompilationError from buildhistory import BuildHistory
@@ -104,31 +106,87 @@ def devtool_upgrade(devtool, bb, git, opts, group): for p in group['pkgs']: _devtool_upgrade(devtool, bb, git, opts, p) -def _compile(bb, pkg, machine, workdir): - try: - bb.complete(pkg, machine) - except Error as e: - with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: - f.write(e.stdout + e.stderr) - for line in e.stdout.split("\n") + e.stderr.split("\n"): - # version going backwards is not a real error - if re.match(".* went backwards which would break package feeds .*", line): - break - # 'not in COMPATIBLE_HOST/MACHINE is not a real error - if re.match(".*not in COMPATIBLE.*", line): - break - else: - raise CompilationError() +def _compile_worker(args): + """Worker function for multiprocess compilation. Runs in a separate process.""" + pkg, machine, workdir, orig_builddir, use_orig = args + from utils.bitbake import Bitbake + + if use_orig: + builddir = orig_builddir + else: + builddir = "{}-{}".format(orig_builddir, machine) + os.makedirs(builddir, exist_ok=True) + + new_conf = os.path.join(builddir, "conf") + if os.path.isdir(new_conf): + shutil.rmtree(new_conf) + shutil.copytree(os.path.join(orig_builddir, "conf"), new_conf) + + os.environ["BUILDDIR"] = builddir + os.environ["BBPATH"] = builddir + + bb = Bitbake(builddir) + try: + bb.complete(pkg, machine) + return (machine, True, None) + except Error as e: + with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: + f.write(e.stdout + e.stderr) + for line in e.stdout.split("\n") + e.stderr.split("\n"): + if re.match(".* went backwards which would break package feeds .*", line): + return (machine, True, None) + if re.match(".*not in COMPATIBLE.*", line): + return (machine, True, None) + return (machine, False, e.stdout + e.stderr) + +def _compile_serial(bb, pkg, machine, workdir): + """Compile a single machine serially (original behaviour).""" + try: + bb.complete(pkg, machine) + except Error as e: + with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: + f.write(e.stdout + e.stderr) + for line in e.stdout.split("\n") + e.stderr.split("\n"): + if re.match(".* went backwards which would break package feeds .*", line): + break + if re.match(".*not in COMPATIBLE.*", line): + break + else: + raise CompilationError() def compile(devtool, bb, git, opts, group): if opts['skip_compilation']: W(" %s: Compilation was skipped by user choice!" % group['name']) return - for machine in opts['machines']: - I(" %s: compiling upgraded version for %s ..." % (group['name'], machine)) - _compile(bb, " ".join([pkg_ctx['PN'] for pkg_ctx in group['pkgs']]), machine, group['workdir']) - if opts['buildhistory'] and machine == opts['machines'][0]: + pkg = " ".join([pkg_ctx['PN'] for pkg_ctx in group['pkgs']]) + machines = opts['machines'] + + if opts.get('serial_build'): + for machine in machines: + I(" %s: compiling upgraded version for %s ..." % (group['name'], machine)) + _compile_serial(bb, pkg, machine, group['workdir']) + if opts['buildhistory'] and machine == machines[0]: + I(" %s: Checking buildhistory ..." % group['name']) + group['buildhistory'].diff() + else: + orig_builddir = os.environ.get("BUILDDIR", "") + worker_args = [(pkg, m, group['workdir'], orig_builddir, i == 0) for i, m in enumerate(machines)] + + I(" %s: compiling upgraded version for %s in parallel ..." % (group['name'], machines)) + with multiprocessing.Pool(processes=len(machines)) as pool: + results = pool.map(_compile_worker, worker_args) + + os.environ["BUILDDIR"] = orig_builddir + os.environ["BBPATH"] = orig_builddir + + failed = [(m, err) for m, ok, err in results if not ok] + if failed: + for m, err in failed: + E(" %s: compilation failed for %s" % (group['name'], m)) + raise CompilationError() + + if opts['buildhistory'] and machines: I(" %s: Checking buildhistory ..." % group['name']) group['buildhistory'].diff()
upgrade-helper.py +3 -0
@@ -108,6 +108,8 @@ def parse_cmdline(): help="layers to include in the upgrade research") parser.add_argument("--layer-dir", action="store", default='', help="the layers root directory") + parser.add_argument("--serial-build", action="store_true", default=False, + help="compile for each machine serially instead of in parallel") return parser.parse_args() def parse_config_file(config_file):
@@ -193,6 +195,7 @@ class Updater(object): self.opts['author'] = "Upgrade Helper <%s>" % \ settings.get('from', '[email protected]') self.opts['skip_compilation'] = self.args.skip_compilation + self.opts['serial_build'] = self.args.serial_build self.opts['buildhistory'] = self._buildhistory_is_enabled() self.opts['testimage'] = self._testimage_is_enabled()
Signature · 1 line
2.49.0
Re: [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
I understand the idea, but unfortunately I do not think we can put this (and the following patch) on the Yocto autobuilder. The expectation for the builder jobs is that they are expected to run only one Yocto build at a time, and the autobuilder will:
- configure each builder to use carefully adjusted limited resources for that one build (via the fragment in oe-core)
- start several builders on each worker machine to parallelize large builds (e.g. a-full), but no more than a certain upper limit, and queue them if the limit is reached on all workers
This patch breaks that expectation, as auh build is not using an exclusive worker for itself, and shares workers with other builds.
It also isn't clear how this patch was tested and benchmarked. Did you run a substantial upgrade job with it? Did you see a significant speedup?
Alex
Quoted 194 lines · On Tue, 14 Apr 2026 at 07:32, <[email protected]> wrote:
From: Chen Qi <[email protected]>
Reasons:
1. Building recipes one by one consumes little build resource. In practice, when building one recipe machine by machine in a serialized way, the machine load is very very low. This means AUH is not making use of the machines' computing resources.
2. Yocto project has good and mature technique to control the build to not exhaust machines' resource, e.g.,BB_NUMBER_THREADS,BB_PRESSURE_MAX_XXX. So we really don't need to worry about AUH exhausting server resources.Technical Details:
1. Each machine gets its own build directory ($BUILDDIR-{machine}) with BUILDDIR and BBPATH env vars set accordingly, and conf/ copied from the original build directory. All machines compile in parallel via multiprocessing.Pool, with errors collected and reported after all workers complete.
2. Use original build directory for the first machine in parallel build. Skip creating a separate build directory for the first machine and build directly in the original BUILDDIR. Only the remaining machines get their own $BUILDDIR-{machine} copies. The purpose is that if testimage is enabled, then the build can just the previous results.
3. Always refresh per-machine conf/ directory before compilation. This ensure anything new in the main build directory will get into new build directory.
4. Add --serial-build option to use sequential per-machine compilation. When --serial-build is passed, compile each machine serially using the original build logic instead of the parallel multiprocessing approach.
Note:
After this change, AUH will use parallel build by default. The '--serial-build' is there if users really need to use serial build for some reason.
Signed-off-by Chen Qi <[email protected]>--- modules/steps.py | 96 +++++++++++++++++++++++++++++++++++++---------- upgrade-helper.py | 3 ++ 2 files changed, 80 insertions(+), 19 deletions(-)modules/steps.py +77
-19@@ -22,10 +22,12 @@ import os import shutil import re +import multiprocessing from logging import debug as D from logging import info as I from logging import warning as W +from logging import error as E from errors import Error, DevtoolError, CompilationError from buildhistory import BuildHistory@@ -104,31 +106,87 @@ def devtool_upgrade(devtool, bb, git, opts, group): for p in group['pkgs']: _devtool_upgrade(devtool, bb, git, opts, p) -def _compile(bb, pkg, machine, workdir): - try: - bb.complete(pkg, machine) - except Error as e: - with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: - f.write(e.stdout + e.stderr) - for line in e.stdout.split("\n") + e.stderr.split("\n"): - # version going backwards is not a real error - if re.match(".* went backwards which would break package feeds .*", line): - break - # 'not in COMPATIBLE_HOST/MACHINE is not a real error - if re.match(".*not in COMPATIBLE.*", line): - break - else: - raise CompilationError() +def _compile_worker(args): + """Worker function for multiprocess compilation. Runs in a separate process.""" + pkg, machine, workdir, orig_builddir, use_orig = args + from utils.bitbake import Bitbake + + if use_orig: + builddir = orig_builddir + else: + builddir = "{}-{}".format(orig_builddir, machine) + os.makedirs(builddir, exist_ok=True) + + new_conf = os.path.join(builddir, "conf") + if os.path.isdir(new_conf): + shutil.rmtree(new_conf) + shutil.copytree(os.path.join(orig_builddir, "conf"), new_conf) + + os.environ["BUILDDIR"] = builddir + os.environ["BBPATH"] = builddir + + bb = Bitbake(builddir) + try: + bb.complete(pkg, machine) + return (machine, True, None) + except Error as e: + with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: + f.write(e.stdout + e.stderr) + for line in e.stdout.split("\n") + e.stderr.split("\n"): + if re.match(".* went backwards which would break package feeds .*", line): + return (machine, True, None) + if re.match(".*not in COMPATIBLE.*", line): + return (machine, True, None) + return (machine, False, e.stdout + e.stderr) + +def _compile_serial(bb, pkg, machine, workdir): + """Compile a single machine serially (original behaviour).""" + try: + bb.complete(pkg, machine) + except Error as e: + with open("{}/bitbake-output-{}.txt".format(workdir, machine), 'w') as f: + f.write(e.stdout + e.stderr) + for line in e.stdout.split("\n") + e.stderr.split("\n"): + if re.match(".* went backwards which would break package feeds .*", line): + break + if re.match(".*not in COMPATIBLE.*", line): + break + else: + raise CompilationError() def compile(devtool, bb, git, opts, group): if opts['skip_compilation']: W(" %s: Compilation was skipped by user choice!" % group['name']) return - for machine in opts['machines']: - I(" %s: compiling upgraded version for %s ..." % (group['name'], machine)) - _compile(bb, " ".join([pkg_ctx['PN'] for pkg_ctx in group['pkgs']]), machine, group['workdir']) - if opts['buildhistory'] and machine == opts['machines'][0]: + pkg = " ".join([pkg_ctx['PN'] for pkg_ctx in group['pkgs']]) + machines = opts['machines'] + + if opts.get('serial_build'): + for machine in machines: + I(" %s: compiling upgraded version for %s ..." % (group['name'], machine)) + _compile_serial(bb, pkg, machine, group['workdir']) + if opts['buildhistory'] and machine == machines[0]: + I(" %s: Checking buildhistory ..." % group['name']) + group['buildhistory'].diff() + else: + orig_builddir = os.environ.get("BUILDDIR", "") + worker_args = [(pkg, m, group['workdir'], orig_builddir, i == 0) for i, m in enumerate(machines)] + + I(" %s: compiling upgraded version for %s in parallel ..." % (group['name'], machines)) + with multiprocessing.Pool(processes=len(machines)) as pool: + results = pool.map(_compile_worker, worker_args) + + os.environ["BUILDDIR"] = orig_builddir + os.environ["BBPATH"] = orig_builddir + + failed = [(m, err) for m, ok, err in results if not ok] + if failed: + for m, err in failed: + E(" %s: compilation failed for %s" % (group['name'], m)) + raise CompilationError() + + if opts['buildhistory'] and machines: I(" %s: Checking buildhistory ..." % group['name']) group['buildhistory'].diff()upgrade-helper.py +3
-1@@ -108,6 +108,8 @@ def parse_cmdline(): help="layers to include in the upgrade research") parser.add_argument("--layer-dir", action="store", default='', help="the layers root directory") + parser.add_argument("--serial-build", action="store_true", default=False, + help="compile for each machine serially instead of in parallel") return parser.parse_args() def parse_config_file(config_file):@@ -193,6 +195,7 @@ class Updater(object): self.opts['author'] = "Upgrade Helper <%s>" % \ settings.get('from', '[email protected]') self.opts['skip_compilation'] = self.args.skip_compilation + self.opts['serial_build'] = self.args.serial_build self.opts['buildhistory'] = self._buildhistory_is_enabled() self.opts['testimage'] = self._testimage_is_enabled() --2.49.0
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Quoted 12 lines · Alexander Kanavin via lists.yoctoproject.org
I understand the idea, but unfortunately I do not think we can put this (and the following patch) on the Yocto autobuilder. The expectation for the builder jobs is that they are expected to run only
one Yocto build at a time, and the autobuilder will:- configure each builder to use carefully adjusted limited resources for that one build (via the fragment in oe-core)
- start several builders on each worker machine to parallelize large builds (e.g. a-full), but no more than a certain upper limit, and queue them if the limit is reached on all workers
Quoted 7 lines
This patch breaks that expectation, as auh build is not using an exclusive worker for itself, and shares workers with other builds.
It also isn't clear how this patch was tested and benchmarked. Did you
run a substantial upgrade job with it? Did you see a significant speedup?
Whilst we do need to be mindful of this, I suspect there is some room to parallelise a bit. How much parallelism is this expected to have? Some existing jobs like oe-selftest do run a lot in parallel already so this could be not that much different to that...
Cheers,
Richard
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Quoted 4 lines · Richard Purdie via lists.yoctoproject.org
Whilst we do need to be mindful of this, I suspect there is some room to parallelise a bit. How much parallelism is this expected to have? Some existing jobs like oe-selftest do run a lot in parallel already so this could be not that much different to that...
Patch 3/3 sets the parallelism to four:
- self.opts['machines'] = settings.get('machines', 'qemux86-64
qemuarm_musl').split()
+ self.opts['machines'] = settings.get('machines', 'qemux86-64
qemux86 qemuarm64_musl qemuarm_musl').split()
Alex
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Quoted 7 lines · Alexander Kanavin
Quoted 4 lines · Richard Purdie via lists.yoctoproject.org
Whilst we do need to be mindful of this, I suspect there is some room to parallelise a bit. How much parallelism is this expected to have? Some existing jobs like oe-selftest do run a lot in parallel already so this could be not that much different to that...
Quoted 8 lines
Patch 3/3 sets the parallelism to four:
- self.opts['machines'] = settings.get('machines', 'qemux86-64
qemuarm_musl').split()
+ self.opts['machines'] = settings.get('machines', 'qemux86-64
qemux86qemuarm64_muslqemuarm_musl').split()Alex
Yes. The parallelism is determined by the machines setting.
Let me add more explanation for this patch.
The biggest problem of auh is that it's running too slowly. With serial run of builds, you don't dare to specify enough machines because it'll just spend too much time. For example, you upgrade the whole oe-core, in most of the time, what the build is doing is just building one recipe. The first recipe upgrade has already got cross/native tools, glibc, etc., available, and subsequent upgrades will only run its own build. Such builds consume very little resources. On my server, if auh is the only one that is running builds, the PSI data are very close to 0 for most of the time.
We don't need to worry about auh using too much resources, even if it's running with other autobuilder tests. Because Yocto has very good techniques to control this. For example, if we put autobuilder-resource-constraints.conf config fragment in local.conf, we don't need to worry about auh exhaust the servers' resources.
With this parallel build added, Yocto can configure oe-core's upgrade to use machines that oe-core uses as the acceptance criteria. This means a successful auh upgrade will very like to be merged without any change. If there is some failure, e.g., qemuarm64_musl, the developers/maintainers can see it directly without doing more local builds.
Regards,
Qi
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Quoted 5 lines · ChenQi
With this parallel build added, Yocto can configure oe-core's upgrade to use machines that oe-core uses as the acceptance criteria. This means a successful auh upgrade will very like to be merged without any change. If there is some failure, e.g.,
qemuarm64_musl, the developers/maintainers can see it directly without doing more local builds.
For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (qemux86_64 and qemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.
It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.
I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.
Alex
RE: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Hi Alex,
I totally agree with you that "It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds".
Let me first share my local practice and then share my thoughts on AUH improvements.
Here's my local practice for LTS releases:
Upgrade recipes (stable version upgrades only) for each layer using AUH (parallel build enabled) + manually checking the upgrade results + world build
I'm using world build instead of some extra build tests for each upgrade because I'm upgrading quite a few recipes in multiple layers, the upgrades are stable upgrades and the server is powerful enough to finish a world build in a few hours.
What I have in mind for AUH improvement for oe-core master is that, apart from this parallel change, we can also add some other changes to AUH so that it can be more efficient and save developers time.
1. A new step, compile_direct_dependents, could be added after the compile step.
What this step does it building recipes that depend directly on the recipe that was just upgraded. The can be got by using 'bitbake -g world' + some filtering. This step can be enabled by default, but could be also be skipped by settings. When enabled, it only builds those recipes for the first machine.
2. The machines settings could be extended to include fragements
e.g.,
machines = qemux86-64:core/yocto-autobuilder/multilib-x86-lib32 qemuarm64_musl:some-fragment-to-enable-debug-build .... In this way, we're giving AUH more build coverage.
With the above improvements finished, a successful AUH upgrade in oe-core means:
1) it builds successfully for all specified machines and configurations
2) all recipes that directly depend on it build successfully
3) testimage passes (which means if it has ptest, its ptest passes)
When the emails are sent out, the maintainers really do not have much to do with a successful upgrade. The maintainer just reviews the changes, correct license if there's some license change, put some commit log and send out the patch which is a sign of commitment to fix any further problems. When the AUH upgrade fails, the maintainer can also easily see what the problem is and how to reproduce it.
What do you think about this?
Regards,
Qi
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Monday, April 27, 2026 4:55 PM
To: Chen, Qi <[email protected]>
Cc: [email protected]; MacLeod, Randy <[email protected]>
Subject: Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Quoted 5 lines · ChenQi
With this parallel build added, Yocto can configure oe-core's upgrade to use machines that oe-core uses as the acceptance criteria. This means a successful auh upgrade will very like to be merged without any change. If there is some failure, e.g.,
qemuarm64_musl, the developers/maintainers can see it directly without doing more local builds.
For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (qemux86_64 andqemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.
It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.
I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.
Alex
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Hello Qi,
thanks for the detailed answer. I fully agree those would be useful improvements, my concern is that AUH right now is a very shaky foundation for doing them. Specifically:
1. No test suite. This is becoming a real problem, as I struggle to have confidence in non-trivial patches, and especially when they add new features instead of refactoring code. This code can and does break easily, as the amount of corner cases is particularly large. If I take this patch, you'll expect me to take more patches that add even more code, while still not having any way to test them.
This could be addressed by having 'mock' versions of bitbake and devtool in the test suite. The 'mock' versions would print things that AUH expects, both in 'success' and 'fail' scenarios, and maybe write out 'upgraded' recipes, but otherwise do nothing. That way, all or most code paths in AUH would be exercised, without having to run a 'real' upgrade, which is heavy, takes forever, and not really suitable for testing changes.
2. Which brings me to the other issue, and it probably should be addressed first: the brittle interface with devtool and bitbake. AUH makes decisions based on finding strings in human readable output of these tools. This isn't great. One way to address that is to move at least the upgrade/finish/build/test/commit loop into a new devtool plugin, so that it would use devtool libraries directly, and interface with bitbake via tinfoil. Then AUH's job would be simply to determine what needs to be updated, call devtool with appropriate command line arguments (or maybe json config with the 'upgrade job' specification) and to make reports/emails from resulting patches and logs.
Once we have these things, adding more features becomes a lot more sustainable. I understand such a refactor is non-trivial, but equally AUH as it is can't take much more extensions.
I also have a side question: when you say "Upgrade recipes (stable version upgrades only) ", how do you determine the stable versions? There's an ongoing discussion about this with some patches; figuring out what the stable versions are isn't trivial.
Alex
Quoted 59 lines · Qi
Hi Alex,
I totally agree with you that "It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds".
Let me first share my local practice and then share my thoughts on AUH improvements.
Here's my local practice for LTS releases:
Upgrade recipes (stable version upgrades only) for each layer using AUH (parallel build enabled) + manually checking the upgrade results + world buildI'm using world build instead of some extra build tests for each upgrade because I'm upgrading quite a few recipes in multiple layers, the upgrades are stable upgrades and the server is powerful enough to finish a world build in a few hours.
What I have in mind for AUH improvement for oe-core master is that, apart from this parallel change, we can also add some other changes to AUH so that it can be more efficient and save developers time.
1. A new step,
compile_direct_dependents, could be added after the compile step.
What this step does it building recipes that depend directly on the recipe that was just upgraded. The can be got by using 'bitbake -g world' + some filtering. This step can be enabled by default, but could be also be skipped by settings. When enabled, it only builds those recipes for the first machine.2. The machines settings could be extended to include fragements
e.g.,
machines = qemux86-64:core/yocto-autobuilder/multilib-x86-lib32 qemuarm64_musl:some-fragment-to-enable-debug-build .... In this way, we're giving AUH more build coverage.With the above improvements finished, a successful AUH upgrade in oe-core means:
1) it builds successfully for all specified machines and configurations
2) all recipes that directly depend on it build successfully
3) testimage passes (which means if it has ptest, its ptest passes)When the emails are sent out, the maintainers really do not have much to do with a successful upgrade. The maintainer just reviews the changes, correct license if there's some license change, put some commit log and send out the patch which is a sign of commitment to fix any further problems. When the AUH upgrade fails, the maintainer can also easily see what the problem is and how to reproduce it.
What do you think about this?
Regards,
Qi-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Monday, April 27, 2026 4:55 PM
To: Chen, Qi <[email protected]>
Cc: [email protected]; MacLeod, Randy <[email protected]>
Subject: Re: [yocto-patches] [auh][PATCH 2/3]modules/steps.py: refactorcompile()to use parallel per-machine buildsQuoted 5 lines · ChenQi
With this parallel build added, Yocto can configure oe-core's upgrade to use machines that oe-core uses as the acceptance criteria. This means a successful auh upgrade will very like to be merged without any change. If there is some failure, e.g.,
qemuarm64_musl, the developers/maintainers can see it directly without doing more local builds.For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (
qemux86_64andqemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.
I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.
Alex
Re: [yocto-patches] [auh][PATCH 2/3] modules/steps.py: refactor compile() to use parallel per-machine builds
Hi Alex,
See comments inline.
Quoted 21 lines · Alexander Kanavin
Hello Qi,
thanks for the detailed answer. I fully agree those would be useful improvements, my concern is that AUH right now is a very shaky foundation for doing them. Specifically:
1. No test suite. This is becoming a real problem, as I struggle to have confidence in non-trivial patches, and especially when they add new features instead of refactoring code. This code can and does break easily, as the amount of corner cases is particularly large. If I take this patch, you'll expect me to take more patches that add even more code, while still not having any way to test them.
This could be addressed by having 'mock' versions of bitbake and devtool in the test suite. The 'mock' versions would print things that AUH expects, both in 'success' and 'fail' scenarios, and maybe write out 'upgraded' recipes, but otherwise do nothing. That way, all or most code paths in AUH would be exercised, without having to run a 'real' upgrade, which is heavy, takes forever, and not really suitable for testing changes.
Quoted 10 lines
2. Which brings me to the other issue, and it probably should be addressed first: the brittle interface with devtool and bitbake. AUH makes decisions based on finding strings in human readable output of these tools. This isn't great. One way to address that is to move at least the upgrade/finish/build/test/commit loop into a new devtool plugin, so that it would use devtool libraries directly, and interface with bitbake via tinfoil. Then AUH's job would be simply to determine what needs to be updated, call devtool with appropriate command line arguments (or maybe json config with the 'upgrade job' specification) and to make reports/emails from resulting patches and logs.
I believe this is the correct path.
This will also address the above testsuite concern. When the major functionalities are in devtool/bitbake, things can be reliably tested by oeqa.
Quoted 4 lines
Once we have these things, adding more features becomes a lot more sustainable. I understand such a refactor is non-trivial, but equally AUH as it is can't take much more extensions.
I agree. When reading source codes of AUH, I found that it was designed to be a simple tool. It mainly relies on oe.recipeutils.get_recipe_upgrade_status and 'devtool upgrade/finish' to do the job. It even does not use tinfoil and specially uses layer_mode/layer_dir. If AUH can use tinfoil, it really does not need layer_mode/layer_dir design, as it can easily know where the recipe is. I'm not sure if this is a deliberate design, but in practice, this turned out to be a robust implementation.
Quoted 7 lines
I also have a side question: when you say "Upgrade recipes (stable version upgrades only) ", how do you determine the stable versions? There's an ongoing discussion about this with some patches; figuring out what the stable versions are isn't trivial.
Alex
I modified recipeutils.py and bitbake fetchers to add an optional parameter to them.
e.g.,
recipesutils:
-def get_recipe_upgrade_status(recipes=None):
+def get_recipe_upgrade_status(recipes=None, same_major_version=False):
bitbake fetchers:
- def latest_versionstring(self, ud, d):
+ def latest_versionstring(self, ud, d, major_version=None)
I used this method because, as I said above, I found AUH is a simple tool, and the major functionalities should remain in devtool/bitbake.
With the changes in recipeutils and bitbake, what AUH needs is just to pass same_major_version=True to the oe.recipeutils.get_recipe_upgrade_status call when it gets some option (I used --lts locally).
My initial plan was to send out these patches after at least two local runs, so that I can see in advance if anything's not working.
My first round of the AUH stable upgrades was finished about two weeks ago, things were working pretty well. The second round was scheduled to happen around 5/20.
Now I know that there's an on-going discussion. I'll send out these bitbake/recipeutils patches as RFC for more discussion.
Regards,
Qi
Quoted 60 lines · Qi
Quoted 57 lines
Hi Alex,
I totally agree with you that "It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds".
Let me first share my local practice and then share my thoughts on AUH improvements.
Here's my local practice for LTS releases:
Upgrade recipes (stable version upgrades only) for each layer using AUH (parallel build enabled) + manually checking the upgrade results + world buildI'm using world build instead of some extra build tests for each upgrade because I'm upgrading quite a few recipes in multiple layers, the upgrades are stable upgrades and the server is powerful enough to finish a world build in a few hours.
What I have in mind for AUH improvement for oe-core master is that, apart from this parallel change, we can also add some other changes to AUH so that it can be more efficient and save developers time.
1. A new step,
compile_direct_dependents, could be added after the compile step.
What this step does it building recipes that depend directly on the recipe that was just upgraded. The can be got by using 'bitbake -g world' + some filtering. This step can be enabled by default, but could be also be skipped by settings. When enabled, it only builds those recipes for the first machine.2. The machines settings could be extended to include fragements
e.g.,
machines = qemux86-64:core/yocto-autobuilder/multilib-x86-lib32 qemuarm64_musl:some-fragment-to-enable-debug-build .... In this way, we're giving AUH more build coverage.With the above improvements finished, a successful AUH upgrade in oe-core means:
1) it builds successfully for all specified machines and configurations
2) all recipes that directly depend on it build successfully
3) testimage passes (which means if it has ptest, its ptest passes)When the emails are sent out, the maintainers really do not have much to do with a successful upgrade. The maintainer just reviews the changes, correct license if there's some license change, put some commit log and send out the patch which is a sign of commitment to fix any further problems. When the AUH upgrade fails, the maintainer can also easily see what the problem is and how to reproduce it.
What do you think about this?
Regards,
Qi-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Monday, April 27, 2026 4:55 PM
To: Chen, Qi <[email protected]>
Cc: [email protected]; MacLeod, Randy <[email protected]>
Subject: Re: [yocto-patches] [auh][PATCH 2/3]modules/steps.py: refactorcompile()to use parallel per-machine buildsQuoted 5 lines · ChenQi
With this parallel build added, Yocto can configure oe-core's upgrade to use machines that oe-core uses as the acceptance criteria. This means a successful auh upgrade will very like to be merged without any change. If there is some failure, e.g.,
qemuarm64_musl, the developers/maintainers can see it directly without doing more local builds.For the purpose of updating oe-core, I feel a bit different about benefits of adding more machines actually. I think it's unlikely that adding more entries to the two existing ones (
qemux86_64andqemuarm_musl) will help uncover more issues - if 32 bit and musl variants are already covered between these two, then it's unlikely there will a particular 32 bit x86 issue, or 64 bit arm issue, or a risc-v issue, e.g. something that those additional machines would catch.It's far more likely that an update builds fine by itself, but breaks something not covered by AUH recipe builds, e.g. a dependent recipe, ptests, oe-selftest, or runtime behavior. Such things are unrealistic to test for each update, and so they're done after the fact, in a-full.
I agree that speeding up AUH would be beneficial, but that is better done in the outermoost loop, which takes recipe groups and updates them one by one. Parallelizing that is more complicated, as each update operation would need its own copy of oe-core repo in addition to its own build directory, but it would really run everything in parallel, including devtool upgrade (which can be very sluggish on large source trees or large commit histories), buildhistory, and test builds. And of course, lack of AUH tests hinders such large refactor efforts as well.
Alex
PATCH 3/3 upgrade-helper.py: use more combinations for default machines setting
In reply to [email protected]From: Chen Qi <[email protected]>
Now that we've changed to use parallel build for machines, we can change the default machines setting to cover more cases. This is because such change will not result in significant build time increase, and maintainers can have more data.
--- upgrade-helper.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
upgrade-helper.py +1 -1
@@ -189,7 +189,7 @@ class Updater(object): else: # use scripts_path to get oe-core directory self.git = Git(os.path.dirname(scripts_path)) - self.opts['machines'] = settings.get('machines', 'qemux86-64 qemuarm_musl').split() + self.opts['machines'] = settings.get('machines', 'qemux86-64 qemux86 qemuarm64_musl qemuarm_musl').split() self.opts['send_email'] = self.args.send_emails self.opts['author'] = "Upgrade Helper <%s>" % \
Signature · 1 line
2.49.0