Bug fix: added support for ABHELPER_JSON environ.
PATCH 1/3 Bug fix: added support for ABHELPER_JSON environ.
The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to extra_env.
--- builders.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
builders.py +2 -1
@@ -15,7 +15,8 @@ maxsteps = 9 # Environment to pass into the workers, e.g. to load further local configuration # fragments extra_env = {} -#extra_env = {"ABHELPER_JSON" : "config.json /home/pokybuild/config-local.json"} +if os.environ.get('ABHELPER_JSON'): + extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON'] @util.renderer def get_sstate_release_number(props):
Signature · 1 line
2.17.1
PATCH 2/3 Added remote distribution of config JSON files
This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.
--- builders.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
builders.py +22 -0
@@ -18,6 +18,26 @@ extra_env = {} if os.environ.get('ABHELPER_JSON'): extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON'] +def add_abhelper_steps(factory): + """ + Adds FileDownload steps to the factory for each specified autobuilder + helper JSON file and installs the file in the worker's + builddir/yocto-autobuilder-helper/ path. + """ + abhelper_dir = os.path.expanduser("~/yocto-autobuilder-helper") + abhelper_configs = extra_env.get("ABHELPER_JSON", "config.json").split(" ") + for helper in abhelper_configs: + helper_name = os.path.basename(helper) + worker_helper = util.Interpolate("%(prop:builddir)s/yocto-autobuilder-helper/" + helper_name) + master_helper = helper + if not helper.startswith("/"): + master_helper = os.path.join(abhelper_dir, helper) + factory.addStep(steps.FileDownload( + name="Download '{}'".format(helper_name), + mastersrc=master_helper, + workerdest=worker_helper, + haltOnFailure=True )) + @util.renderer def get_sstate_release_number(props): """
@@ -145,6 +165,7 @@ def create_builder_factory(): mode='incremental', haltOnFailure=True, name='Fetch yocto-autobuilder-helper')) + add_abhelper_steps(f) f.addStep(steps.SetProperties(properties=ensure_props_set)) f.addStep(WriteLayerInfo(name='Write main layerinfo.json', haltOnFailure=True)) f.addStep(steps.ShellCommand(
@@ -213,6 +234,7 @@ def create_parent_builder_factory(buildername, waitname): mode='incremental', haltOnFailure=True, name='Fetch yocto-autobuilder-helper')) + add_abhelper_steps(factory) factory.addStep(WriteLayerInfo(name='Write main layerinfo.json', haltOnFailure=True)) factory.addStep(steps.ShellCommand( command=[
Signature · 1 line
2.17.1
Re: [yocto-autobuilder2][PATCH 2/3] Added remote distribution of config JSON files
Quoted 11 lines · Thomas Goodwin
This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.
Signed-off-by Thomas Goodwin <[email protected]>--- builders.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
I like the idea however this is going to break the autobuilders as they already use full paths in the variable.
I've merged some of the other patches but this one will need a little further thought so we don't break the existing install.
The existing code also allows for different path locations on individual workers (e.g. different username) which this would break.
Cheers,
Richard
Re: [yocto-autobuilder2][PATCH 2/3] Added remote distribution of config JSON files
I'm not sure I follow. This collects everything into the worker's "builddir/yocto-autobuilder-helper" path (I think; I'm new to buildbot).
My aim was to let the buildbot master the chance to SCM the configuration files apart from the helper's behavior which should otherwise be consistent whether we're rebuilding for warrior or morty. For us this is turning out to not be "enough" since certain build targets don't exist between different releases of Poky and we're held back by BSPs to certain releases, so we need to decouple a bit further than this patch. The issue could also be solved with templates that dynamically point to other templates based on the Force Build branch, revision, etc., but I'm looking for a more simple approach than a recursive overlay.
Thanks again for all your help and feedback.
Cheers,
Thomas
Quoted 27 lines · Richard Purdie
Quoted 11 lines · Thomas Goodwin
This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.
Signed-off-by Thomas Goodwin <[email protected]>--- builders.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)I like the idea however this is going to break the autobuilders as they already use full paths in the variable.
I've merged some of the other patches but this one will need a little further thought so we don't break the existing install.
The existing code also allows for different path locations on individual workers (e.g. different username) which this would break.
Cheers,
Richard
PATCH 3/3 Support for absolute paths to config files
In reply to Thomas GoodwinUsing absolute paths in ABHELPER_JSON with remote-loading now works; the paths are stripped to os.path.basename() before being provided to the builders.
--- builders.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
builders.py +11 -3
@@ -4,6 +4,7 @@ from yoctoabb import config from yoctoabb.steps.writelayerinfo import WriteLayerInfo from yoctoabb.steps.observer import RunConfigLogObserver from datetime import datetime +import copy import os import json
@@ -38,6 +39,13 @@ def add_abhelper_steps(factory): workerdest=worker_helper, haltOnFailure=True )) +# At the worker, all paths are relative to builddir/yocto-autobuilder-helper +# once the add_abhelper_steps run. +worker_env = copy.deepcopy(extra_env) +if worker_env.get('ABHELPER_JSON'): + configs = [os.path.basename(c) for c in worker_env['ABHELPER_JSON'].split(" ")] + worker_env['ABHELPER_JSON'] = " ".join(configs) + @util.renderer def get_sstate_release_number(props): """
@@ -216,7 +224,7 @@ for builder in config.subbuilders: workers = config.builder_to_workers['default'] builders.append(util.BuilderConfig(name=builder, workernames=workers, - factory=f, env=extra_env)) + factory=f, env=worker_env)) def create_parent_builder_factory(buildername, waitname): factory = util.BuildFactory()
@@ -337,5 +345,5 @@ def create_parent_builder_factory(buildername, waitname): return factory -builders.append(util.BuilderConfig(name="a-quick", workernames=config.workers, factory=create_parent_builder_factory("a-quick", "wait-quick"), env=extra_env)) -builders.append(util.BuilderConfig(name="a-full", workernames=config.workers, factory=create_parent_builder_factory("a-full", "wait-full"), env=extra_env)) +builders.append(util.BuilderConfig(name="a-quick", workernames=config.workers, factory=create_parent_builder_factory("a-quick", "wait-quick"), env=worker_env)) +builders.append(util.BuilderConfig(name="a-full", workernames=config.workers, factory=create_parent_builder_factory("a-full", "wait-full"), env=worker_env))
Signature · 1 line
2.17.1
Quoted 4 lines · Thomas Goodwin
The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to
extra_env.
Any chance of updating documentation on how to use this?
- armin
Quoted 20 lines
Signed-off-by Thomas Goodwin <[email protected]>--- builders.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)builders.py +2
-1@@ -15,7 +15,8 @@ maxsteps = 9 # Environment to pass into the workers, e.g. to load further local configuration # fragments extra_env = {} -#extra_env = {"ABHELPER_JSON" : "config.json /home/pokybuild/config-local.json"} +if os.environ.get('ABHELPER_JSON'): + extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON'] @util.renderer def get_sstate_release_number(props):
It's already there, this patch is to make it work as described:
http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder2/tree/README-WALKTHROUGHS.md?h=contrib/tgoodwin/config-json-fixes#n79
Consequently, having the variable in your environment is the only way for the janitor to parse and use the same configuration files as the controller.
Thomas
Quoted 35 lines · akuster
Quoted 4 lines · Thomas Goodwin
The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to
extra_env.Any chance of updating documentation on how to use this?
- armin
builders.py maxsteps = 9 · line 15
Quoted 12 lines
Signed-off-by Thomas Goodwin <[email protected]>--- builders.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)builders.py +0
-0@@ -15,7 +15,8 @@ maxsteps = 9 # Environment to pass into the workers, e.g. to load further localconfiguration
builders.py maxsteps = 9
Quoted 3 lines
builders.py +0
-1# fragments extra_env = {} -#extra_env = {"ABHELPER_JSON" : "config.json
/home/pokybuild/config-local.json"}Quoted 5 lines
builders.py +2
-0+if os.environ.get('ABHELPER_JSON'): + extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON'] @util.renderer def get_sstate_release_number(props):