Skip to contentSkip to conversation

Bug fix: added support for ABHELPER_JSON environ.

PATCH In review Thomas Goodwin 3 participants 7 messages Jul 30, 2019 – Aug 2, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 Bug fix: added support for ABHELPER_JSON environ.

The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to extra_env.

---
 builders.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builders.py +2 -1
index 5765040..3d12969 100644
@@ -15,7 +15,8 @@ maxsteps = 9
 # Environment to pass into the workers, e.g. to load further local configuration
 # fragments
 extra_env = {}
-#extra_env = {"ABHELPER_JSON" : "config.json /home/pokybuild/config-local.json"}
+if os.environ.get('ABHELPER_JSON'):
+    extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON']
 
 @util.renderer
 def get_sstate_release_number(props):
Signature · 1 line
2.17.1
#1 lore raw

PATCH 2/3 Added remote distribution of config JSON files

This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.

---
 builders.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
builders.py +22 -0
index 3d12969..35903de 100644
@@ -18,6 +18,26 @@ extra_env = {}
 if os.environ.get('ABHELPER_JSON'):
     extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON']
 
+def add_abhelper_steps(factory):
+    """
+    Adds FileDownload steps to the factory for each specified autobuilder
+    helper JSON file and installs the file in the worker's
+    builddir/yocto-autobuilder-helper/ path.
+    """
+    abhelper_dir = os.path.expanduser("~/yocto-autobuilder-helper")
+    abhelper_configs = extra_env.get("ABHELPER_JSON", "config.json").split(" ")
+    for helper in abhelper_configs:
+        helper_name = os.path.basename(helper)
+        worker_helper = util.Interpolate("%(prop:builddir)s/yocto-autobuilder-helper/" + helper_name)
+        master_helper = helper
+        if not helper.startswith("/"):
+            master_helper = os.path.join(abhelper_dir, helper)
+        factory.addStep(steps.FileDownload(
+            name="Download '{}'".format(helper_name),
+            mastersrc=master_helper,
+            workerdest=worker_helper,
+            haltOnFailure=True ))
+
 @util.renderer
 def get_sstate_release_number(props):
     """
@@ -145,6 +165,7 @@ def create_builder_factory():
         mode='incremental',
         haltOnFailure=True,
         name='Fetch yocto-autobuilder-helper'))
+    add_abhelper_steps(f)
     f.addStep(steps.SetProperties(properties=ensure_props_set))
     f.addStep(WriteLayerInfo(name='Write main layerinfo.json', haltOnFailure=True))
     f.addStep(steps.ShellCommand(
@@ -213,6 +234,7 @@ def create_parent_builder_factory(buildername, waitname):
         mode='incremental',
         haltOnFailure=True,
         name='Fetch yocto-autobuilder-helper'))
+    add_abhelper_steps(factory)
     factory.addStep(WriteLayerInfo(name='Write main layerinfo.json', haltOnFailure=True))
     factory.addStep(steps.ShellCommand(
         command=[
Signature · 1 line
2.17.1
#2 lore raw

Re: [yocto-autobuilder2][PATCH 2/3] Added remote distribution of config JSON files

Quoted 11 lines · Thomas Goodwin

This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.

---
 builders.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

I like the idea however this is going to break the autobuilders as they already use full paths in the variable.

I've merged some of the other patches but this one will need a little further thought so we don't break the existing install.

The existing code also allows for different path locations on individual workers (e.g. different username) which this would break.

Cheers,

Richard

#3 lore raw

Re: [yocto-autobuilder2][PATCH 2/3] Added remote distribution of config JSON files

I'm not sure I follow. This collects everything into the worker's "builddir/yocto-autobuilder-helper" path (I think; I'm new to buildbot).

My aim was to let the buildbot master the chance to SCM the configuration files apart from the helper's behavior which should otherwise be consistent whether we're rebuilding for warrior or morty. For us this is turning out to not be "enough" since certain build targets don't exist between different releases of Poky and we're held back by BSPs to certain releases, so we need to decouple a bit further than this patch. The issue could also be solved with templates that dynamically point to other templates based on the Force Build branch, revision, etc., but I'm looking for a more simple approach than a recursive overlay.

Thanks again for all your help and feedback.

Cheers,

Thomas

Quoted 27 lines · Richard Purdie
Quoted 11 lines · Thomas Goodwin

This feature/fix allows the worker to download the autobuilder helper JSON files from the controller rather than having to store them in the yocto-autobuilder-helper repository. The process installs the file into the worker's builddir/yocto-autobuilder-helper path.

---
 builders.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

I like the idea however this is going to break the autobuilders as they already use full paths in the variable.

I've merged some of the other patches but this one will need a little further thought so we don't break the existing install.

The existing code also allows for different path locations on individual workers (e.g. different username) which this would break.

Cheers,

Richard

#4 lore raw

PATCH 3/3 Support for absolute paths to config files

In reply to Thomas Goodwin

Using absolute paths in ABHELPER_JSON with remote-loading now works; the paths are stripped to os.path.basename() before being provided to the builders.

---
 builders.py | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
builders.py +11 -3
index 35903de..16c1344 100644
@@ -4,6 +4,7 @@ from yoctoabb import config
 from yoctoabb.steps.writelayerinfo import WriteLayerInfo
 from yoctoabb.steps.observer import RunConfigLogObserver
 from datetime import datetime
+import copy
 
 import os
 import json
@@ -38,6 +39,13 @@ def add_abhelper_steps(factory):
             workerdest=worker_helper,
             haltOnFailure=True ))
 
+# At the worker, all paths are relative to builddir/yocto-autobuilder-helper
+# once the add_abhelper_steps run.
+worker_env = copy.deepcopy(extra_env)
+if worker_env.get('ABHELPER_JSON'):
+    configs = [os.path.basename(c) for c in worker_env['ABHELPER_JSON'].split(" ")]
+    worker_env['ABHELPER_JSON'] = " ".join(configs)
+
 @util.renderer
 def get_sstate_release_number(props):
     """
@@ -216,7 +224,7 @@ for builder in config.subbuilders:
         workers = config.builder_to_workers['default']
     builders.append(util.BuilderConfig(name=builder,
                                        workernames=workers,
-                                       factory=f, env=extra_env))
+                                       factory=f, env=worker_env))
 
 def create_parent_builder_factory(buildername, waitname):
     factory = util.BuildFactory()
@@ -337,5 +345,5 @@ def create_parent_builder_factory(buildername, waitname):
 
     return factory
 
-builders.append(util.BuilderConfig(name="a-quick", workernames=config.workers, factory=create_parent_builder_factory("a-quick", "wait-quick"), env=extra_env))
-builders.append(util.BuilderConfig(name="a-full", workernames=config.workers, factory=create_parent_builder_factory("a-full", "wait-full"), env=extra_env))
+builders.append(util.BuilderConfig(name="a-quick", workernames=config.workers, factory=create_parent_builder_factory("a-quick", "wait-quick"), env=worker_env))
+builders.append(util.BuilderConfig(name="a-full", workernames=config.workers, factory=create_parent_builder_factory("a-full", "wait-full"), env=worker_env))
Signature · 1 line
2.17.1
#5 lore raw
In reply to Thomas Goodwin
Quoted 4 lines · Thomas Goodwin

The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to extra_env.

Any chance of updating documentation on how to use this?

- armin

Quoted 20 lines
---
 builders.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builders.py +2 -1
index 5765040..3d12969 100644
@@ -15,7 +15,8 @@ maxsteps = 9
 # Environment to pass into the workers, e.g. to load further local configuration
 # fragments
 extra_env = {}
-#extra_env = {"ABHELPER_JSON" : "config.json /home/pokybuild/config-local.json"}
+if os.environ.get('ABHELPER_JSON'):
+    extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON']
 
 @util.renderer
 def get_sstate_release_number(props):
#6 lore raw

It's already there, this patch is to make it work as described:
http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder2/tree/README-WALKTHROUGHS.md?h=contrib/tgoodwin/config-json-fixes#n79

Consequently, having the variable in your environment is the only way for the janitor to parse and use the same configuration files as the controller.

Thomas

Quoted 35 lines · akuster
Quoted 4 lines · Thomas Goodwin

The readme suggests exporting this variable from the environment to expose multiple configuration JSON files but the intended behavior doesn't work without adding the variable to extra_env.

Any chance of updating documentation on how to use this?

- armin

builders.py maxsteps = 9 · line 15

Quoted 12 lines
---
 builders.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
builders.py +0 -0
index 5765040..3d12969 100644
@@ -15,7 +15,8 @@ maxsteps = 9
 # Environment to pass into the workers, e.g. to load further local

configuration

builders.py maxsteps = 9

Quoted 3 lines
builders.py +0 -1
 # fragments
 extra_env = {}
-#extra_env = {"ABHELPER_JSON" : "config.json

/home/pokybuild/config-local.json"}

Quoted 5 lines
builders.py +2 -0
+if os.environ.get('ABHELPER_JSON'):
+    extra_env['ABHELPER_JSON'] = os.environ['ABHELPER_JSON']
 
 @util.renderer
 def get_sstate_release_number(props):