Skip to contentSkip to conversation

image.bbclass: deploy image artifacts in stages

PATCH In review Bartosz Golaszewski 2 participants 5 messages May 12, 2020 – Aug 7, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 image.bbclass: deploy image artifacts in stages

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.

In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.

---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistency

 meta/classes/image.bbclass  | 17 +++++++++++++++++
 meta/classes/sstate.bbclass |  4 ++++
 2 files changed, 21 insertions(+)
meta/classes/image.bbclass +17 -0
index 694b58fc9f..3cf8ceb2d6 100644
@@ -425,10 +425,12 @@ python () {
         # date/time values. It will get expanded at execution time.
         # Similarly TMPDIR since otherwise we see QA stamp comparision problems
         # Expand PV else it can trigger get_srcrev which can fail due to these variables being unset
+        # Delete IMGDEPLOYDIR so that each task gets its own, overriden value
         localdata.setVar('PV', d.getVar('PV'))
         localdata.delVar('DATETIME')
         localdata.delVar('DATE')
         localdata.delVar('TMPDIR')
+        localdata.delVar('IMGDEPLOYDIR')
         vardepsexclude = (d.getVarFlag('IMAGE_CMD_' + realt, 'vardepsexclude', True) or '').split()
         for dep in vardepsexclude:
             localdata.delVar(dep)
@@ -499,6 +501,21 @@ python () {
 
         bb.debug(2, "Adding task %s before %s, after %s" % (task, 'do_image_complete', after))
         bb.build.addtask(task, 'do_image_complete', after, d)
+
+        imgdeploydir = d.getVar('IMGDEPLOYDIR')
+        task_override = task[3:].replace('_', '-') # 'do_image_ext4' becomes 'image-ext4'
+        taskdeploydir = '%s/deploy-%s-%s' % (os.path.dirname(imgdeploydir),
+                                             d.getVar('PN'), task_override)
+
+        # Each image task gets its own IMGDEPLOYDIR directory and is added to
+        # SSTATETASKS. This way every set of artifacts gets deployed right after
+        # the do_image_foo task completes.
+        d.setVar('IMGDEPLOYDIR_task-%s' % task_override, taskdeploydir)
+        d.appendVar('SSTATETASKS', ' %s' % task)
+        d.setVarFlag(task, 'sstate-inputdirs', taskdeploydir)
+        d.setVarFlag(task, 'sstate-outputdirs', d.getVar('DEPLOY_DIR_IMAGE'))
+        d.setVarFlag(task, 'cleandirs', taskdeploydir)
+        d.setVar('SSTATE_SKIP_CREATION_task-%s' % task_override, '1')
 }
 
 #
meta/classes/sstate.bbclass +4 -0
index aa9c30b4e1..5bdada673f 100644
@@ -138,13 +138,17 @@ python () {
         d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/")
         d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}")
         d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/")
+}
 
+python sstate_setup_tasks() {
     unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split()))
     d.setVar('SSTATETASKS', " ".join(unique_tasks))
     for task in unique_tasks:
         d.prependVarFlag(task, 'prefuncs', "sstate_task_prefunc ")
         d.appendVarFlag(task, 'postfuncs', " sstate_task_postfunc")
 }
+addhandler sstate_setup_tasks
+sstate_setup_tasks[eventmask] = "bb.event.RecipeTaskPreProcess"
 
 def sstate_init(task, d):
     ss = {}
Signature · 1 line
2.25.0
#1 lore raw

wt., 12 maj 2020 o 16:05 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 13 lines

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.

Quoted 8 lines

In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.

---

Hi all!

I just noticed I tagged this patch as [meta-OE] instead of [OE-core] - hope it's not an issue? Should I resend it with a correct tag?

Bartosz

#2 lore raw
Quoted 28 lines · Bartosz Golaszewski

wt., 12 maj 2020 o 16:05 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 26 lines

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon
as it's ready instead of the do_image_complete task handling the entire
deployement. This allows us to better fine-tune dependencies e.g. we
can make do_image_wic depend on fitImage task which can in turn depend
on do_image_ext4.

We need delete the IMGDEPLOYDIR variable from the data object passed
to each image task so that it gets expanded with the correct override.

In order to make sure that tasks added to SSTATETASKS in anonymous python
functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess
in sstate.bbclass. This event is fired right after the anonymous functions.

Quoted 8 lines
Quoted 2 lines

---

Hi all!

I just noticed I tagged this patch as [meta-OE] instead of [OE-core] -
hope it's not an issue? Should I resend it with a correct tag?

I did notice but no need to resend. I was planning to queue the patches for testing.

Cheers,

Richard

#3 lore raw
In reply to Bartosz Golaszewski
Quoted 18 lines · Bartosz Golaszewski

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.

In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.

Quoted 5 lines

---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistency

Thanks, I included this in automated testing and its shown a few issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/943

"oe-selftest -r sstatetests.SStateTests.test_sstate_nativelsbstring_same_hash " should reproduce

I think multiconfig.MultiConfig.test_multiconfig and multiconfig.MultiConfig.test_multiconfig are also related but its harder to be sure.

package.PackageTests.test_gdb_hardlink_debug isn't related in there. runqemu.RunqemuTests.test_boot_deploy_hddimg and runqemu.RunqemuTests.test_boot_machine_iso could be, not sure.

Cheers,

Richard

#4 lore raw
Quoted 35 lines · Richard Purdie
Quoted 18 lines · Bartosz Golaszewski

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.

In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.

Quoted 5 lines

---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistency

Thanks, I included this in automated testing and its shown a few issues:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/943

"oe-selftest -r sstatetests.SStateTests.test_sstate_nativelsbstring_same_hash " should reproduce

Quoted 8 lines

I think multiconfig.MultiConfig.test_multiconfig and multiconfig.MultiConfig.test_multiconfig are also related but its harder to be sure.

package.PackageTests.test_gdb_hardlink_debug isn't related in there. runqemu.RunqemuTests.test_boot_deploy_hddimg and runqemu.RunqemuTests.test_boot_machine_iso could be, not sure.

Hi Richard,

I've been trying to figure out what's wrong on and off for some time now but I'm afraid I don't get the idea behind the NATIVELSBSTRING variable and how it affects generating the hashes. Could you maybe point me in the right direction because so far I've been unable to find a way to fix these tests.

Best regards,
Bartosz