image.bbclass: deploy image artifacts in stages
PATCH v3 1/1 image.bbclass: deploy image artifacts in stages
From: Bartosz Golaszewski <[email protected]>
Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.
We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.
In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.
---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistency
meta/classes/image.bbclass | 17 +++++++++++++++++ meta/classes/sstate.bbclass | 4 ++++ 2 files changed, 21 insertions(+)
meta/classes/image.bbclass +17 -0
@@ -425,10 +425,12 @@ python () { # date/time values. It will get expanded at execution time. # Similarly TMPDIR since otherwise we see QA stamp comparision problems # Expand PV else it can trigger get_srcrev which can fail due to these variables being unset + # Delete IMGDEPLOYDIR so that each task gets its own, overriden value localdata.setVar('PV', d.getVar('PV')) localdata.delVar('DATETIME') localdata.delVar('DATE') localdata.delVar('TMPDIR') + localdata.delVar('IMGDEPLOYDIR') vardepsexclude = (d.getVarFlag('IMAGE_CMD_' + realt, 'vardepsexclude', True) or '').split() for dep in vardepsexclude: localdata.delVar(dep)
@@ -499,6 +501,21 @@ python () { bb.debug(2, "Adding task %s before %s, after %s" % (task, 'do_image_complete', after)) bb.build.addtask(task, 'do_image_complete', after, d) + + imgdeploydir = d.getVar('IMGDEPLOYDIR') + task_override = task[3:].replace('_', '-') # 'do_image_ext4' becomes 'image-ext4' + taskdeploydir = '%s/deploy-%s-%s' % (os.path.dirname(imgdeploydir), + d.getVar('PN'), task_override) + + # Each image task gets its own IMGDEPLOYDIR directory and is added to + # SSTATETASKS. This way every set of artifacts gets deployed right after + # the do_image_foo task completes. + d.setVar('IMGDEPLOYDIR_task-%s' % task_override, taskdeploydir) + d.appendVar('SSTATETASKS', ' %s' % task) + d.setVarFlag(task, 'sstate-inputdirs', taskdeploydir) + d.setVarFlag(task, 'sstate-outputdirs', d.getVar('DEPLOY_DIR_IMAGE')) + d.setVarFlag(task, 'cleandirs', taskdeploydir) + d.setVar('SSTATE_SKIP_CREATION_task-%s' % task_override, '1') } #
meta/classes/sstate.bbclass +4 -0
@@ -138,13 +138,17 @@ python () { d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/") d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}") d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/") +} +python sstate_setup_tasks() { unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split())) d.setVar('SSTATETASKS', " ".join(unique_tasks)) for task in unique_tasks: d.prependVarFlag(task, 'prefuncs', "sstate_task_prefunc ") d.appendVarFlag(task, 'postfuncs', " sstate_task_postfunc") } +addhandler sstate_setup_tasks +sstate_setup_tasks[eventmask] = "bb.event.RecipeTaskPreProcess" def sstate_init(task, d): ss = {}
Signature · 1 line
2.25.0
wt., 12 maj 2020 o 16:05 Bartosz Golaszewski <[email protected]> napisał(a):
Quoted 13 lines
From: Bartosz Golaszewski <[email protected]>
Make each
IMAGE_CMDtask an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of thedo_image_completetask handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can makedo_image_wicdepend on fitImage task which can in turn depend ondo_image_ext4.We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.
Quoted 8 lines
In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.
Signed-off-by Bartosz Golaszewski <[email protected]>---
Hi all!
I just noticed I tagged this patch as [meta-OE] instead of [OE-core] - hope it's not an issue? Should I resend it with a correct tag?
Bartosz
Quoted 28 lines · Bartosz Golaszewski
wt., 12 maj 2020 o 16:05 Bartosz Golaszewski <[email protected]> napisał(a):
Quoted 26 lines
From: Bartosz Golaszewski <[email protected]>
Make each
IMAGE_CMDtask an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon
as it's ready instead of thedo_image_completetask handling the entire
deployement. This allows us to better fine-tune dependencies e.g. we
can makedo_image_wicdepend on fitImage task which can in turn depend
ondo_image_ext4.We need delete the IMGDEPLOYDIR variable from the data object passed
to each image task so that it gets expanded with the correct override.In order to make sure that tasks added to SSTATETASKS in anonymous python
functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess
in sstate.bbclass. This event is fired right after the anonymous functions.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Bartosz Golaszewski <[email protected]>---
Hi all!
I just noticed I tagged this patch as [meta-OE] instead of [OE-core] -
hope it's not an issue? Should I resend it with a correct tag?
I did notice but no need to resend. I was planning to queue the patches for testing.
Cheers,
Richard
Quoted 18 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Make each
IMAGE_CMDtask an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of thedo_image_completetask handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can makedo_image_wicdepend on fitImage task which can in turn depend ondo_image_ext4.We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.
In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.
Quoted 5 lines
Signed-off-by Bartosz Golaszewski <[email protected]>---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistency
Thanks, I included this in automated testing and its shown a few issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/943
"oe-selftest -r sstatetests.SStateTests.test_sstate_nativelsbstring_same_hash " should reproduce
I think multiconfig.MultiConfig.test_multiconfig and multiconfig.MultiConfig.test_multiconfig are also related but its harder to be sure.
package.PackageTests.test_gdb_hardlink_debug isn't related in there. runqemu.RunqemuTests.test_boot_deploy_hddimg and runqemu.RunqemuTests.test_boot_machine_iso could be, not sure.
Cheers,
Richard
Quoted 35 lines · Richard Purdie
Quoted 18 lines · Bartosz Golaszewski
From: Bartosz Golaszewski <[email protected]>
Make each
IMAGE_CMDtask an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of thedo_image_completetask handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can makedo_image_wicdepend on fitImage task which can in turn depend ondo_image_ext4.We need delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it gets expanded with the correct override.
In order to make sure that tasks added to SSTATETASKS in anonymous python functions are correctly setup, move the code that assigns pre- and postfuncs to an event handler invoked on bb.event.RecipeTaskPreProcess in sstate.bbclass. This event is fired right after the anonymous functions.
Quoted 5 lines
Signed-off-by Bartosz Golaszewski <[email protected]>---
Changes since v2:
- dropped the qemuboot patch (already upstream)
- switched to % string formatting instead of using .format() for consistencyThanks, I included this in automated testing and its shown a few issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/943
"oe-selftest -r sstatetests.SStateTests.test_sstate_nativelsbstring_same_hash " should reproduce
Quoted 8 lines
I think multiconfig.MultiConfig.test_multiconfig and multiconfig.MultiConfig.test_multiconfig are also related but its harder to be sure.
package.PackageTests.test_gdb_hardlink_debug isn't related in there. runqemu.RunqemuTests.test_boot_deploy_hddimg and runqemu.RunqemuTests.test_boot_machine_iso could be, not sure.
Hi Richard,
I've been trying to figure out what's wrong on and off for some time now but I'm afraid I don't get the idea behind the NATIVELSBSTRING variable and how it affects generating the hashes. Could you maybe point me in the right direction because so far I've been unable to find a way to fix these tests.
Best regards,
Bartosz