Skip to contentSkip to conversation

introduce multi-state image deployment

RFC PATCH Superseded Bartosz Golaszewski 3 participants 8 messages Mar 23, 2020 – Apr 2, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover introduce multi-state image deployment

From: Bartosz Golaszewski <[email protected]>

Hi Richard,

please take a look at the following patch series. It's a follow up to my previous attempt[1] and is also related to the dm-verity problem I described on the list[2].

This time instead of having hardcoded two stages of deployment for simple and composed image types, each image task becomes an sstate task and handles the deployment of the artifacts it generates. Files created by tasks other than IMAGE_CMD are still deployed by do_image_complete.

I'm tagging this series as RFC but actually the code turned out to be surprisingly clean and straightforward. Please let me know what you think.

[1] https://lists.openembedded.org/g/openembedded-core/message/136526
[2] https://lists.openembedded.org/g/openembedded-core/message/136203

Bartosz Golaszewski (2):
qemuboot.bbclass: don't redefine IMGDEPLOYDIR
image.bbclass: deploy image artifacts in stages

 meta/classes/image.bbclass    | 26 +++++++++++++++++++++++++-
 meta/classes/qemuboot.bbclass |  1 -
 2 files changed, 25 insertions(+), 2 deletions(-)
Signature · 1 line
2.19.1
#1 lore raw

RFC PATCH 1/2 qemuboot.bbclass: don't redefine IMGDEPLOYDIR

From: Bartosz Golaszewski <[email protected]>

This variable is already defined in image.bbclass and there's not need to redefine it here.

---
 meta/classes/qemuboot.bbclass | 1 -
 1 file changed, 1 deletion(-)
meta/classes/qemuboot.bbclass +0 -1
index 54044c38da..68f7a03619 100644
@@ -83,7 +83,6 @@ QB_DRIVE_TYPE ?= "/dev/sd"
 
 # Create qemuboot.conf
 addtask do_write_qemuboot_conf after do_rootfs before do_image
-IMGDEPLOYDIR ?= "${WORKDIR}/deploy-${PN}-image-complete"
 
 def qemuboot_vars(d):
     build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE',
Signature · 1 line
2.19.1
#3 lore raw

RFC PATCH 2/2 image.bbclass: deploy image artifacts in stages

In reply to Bartosz Golaszewski

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

We need to completely delete the do_package task in order to avoid problems with task hash changes as well as delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it's expanded with the correct override.

---
 meta/classes/image.bbclass | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)
meta/classes/image.bbclass +25 -1
index 07aa1f1fa5..3487e2abe4 100644
@@ -428,10 +428,12 @@ python () {
         # date/time values. It will get expanded at execution time.
         # Similarly TMPDIR since otherwise we see QA stamp comparision problems
         # Expand PV else it can trigger get_srcrev which can fail due to these variables being unset
+        # Delete IMGDEPLOYDIR so that each task gets its own, overriden value
         localdata.setVar('PV', d.getVar('PV'))
         localdata.delVar('DATETIME')
         localdata.delVar('DATE')
         localdata.delVar('TMPDIR')
+        localdata.delVar('IMGDEPLOYDIR')
         vardepsexclude = (d.getVarFlag('IMAGE_CMD_' + realt, 'vardepsexclude', True) or '').split()
         for dep in vardepsexclude:
             localdata.delVar(dep)
@@ -502,6 +504,28 @@ python () {
 
         bb.debug(2, "Adding task %s before %s, after %s" % (task, 'do_image_complete', after))
         bb.build.addtask(task, 'do_image_complete', after, d)
+
+        imgdeploydir = d.getVar('IMGDEPLOYDIR')
+        task_override = task[3:].replace('_', '-') # 'do_image_ext4' becomes 'image-ext4'
+        taskdeploydir = '{}/deploy-{}-{}'.format(os.path.dirname(imgdeploydir),
+                                                 d.getVar('PN'), task_override)
+
+        # Each image task gets its own IMGDEPLOYDIR and becomes part of SSTATETASKS
+        # so that each set of artifacts be deployed right after the task completes.
+        d.setVar('IMGDEPLOYDIR_task-{}'.format(task_override), taskdeploydir)
+        d.appendVar('SSTATETASKS', ' {}'.format(task))
+        d.setVarFlag(task, 'sstate-inputdirs', taskdeploydir)
+        d.setVarFlag(task, 'sstate-outputdirs', d.getVar('DEPLOY_DIR_IMAGE'))
+        d.setVarFlag(task, 'cleandirs', taskdeploydir)
+        d.setVar('SSTATE_SKIP_CREATION_task-{}'.format(task_override), '1')
+
+        # FIXME We need to do this manually if we want to set the sstate flags from
+        # python code. The reason for this is that sstate.bbclass does this from an
+        # annonymous python function too and we can't guarantee correct ordering.
+        # This isn't a problem for most recipes as they define sstate tasks statically
+        # with bitbake variables.
+        d.prependVarFlag(task, 'prefuncs', 'sstate_task_prefunc ')
+        d.appendVarFlag(task, 'postfuncs', ' sstate_task_postfunc')
 }
 
 #
@@ -611,7 +635,7 @@ do_compile[noexec] = "1"
 do_install[noexec] = "1"
 deltask do_populate_lic
 deltask do_populate_sysroot
-do_package[noexec] = "1"
+deltask do_package
 deltask do_package_qa
 do_packagedata[noexec] = "1"
 deltask do_package_write_ipk
Signature · 1 line
2.19.1
#4 lore raw

Re: [OE-core][RFC PATCH 2/2] image.bbclass: deploy image artifacts in stages

pon., 23 mar 2020 o 18:28 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 10 lines

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon as it's ready instead of the do_image_complete task handling the entire deployement. This allows us to better fine-tune dependencies e.g. we can make do_image_wic depend on fitImage task which can in turn depend on do_image_ext4.

Quoted 6 lines

We need to completely delete the do_package task in order to avoid problems with task hash changes as well as delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it's expanded with the correct override.

Hi all!

It's been a week. Can I get some feedback on this? Is the idea at least remotely acceptable/can we build upon it?

Bart

#5 lore raw

Re: [OE-core][RFC PATCH 2/2] image.bbclass: deploy image artifacts in stages

Quoted 21 lines · Bartosz Golaszewski

pon., 23 mar 2020 o 18:28 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 19 lines

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon
as it's ready instead of the do_image_complete task handling the entire
deployement. This allows us to better fine-tune dependencies e.g. we
can make do_image_wic depend on fitImage task which can in turn depend
on do_image_ext4.

We need to completely delete the do_package task in order to avoid problems with task hash changes as well as delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it's
expanded with the correct override.

Quoted 4 lines
Quoted 1 line

It's been a week. Can I get some feedback on this? Is the idea at least remotely acceptable/can we build upon it?

FWIW at a quick glance I didn't think it was too bad.

I think there will be corner cases to resolve which I was hoping to look into and give some pointers to but I haven't had the time.

I'm hoping somehow we can improve the FIXME you mention too.

The do_package change should probably be separated out - I'm guessing we did noexec there for a reason though?

Also, you used {}.format which I'm torn on since most of the codebase uses the other approach.

Its too late to get this into 3.1 and that is where I'm focusing my effort right now but I think this is probably going the right way.

Cheers,

Richard

#6 lore raw

Re: [OE-core][RFC PATCH 2/2] image.bbclass: deploy image artifacts in stages

pon., 30 mar 2020 o 18:31 Richard Purdie
<[email protected]> napisał(a):

Quoted 28 lines · Bartosz Golaszewski
Quoted 21 lines

pon., 23 mar 2020 o 18:28 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 19 lines

From: Bartosz Golaszewski <[email protected]>

Make each IMAGE_CMD task an sstate task with its own IMGDEPLOYDIR override. This way each generated set of artifacts is deployed as soon
as it's ready instead of the do_image_complete task handling the entire
deployement. This allows us to better fine-tune dependencies e.g. we
can make do_image_wic depend on fitImage task which can in turn depend
on do_image_ext4.

We need to completely delete the do_package task in order to avoid problems with task hash changes as well as delete the IMGDEPLOYDIR variable from the data object passed to each image task so that it's
expanded with the correct override.

Quoted 4 lines
Quoted 1 line

It's been a week. Can I get some feedback on this? Is the idea at least remotely acceptable/can we build upon it?

Quoted 7 lines

FWIW at a quick glance I didn't think it was too bad.

I think there will be corner cases to resolve which I was hoping to look into and give some pointers to but I haven't had the time.

I'm hoping somehow we can improve the FIXME you mention too.

Yeah, after thinking about it more, I figured that if a task signature changes, then something is wrong. I guess it's because we end up assigning pre- and postfuncs to tasks twice: once in anonymous python code from image.bbclass and then from sstate.bbclass. Maybe we should provide a python helper for explicitly making a task an sstate task and - when using it - not add said task to SSTATETASKS, so that sstate.bbclass code ignores it? Or maybe this anonymous function should become an event handler invoked at event.RecipeParsed?

Quoted 3 lines

The do_package change should probably be separated out - I'm guessing we did noexec there for a reason though?

I couldn't find a reason. Maybe it's a leftover from some previous, now cleaned up change.

Quoted 3 lines

Also, you used {}.format which I'm torn on since most of the codebase uses the other approach.

I couldn't find any official guidelines for that. {} is the preferred way in Python. Maybe it's time to start slowly converting the bitbake codebase?

Quoted 3 lines

Its too late to get this into 3.1 and that is where I'm focusing my effort right now but I think this is probably going the right way.

Sure, I didn't expect it to go into the next release. If you're happy with this direction I'll try to improve it and send a v2.

Bartosz

#7 lore raw

Re: [OE-core][RFC PATCH 2/2] image.bbclass: deploy image artifacts in stages

Quoted 15 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Bartosz Golaszewski
Sent: den 31 mars 2020 11:43
To: Richard Purdie <[email protected]>
Cc: Khem Raj <[email protected]>; Armin Kuster <[email protected]>; Jerome Neanne <[email protected]>; Quentin Schulz <[email protected]>; Patches and discussions about the oe-core layer <[email protected]>; Bartosz Golaszewski <[email protected]>
Subject: Re: [OE-core][RFC PATCH 2/2] image.bbclass: deploy image artifacts in stages

pon., 30 mar 2020 o 18:31 Richard Purdie
<[email protected]> napisał(a):

[cut]

Quoted 6 lines
Quoted 2 lines

Also, you used {}.format which I'm torn on since most of the codebase uses the other approach.

I couldn't find any official guidelines for that. {} is the preferred way in Python. Maybe it's time to start slowly converting the bitbake codebase?

If we have waited so long to change string formatting, why not wait till we require Python 3.6 and go for f-strings directly? With Tim's proposed patch we will soon require Python 3.5 as a minimum, so 3.6 as a minimum requirement is probably not too long in the future...

Anyway, since I had fun generating it, I have included a graph depicting the use of % vs .format() for all Poky releases (and the script used to generate it; run it in a pristine clone of Poky for an interactive experience). Enjoy. :)

(For the curious, the first use of .format() was introduced to bitbake almost exactly 10 years ago in the Laverne release.)

//Peter

#2 lore raw

Re: [OE-core][RFC PATCH 1/2] qemuboot.bbclass: don't redefine IMGDEPLOYDIR

In reply to Bartosz Golaszewski

pon., 23 mar 2020 o 18:28 Bartosz Golaszewski <[email protected]> napisał(a):

Quoted 24 lines

From: Bartosz Golaszewski <[email protected]>

This variable is already defined in image.bbclass and there's not need to redefine it here.

---
 meta/classes/qemuboot.bbclass | 1 -
 1 file changed, 1 deletion(-)
meta/classes/qemuboot.bbclass +0 -2
index 54044c38da..68f7a03619 100644
@@ -83,7 +83,6 @@ QB_DRIVE_TYPE ?= "/dev/sd"
 
 # Create qemuboot.conf
 addtask do_write_qemuboot_conf after do_rootfs before do_image
-IMGDEPLOYDIR ?= "${WORKDIR}/deploy-${PN}-image-complete"
 
 def qemuboot_vars(d):
     build_vars = ['MACHINE', 'TUNE_ARCH', 'DEPLOY_DIR_IMAGE',
--
Quoted 2 lines

2.19.1

Hi Richard,

this patch can be applied without waiting for the v2 of the image deployment patch. Would you mind picking it up into your tree?

Bartosz