Skip to contentSkip to conversation

rm_work: Improve handling for addto_recipe_sysroot

PATCH New Richard Purdie 1 participant 5 messages Jul 30, 2017

lore
Share & tools
lore
#0 lore raw

PATCH 1/5 rm_work: Improve handling for addto_recipe_sysroot

Rather than requiring each user to handle this individually, handle addto_recipe_sysroot in the core class. As well as preserving the sysroot directory, this also ensures the stamp is preserved rather than rerunning the task every time as currently happens.

---
 meta/classes/rm_work.bbclass                       | 32 ++++++++++++++--------
 .../qemu/qemu-helper-native_1.0.bb                 |  1 -
 .../systemtap/systemtap-native_git.bb              |  1 -
 3 files changed, 20 insertions(+), 14 deletions(-)
meta/classes/rm_work.bbclass +20 -12
index badeaeb..2a4e808 100644
@@ -35,22 +35,12 @@ do_rm_work () {
         fi
     done
 
-    cd ${WORKDIR}
-    for dir in *
-    do
-        # Retain only logs and other files in temp, safely ignore
-        # failures of removing pseudo folers on NFS2/3 server.
-        if [ $dir = 'pseudo' ]; then
-            rm -rf $dir 2> /dev/null || true
-        elif ! echo '${RM_WORK_EXCLUDE_ITEMS}' | grep -q -w "$dir"; then
-            rm -rf $dir
-        fi
-    done
-
     # Need to add pseudo back or subsqeuent work in this workdir
     # might fail since setscene may not rerun to recreate it
     mkdir -p ${WORKDIR}/pseudo/
 
+    excludes='${RM_WORK_EXCLUDE_ITEMS}'
+
     # Change normal stamps into setscene stamps as they better reflect the
     # fact WORKDIR is now empty
     # Also leave noexec stamps since setscene stamps don't cover them
@@ -79,6 +69,12 @@ do_rm_work () {
                 i=dummy
                 break
                 ;;
+            *do_addto_recipe_sysroot*)
+                # Preserve recipe-sysroot-native if do_addto_recipe_sysroot has been used
+                excludes="$excludes recipe-sysroot-native"
+                i=dummy
+                break
+                ;;
             # We remove do_package entirely, including any
             # sstate version since otherwise we'd need to leave 'plaindirs' around
             # such as 'packages' and 'packages-split' and these can be large. No end
@@ -101,6 +97,18 @@ do_rm_work () {
         done
         rm -f $i
     done
+
+    cd ${WORKDIR}
+    for dir in *
+    do
+        # Retain only logs and other files in temp, safely ignore
+        # failures of removing pseudo folers on NFS2/3 server.
+        if [ $dir = 'pseudo' ]; then
+            rm -rf $dir 2> /dev/null || true
+        elif ! echo "$excludes" | grep -q -w "$dir"; then
+            rm -rf $dir
+        fi
+    done
 }
 do_rm_work_all () {
     :
meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb +0 -1
index 27d5315..d86b155 100644
@@ -20,6 +20,5 @@ do_install() {
 	install tunctl ${D}${bindir}/
 }
 
-RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
 DEPENDS += "qemu-native"
 addtask addto_recipe_sysroot after do_populate_sysroot before do_build
meta/recipes-kernel/systemtap/systemtap-native_git.bb +0 -1
index c3da77c..19cc1cf 100644
@@ -3,5 +3,4 @@ require systemtap_git.bb
 
 inherit native
 
-RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
 addtask addto_recipe_sysroot after do_populate_sysroot before do_build
Signature · 1 line
2.7.4
#1 lore raw

PATCH 2/5 sstate: Improve SSTATE_SKIP_CREATION

Currently, dummy archives are created for skipped sstate tasks. We relied on these never being installed (the setscene task is missing) however this leads to odd behaviour as for example the setscene stamps are never looked at.

Rather than trying to continue with the two separate behaviours, really skip package creation. We do need the file manipulation steps to install files under sstate control from a manifest perspective so we only skip at the final creation step.

---
 meta/classes/sstate.bbclass | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
meta/classes/sstate.bbclass +4 -6
index f446c3d..6af0d38 100644
@@ -346,8 +346,6 @@ def sstate_installpkgdir(ss, d):
         oe.path.remove(dir)
 
     for state in ss['dirs']:
-        if d.getVar('SSTATE_SKIP_CREATION') == '1':
-            continue
         prepdir(state[1])
         os.rename(sstateinst + state[0], state[1])
     sstate_install(ss, d)
@@ -596,8 +594,6 @@ def sstate_package(ss, d):
     for state in ss['dirs']:
         if not os.path.exists(state[1]):
             continue
-        if d.getVar('SSTATE_SKIP_CREATION') == '1':
-            continue
         srcbase = state[0].rstrip("/").rsplit('/', 1)[0]
         # Find and error for absolute symlinks. We could attempt to relocate but its not
         # clear where the symlink is relative to in this context. We could add that markup
@@ -625,6 +621,10 @@ def sstate_package(ss, d):
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_PKG', sstatepkg)
+    d.setVar('SSTATE_INSTDIR', sstatebuild)
+
+    if d.getVar('SSTATE_SKIP_CREATION') == '1':
+        return
 
     for f in (d.getVar('SSTATECREATEFUNCS') or '').split() + \
              ['sstate_create_package', 'sstate_sign_package'] + \
@@ -634,8 +634,6 @@ def sstate_package(ss, d):
 
     bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
 
-    d.setVar('SSTATE_INSTDIR', sstatebuild)
-
     return
 
 def pstaging_fetch(sstatefetch, sstatepkg, d):
Signature · 1 line
2.7.4
#2 lore raw

PATCH 3/5 image: Drop before do_build from do_rootfs and do_image

In reply to Richard Purdie

do_image_complete has the do_build dependency, these other two tasks don't need it. If this is present, it complicates the dependency tree and convinces rm_work that the tasks need to rerun even if they already did since the stamps are removed.

Drop the unneeded dependency to help resolve this issue.

---
 meta/classes/image.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +2 -2
index 40bd614..0885fdb 100644
@@ -261,7 +261,7 @@ fakeroot python do_rootfs () {
 do_rootfs[dirs] = "${TOPDIR}"
 do_rootfs[cleandirs] += "${S} ${IMGDEPLOYDIR}"
 do_rootfs[umask] = "022"
-addtask rootfs before do_build after do_prepare_recipe_sysroot
+addtask rootfs after do_prepare_recipe_sysroot
 
 fakeroot python do_image () {
     from oe.utils import execute_pre_post_process
@@ -272,7 +272,7 @@ fakeroot python do_image () {
 }
 do_image[dirs] = "${TOPDIR}"
 do_image[umask] = "022"
-addtask do_image after do_rootfs before do_build
+addtask do_image after do_rootfs
 
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
Signature · 1 line
2.7.4
#3 lore raw

PATCH 4/5 image/rm_work: Promote do_image_complete to be more sstate like

In reply to Richard Purdie

We relied on the missing do_image_complete_setscene task to ensure the dummy sstate tarball that was created would never be used. This lead to its own issues and a better fix for SSTATE_SKIP_CREATION has now been merged.

We can therefore make do_image_complete look like a more standard sstate task which means image generation doesn't keep rerunning when using rm_work.

We do need to turn do_image_complete's stamp into an sstate version to handle this (it otherwise matches the do_image_* glob).

---
 meta/classes/image.bbclass   | 4 ++++
 meta/classes/rm_work.bbclass | 5 +++++
 2 files changed, 9 insertions(+)
meta/classes/image.bbclass +4 -0
index 0885fdb..696aacd 100644
@@ -289,6 +289,10 @@ do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_image_complete[stamp-extra-info] = "${MACHINE}"
 addtask do_image_complete after do_image before do_build
+python do_image_complete_setscene () {
+    sstate_setscene(d)
+}
+addtask do_image_complete_setscene
 
 # Add image-level QA/sanity checks to IMAGE_QA_COMMANDS
 #
meta/classes/rm_work.bbclass +5 -0
index 2a4e808..71c21a0 100644
@@ -61,6 +61,11 @@ do_rm_work () {
                 i=dummy
                 break
                 ;;
+            *do_image_complete*)
+                mv $i `echo $i | sed -e "s#${j}#${j}_setscene#"`
+                i=dummy
+                break
+                ;;
             *do_rootfs*|*do_image*|*do_bootimg*|*do_bootdirectdisk*|*do_vmimg*|*do_write_qemuboot_conf*)
                 i=dummy
                 break
Signature · 1 line
2.7.4
#4 lore raw

PATCH 5/5 image: Make do_image_qa a dummy sstate task to match do_image_complete

In reply to Richard Purdie

Similar to do_image_complete, make do_image_qa a dummy sstate task so that rm_work doesn't cause image generation to repeat on every new build command.

---
 meta/classes/image.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)
meta/classes/image.bbclass +9 -0
index 696aacd..2c1dc81 100644
@@ -324,6 +324,15 @@ fakeroot python do_image_qa () {
 }
 addtask do_image_qa after do_image_complete before do_build
 
+SSTATETASKS += "do_image_qa"
+SSTATE_SKIP_CREATION_task-image-qa = '1'
+do_image_qa[sstate-inputdirs] = ""
+do_image_qa[sstate-outputdirs] = ""
+python do_image_qa_setscene () {
+    sstate_setscene(d)
+}
+addtask do_image_qa_setscene
+
 def setup_debugfs_variables(d):
     d.appendVar('IMAGE_ROOTFS', '-dbg')
     d.appendVar('IMAGE_LINK_NAME', '-dbg')
Signature · 1 line
2.7.4