Fix problem with leftover files in tmp/sysroots-components

We have seen problems where sometimes files are leftover in tmp/sysroots-components when changing PACKAGE_ARCH for a package. This could then lead to those files being picked up instead of the real files when the RSS for another package was created.

It turned out that this happened if the original architecture of the package was one that was added using PACKAGE_EXTRA_ARCHS. Those extra architectures were not handled in the sstate_eventhandler2() function in sstate.bbclass that does the clean up of tmp/sysroots-components.

During the course of debugging this problem I also noticed that there were two index files in tmp/sstate-control for machines that contain a dash in their names. This turned out to be due to the use of the stamp-extra-info task flag for some tasks, which were using ${MACHINE} rather than ${MACHINE_ARCH} to specify the architecture. Since this seemed inconsistent and confusing, I fixed that as well.

//Peter

The following changes since commit 312a5a93cac95fe62c56e374db728c825b272e64:

layer.conf: Update LAYERSERIES rocko -> sumo (2018-04-06 11:39:49 +0100)

are available in the git repository at:

git://push.yoctoproject.org/poky-contrib pkj/extra-archs-in-sstate

Peter Kjellerstedt (3):
Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task
flag
sstate.bbclass: Add ${PACKAGE_EXTRA_ARCHS} to SSTATE_ARCHS license.bbclass: Minor simplification of get_deployed_dependencies()

 meta/classes/deploy.bbclass            | 2 +-
 meta/classes/image.bbclass             | 2 +-
 meta/classes/license.bbclass           | 4 +---
 meta/classes/package.bbclass           | 2 +-
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/classes/populate_sdk_ext.bbclass  | 2 +-
 meta/classes/sstate.bbclass            | 5 +++--
 meta/recipes-core/meta/signing-keys.bb | 3 ++-
 8 files changed, 11 insertions(+), 11 deletions(-)

2.12.0

Without this change, there will be two sstate index files in tmp/sstate-control for any machine that contains a dash in the name.

---
 meta/classes/deploy.bbclass            | 2 +-
 meta/classes/image.bbclass             | 2 +-
 meta/classes/package.bbclass           | 2 +-
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/classes/populate_sdk_ext.bbclass  | 2 +-
 meta/recipes-core/meta/signing-keys.bb | 3 ++-
 6 files changed, 7 insertions(+), 6 deletions(-)

index 8ad07da015..6d52908783 100644
@@ -8,4 +8,4 @@ python do_deploy_setscene () {
 }
 addtask do_deploy_setscene
 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
-do_deploy[stamp-extra-info] = "${MACHINE}"
+do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"

index 1636eed6b0..2247b305da 100644
@@ -289,7 +289,7 @@ SSTATETASKS += "do_image_complete"
 SSTATE_SKIP_CREATION_task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
-do_image_complete[stamp-extra-info] = "${MACHINE}"
+do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
 addtask do_image_complete after do_image before do_build
 python do_image_complete_setscene () {
     sstate_setscene(d)

index 504c0232ef..9bba021efb 100644
@@ -2234,7 +2234,7 @@ SSTATETASKS += "do_packagedata"
 do_packagedata[sstate-inputdirs] = "${PKGDESTWORK}"
 do_packagedata[sstate-outputdirs] = "${PKGDATA_DIR}"
 do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
-do_packagedata[stamp-extra-info] = "${MACHINE}"
+do_packagedata[stamp-extra-info] = "${MACHINE_ARCH}"
 
 python do_packagedata_setscene () {
     sstate_setscene(d)

index 77ec8aaec2..3560ef143c 100644
@@ -156,7 +156,7 @@ SSTATE_SKIP_CREATION_task-populate-sdk = '1'
 do_populate_sdk[cleandirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-inputdirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
-do_populate_sdk[stamp-extra-info] = "${MACHINE}${SDKMACHINE}"
+do_populate_sdk[stamp-extra-info] = "${MACHINE_ARCH}${SDKMACHINE}"
 
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/

index 655375416b..e1bba49eaf 100644
@@ -722,6 +722,6 @@ SSTATE_SKIP_CREATION_task-populate-sdk-ext = '1'
 do_populate_sdk_ext[cleandirs] = "${SDKEXTDEPLOYDIR}"
 do_populate_sdk_ext[sstate-inputdirs] = "${SDKEXTDEPLOYDIR}"
 do_populate_sdk_ext[sstate-outputdirs] = "${SDK_DEPLOY}"
-do_populate_sdk_ext[stamp-extra-info] = "${MACHINE}"
+do_populate_sdk_ext[stamp-extra-info] = "${MACHINE_ARCH}"
 
 addtask populate_sdk_ext after do_sdk_depends

index 6387d90d47..1e1c7e3459 100644
@@ -69,7 +69,8 @@ do_deploy () {
 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_RPM}"
 # cleandirs should possibly be in deploy.bbclass but we need it
 do_deploy[cleandirs] = "${DEPLOYDIR}"
-# clear stamp-extra-info since MACHINE is normally put there by deploy.bbclass
+# clear stamp-extra-info since MACHINE_ARCH is normally put there by
+# deploy.bbclass
 do_deploy[stamp-extra-info] = ""
 addtask deploy after do_get_public_keys
 

2.12.0

This makes sure files provided by packages that use any of the extra architectures defined using ${PACKAGE_EXTRA_ARCHS} are cleaned from tmp/sysroot-components when sstate_eventhandler2() executes.

Without this, changing a package from using one of the extra architectures to another architecture would lead to files being leftover in tmp/sysroot-components, which could later be picked up when creating the RSS for another package rather than the files that belonged to the updated package.

---
 meta/classes/sstate.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index 68089421f5..0b28850140 100644
@@ -48,7 +48,8 @@ SSTATE_ARCHS = " \
     ${SDK_ARCH}_${PACKAGE_ARCH} \
     allarch \
     ${PACKAGE_ARCH} \
-    ${MACHINE}"
+    ${PACKAGE_EXTRA_ARCHS} \
+    ${MACHINE_ARCH}"
 
 SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
 

@@ -1026,7 +1027,7 @@ python sstate_eventhandler2() {
         with open(preservestampfile, 'r') as f:
             preservestamps = f.readlines()
     seen = []
-    for a in d.getVar("SSTATE_ARCHS").split():
+    for a in sorted(list(set(d.getVar("SSTATE_ARCHS").split()))):
         toremove = []
         i = d.expand("${SSTATE_MANIFESTS}/index-" + a)
         if not os.path.exists(i):

2.12.0

Since ${SSTATE_ARCHS} now contains ${PACKAGE_EXTRA_ARCHS} there is no longer any need to add those extra architectures to the list of architectures handled in get_deployed_dependencies().

---
 meta/classes/license.bbclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

index d353110464..9ac7e0bef8 100644
@@ -226,9 +226,7 @@ def get_deployed_dependencies(d):
     # The manifest file name contains the arch. Because we are not running
     # in the recipe context it is necessary to check every arch used.
     sstate_manifest_dir = d.getVar("SSTATE_MANIFESTS")
-    sstate_archs = d.getVar("SSTATE_ARCHS")
-    extra_archs = d.getVar("PACKAGE_EXTRA_ARCHS")
-    archs = list(set(("%s %s" % (sstate_archs, extra_archs)).split()))
+    archs = list(set(d.getVar("SSTATE_ARCHS").split()))
     for dep in depends:
         # Some recipes have an arch on their own, so we try that first.
         special_arch = d.getVar("PACKAGE_ARCH_pn-%s" % dep)

2.12.0

== Series Details ==

Series: Fix problem with leftover files in tmp/sysroots-components
Revision: 1
URL : https://patchwork.openembedded.org/series/11706/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Patch            [1/3] Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task flag
 Issue             Shortlog does not follow expected format [test_shortlog_format] 
  Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Which do you prefer:
* Leave it as it is
* Prefix it with a generic "classes:" (which isn't all true as the last hunk is a comment in a recipe about those classes)
* Split it in six almost identical commits

//Peter