Skip to contentSkip to conversation

image.bbclass: deltask do_packagedata

PATCH In review Robert Yang 5 participants 13 messages Mar 11, 2021 – Mar 15, 2021 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover image.bbclass: deltask do_packagedata

The following changes since commit b27bff26a9640dfdfba4fc711a9b28acb4501d2b:

sstate-diff-machines.sh: support rpm, deb and tar package types in analysis (2021-03-10 10:29:44 +0000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/image
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/image

Robert Yang (1):
image.bbclass: deltask do_packagedata

 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 image.bbclass: deltask do_packagedata

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image.bbclass +1 -1
index 41fc3291781..b5586536355 100644
@@ -612,7 +612,7 @@ deltask do_populate_lic
 deltask do_populate_sysroot
 do_package[noexec] = "1"
 deltask do_package_qa
-do_packagedata[noexec] = "1"
+deltask do_packagedata
 deltask do_package_write_ipk
 deltask do_package_write_deb
 deltask do_package_write_rpm
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#2 lore raw
Quoted 13 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

Quoted 8 lines

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 24 lines
---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image.bbclass +1 -1
index 41fc3291781..b5586536355 100644
@@ -612,7 +612,7 @@ deltask do_populate_lic
 deltask do_populate_sysroot
 do_package[noexec] = "1"
 deltask do_package_qa
-do_packagedata[noexec] = "1"
+deltask do_packagedata
 deltask do_package_write_ipk
 deltask do_package_write_deb
 deltask do_package_write_rpm
Signature · 1 line
2.17.1
#3 lore raw
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

The differences were:

@@ -11036,7 +11036,6 @@
"core-image-sato.do_build" -> "connman.do_build"
"core-image-sato.do_build" -> "consolekit.do_build"
"core-image-sato.do_build" -> "core-image-sato.do_image_complete"
-"core-image-sato.do_build" -> "core-image-sato.do_packagedata"
"core-image-sato.do_build" -> "core-image-sato.do_populate_lic_deploy"
"core-image-sato.do_build" -> "coreutils.do_build"
"core-image-sato.do_build" -> "cross-localedef-native.do_build"
@@ -11459,14 +11458,6 @@
"core-image-sato.do_build" -> "zip.do_build"
"core-image-sato.do_build" -> "zlib-native.do_build"
"core-image-sato.do_build" -> "zlib.do_build"
-"core-image-sato.do_compile" [label="core-image-sato do_compile\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_compile" -> "core-image-sato.do_configure"
-"core-image-sato.do_configure" [label="core-image-sato do_configure\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_configure" -> "core-image-sato.do_deploy_source_date_epoch"
-"core-image-sato.do_configure" -> "core-image-sato.do_patch"
-"core-image-sato.do_configure" -> "core-image-sato.do_prepare_recipe_sysroot"
-"core-image-sato.do_deploy_source_date_epoch" [label="core-image-sato do_deploy_source_date_epoch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_deploy_source_date_epoch" -> "core-image-sato.do_patch"
"core-image-sato.do_fetch" [label="core-image-sato do_fetch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] "core-image-sato.do_flush_pseudodb" [label="core-image-sato do_flush_pseudodb\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] "core-image-sato.do_flush_pseudodb" -> "core-image-sato.do_rootfs"
@@ -11489,18 +11480,6 @@
"core-image-sato.do_image_qa" -> "core-image-sato.do_rootfs"
"core-image-sato.do_image_tar" [label="core-image-sato do_image_tar\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] "core-image-sato.do_image_tar" -> "core-image-sato.do_image"
-"core-image-sato.do_install" [label="core-image-sato do_install\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_install" -> "core-image-sato.do_compile"
-"core-image-sato.do_install" -> "pseudo-native.do_populate_sysroot"
-"core-image-sato.do_package" [label="core-image-sato do_package\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_package" -> "core-image-sato.do_install"
-"core-image-sato.do_package" -> "pseudo-native.do_populate_sysroot"
-"core-image-sato.do_packagedata" [label="core-image-sato do_packagedata\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_packagedata" -> "core-image-sato.do_package"
-"core-image-sato.do_patch" [label="core-image-sato do_patch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_patch" -> "core-image-sato.do_unpack"
-"core-image-sato.do_patch" -> "patch-native.do_populate_sysroot"
-"core-image-sato.do_patch" -> "quilt-native.do_populate_sysroot"
"core-image-sato.do_populate_lic_deploy" [label="core-image-sato do_populate_lic_deploy\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] "core-image-sato.do_populate_lic_deploy" -> "acl-native.do_populate_lic"
"core-image-sato.do_populate_lic_deploy" -> "acl.do_populate_lic"
@@ -12036,7 +12015,6 @@
"core-image-sato.do_rootfs" -> "consolekit.do_package_write_ipk"
"core-image-sato.do_rootfs" -> "consolekit.do_packagedata"
"core-image-sato.do_rootfs" -> "consolekit.do_populate_lic"
-"core-image-sato.do_rootfs" -> "core-image-sato.do_packagedata"
"core-image-sato.do_rootfs" -> "core-image-sato.do_prepare_recipe_sysroot"
"core-image-sato.do_rootfs" -> "coreutils.do_package_qa"
"core-image-sato.do_rootfs" -> "coreutils.do_package_write_ipk"
@@ -13257,8 +13235,6 @@
"core-image-sato.do_rootfs" -> "zlib.do_package_write_ipk"
"core-image-sato.do_rootfs" -> "zlib.do_packagedata"
"core-image-sato.do_rootfs" -> "zlib.do_populate_lic"
-"core-image-sato.do_unpack" [label="core-image-sato do_unpack\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] -"core-image-sato.do_unpack" -> "core-image-sato.do_fetch"
"core-image-sato.do_write_qemuboot_conf" [label="core-image-sato do_write_qemuboot_conf\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"] "core-image-sato.do_write_qemuboot_conf" -> "core-image-sato.do_rootfs"
"coreutils.do_build" [label="coreutils do_build\n:8.32-r0\n/media/build1/poky/meta/recipes-core/coreutils/coreutils_8.32.bb"]

i.e. it "disconnects" do_rootfs from the existing unpack/patch/configure/compile /install/package tasks. I know last time we tried this I got several complaints from users who use these tasks for things.

I'm not sure how we fix this but it isn't straightforward, particularly for something which people are going to want backports into the LTS for.

Cheers,

Richard

#4 lore raw
Quoted 31 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 4 lines

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

The trouble for me is that seeing this "bitbake -g" again, doesn't look wrong to me and:

Quoted 2 lines

I know last time we tried this I got several complaints from users who use these tasks for things.

is the first time I've heard about people using these tasks in image recipes. I haven't found any reference to this in the patches which previously touched this area. Complaining that people don't remember whole history of OE like you isn't fair :).

Cheers,

#5 lore raw
Quoted 32 lines · Martin Jansa
Quoted 31 lines · Richard Purdie
Quoted 29 lines · Martin Jansa
Quoted 13 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

Quoted 8 lines

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 7 lines
Quoted 4 lines

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

The trouble for me is that seeing this "bitbake -g" again, doesn't look wrong to me and:

It is a change of behaviour and it is not mentioned in the commit message so it would silently break anyone using SRC_URI in image recipes (amongst other possible failure modes). Whether they should be doing that is a different question.

Quoted 9 lines
Quoted 4 lines

I know last time we tried this I got several complaints from users who use these tasks for things.

is the first time I've heard about people using these tasks in image recipes. I haven't found any reference to this in the patches which previously touched this area. Complaining that people don't remember whole history of OE like you isn't fair :).

I just remember changing this area of code and getting complaints. The information probably isn't in the commits but in the mailing list discussion surrounding them. Sadly my memory is far from perfect on any of this, seeing that diff did spark something though :/.

Cheers,

Richard

#7 lore raw
In reply to a message not in the archive
Quoted 38 lines · Richard Purdie via lists.openembedded.org
Quoted 37 lines · Martin Jansa
Quoted 35 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

Quoted 7 lines
Quoted 2 lines

The trouble for me is that seeing this "bitbake -g" again, doesn't look wrong to me and:

It is a change of behaviour and it is not mentioned in the commit message so it would silently break anyone using SRC_URI in image recipes (amongst other possible failure modes). Whether they should be doing that is a different question.

I would add that anyone actually depending on that behaviour actually already has a race issue since do_install and friends only has to happen before do_build so can happen long after do_image_complete...

Cheers,

Richard

Quoted 1 line
#8 lore raw
Quoted 42 lines · Richard Purdie
Quoted 41 lines · Richard Purdie via lists.openembedded.org
Quoted 39 lines · Martin Jansa
Quoted 31 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 4 lines

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

The trouble for me is that seeing this "bitbake -g" again, doesn't look wrong to me and:

Quoted 8 lines
Quoted 4 lines

It is a change of behaviour and it is not mentioned in the commit message so it would silently break anyone using SRC_URI in image recipes (amongst other possible failure modes). Whether they should be doing that is a different question.

I would add that anyone actually depending on that behaviour actually already has a race issue since do_install and friends only has to happen before do_build so can happen long after do_image_complete...

And according to your answer here:
https://stackoverflow.com/questions/51952495/yocto-image-recipe-and-src-uri
they also need to do few more steps to fetch the files.

#9 lore raw
In reply to Richard Purdie

Hi RP,

Quoted 31 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 4 lines

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

If we want to avoid disconnecting them, it seems that a better way is fixing this in the recipes which has this problem? For example, I got this error in secure-core-image-initramfs.bb, then we can add "deltask do_packagedata" in the
recipe to fix the problem.

@Martin maybe we shouldn't backport this patch to gatesgarth or dunfell since it may suprise the users.

// Robert

Quoted 75 lines

The differences were:

(quoted diff) +0 -24
@@ -11036,7 +11036,6 @@
  "core-image-sato.do_build" -> "connman.do_build"
  "core-image-sato.do_build" -> "consolekit.do_build"
  "core-image-sato.do_build" -> "core-image-sato.do_image_complete"
-"core-image-sato.do_build" -> "core-image-sato.do_packagedata"
  "core-image-sato.do_build" -> "core-image-sato.do_populate_lic_deploy"
  "core-image-sato.do_build" -> "coreutils.do_build"
  "core-image-sato.do_build" -> "cross-localedef-native.do_build"
@@ -11459,14 +11458,6 @@
  "core-image-sato.do_build" -> "zip.do_build"
  "core-image-sato.do_build" -> "zlib-native.do_build"
  "core-image-sato.do_build" -> "zlib.do_build"
-"core-image-sato.do_compile" [label="core-image-sato do_compile\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_compile" -> "core-image-sato.do_configure"
-"core-image-sato.do_configure" [label="core-image-sato do_configure\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_configure" -> "core-image-sato.do_deploy_source_date_epoch"
-"core-image-sato.do_configure" -> "core-image-sato.do_patch"
-"core-image-sato.do_configure" -> "core-image-sato.do_prepare_recipe_sysroot"
-"core-image-sato.do_deploy_source_date_epoch" [label="core-image-sato do_deploy_source_date_epoch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_deploy_source_date_epoch" -> "core-image-sato.do_patch"
  "core-image-sato.do_fetch" [label="core-image-sato do_fetch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
  "core-image-sato.do_flush_pseudodb" [label="core-image-sato do_flush_pseudodb\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
  "core-image-sato.do_flush_pseudodb" -> "core-image-sato.do_rootfs"
@@ -11489,18 +11480,6 @@
  "core-image-sato.do_image_qa" -> "core-image-sato.do_rootfs"
  "core-image-sato.do_image_tar" [label="core-image-sato do_image_tar\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
  "core-image-sato.do_image_tar" -> "core-image-sato.do_image"
-"core-image-sato.do_install" [label="core-image-sato do_install\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_install" -> "core-image-sato.do_compile"
-"core-image-sato.do_install" -> "pseudo-native.do_populate_sysroot"
-"core-image-sato.do_package" [label="core-image-sato do_package\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_package" -> "core-image-sato.do_install"
-"core-image-sato.do_package" -> "pseudo-native.do_populate_sysroot"
-"core-image-sato.do_packagedata" [label="core-image-sato do_packagedata\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_packagedata" -> "core-image-sato.do_package"
-"core-image-sato.do_patch" [label="core-image-sato do_patch\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_patch" -> "core-image-sato.do_unpack"
-"core-image-sato.do_patch" -> "patch-native.do_populate_sysroot"
-"core-image-sato.do_patch" -> "quilt-native.do_populate_sysroot"
  "core-image-sato.do_populate_lic_deploy" [label="core-image-sato do_populate_lic_deploy\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
  "core-image-sato.do_populate_lic_deploy" -> "acl-native.do_populate_lic"
  "core-image-sato.do_populate_lic_deploy" -> "acl.do_populate_lic"
@@ -12036,7 +12015,6 @@
  "core-image-sato.do_rootfs" -> "consolekit.do_package_write_ipk"
  "core-image-sato.do_rootfs" -> "consolekit.do_packagedata"
  "core-image-sato.do_rootfs" -> "consolekit.do_populate_lic"
-"core-image-sato.do_rootfs" -> "core-image-sato.do_packagedata"
  "core-image-sato.do_rootfs" -> "core-image-sato.do_prepare_recipe_sysroot"
  "core-image-sato.do_rootfs" -> "coreutils.do_package_qa"
  "core-image-sato.do_rootfs" -> "coreutils.do_package_write_ipk"
@@ -13257,8 +13235,6 @@
  "core-image-sato.do_rootfs" -> "zlib.do_package_write_ipk"
  "core-image-sato.do_rootfs" -> "zlib.do_packagedata"
  "core-image-sato.do_rootfs" -> "zlib.do_populate_lic"
-"core-image-sato.do_unpack" [label="core-image-sato do_unpack\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
-"core-image-sato.do_unpack" -> "core-image-sato.do_fetch"
  "core-image-sato.do_write_qemuboot_conf" [label="core-image-sato do_write_qemuboot_conf\n:1.0-r0\n/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb"]
  "core-image-sato.do_write_qemuboot_conf" -> "core-image-sato.do_rootfs"
  "coreutils.do_build" [label="coreutils do_build\n:8.32-r0\n/media/build1/poky/meta/recipes-core/coreutils/coreutils_8.32.bb"]

i.e. it "disconnects" do_rootfs from the existing unpack/patch/configure/compile /install/package tasks. I know last time we tried this I got several complaints from users who use these tasks for things.

I'm not sure how we fix this but it isn't straightforward, particularly for something which people are going to want backports into the LTS for.

Cheers,

Richard

#10 lore raw
Quoted 40 lines · Robert Yang

Hi RP,

Quoted 31 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

Quoted 4 lines

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

Quoted 4 lines

If we want to avoid disconnecting them, it seems that a better way is fixing this in the recipes which has this problem? For example, I got this error in secure-core-image-initramfs.bb, then we can add "deltask do_packagedata" in the recipe to fix the problem.

That's what I did in initramfs-android-image (which was triggering this for me as reported on ML)
https://github.com/shr-distribution/meta-smartphone/commit/732dd29325d882a9b3414764ad4c7a254dae3f03

Quoted 2 lines

@Martin maybe we shouldn't backport this patch to gatesgarth or dunfell since it may suprise the users.

Unfortunately
https://git.openembedded.org/openembedded-core/commit/?h=dunfell&id=e0c1db170fdd6c1d78fdfce017feae26c96fac29
was already backported to both gatesgarth and dunfell.

#11 lore raw
Quoted 40 lines · Martin Jansa
Quoted 39 lines · Robert Yang

Hi RP,

Quoted 35 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

Quoted 8 lines
Quoted 4 lines

If we want to avoid disconnecting them, it seems that a better way is fixing this in the recipes which has this problem? For example, I got this error in secure-core-image-initramfs.bb, then we can add "deltask do_packagedata" in the recipe to fix the problem.

That's what I did in initramfs-android-image (which was triggering this for me as reported on ML)
https://github.com/shr-distribution/meta-smartphone/commit/732dd29325d882a9b3414764ad4c7a254dae3f03

The issue of doing this in each and every possible initramfs image recipe is that it does not scale. For example, this code in the original patch is very common:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

And INITRAMFS_IMAGE can be set anywhere - in machine config, distro config or even by end user in local.conf

Quoted 6 lines
Quoted 2 lines

@Martin maybe we shouldn't backport this patch to gatesgarth or dunfell since it may suprise the users.

Unfortunately
https://git.openembedded.org/openembedded-core/commit/?h=dunfell&id=e0c1db170fdd6c1d78fdfce017feae26c96fac29
was already backported to both gatesgarth and dunfell.

Yeah, it's unfortunate. Our LTS maintainer is too good! :) Seriously though, not sure why changing warning to error was even considered for backporting to LTS! Making things stricter doesn't usually fall under "bugfix" category...

Signature · 1 line
Denys
#12 lore raw
Quoted 63 lines · Denys Dmytriyenko
Quoted 40 lines · Martin Jansa
Quoted 39 lines · Robert Yang

Hi RP,

Quoted 35 lines · Richard Purdie
Quoted 21 lines · Martin Jansa
Quoted 20 lines · Robert Yang

This can fix a do_package error when compile with meta-secure-core layer:
http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest
build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata
not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch x86_64_x86_64-nativesdk (variant '')?
[snip]

This is because kernel-initramfs wants to pack an initramfs image into kernel-initramfs.rpm which adds a dependency in kernel-initramfs.bb to do this:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we will get the error. Delete do_packagedata as other do_package relelated tasks for the image recipe will fix the error.

Quoted 8 lines
Quoted 1 line

I went a bit further and inheritted nopackages, but RP mentioned that it might be removing too much. Nothing failed with nopackages over night, but still this smaller version will be safer as this needs to got to gatesgarth and dunfell as well now.

This has side effects and there was a reason this wasn't done originally. The trouble is that nobody is testing for that, just wanting to fix the problem they see. To prove this isn't straightforward, I tested this locally, comparing task-depends.dot from "bitbake core-image-sato -g" before and after this patch.

Quoted 8 lines
Quoted 4 lines

If we want to avoid disconnecting them, it seems that a better way is fixing this in the recipes which has this problem? For example, I got this error in secure-core-image-initramfs.bb, then we can add "deltask do_packagedata" in the recipe to fix the problem.

That's what I did in initramfs-android-image (which was triggering this for me as reported on ML)
https://github.com/shr-distribution/meta-smartphone/commit/732dd29325d882a9b3414764ad4c7a254dae3f03

The issue of doing this in each and every possible initramfs image recipe is that it does not scale. For example, this code in the original patch is very common:

d.appendVarFlag('do_install', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')

And INITRAMFS_IMAGE can be set anywhere - in machine config, distro config or even by end user in local.conf

Quoted 3 lines
Quoted 2 lines

@Martin maybe we shouldn't backport this patch to gatesgarth or dunfell since it may suprise the users.

Quoted 8 lines
Quoted 3 lines

Unfortunately
https://git.openembedded.org/openembedded-core/commit/?h=dunfell&id=e0c1db170fdd6c1d78fdfce017feae26c96fac29
was already backported to both gatesgarth and dunfell.

Yeah, it's unfortunate. Our LTS maintainer is too good! :) Seriously though, not sure why changing warning to error was even considered for backporting to LTS! Making things stricter doesn't usually fall under "bugfix" category...

FWIW I'm leaning to merging a deltask to image.bbclass in master and suggesting we revert this and live with the warning in older branches...

The deltask will probably break things but I'm starting to think that is the lesser of several issues. Its not appropriate for the stable branches though.

Cheers,

Richard

#13 lore raw
Quoted 96 lines · Richard Purdie
Quoted 8 lines · Denys Dmytriyenko
Quoted 7 lines · Martin Jansa
Quoted 6 lines · Robert Yang

Hi RP,

Quoted 3 lines · Richard Purdie
Quoted 2 lines · Martin Jansa
Quoted 1 line · Robert Yang

This can fix a do_package error when compile with

meta-secure-core layer:

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

http://layers.openembedded.org/layerindex/branch/master/layer/meta-secure-core/

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
$ bitbake kernel-initramfs
[snip]
WARNING:kernel-initramfs-1.0-r0 do_package: Manifest

build/tmp-glibc/sstate-control/manifest-x86_64_x86_64-nativesdk-secure-core-image-initramfs.packagedata

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

not found in intel_x86_64 corei7-64 core2-64 x86_64 allarch

x86_64_x86_64-nativesdk (variant '')?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

[snip]

This is because kernel-initramfs wants to pack an initramfs

image into

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel-initramfs.rpm which adds a dependency in

kernel-initramfs.bb to do this:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

d.appendVarFlag('do_install', 'depends', '

${INITRAMFS_IMAGE}:do_image_complete')

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This causes kernel-initramfs' do_package depends on ${INITRAMFS_IMAGE}:do_image_complete's do_packagedata, then we

will get the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

error. Delete do_packagedata as other do_package relelated

tasks for the image

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

recipe will fix the error.

I went a bit further and inheritted nopackages, but RP mentioned

that it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

might be removing too much. Nothing failed with nopackages over

night,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

but still this smaller version will be safer as this needs to

got to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

gatesgarth and dunfell as well now.

This has side effects and there was a reason this wasn't done

originally. The

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

trouble is that nobody is testing for that, just wanting to fix

the problem

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

they see. To prove this isn't straightforward, I tested this

locally, comparing

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

task-depends.dot from "bitbake core-image-sato -g" before and

after this patch.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

If we want to avoid disconnecting them, it seems that a better way

is fixing

Quoted 1 line
Quoted 1 line
Quoted 1 line

this in the recipes which has this problem? For example, I got this

error in

Quoted 1 line
Quoted 1 line
Quoted 1 line

secure-core-image-initramfs.bb, then we can add "deltask

do_packagedata" in the

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

recipe to fix the problem.

That's what I did in initramfs-android-image (which was triggering this for me as reported on ML)

https://github.com/shr-distribution/meta-smartphone/commit/732dd29325d882a9b3414764ad4c7a254dae3f03

Quoted 2 lines

The issue of doing this in each and every possible initramfs image

recipe is

Quoted 1 line

that it does not scale. For example, this code in the original patch is

very

Quoted 3 lines

common:

d.appendVarFlag('do_install', 'depends', '

${INITRAMFS_IMAGE}:do_image_complete')

Quoted 2 lines

And INITRAMFS_IMAGE can be set anywhere - in machine config, distro

config or

Quoted 4 lines

even by end user in local.conf

Quoted 1 line
Quoted 1 line

@Martin maybe we shouldn't backport this patch to gatesgarth or

dunfell since

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

it may suprise the users.

Unfortunately

https://git.openembedded.org/openembedded-core/commit/?h=dunfell&id=e0c1db170fdd6c1d78fdfce017feae26c96fac29

Quoted 4 lines
Quoted 1 line

was already backported to both gatesgarth and dunfell.

Yeah, it's unfortunate. Our LTS maintainer is too good! :) Seriously though, not sure why changing warning to error was even

Quoted 8 lines

considered

Quoted 1 line

for backporting to LTS! Making things stricter doesn't usually fall

under

Quoted 1 line

"bugfix" category...

FWIW I'm leaning to merging a deltask to image.bbclass in master and suggesting we revert this and live with the warning in older branches...

Agreed, I'll add a revert to my next set of patches.

Steve

Quoted 11 lines

The deltask will probably break things but I'm starting to think that is the lesser of several issues. Its not appropriate for the stable branches though.

Cheers,

Richard