Skip to contentSkip to conversation

rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work

PATCH In review Chen Qi 3 participants 6 messages Dec 5, 2022 – Jan 18, 2023 1 message not in the archive

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

PATCH 1/1 rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work

For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.

Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.

Below are the detailed explanation of why this would work.

Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]

                              |

[do_rm_work_all] -------------+

With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]

Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.

With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.

---
 meta/classes/rm_work.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +1 -1
index 4121a13279..1f28bc7187 100644
@@ -180,7 +180,7 @@ python inject_rm_work() {
         # other recipes and thus will typically run much later than completion of
         # work in the recipe itself.
         # In practice, addtask() here merely updates the dependencies.
-        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
+        bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d)
 
     # Always update do_build_without_rm_work dependencies.
     bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
Signature · 1 line
2.37.1
#1 lore raw

Hi Chen,

Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.

I am investigating it more deeply and will come back with my findings.

Jose

Chen Qi <[email protected]> escreveu no dia segunda, 5/12/2022 à(s) 04:59:

Quoted 64 lines

For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.

Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.

Below are the detailed explanation of why this would work.

Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]

                              |

[do_rm_work_all] -------------+

With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]

Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.

With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.

---
 meta/classes/rm_work.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +0 -0
index 4121a13279..1f28bc7187 100644
@@ -180,7 +180,7 @@ python inject_rm_work() {
         # other recipes and thus will typically run much later than

completion of

         # work in the recipe itself.
         # In practice, addtask() here merely updates the dependencies.
meta/classes/rm_work.bbclass +1 -1
-        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
+        bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', '

'.join(deps), d)

# Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
--
2.37.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#174276):
https://lists.openembedded.org/g/openembedded-core/message/174276
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Hi Chen,

Jose Quaresma via lists.openembedded.org <quaresma.jose= [email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:

Quoted 8 lines

Hi Chen,

Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.

I am investigating it more deeply and will come back with my findings.

I don't understand the reason for this regression but I am thinking of sending a revert for this patch.

Here are the steps to reproduce:

git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf

MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal

At least these two machines are broken.

Jose

Quoted 80 lines

Jose

Chen Qi <[email protected]> escreveu no dia segunda, 5/12/2022 à(s) 04:59:

Quoted 47 lines

For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.

Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.

Below are the detailed explanation of why this would work.

Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]

                              |

[do_rm_work_all] -------------+

With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]

Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.

With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.

---
 meta/classes/rm_work.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +0 -0
index 4121a13279..1f28bc7187 100644
@@ -180,7 +180,7 @@ python inject_rm_work() {
         # other recipes and thus will typically run much later than

completion of

         # work in the recipe itself.
         # In practice, addtask() here merely updates the dependencies.
meta/classes/rm_work.bbclass +1 -1
-        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
+        bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', '

'.join(deps), d)

Quoted 9 lines

# Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
--
2.37.1

--
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#175835):
https://lists.openembedded.org/g/openembedded-core/message/175835
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma
author
#4 lore raw

You may need to check with the layer maintainers to find out why these two layers are not working. It does not feel right if you revert a patch but cannot prove it’s wrong.

Regards,
Qi

From: Jose Quaresma <[email protected]>
Sent: Wednesday, January 18, 2023 9:10 PM
To: [email protected]
Cc: Chen, Qi <[email protected]>; [email protected]
Subject: Re: [OE-core][PATCH] rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work

Hi Chen,

Jose Quaresma via lists.openembedded.org<https://urldefense.com/v3/__http:/lists.openembedded.org__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7ewoiqb7_$> <[email protected]<mailto:[email protected]>> escreveu no dia sexta, 13/01/2023 à(s) 15:39:
Hi Chen,

Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.

I am investigating it more deeply and will come back with my findings.

I don't understand the reason for this regression but I am thinking of sending a revert for this patch.

Here are the steps to reproduce:

git clone https://git.yoctoproject.org/git/poky<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/poky__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-cya_bF$>
cd poky
git clone https://git.yoctoproject.org/git/meta-arm<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-arm__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-3E8fSx$>
git clone https://git.yoctoproject.org/git/meta-ti<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-ti__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e3bU6PC9$>
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain
bitbake-layers add-layer ../meta-arm/meta-arm
bitbake-layers add-layer ../meta-ti/meta-ti-bsp
bitbake-layers add-layer ../meta-ti/meta-ti-extras
echo 'INHERIT += "rm_work"' >>conf/local.conf
echo 'INHERIT += "create-spdx"' >>conf/local.conf

MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal

At least these two machines are broken.

Jose

Jose

Chen Qi <[email protected]<mailto:[email protected]>> escreveu no dia segunda, 5/12/2022 à(s) 04:59:
For now, if we use rm_work and `bitbake core-image-minimal', some
recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.

Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.

Below are the detailed explanation of why this would work.

Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]

                              |

[do_rm_work_all] -------------+

With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]

Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.

With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.

---
 meta/classes/rm_work.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +1 -1
index 4121a13279..1f28bc7187 100644
@@ -180,7 +180,7 @@ python inject_rm_work() {
         # other recipes and thus will typically run much later than completion of
         # work in the recipe itself.
         # In practice, addtask() here merely updates the dependencies.
-        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
+        bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d)
 
     # Always update do_build_without_rm_work dependencies.
     bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)

--
2.37.1

--
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#175835): https://lists.openembedded.org/g/openembedded-core/message/175835<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/message/175835__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-J3aJAB$>
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612<https://urldefense.com/v3/__https:/lists.openembedded.org/mt/95462765/5052612__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4RCh9cz$>
Group Owner: [email protected]<mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4wWPBlU$> [[email protected]<mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-

--
Best regards,

José Quaresma

#5 lore raw
In reply to Jose Quaresma
Quoted 33 lines · Jose Quaresma

Hi Chen,

Jose Quaresma via lists.openembedded.org <[email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:

Quoted 9 lines

Hi Chen,

Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.

I am investigating it more deeply and will come back with my findings.

I don't understand the reason for this regression but I am thinking of sending a revert for this patch.

Here are the steps to reproduce:

git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf

Quoted 5 lines

MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal

At least these two machines are broken.

Do you have a pointer to the actual errors that result?

I'm not sure I would take a revert as I did carefully check this patch and I think it does fix a number of problems.

Cheers,

Richard

#6 lore raw

Richard Purdie <[email protected]> escreveu no dia quarta, 18/01/2023 à(s) 13:39:

Quoted 34 lines · Jose Quaresma
Quoted 33 lines

Hi Chen,

Jose Quaresma via lists.openembedded.org <[email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:

Quoted 9 lines

Hi Chen,

Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.

I am investigating it more deeply and will come back with my findings.

I don't understand the reason for this regression but I am thinking of sending a revert for this patch.

Here are the steps to reproduce:

git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf

Quoted 8 lines
Quoted 5 lines

MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal

At least these two machines are broken.

Do you have a pointer to the actual errors that result?

The issue can be reproduced quite fast in less than 1 minute, the error is the following:

ERROR: mc:k3r5-gp:quilt-native-0.67-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_create_spdx(d)
     0003:

File: '/home/jq/foundries/git/poky/meta/classes/create-spdx-2.2.bbclass',
lineno: 525, function: do_create_spdx
0521: recipe.packageFileName = str(recipe_archive.name)
0522:
0523: dep_recipes = collect_dep_recipes(d, doc, recipe)
0524:
*** 0525: doc_sha1 = oe.sbom.write_doc(d, doc, "recipes",
indent=get_json_indent(d))
0526: dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
0527:
0528: recipe_ref = oe.spdx.SPDXExternalDocumentRef()
0529: recipe_ref.externalDocumentId = "DocumentRef-recipe-" +
recipe.name
File: '/home/jq/foundries/git/poky/meta/lib/oe/sbom.py', lineno: 50,
function: write_doc

     0046:        doc_sha1 = spdx_doc.to_json(f, sort_keys=True,
indent=indent)
     0047:
     0048:    l = spdx_deploy / "by-namespace" /
spdx_doc.documentNamespace.replace("/", "_")
     0049:    l.parent.mkdir(exist_ok=True, parents=True)
 *** 0050:    l.symlink_to(os.path.relpath(dest, l.parent))
     0051:
     0052:    return doc_sha1
     0053:
     0054:
File: '/usr/lib/python3.10/pathlib.py', lineno: 1255, function: symlink_to
     1251:        """
     1252:        Make this path a symlink pointing to the target path.
     1253:        Note the order of arguments (link, target) is the reverse
of os.symlink.
     1254:        """
 *** 1255:        self._accessor.symlink(target, self, target_is_directory)
     1256:
     1257:    def hardlink_to(self, target):
     1258:        """
     1259:        Make this path a hard link pointing to the same file as

*target*.
Exception: FileExistsError: [Errno 17] File exists:
'../recipes/recipe-quilt-native.spdx.json' -> '/home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/spdx/deploy/by-namespace/http:__spdx.org_spdxdoc_recipe-quilt-native-16737c05-483f-569e-ab20-5a7a779c8317'

ERROR: Logfile of failure stored in:
/home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/temp/log.do_create_spdx.1386095
ERROR: Task (mc:k3r5-gp:/home/jq/foundries/git/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_create_spdx) failed with exit code '1'

Quoted 4 lines

I'm not sure I would take a revert as I did carefully check this patch and I think it does fix a number of problems.

I understand and use this patch for a bunch of other machines and it works in all of them.
When writing this last sentence I remembered that I use the patch as well in some other multiconfig machines
but these ones are from Xilinx that has one TMP folder for each machine.

Is not the first time I have an issue with multiconfig machines that uses the same TMP
folder and looks like this can be the issue with the TI machines.

Jose

Quoted 6 lines

Cheers,

Richard

Signature · 3 lines
Best regards,

José Quaresma