rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work
PATCH 1/1 rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work
For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.
Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.
Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]
|
[do_rm_work_all] -------------+
With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]
Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.
With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.
--- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +1 -1
@@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later than completion of # work in the recipe itself. # In practice, addtask() here merely updates the dependencies. - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d) # Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
Signature · 1 line
2.37.1
Hi Chen,
Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.
I am investigating it more deeply and will come back with my findings.
Jose
Chen Qi <[email protected]> escreveu no dia segunda, 5/12/2022 à(s) 04:59:
Quoted 64 lines
For now, if we use
rm_workand `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.Adjust the dependency to make
do_rm_work_alldepend ondo_rm_workto solve this problem.Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]|[
do_rm_work_all] -------------+With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]Such dependency chain adjustment fixes the issue because
do_rm_work_allnow depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment,do_rm_work_alldoes not have any relationship withdo_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting ofdo_rm_work_allextend to makedevs-native. With this patch, thedo_rm_work_allnow depends ondo_rm_workwhich in turn depends ondo_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.
Signed-off-by Chen Qi <[email protected]>--- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/rm_work.bbclass +0
-0@@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later thancompletion of
# work in the recipe itself. # In practice, addtask() here merely updates the dependencies.meta/classes/rm_work.bbclass +1
-1- bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ''.join(deps), d)
# Always update
do_build_without_rm_workdependencies.bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
--
2.37.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#174276):
https://lists.openembedded.org/g/openembedded-core/message/174276
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Best regards, José Quaresma
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Chen,
Jose Quaresma via lists.openembedded.org <quaresma.jose= [email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:
Quoted 8 lines
Hi Chen,
Just for your information,
This patch introduces a regression that breaks thedo_create_spdxtasks when building with multiconfig machines.I am investigating it more deeply and will come back with my findings.
I don't understand the reason for this regression but I am thinking of sending a revert for this patch.
Here are the steps to reproduce:
git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf
MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal
At least these two machines are broken.
Jose
Quoted 80 lines
Jose
Chen Qi <[email protected]> escreveu no dia segunda, 5/12/2022 à(s) 04:59:
Quoted 47 lines
For now, if we use
rm_workand `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.Adjust the dependency to make
do_rm_work_alldepend ondo_rm_workto solve this problem.Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]|[
do_rm_work_all] -------------+With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]Such dependency chain adjustment fixes the issue because
do_rm_work_allnow depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment,do_rm_work_alldoes not have any relationship withdo_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting ofdo_rm_work_allextend to makedevs-native. With this patch, thedo_rm_work_allnow depends ondo_rm_workwhich in turn depends ondo_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.
Signed-off-by Chen Qi <[email protected]>--- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/rm_work.bbclass +0
-0@@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later thancompletion of
# work in the recipe itself. # In practice, addtask() here merely updates the dependencies.meta/classes/rm_work.bbclass +1
-1- bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ''.join(deps), d)
Quoted 9 lines
# Always update
do_build_without_rm_workdependencies.bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
--
2.37.1--
Best regards,José Quaresma
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#175835):
https://lists.openembedded.org/g/openembedded-core/message/175835
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Best regards, José Quaresma
You may need to check with the layer maintainers to find out why these two layers are not working. It does not feel right if you revert a patch but cannot prove it’s wrong.
Regards,
Qi
From: Jose Quaresma <[email protected]>
Sent: Wednesday, January 18, 2023 9:10 PM
To: [email protected]
Cc: Chen, Qi <[email protected]>; [email protected]
Subject: Re: [OE-core][PATCH] rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work
Hi Chen,
Jose Quaresma via lists.openembedded.org<https://urldefense.com/v3/__http:/lists.openembedded.org__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7ewoiqb7_$> <[email protected]<mailto:[email protected]>> escreveu no dia sexta, 13/01/2023 à(s) 15:39:
Hi Chen,
Just for your information,
This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines.
I am investigating it more deeply and will come back with my findings.
I don't understand the reason for this regression but I am thinking of sending a revert for this patch.
Here are the steps to reproduce:
git clone https://git.yoctoproject.org/git/poky<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/poky__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-cya_bF$>
cd poky
git clone https://git.yoctoproject.org/git/meta-arm<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-arm__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-3E8fSx$>
git clone https://git.yoctoproject.org/git/meta-ti<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-ti__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e3bU6PC9$>
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain
bitbake-layers add-layer ../meta-arm/meta-arm
bitbake-layers add-layer ../meta-ti/meta-ti-bsp
bitbake-layers add-layer ../meta-ti/meta-ti-extras
echo 'INHERIT += "rm_work"' >>conf/local.conf
echo 'INHERIT += "create-spdx"' >>conf/local.conf
MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimal
At least these two machines are broken.
Jose
Jose
Chen Qi <[email protected]<mailto:[email protected]>> escreveu no dia segunda, 5/12/2022 à(s) 04:59:
For now, if we use rm_work and `bitbake core-image-minimal', some
recipes' WORKDIRs are not cleaned up, e.g., makedevs-native.
Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem.
Below are the detailed explanation of why this would work.
Without this patch, the dependency chain is like:
[other deps] -> [do_rm_work] -+-> [do_build]
|
[do_rm_work_all] -------------+
With this patch, the depedency chain is like:
[other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build]
Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native.
With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native.
--- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/rm_work.bbclass +1 -1
@@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later than completion of # work in the recipe itself. # In practice, addtask() here merely updates the dependencies. - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d) # Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
--
2.37.1
--
Best regards,
José Quaresma
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#175835): https://lists.openembedded.org/g/openembedded-core/message/175835<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/message/175835__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-J3aJAB$>
Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612<https://urldefense.com/v3/__https:/lists.openembedded.org/mt/95462765/5052612__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4RCh9cz$>
Group Owner: [email protected]<mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4wWPBlU$> [[email protected]<mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-
--
Best regards,
José Quaresma
Quoted 33 lines · Jose Quaresma
Hi Chen,
Jose Quaresma via lists.openembedded.org <[email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:
Quoted 9 lines
Hi Chen,
Just for your information,
This patch introduces a regression that breaks thedo_create_spdxtasks when building with multiconfig machines.I am investigating it more deeply and will come back with my findings.
I don't understand the reason for this regression but I am thinking of sending a revert for this patch.
Here are the steps to reproduce:
git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf
Quoted 5 lines
MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimalAt least these two machines are broken.
Do you have a pointer to the actual errors that result?
I'm not sure I would take a revert as I did carefully check this patch and I think it does fix a number of problems.
Cheers,
Richard
Richard Purdie <[email protected]> escreveu no dia quarta, 18/01/2023 à(s) 13:39:
Quoted 34 lines · Jose Quaresma
Quoted 33 lines
Hi Chen,
Jose Quaresma via lists.openembedded.org <[email protected]> escreveu no dia sexta, 13/01/2023 à(s) 15:39:
Quoted 9 lines
Hi Chen,
Just for your information,
This patch introduces a regression that breaks thedo_create_spdxtasks when building with multiconfig machines.I am investigating it more deeply and will come back with my findings.
I don't understand the reason for this regression but I am thinking of sending a revert for this patch.
Here are the steps to reproduce:
git clone https://git.yoctoproject.org/git/poky
cd poky
git clone https://git.yoctoproject.org/git/meta-arm
git clone https://git.yoctoproject.org/git/meta-ti
source oe-init-build-env
bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf
Quoted 8 lines
Quoted 5 lines
MACHINE=am64xx-evm bitbake core-image-minimal
MACHINE=am62xx-evm bitbake core-image-minimalAt least these two machines are broken.
Do you have a pointer to the actual errors that result?
The issue can be reproduced quite fast in less than 1 minute, the error is the following:
ERROR: mc:k3r5-gp:quilt-native-0.67-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_create_spdx(d)
0003:File: '/home/jq/foundries/git/poky/meta/classes/create-spdx-2.2.bbclass',
lineno: 525, function: do_create_spdx
0521: recipe.packageFileName = str(recipe_archive.name)
0522:
0523: dep_recipes = collect_dep_recipes(d, doc, recipe)
0524:
*** 0525: doc_sha1 = oe.sbom.write_doc(d, doc, "recipes",
indent=get_json_indent(d))
0526: dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
0527:
0528: recipe_ref = oe.spdx.SPDXExternalDocumentRef()
0529: recipe_ref.externalDocumentId = "DocumentRef-recipe-" +
recipe.name
File: '/home/jq/foundries/git/poky/meta/lib/oe/sbom.py', lineno: 50,
function: write_doc
0046: doc_sha1 = spdx_doc.to_json(f, sort_keys=True,
indent=indent)
0047:
0048: l = spdx_deploy / "by-namespace" /
spdx_doc.documentNamespace.replace("/", "_")
0049: l.parent.mkdir(exist_ok=True, parents=True)
*** 0050: l.symlink_to(os.path.relpath(dest, l.parent))
0051:
0052: return doc_sha1
0053:
0054:
File: '/usr/lib/python3.10/pathlib.py', lineno: 1255, function: symlink_to
1251: """
1252: Make this path a symlink pointing to the target path.
1253: Note the order of arguments (link, target) is the reverse
of os.symlink.
1254: """
*** 1255: self._accessor.symlink(target, self, target_is_directory)
1256:
1257: def hardlink_to(self, target):
1258: """
1259: Make this path a hard link pointing to the same file as*target*.
Exception: FileExistsError: [Errno 17] File exists:
'../recipes/recipe-quilt-native.spdx.json' -> '/home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/spdx/deploy/by-namespace/http:__spdx.org_spdxdoc_recipe-quilt-native-16737c05-483f-569e-ab20-5a7a779c8317'
ERROR: Logfile of failure stored in:/home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/temp/log.do_create_spdx.1386095
ERROR: Task (mc:k3r5-gp:/home/jq/foundries/git/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_create_spdx) failed with exit code '1'
Quoted 4 lines
I'm not sure I would take a revert as I did carefully check this patch and I think it does fix a number of problems.
I understand and use this patch for a bunch of other machines and it works in all of them.
When writing this last sentence I remembered that I use the patch as well in some other multiconfig machines
but these ones are from Xilinx that has one TMP folder for each machine.
Is not the first time I have an issue with multiconfig machines that uses the same TMP
folder and looks like this can be the issue with the TI machines.
Jose
Quoted 6 lines
Cheers,
Richard
Signature · 3 lines
Best regards, José Quaresma