kernel.bbclass: make do_symlink_kernelsrc reentrant
PATCH v2 1/1 kernel.bbclass: make do_symlink_kernelsrc reentrant
From: Etienne Cordonnier <[email protected]>
The function do_symlink_kernsrc is not reentrant in the case where S is defined to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.
Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes-recipe/kernel.bbclass +5 -0
@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc) else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces symlink_kernsrc to run again + # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernel sources" % s) + return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }
Signature · 1 line
2.34.1
Quoted 19 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where S is defined to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/kernel.bbclass +0
-0
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 200
Quoted 7 lines
meta/classes-recipe/kernel.bbclass +4
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc) else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces symlink_kernsrc to run again + # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernel sources" % s)
I don't think this is a warning situation, perhaps bb.info() is better.
Since nothing is wrong, no unexpected results will occur and I really don't see any side effects .. is why I don't see it as a warning.
Bruce
Quoted 16 lines
meta/classes-recipe/kernel.bbclass +1
-1+ return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) } --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192564): https://lists.openembedded.org/g/openembedded-core/message/192564
Mute This Topic: https://lists.openembedded.org/mt/103208527/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 19 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where S is defined to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.Bugzilla: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/kernel.bbclass +0
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc)
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 204
Quoted 10 lines
meta/classes-recipe/kernel.bbclass +5
-0else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces symlink_kernsrc to run again + # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernel sources" % s) + return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }
Usually what we recommend is that the first pass does something if "if not exists X.org; cp X X.orig; fi", then you can always copy from X.orig.
As Bruce says, it definitely shouldn't be a warning, this can happen quite normally.
Cheers,
Richard
My thinking was that in my local build I was updating from a build without
https://github.com/yoctoproject/poky/commit/3b8d0acca30c806fb69fc3096d41593cc76ed22c
to a build containing this commit, so the behavior of do_symlink_kernsrc got modified, but my incremental build is not executing the new version of the code since it detects that the symlink already exists and then the rest of the function gets skipped, so I thought this justified a warning.
I'll lower the warning to info, but then this assumes that we will never modify do_symlink_kernsrc in a way which is not backwards-compatible.
@Richard: the reason I didn't go for the copy to "orig" is that I don't see a way to add this without breaking incremental builds (where s is a symlink to kernsrc which is potentially modified already by do_patch etc.). However I think it would be a cleaner solution than simply skipping the code like I did:
- if d.getVar("EXTERNALSRC"):
+ if d.getVar("EXTERNALSRC") and not os.path.islink(s):
# With EXTERNALSRC S will not be wiped so we can symlink to it os.symlink(s, kernsrc)
else:
import shutil
+ s_copy = s + ".orig"
+ if not os.path.isdir(s_copy):
+ shutil.copytree(s, s_copy)
+ bb.utils.remove(s, recurse=True)
+ shutil.copytree(s_copy, s)
Etienne
Quoted 50 lines · Richard Purdie
Quoted 3 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where S isdefined
Quoted 1 line
to a non-default value. This causes build-failures e.g. when building
linux-yocto, then updating
Quoted 1 line
poky to a commit which modifies kernel.bbclass, and then building
linux-yocto again.
Quoted 2 lines
Bugzilla:
Review
Quoted 7 lines
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)(quoted diff) +0
-0b/meta/classes-recipe/kernel.bbclass
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 198
Quoted 9 lines
index
9ff37f5c38..130bedd901100644meta/classes-recipe/kernel.bbclass +2
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc) else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces
symlink_kernsrcto run againmeta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 200
Quoted 2 lines
meta/classes-recipe/kernel.bbclass +2
-0+ # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernelsources" % s)
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 204
Quoted 4 lines
meta/classes-recipe/kernel.bbclass +1
-0+ return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }Usually what we recommend is that the first pass does something if "if not exists X.org; cp X X.orig; fi", then you can always copy from X.orig.
As Bruce says, it definitely shouldn't be a warning, this can happen quite normally.
Cheers,
Richard
Quoted 6 lines · Etienne Cordonnier via lists.openembedded.org
My thinking was that in my local build I was updating from a build without https://github.com/yoctoproject/poky/commit/3b8d0acca30c806fb69fc3096d41593cc76ed22c to a build containing this commit, so the behavior of
do_symlink_kernsrcgot modified, but my incremental build is not executing the new version of the code since it detects that the symlink already exists and then the rest of the function gets skipped, so I thought this justified a warning.I'll lower the warning to info, but then this assumes that we will never modify
do_symlink_kernsrcin a way which is not backwards-compatible.@Richard: the reason I didn't go for the copy to "orig" is that I don't see a way to add this without breaking incremental builds (where s is a symlink to kernsrc which is potentially modified already by
do_patchetc.). However I think it would be a cleaner solution than simply skipping the code like I did:
If the only way to trigger the issue is through really rare pre-<some commit> to post
that commit that happens to modify the detection/manipulation/other of S and the symlink, then it is rare enough to just drop an info message breadcrumb and keep moving.
But if it really is a rare situation like you ran into, or some sort of forced execution
of the task, I don't think that a copy would be an unexpected thing to happen.
Bruce
Quoted 62 lines
(quoted diff) +6
-1- if d.getVar("EXTERNALSRC"): + if d.getVar("EXTERNALSRC") and not os.path.islink(s): # With EXTERNALSRC S will not be wiped so we can symlink to it os.symlink(s, kernsrc) else: import shutil + s_copy = s + ".orig" + if not os.path.isdir(s_copy): + shutil.copytree(s, s_copy) + bb.utils.remove(s, recurse=True) + shutil.copytree(s_copy, s)Etienne
Quoted 37 lines · Richard Purdie
Quoted 19 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where S is defined to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/kernel.bbclass +0
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc)meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 204
Quoted 10 lines
meta/classes-recipe/kernel.bbclass +5
-0else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces symlink_kernsrc to run again + # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernel sources" % s) + return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }Usually what we recommend is that the first pass does something if "if not exists X.org; cp X X.orig; fi", then you can always copy from X.orig.
Quoted 6 lines
As Bruce says, it definitely shouldn't be a warning, this can happen quite normally.
Cheers,
Richard
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192635): https://lists.openembedded.org/g/openembedded-core/message/192635
Mute This Topic: https://lists.openembedded.org/mt/103208527/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
can you please clarify? Are you saying that you prefer the version of the patch which I pasted with shutil.copytree?
The condition to trigger the issue is not that rare, e.g. anybody updating from kirkstone 4.0.12 to 4.0.13 and doing an incremental build will get a build failure if S doesn't have the default value (in my case the BSP layer overrides S). In the same way, if we merge the version with shutil.copytree, people will need to perform a manual clean, or they may get build failures.
Etienne
Quoted 126 lines · Bruce Ashfield
Quoted 2 lines · Etienne Cordonnier via lists.openembedded.org
My thinking was that in my local build I was updating from a build
without
https://github.com/yoctoproject/poky/commit/3b8d0acca30c806fb69fc3096d41593cc76ed22c
to a build containing this commit, so the behavior ofdo_symlink_kernsrcgot modified, but my incremental build is not executing the new version of the code since it detects that the symlink already exists and then the rest of the function gets skipped, so I thought this justified a warning.Quoted 2 lines
I'll lower the warning to info, but then this assumes that we will never
modify
do_symlink_kernsrcin a way which is not backwards-compatible.Quoted 2 lines
@Richard: the reason I didn't go for the copy to "orig" is that I don't
see a way to add this without breaking incremental builds (where s is a symlink to kernsrc which is potentially modified already by
do_patchetc.). However I think it would be a cleaner solution than simply skipping the code like I did:If the only way to trigger the issue is through really rare pre-<some commit> to post
that commit that happens to modify the detection/manipulation/other of S and the
symlink, then it is rare enough to just drop an info message breadcrumb and keep
moving.But if it really is a rare situation like you ran into, or some sort of forced execution
of the task, I don't think that a copy would be an unexpected thing to happen.Bruce
Review
Quoted 4 lines
(quoted diff) +1
-1- if d.getVar("EXTERNALSRC"): + if d.getVar("EXTERNALSRC") and not os.path.islink(s): # With EXTERNALSRC S will not be wiped so we can symlink toit
Quoted 13 lines
os.symlink(s, kernsrc)
else:
import shutil(quoted diff) +5
-0+ s_copy = s + ".orig" + if not os.path.isdir(s_copy): + shutil.copytree(s, s_copy) + bb.utils.remove(s, recurse=True) + shutil.copytree(s_copy, s)Etienne
On Sun, Dec 17, 2023 at 8:52 PM Richard Purdie <
Quoted 6 lines · [email protected]> wrote:
Quoted 6 lines
Quoted 3 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where Sis defined
Quoted 1 line
Quoted 1 line
Quoted 1 line
to a non-default value. This causes build-failures e.g. when building
linux-yocto, then updating
Quoted 1 line
Quoted 1 line
Quoted 1 line
poky to a commit which modifies kernel.bbclass, and then building
linux-yocto again.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Bugzilla:
Quoted 7 lines
Quoted 7 lines
Review
Quoted 7 lines
Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)(quoted diff) +0
-0b/meta/classes-recipe/kernel.bbclass
Quoted 9 lines
Quoted 9 lines
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 198
Quoted 9 lines
index
9ff37f5c38..130bedd901100644meta/classes-recipe/kernel.bbclass +2
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc) else: import shutil + if os.path.islink(s): + # this happens for instance when a poky updateforces
symlink_kernsrcto run againQuoted 2 lines
Quoted 2 lines
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 200
Quoted 2 lines
meta/classes-recipe/kernel.bbclass +2
-0+ # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinkingkernel sources" % s)
Quoted 12 lines
Quoted 12 lines
meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 204
Quoted 4 lines
meta/classes-recipe/kernel.bbclass +1
-0+ return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }Usually what we recommend is that the first pass does something if "if not exists X.org; cp X X.orig; fi", then you can always copy from X.orig.
As Bruce says, it definitely shouldn't be a warning, this can happen quite normally.
Quoted 8 lines
Quoted 3 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#192635):Quoted 1 line
Mute This Topic:
Quoted 2 lines
Group Owner: [email protected] Unsubscribe:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=82QaxN_7mSeykh83DV-qNXJc9JezHaPCHv4jG2rrq55pywmZCA6Nz75KaUoSuFn8&s=roqLGNR0ewaxjmAl_ZE8nSZf_6sCh3yj7LXuITei5EA&e=
[[email protected]]Quoted 2 lines
-=-=-=-=-=-=-=-=-=-=-=-
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Quoted 5 lines · Etienne Cordonnier
Hi Bruce,
can you please clarify? Are you saying that you prefer the version of the patch which I pasted with shutil.copytree?The condition to trigger the issue is not that rare, e.g. anybody updating from kirkstone 4.0.12 to 4.0.13 and doing an incremental build will get a build failure if S doesn't have the default value (in my case the BSP layer overrides S). In the same way, if we merge the version with shutil.copytree, people will need to perform a manual clean, or they may get build failures.
For my rarity comment, I meant that it isn't within a typical flow of bitbaking an image, forcing a task, and bitbaking again. If someone is doing a meta-layer update, and re-using their build (I have quite literally 7 year old directories like this) then they will run into some issues from time to time (and folks doing this, normally know how to get themselves out of trouble).
Taking that into consideration, I just mean that I'd be ok with a copy of the files, as it is really an update to "the latest structure" and if you are doing it on an existing build, that's one of the consequences.
Bruce
Quoted 103 lines
Etienne
Quoted 99 lines · Bruce Ashfield
Quoted 6 lines · Etienne Cordonnier via lists.openembedded.org
My thinking was that in my local build I was updating from a build without https://github.com/yoctoproject/poky/commit/3b8d0acca30c806fb69fc3096d41593cc76ed22c to a build containing this commit, so the behavior of
do_symlink_kernsrcgot modified, but my incremental build is not executing the new version of the code since it detects that the symlink already exists and then the rest of the function gets skipped, so I thought this justified a warning.I'll lower the warning to info, but then this assumes that we will never modify
do_symlink_kernsrcin a way which is not backwards-compatible.@Richard: the reason I didn't go for the copy to "orig" is that I don't see a way to add this without breaking incremental builds (where s is a symlink to kernsrc which is potentially modified already by
do_patchetc.). However I think it would be a cleaner solution than simply skipping the code like I did:If the only way to trigger the issue is through really rare pre-<some commit> to post
that commit that happens to modify the detection/manipulation/other of S and the symlink, then it is rare enough to just drop an info message breadcrumb and keep moving.But if it really is a rare situation like you ran into, or some sort of forced execution
of the task, I don't think that a copy would be an unexpected thing to happen.Bruce
Quoted 62 lines
(quoted diff) +6
-1- if d.getVar("EXTERNALSRC"): + if d.getVar("EXTERNALSRC") and not os.path.islink(s): # With EXTERNALSRC S will not be wiped so we can symlink to it os.symlink(s, kernsrc) else: import shutil + s_copy = s + ".orig" + if not os.path.isdir(s_copy): + shutil.copytree(s, s_copy) + bb.utils.remove(s, recurse=True) + shutil.copytree(s_copy, s)Etienne
Quoted 37 lines · Richard Purdie
Quoted 19 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The function
do_symlink_kernsrcis not reentrant in the case where S is defined to a non-default value. This causes build-failures e.g. when building linux-yocto, then updating poky to a commit which modifies kernel.bbclass, and then building linux-yocto again.Signed-off-by Etienne Cordonnier <[email protected]>--- meta/classes-recipe/kernel.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/kernel.bbclass +0
-0@@ -194,6 +194,11 @@ python do_symlink_kernsrc () { os.symlink(s, kernsrc)meta/classes-recipe/kernel.bbclass python do_symlink_kernsrc () { · line 204
Quoted 10 lines
meta/classes-recipe/kernel.bbclass +5
-0else: import shutil + if os.path.islink(s): + # this happens for instance when a poky update forces symlink_kernsrc to run again + # after s was already moved to kernsrc + bb.warn("%s is already a symlink! Not symlinking kernel sources" % s) + return 0 shutil.move(s, kernsrc) os.symlink(kernsrc, s) }Usually what we recommend is that the first pass does something if "if not exists X.org; cp X X.orig; fi", then you can always copy from X.orig.
Quoted 6 lines
As Bruce says, it definitely shouldn't be a warning, this can happen quite normally.
Cheers,
Richard
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192635): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_192635&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=82QaxN_7mSeykh83DV-qNXJc9JezHaPCHv4jG2rrq55pywmZCA6Nz75KaUoSuFn8&s=xGE3OScznEMBXLhPR_DdKU6FPE6dbxsHXhKnfksbT8U&e=
Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_103208527_1050810&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=82QaxN_7mSeykh83DV-qNXJc9JezHaPCHv4jG2rrq55pywmZCA6Nz75KaUoSuFn8&s=GX9x-WmhPnkZRNBnirDhR76l6Laww76WDH7ZwKCFSKQ&e=
Group Owner: [email protected]
Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=82QaxN_7mSeykh83DV-qNXJc9JezHaPCHv4jG2rrq55pywmZCA6Nz75KaUoSuFn8&s=roqLGNR0ewaxjmAl_ZE8nSZf_6sCh3yj7LXuITei5EA&e= [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II