Skip to contentSkip to conversation

kernel: fix do_symlink_kernsrc function

PATCH In review Julien Stephan 3 participants 5 messages Jul 5, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel: fix do_symlink_kernsrc function

According to the comment in do_symlink_kernsrc, this function exists for compatibility with old style kernel recipes:

    # Old style kernels may set ${S} = ${WORKDIR}/git for example

For such recipes S will always be different from STAGING_KERNEL_DIR. This is fine for the first build or when unpack is rerun because new sources are in S. However the following command breaks the build:

bitbake -C do_symlink_kernsrc virtual/kernel

At this point, S is a symlink to STAGING_KERNEL_DIR, (meaning S != STAGING_KERNEL_DIR). We first remove the contents of STAGING_KERNEL_DIR without removing the folder itself causing us to lose kernel sources. Then we create a symlink from S to STAGING_KERNEL_DIR which results in the following broken symlinks:

    ${WORKDIR}/git -> <...>/build/tmp/work-shared/<machine>/kernel-source
    kernel-source -> <...>/build/tmp/work-shared/<machine>/kernel-source

The build fails with the following error:

ERROR: <linux_recipe> do_kernel_checkout: FileExistsError(17, 'File exists')
ERROR: Task (<linux_recipe>:do_kernel_checkout)
failed with exit code '1'

Attempting to access the kernel-source directory results in:

ls: cannot access 'kernel-source': Too many levels of symbolic links

Fix this by checking if S is a symlink, and if so, verifying whether the symlink points to STAGING_KERNEL_DIR to avoid losing sources

---
 meta/classes-recipe/kernel.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/kernel.bbclass +6 -1
index 89badd90f18..e328151cb59 100644
@@ -184,7 +184,12 @@ do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B} ${STAGING_KERNEL_BUILDD
 python do_symlink_kernsrc () {
     s = d.getVar("S")
     kernsrc = d.getVar("STAGING_KERNEL_DIR")
-    if s != kernsrc:
+    if os.path.islink(s):
+        _s = os.readlink(s)
+    else:
+        _s = s
+
+    if _s != kernsrc:
         bb.utils.mkdirhier(kernsrc)
         bb.utils.remove(kernsrc, recurse=True)
         if s[-1] == '/':
Signature · 1 line
2.45.1
#1 lore raw

Julien Stephan via lists.openembedded.org <jstephan= [email protected]> escreveu (sexta, 5/07/2024 à(s) 12:31):

Quoted 61 lines

According to the comment in do_symlink_kernsrc, this function exists for compatibility with old style kernel recipes:

    # Old style kernels may set ${S} = ${WORKDIR}/git for example

For such recipes S will always be different from STAGING_KERNEL_DIR. This is fine for the first build or when unpack is rerun because new sources are in S. However the following command breaks the build:

bitbake -C do_symlink_kernsrc virtual/kernel

At this point, S is a symlink to STAGING_KERNEL_DIR, (meaning S != STAGING_KERNEL_DIR). We first remove the contents of STAGING_KERNEL_DIR without removing the folder itself causing us to lose kernel sources. Then we create a symlink from S to STAGING_KERNEL_DIR which results in the following broken symlinks:

    ${WORKDIR}/git -> <...>/build/tmp/work-shared/<machine>/kernel-source
    kernel-source -> <...>/build/tmp/work-shared/<machine>/kernel-source

The build fails with the following error:

ERROR: <linux_recipe> do_kernel_checkout: FileExistsError(17, 'File
exists')
ERROR: Task (<linux_recipe>:do_kernel_checkout)
failed with exit code '1'

Attempting to access the kernel-source directory results in:

ls: cannot access 'kernel-source': Too many levels of symbolic links

Fix this by checking if S is a symlink, and if so, verifying whether the symlink
points to STAGING_KERNEL_DIR to avoid losing sources

---
 meta/classes-recipe/kernel.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/kernel.bbclass +0 -0
index 89badd90f18..e328151cb59 100644
@@ -184,7 +184,12 @@ do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR}

${B} ${STAGING_KERNEL_BUILDD
python do_symlink_kernsrc () {
s = d.getVar("S")
kernsrc = d.getVar("STAGING_KERNEL_DIR")

meta/classes-recipe/kernel.bbclass +6 -2
-    if s != kernsrc:
+    if os.path.islink(s):
+        _s = os.readlink(s)
+    else:
+        _s = s
+
+    if _s != kernsrc:
         bb.utils.mkdirhier(kernsrc)
         bb.utils.remove(kernsrc, recurse=True)
         if s[-1] == '/':
--
Quoted 2 lines

2.45.1

What kernel bbclass are you using to build your kernel? Please note that a very similar logic was replicated on the kernel-yocto.bbclass do_kernel_checkout function
so it will be good to test with and without the kernel-yocto.bbclass.
https://git.yoctoproject.org/poky/tree/meta/classes-recipe/kernel-yocto.bbclass#n376

I recently came across this problem in a change that I ended up having to revert.
https://github.com/Freescale/meta-freescale/pull/1843
https://github.com/Freescale/meta-freescale/pull/1857

Jose

Quoted 13 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201601):
https://lists.openembedded.org/g/openembedded-core/message/201601
Mute This Topic: https://lists.openembedded.org/mt/107052142/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Best regards,

José Quaresma
#2 lore raw
Quoted 70 lines · Jose Quaresma

Julien Stephan via lists.openembedded.org <[email protected]> escreveu (sexta, 5/07/2024 à(s) 12:31):

Quoted 58 lines

According to the comment in do_symlink_kernsrc, this function exists for compatibility with old style kernel recipes:

    # Old style kernels may set ${S} = ${WORKDIR}/git for example

For such recipes S will always be different from STAGING_KERNEL_DIR. This is fine for the first build or when unpack is rerun because new sources are in S. However the following command breaks the build:

bitbake -C do_symlink_kernsrc virtual/kernel

At this point, S is a symlink to STAGING_KERNEL_DIR, (meaning S != STAGING_KERNEL_DIR). We first remove the contents of STAGING_KERNEL_DIR without removing the folder itself causing us to lose kernel sources. Then we create a symlink from S to STAGING_KERNEL_DIR which results in the following broken symlinks:

    ${WORKDIR}/git -> <...>/build/tmp/work-shared/<machine>/kernel-source
    kernel-source -> <...>/build/tmp/work-shared/<machine>/kernel-source

The build fails with the following error:

ERROR: <linux_recipe> do_kernel_checkout: FileExistsError(17, 'File exists')
ERROR: Task (<linux_recipe>:do_kernel_checkout)
failed with exit code '1'

Attempting to access the kernel-source directory results in:

ls: cannot access 'kernel-source': Too many levels of symbolic links

Fix this by checking if S is a symlink, and if so, verifying whether the symlink points to STAGING_KERNEL_DIR to avoid losing sources

---
 meta/classes-recipe/kernel.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/kernel.bbclass +6 -2
index 89badd90f18..e328151cb59 100644
@@ -184,7 +184,12 @@ do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B} ${STAGING_KERNEL_BUILDD
 python do_symlink_kernsrc () {
     s = d.getVar("S")
     kernsrc = d.getVar("STAGING_KERNEL_DIR")
-    if s != kernsrc:
+    if os.path.islink(s):
+        _s = os.readlink(s)
+    else:
+        _s = s
+
+    if _s != kernsrc:
         bb.utils.mkdirhier(kernsrc)
         bb.utils.remove(kernsrc, recurse=True)
         if s[-1] == '/':
--
Quoted 1 line

2.45.1

What kernel bbclass are you using to build your kernel?
Please note that a very similar logic was replicated on the kernel-yocto.bbclass do_kernel_checkout function so it will be good to test with and without the kernel-yocto.bbclass.
https://git.yoctoproject.org/poky/tree/meta/classes-recipe/kernel-yocto.bbclass#n376

Quoted 5 lines

I recently came across this problem in a change that I ended up having to revert.
https://github.com/Freescale/meta-freescale/pull/1843
https://github.com/Freescale/meta-freescale/pull/1857

Jose

Interesting, I'll check this. Thank you for reporting :)

Julien

Quoted 18 lines
Quoted 4 lines
Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201601): https://lists.openembedded.org/g/openembedded-core/message/201601
Mute This Topic: https://lists.openembedded.org/mt/107052142/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Best regards,

José Quaresma

#3 lore raw
In reply to Julien Stephan

Hi Julien,
did you see my previous patch and the follow-up discussion?
Ticket: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
Previous discussion:
https://lists.openembedded.org/g/openembedded-core/topic/103308574#msg195238

I think Richard wanted a solution where this errors out:

Quoted 9 lines

I'm very tempted to say the "using a non default S value" just error out.

Whilst I understand how we got to this level of complexity, I think it will just end up causing us pain in the long run and I'd rather just not support it.

Cheers,
Richard

Étienne

Quoted 77 lines · Julien Stephan via lists.openembedded.org

According to the comment in do_symlink_kernsrc, this function exists for compatibility with old style kernel recipes:

    # Old style kernels may set ${S} = ${WORKDIR}/git for example

For such recipes S will always be different from STAGING_KERNEL_DIR. This is fine for the first build or when unpack is rerun because new sources are in S. However the following command breaks the build:

bitbake -C do_symlink_kernsrc virtual/kernel

At this point, S is a symlink to STAGING_KERNEL_DIR, (meaning S != STAGING_KERNEL_DIR). We first remove the contents of STAGING_KERNEL_DIR without removing the folder itself causing us to lose kernel sources. Then we create a symlink from S to STAGING_KERNEL_DIR which results in the following broken symlinks:

    ${WORKDIR}/git -> <...>/build/tmp/work-shared/<machine>/kernel-source
    kernel-source -> <...>/build/tmp/work-shared/<machine>/kernel-source

The build fails with the following error:

ERROR: <linux_recipe> do_kernel_checkout: FileExistsError(17, 'File
exists')
ERROR: Task (<linux_recipe>:do_kernel_checkout)
failed with exit code '1'

Attempting to access the kernel-source directory results in:

ls: cannot access 'kernel-source': Too many levels of symbolic links

Fix this by checking if S is a symlink, and if so, verifying whether the symlink
points to STAGING_KERNEL_DIR to avoid losing sources

---
 meta/classes-recipe/kernel.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/kernel.bbclass +0 -0
index 89badd90f18..e328151cb59 100644
@@ -184,7 +184,12 @@ do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR}

${B} ${STAGING_KERNEL_BUILDD
python do_symlink_kernsrc () {
s = d.getVar("S")
kernsrc = d.getVar("STAGING_KERNEL_DIR")

meta/classes-recipe/kernel.bbclass +6 -2
-    if s != kernsrc:
+    if os.path.islink(s):
+        _s = os.readlink(s)
+    else:
+        _s = s
+
+    if _s != kernsrc:
         bb.utils.mkdirhier(kernsrc)
         bb.utils.remove(kernsrc, recurse=True)
         if s[-1] == '/':
--

2.45.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201601):
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_201601&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=N3u4aN5Ob1LBtzYAPrUgW7w1Yk_vN3KvuImWnp5g6LA&e=
Mute This Topic:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_107052142_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=VbM00DcAJR0ZUsuhhPWtLEKKZ5MiBkhgS_Q4Aq3X3KM&e=
Group Owner: [email protected] Unsubscribe:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=7EbQmI1Lz7r_bu1RArY3qabfEim4bh48mxJPm1oN_N4&e=
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
Quoted 10 lines · Etienne Cordonnier

Hi Julien,
did you see my previous patch and the follow-up discussion?
Ticket: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15325
Previous discussion: https://lists.openembedded.org/g/openembedded-core/topic/103308574#msg195238

I think Richard wanted a solution where this errors out:

Quoted 3 lines

I'm very tempted to say the "using a non default S value" just error out.

Quoted 9 lines
Quoted 6 lines

Whilst I understand how we got to this level of complexity, I think it will just end up causing us pain in the long run and I'd rather just not support it.

Cheers,
Richard

Étienne

I didn't see it. Let me check this thread :)

Cheers
Julien

Quoted 70 lines · Julien Stephan via lists.openembedded.org
Quoted 69 lines

According to the comment in do_symlink_kernsrc, this function exists for compatibility with old style kernel recipes:

    # Old style kernels may set ${S} = ${WORKDIR}/git for example

For such recipes S will always be different from STAGING_KERNEL_DIR. This is fine for the first build or when unpack is rerun because new sources are in S. However the following command breaks the build:

bitbake -C do_symlink_kernsrc virtual/kernel

At this point, S is a symlink to STAGING_KERNEL_DIR, (meaning S != STAGING_KERNEL_DIR). We first remove the contents of STAGING_KERNEL_DIR without removing the folder itself causing us to lose kernel sources. Then we create a symlink from S to STAGING_KERNEL_DIR which results in the following broken symlinks:

    ${WORKDIR}/git -> <...>/build/tmp/work-shared/<machine>/kernel-source
    kernel-source -> <...>/build/tmp/work-shared/<machine>/kernel-source

The build fails with the following error:

ERROR: <linux_recipe> do_kernel_checkout: FileExistsError(17, 'File exists')
ERROR: Task (<linux_recipe>:do_kernel_checkout)
failed with exit code '1'

Attempting to access the kernel-source directory results in:

ls: cannot access 'kernel-source': Too many levels of symbolic links

Fix this by checking if S is a symlink, and if so, verifying whether the symlink points to STAGING_KERNEL_DIR to avoid losing sources

---
 meta/classes-recipe/kernel.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/kernel.bbclass +6 -2
index 89badd90f18..e328151cb59 100644
@@ -184,7 +184,12 @@ do_clean[cleandirs] += " ${S} ${STAGING_KERNEL_DIR} ${B} ${STAGING_KERNEL_BUILDD
 python do_symlink_kernsrc () {
     s = d.getVar("S")
     kernsrc = d.getVar("STAGING_KERNEL_DIR")
-    if s != kernsrc:
+    if os.path.islink(s):
+        _s = os.readlink(s)
+    else:
+        _s = s
+
+    if _s != kernsrc:
         bb.utils.mkdirhier(kernsrc)
         bb.utils.remove(kernsrc, recurse=True)
         if s[-1] == '/':
--

2.45.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201601): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_201601&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=N3u4aN5Ob1LBtzYAPrUgW7w1Yk_vN3KvuImWnp5g6LA&e=
Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_107052142_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=VbM00DcAJR0ZUsuhhPWtLEKKZ5MiBkhgS_Q4Aq3X3KM&e=
Group Owner: [email protected]
Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=ADx-AJwK4hclmY2pDBh97bnassJ0KI9f_OAKb9D6WpFFcYvO-chBh9k0SVobjYMK&s=7EbQmI1Lz7r_bu1RArY3qabfEim4bh48mxJPm1oN_N4&e= [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-