Skip to contentSkip to conversation

Enable package.bbclass to process kernel packages

PATCH In review Saul Wold 3 participants 10 messages Dec 21, 2021 – Jan 5, 2022

lore
Share & tools
lore

Saul Wold

author
#0 lore raw

cover Enable package.bbclass to process kernel packages

These changes allow the package bbclass to process the kernel and kernel modules so that they that extended packaging info can be used by the create-spdx bbclass.

Tested with qemux86-64

Saul Wold (2):
kernel: add -dbg package
package.bbclass: don't skip kernel and kernel modules

 meta/classes/kernel.bbclass  | 2 +-
 meta/classes/package.bbclass | 8 ++------
 2 files changed, 3 insertions(+), 7 deletions(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH 1/2 kernel: add -dbg package

Adding the dbg package allows the package bbclass to parse the debug information which can then be used by the create_spdx bbclass

---
 meta/classes/kernel.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/kernel.bbclass +1 -1
index 2d219cb5e5..bffe264380 100644
@@ -625,7 +625,7 @@ EXPORT_FUNCTIONS do_compile do_install do_configure
 
 # kernel-base becomes kernel-${KERNEL_VERSION}
 # kernel-image becomes kernel-image-${KERNEL_VERSION}
-PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules"
+PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules ${KERNEL_PACKAGE_NAME}-dbg"
 FILES:${PN} = ""
 FILES:${KERNEL_PACKAGE_NAME}-base = "${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.order ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin.modinfo"
 FILES:${KERNEL_PACKAGE_NAME}-image = ""
Signature · 1 line
2.31.1
author
#2 lore raw

PATCH 2/2 package.bbclass: don't skip kernel and kernel modules

In reply to Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
 meta/classes/package.bbclass | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
     dvar = d.getVar('PKGD')
     objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
     newmode = None
     if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
         origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
-                    continue
+
                 if oe.package.is_static_lib(file):
                     staticlibs.append(file)
                     continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                     continue
                 # Check its an executable
                 if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
                     if cpath.islink(file):
                         checkelflinks[file] = ltarget
Signature · 1 line
2.31.1
#3 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

Quoted 34 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
 meta/classes/package.bbclass | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +1 -5
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
     dvar = d.getVar('PKGD')
     objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
     newmode = None
     if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
         origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                     kernmods.append(file)
-                    continue
+
                 if oe.package.is_static_lib(file):
                     staticlibs.append(file)
                     continue

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 8 lines
meta/classes/package.bbclass +1 -1
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                     continue
                 # Check its an executable
                 if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
                     if cpath.islink(file):
                         checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

qemux86 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/4512/steps/11/logs/stdio

qemux86-64 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/4526

qemux86 cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/4476/steps/11/logs/stdio

selftest failure in linux-yocto:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2981/steps/14/logs/stdio
(file truncated makes it sound like a race?)

stap kernel module failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3395/steps/13/logs/stdio
(race/intermittent?)

another kernel module race:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4480/steps/16/logs/stdio

Cheers,

Richard

author
#4 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
  meta/classes/package.bbclass | 8 ++------
  1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
      dvar = d.getVar('PKGD')
      objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
      newmode = None
      if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
          origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                  if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                      kernmods.append(file)
-                    continue
+
                  if oe.package.is_static_lib(file):
                      staticlibs.append(file)
                      continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                      continue
                  # Check its an executable
                  if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                      if cpath.islink(file):
                          checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

4) Change error to warning in packaging.bbclass for the kernel only
- This would explain that a kernel image (vmlinux) is already
stripped and extended package data would not be available for for SPDX creation.

RP, Bruce, Joshua: Thoughts?

Sau!

[0] https://bugzilla.yoctoproject.org/show_bug.cgi?id=3515

Quoted 9 lines

qemux86 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/4512/steps/11/logs/stdio

qemux86-64 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/4526

qemux86 cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/4476/steps/11/logs/stdio

I tried these and have not been able to reproduce the failure.

Quoted 11 lines

selftest failure in linux-yocto:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2981/steps/14/logs/stdio
(file truncated makes it sound like a race?)

stap kernel module failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3395/steps/13/logs/stdio
(race/intermittent?)

another kernel module race:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4480/steps/16/logs/stdio

Might need to try this on the AB once I resolve the kernel stripping issue above.

Sau!

Quoted 6 lines

Cheers,

Richard

Signature · 1 line
Sau!
#5 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

Quoted 52 lines · Saul Wold
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
  meta/classes/package.bbclass | 8 ++------
  1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
      dvar = d.getVar('PKGD')
      objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
      newmode = None
      if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
          origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                  if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                      kernmods.append(file)
-                    continue
+
                  if oe.package.is_static_lib(file):
                      staticlibs.append(file)
                      continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                      continue
                  # Check its an executable
                  if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                      if cpath.islink(file):
                          checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

Quoted 6 lines

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

I recall when we added that! It was used for some other boards as well, but most of them aren't around anymore.

Quoted 4 lines

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

Having some way to have a custom set of sections to strip (along with skipping stripping (but that can be done via the standard inhibit)) is something I'd suggest we preserve. But I suppose if you inhibit stripping, you'll stop both the packaging one and the kernel custom one ?

Quoted 4 lines

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

What are you considering the general case in this instance ? Meaning a non-spdx user of that same board, will run into issues with the already stripped ? If they can inhibit the do_package stripping, there is a way around it.

Quoted 6 lines

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

See above. There are some machines, and even if not common, it is something I'd like to preserve.

Quoted 3 lines

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

I'd prefer to not do #3.

Quoted 4 lines

4) Change error to warning in packaging.bbclass for the kernel only
- This would explain that a kernel image (vmlinux) is already
stripped and extended package data would not be available for for SPDX creation.

#4 is what came to mind for me. We already have special cases for the kernel, so this isn't making things more complex .. or maybe there's a more elegant "co-operative" section removal flag that the kernel bbclass can set, and then the packaging not error or automatically inhibit the QA check ?

But #1 is my second choice.

Bruce

Quoted 47 lines

RP, Bruce, Joshua: Thoughts?

Sau!

[0] https://bugzilla.yoctoproject.org/show_bug.cgi?id=3515

Quoted 9 lines

qemux86 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/4512/steps/11/logs/stdio

qemux86-64 musl cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/4526

qemux86 cryptodev:
https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/4476/steps/11/logs/stdio

I tried these and have not been able to reproduce the failure.

Quoted 11 lines

selftest failure in linux-yocto:
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2981/steps/14/logs/stdio
(file truncated makes it sound like a race?)

stap kernel module failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/3395/steps/13/logs/stdio
(race/intermittent?)

another kernel module race:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4480/steps/16/logs/stdio

Might need to try this on the AB once I resolve the kernel stripping issue above.

Sau!

Quoted 6 lines

Cheers,

Richard

--
Sau!

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#160172): https://lists.openembedded.org/g/openembedded-core/message/160172
Mute This Topic: https://lists.openembedded.org/mt/87884056/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#6 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

In reply to Saul Wold
Quoted 57 lines · Saul Wold
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
  meta/classes/package.bbclass | 8 ++------
  1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
      dvar = d.getVar('PKGD')
      objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
      newmode = None
      if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
          origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                  if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                      kernmods.append(file)
-                    continue
+
                  if oe.package.is_static_lib(file):
                      staticlibs.append(file)
                      continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                      continue
                  # Check its an executable
                  if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                      if cpath.islink(file):
                          checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

Quoted 7 lines

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

I don't think I like this as it is a side effect that isn't obvious or expected.

Quoted 8 lines

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

I know Bruce has said he doesn't like this, however stepping back, these issues were from a time our stripping code was young and evolving. If we can standardise and have it all work together well in one set of functions, I think that is worth looking at. I'd prefer the kernel wasn't a special case if it no longer needs to be.

That said, I don't remember the details of why we did this.

Quoted 7 lines

4) Change error to warning in packaging.bbclass for the kernel only
- This would explain that a kernel image (vmlinux) is already
stripped and extended package data would not be available for for SPDX creation.

RP, Bruce, Joshua: Thoughts?

If we can simplify and stop the kernel being a special case for this code (or handle kernels generically) that would be worth a bit of effort IMO...

Cheers,

Richard

#7 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

Quoted 78 lines · Richard Purdie
Quoted 57 lines · Saul Wold
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
  meta/classes/package.bbclass | 8 ++------
  1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
      dvar = d.getVar('PKGD')
      objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
      newmode = None
      if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
          origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                  if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                      kernmods.append(file)
-                    continue
+
                  if oe.package.is_static_lib(file):
                      staticlibs.append(file)
                      continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                      continue
                  # Check its an executable
                  if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                      if cpath.islink(file):
                          checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

Quoted 7 lines

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

I don't think I like this as it is a side effect that isn't obvious or expected.

Quoted 8 lines

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

Quoted 7 lines

I know Bruce has said he doesn't like this, however stepping back, these issues were from a time our stripping code was young and evolving. If we can standardise and have it all work together well in one set of functions, I think that is worth looking at. I'd prefer the kernel wasn't a special case if it no longer needs to be.

That said, I don't remember the details of why we did this.

There's a middle ground of debug being possible, and some sections removed to keep the footprint a bit lower. There were also some unwinders, etc, that didn't work when everything was stripped and split into debug. The stripping was too aggressive, and removed some sections that were required.

While I can't exactly point to the use cases for it now, with the 5K options in the kernel, they haven't all been removed, and I'd be very hesitant to remove the capability completely.

Bruce

Quoted 28 lines
Quoted 9 lines

4) Change error to warning in packaging.bbclass for the kernel only
- This would explain that a kernel image (vmlinux) is already
stripped and extended package data would not be available for for SPDX creation.

RP, Bruce, Joshua: Thoughts?

If we can simplify and stop the kernel being a special case for this code (or handle kernels generically) that would be worth a bit of effort IMO...

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#160211): https://lists.openembedded.org/g/openembedded-core/message/160211
Mute This Topic: https://lists.openembedded.org/mt/87884056/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#8 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

Quoted 94 lines · Bruce Ashfield
Quoted 78 lines · Richard Purdie
Quoted 57 lines · Saul Wold
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
  meta/classes/package.bbclass | 8 ++------
  1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
      dvar = d.getVar('PKGD')
      objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
      newmode = None
      if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
          origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                  if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                      kernmods.append(file)
-                    continue
+
                  if oe.package.is_static_lib(file):
                      staticlibs.append(file)
                      continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                      continue
                  # Check its an executable
                  if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                      if cpath.islink(file):
                          checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

Quoted 7 lines

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

I don't think I like this as it is a side effect that isn't obvious or expected.

Quoted 8 lines

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

Quoted 7 lines

I know Bruce has said he doesn't like this, however stepping back, these issues were from a time our stripping code was young and evolving. If we can standardise and have it all work together well in one set of functions, I think that is worth looking at. I'd prefer the kernel wasn't a special case if it no longer needs to be.

That said, I don't remember the details of why we did this.

There's a middle ground of debug being possible, and some sections removed to keep the footprint a bit lower. There were also some unwinders, etc, that didn't work when everything was stripped and split into debug. The stripping was too aggressive, and removed some sections that were required.

Quoted 3 lines

While I can't exactly point to the use cases for it now, with the 5K options in the kernel, they haven't all been removed, and I'd be very hesitant to remove the capability completely.

What may help would be to move the kernel stripping code from kernel.bbclass to live along with the package stripping code, which would then at least solve some of the QA warning issues and the two code blocks not fighting with each other. it would also allow the debug symbols to move to the linked debug files where appropriate more easily.

I think one of the challenges in doing that is the do_deploy of the kernel image though which may not then have the right stripping :(.

We may be able to call the strip function from do_deploy though?

Cheers,

Richard

author
#9 lore raw

Re: [OE-core] [PATCH 2/2] package.bbclass: don't skip kernel and kernel modules

In reply to Bruce Ashfield
Quoted 94 lines · Bruce Ashfield
Quoted 78 lines · Richard Purdie
Quoted 57 lines · Saul Wold
Quoted 41 lines · Richard Purdie
Quoted 40 lines · Saul Wold

Stop ignoring or skipping the kernel and kernel modules code in the split debug and striping functions, this will allow create_spdx to process the kernel and modules.

---
   meta/classes/package.bbclass | 8 ++------
   1 file changed, 2 insertions(+), 6 deletions(-)
meta/classes/package.bbclass +2 -6
index 84eafbd529..4b7fe4f1e1 100644
@@ -390,10 +390,6 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
       dvar = d.getVar('PKGD')
       objcopy = d.getVar("OBJCOPY")
 
-    # We ignore kernel modules, we don't generate debug info files.
-    if file.find("/lib/modules/") != -1 and file.endswith(".ko"):
-        return (file, sources)
-
       newmode = None
       if not os.access(file, os.W_OK) or os.access(file, os.R_OK):
           origmode = os.stat(file)[stat.ST_MODE]
@@ -1147,7 +1143,7 @@ python split_and_strip_files () {
 
                   if file.endswith(".ko") and file.find("/lib/modules/") != -1:
                       kernmods.append(file)
-                    continue
+
                   if oe.package.is_static_lib(file):
                       staticlibs.append(file)
                       continue
@@ -1165,7 +1161,7 @@ python split_and_strip_files () {
                       continue
                   # Check its an executable
                   if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
-                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)):
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and (".so" in f or ".node" in f)) or (f.startswith('vmlinux') or ".ko" in f):
 
Quoted 7 lines

meta/classes/package.bbclass python split_and_strip_files () { · line 1167

Quoted 2 lines
meta/classes/package.bbclass +0 -0
                       if cpath.islink(file):
                           checkelflinks[file] = ltarget

edgerouter:
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/4513
https://autobuilder.yoctoproject.org/typhoon/#/builders/111/builds/2507/steps/11/logs/stdio

So I have been digging into this and it seems that an option was added a decade ago or so to strip the kernel/vmlinux when it's too big, this was done for at least the routerstationpro according to bug #3515 [0], and persists with the edgerouter, although I am not sure if it would still actually be required as the edgerouter also uses the KERNEL_ALT_IMAGETYPE to create a smaller binary kernel image.

Quoted 7 lines

The change I proposed causes the all kernels to be stripped all the time as part of the split_and_strip_files(). As I see it there few different options:

1) Set KERNEL_IMAGE_EXTRA_STRIP_SECTIONS = "" in create_spdx.bbclass
- This solves the problem with create_spdx.bbclass is in use, but not
the general case

I don't think I like this as it is a side effect that isn't obvious or expected.

Quoted 8 lines

2) Remove the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS from edgerouter.conf
- Will solve the edgerouter case but may not solve other usages
unknown to me.
- Does anyone know of other machines/layers usage of this variable?

3) deprecate the kernel.bbclass:do_strip function in favor of using the split_and_strip_files() of package.bbclass

Quoted 7 lines

I know Bruce has said he doesn't like this, however stepping back, these issues were from a time our stripping code was young and evolving. If we can standardise and have it all work together well in one set of functions, I think that is worth looking at. I'd prefer the kernel wasn't a special case if it no longer needs to be.

That said, I don't remember the details of why we did this.

There's a middle ground of debug being possible, and some sections removed to keep the footprint a bit lower. There were also some unwinders, etc, that didn't work when everything was stripped and split into debug. The stripping was too aggressive, and removed some sections that were required.

Quoted 4 lines

While I can't exactly point to the use cases for it now, with the 5K options in the kernel, they haven't all been removed, and I'd be very hesitant to remove the capability completely.

I think this makes the most sense after thinking about it also, having one place where the striping occurs in runstrip() in lib/oe/package.py, seems reasonable. The one neck to ring as it were.

We can extend the is_elf() types to add vmlinux and use the KERNEL_IMAGE_EXTRA_STRIP_SECTIONS there. So this could deprecate the do_strip() from the kernel.bbclass and keep the behavior.

Sau!

Quoted 32 lines

Bruce

Quoted 28 lines
Quoted 9 lines

4) Change error to warning in packaging.bbclass for the kernel only
- This would explain that a kernel image (vmlinux) is already
stripped and extended package data would not be available for for SPDX creation.

RP, Bruce, Joshua: Thoughts?

If we can simplify and stop the kernel being a special case for this code (or handle kernels generically) that would be worth a bit of effort IMO...

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#160211): https://lists.openembedded.org/g/openembedded-core/message/160211
Mute This Topic: https://lists.openembedded.org/mt/87884056/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 1 line
Sau!