kernel/multlib: fixes and performance tweak

Richard/Saul,

I bumped these couple of patches to the front of my queue, since one is for a multilib issue we just ran into, and the other is a performance tweak that I've been running with for a while (and it hasn't caused problems).

The first change is a minor change to the packaging of the kernel to save some time during the install phase. No use cases have been broken here, but it's worth having more eyes on this to see if we've missed something. Every little bit can help.

Note: I haven't checked this against your latest series, so I hope this isn't redundant at this point.

[PATCH 1/2] kernel: avoid copying unnecessary files during do_install

kernel: avoid copying unnecessary files during do_install

kernel_do_install() populates $kerneldir with files needed to build external modules. To accomplish this there are several copy commands to get source from the kernel source tree and build trees after which a 'clean' is performed. Since we are copying from the build tree we get about 1G of .o and .cmd files copied over only to have them removed when we clean. This adds additional IO overhead as well as overhead caused by pseudo. By avoiding copying these files in the first place we get multiple gains:
* avoid initial copy
* avoid file deletes during clean
* reduce pseudo overhead

Additionally we are making use of cpio vs cp which tends to be significantly faster at performing copies.

With these changes I observe a 15-30% decrease in the time to complete the do_install() operation on the kernel.

[YOCTO #3517]

The second change was done partly by myself and the rest by Mark Hatle. We ran into linux-dummy showing up as a multlib prefixed dependency when building in our configurations. The reason was that linux-dummy doesn't inherit kernel.bbclass and hence wasn't being skipped. Rather than brining the extra overhead and potential side effects from kernel.bbclass, I decided to check for recipes that provide virtual/kernel and skip based on that test, versus the include.

** I don't have enough background to know if there's a potential problem with this, but it definitely passes our testing after the change.

Mark also found that we needed to skip kernel-module based packages from being extended with the multib prefix and that makes up the second part of the fix.

[PATCH 2/2] multilib: skip packages that provide virtual/kernel

multilib: skip packages that provide virtual/kernel

Rather than keying on recipes that inherit kernel.bbclass, we should be checking for providers of virtual/kernel when skipping kernel recipes in multlib builds.

Not all providers of virtual/kernel inherit kernel.bbclass (notably linux-dummy), so checking on the provider is a more complete check.

We need to be sure to check for inheritance of module-base as well, this allows for packages that provides modules to avoid the multilib renaming.

Cheers,

Bruce

The following changes since commit 8d80483e2f0f755a61087be1dffaf595fbc467fd:

bitbake: bitbake-layers: print the recipe's depends that crosses a layer boundary (2013-01-31 14:00:03 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib zedd/kernel
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel

Bruce Ashfield (1):
multilib: skip packages that provide virtual/kernel

Mark Asselstine (1):
kernel: avoid copying unnecessary files during do_install

 meta/classes/kernel.bbclass   |    7 +++++--
 meta/classes/multilib.bbclass |    4 +++-
 meta/lib/oe/classextend.py    |    2 +-
 3 files changed, 9 insertions(+), 4 deletions(-)

1.7.10.4

From: Mark Asselstine <[email protected]>

kernel_do_install() populates $kerneldir with files needed to build external modules. To accomplish this there are several copy commands to get source from the kernel source tree and build trees after which a 'clean' is performed. Since we are copying from the build tree we get about 1G of .o and .cmd files copied over only to have them removed when we clean. This adds additional IO overhead as well as overhead caused by pseudo. By avoiding copying these files in the first place we get multiple gains:
* avoid initial copy
* avoid file deletes during clean
* reduce pseudo overhead

Additionally we are making use of cpio vs cp which tends to be significantly faster at performing copies.

With these changes I observe a 15-30% decrease in the time to complete the do_install() operation on the kernel.

[YOCTO #3517]

---
 meta/classes/kernel.bbclass |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

index e2a582b..cc61be6 100644
@@ -143,10 +143,13 @@ kernel_do_install() {
 	# work and sysroots can be on different partitions, so we can't rely on
 	# hardlinking, unfortunately.
 	#
-	cp -fR * $kerneldir
+	find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./.*" -print0 | cpio --null -pdu $kerneldir
 	cp .config $kerneldir
 	if [ "${S}" != "${B}" ]; then
-		cp -fR ${S}/* $kerneldir
+		pwd="$PWD"
+		cd "${S}"
+		find . -depth -not -path "./.*" -print0 | cpio --null -pdu $kerneldir
+		cd "$pwd"
 	fi
 	install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE}
 	install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION}

1.7.10.4

Rather than keying on recipes that inherit kernel.bbclass, we should be checking for providers of virtual/kernel when skipping kernel recipes in multlib builds.

Not all providers of virtual/kernel inherit kernel.bbclass (notably linux-dummy), so checking on the provider is a more complete check.

We need to be sure to check for inheritance of module-base as well, this allows for packages that provides modules to avoid the multilib renaming.

---
 meta/classes/multilib.bbclass |    4 +++-
 meta/lib/oe/classextend.py    |    2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

index f5f3c69..f1696b6 100644
@@ -10,7 +10,9 @@ python multilib_virtclass_handler () {
     e.data.setVar('STAGING_KERNEL_DIR', e.data.getVar('STAGING_KERNEL_DIR', True))
 
     # There should only be one kernel in multilib configs
-    if bb.data.inherits_class('kernel', e.data) or bb.data.inherits_class('module-base', e.data):
+    # We also skip multilib setup for module packages.
+    provides = (e.data.getVar("PROVIDES", True) or "").split()
+    if "virtual/kernel" in provides or bb.data.inherits_class('module-base', e.data):
         raise bb.parse.SkipPackage("We shouldn't have multilib variants for the kernel")
 
     if bb.data.inherits_class('image', e.data):

index 19e1254..61cbb8f 100644
@@ -5,7 +5,7 @@ class ClassExtender(object):
         self.pkgs_mapping = []
 
     def extend_name(self, name):
-        if name.startswith("kernel-module"):
+        if name.startswith("kernel-module") or name == "virtual/kernel":
             return name
         if name.startswith("rtld"):
             return name

1.7.10.4