kernel: Use hardlinks during do_install for speed
PATCH 1/1 kernel: Use hardlinks during do_install for speed
Whilst the comment says we can't do this, its incorrect, install operates on WORKDIR and its hard for different parts of WORKDIR to be on different filesystems. Hardlinking instead of copying is therefore a nice performance gain.
Also, completely skip the Documentation directory (adding a dummy Makefile to keep make happy) and tweak the other cp commands to link since if we don't we'd get "this is the same file" type errors from cp for some kernel versions.
For do_install on linux-yocto, this takes it from 227s -> 84s.
---
meta/classes/kernel.bbclass +9 -9
@@ -236,20 +236,21 @@ kernel_do_install() { # dir. This ensures the original Makefiles are used and not the # redirecting Makefiles in the build directory. # - # work and sysroots can be on different partitions, so we can't rely on - # hardlinking, unfortunately. - # - find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./.*" -print0 | cpio --null -pdu $kerneldir + find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir cp .config $kerneldir if [ "${S}" != "${B}" ]; then pwd="$PWD" cd "${S}" - find . -depth -not -path "./.*" -print0 | cpio --null -pdu $kerneldir + find . -depth -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir cd "$pwd" fi install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE} install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION} + # Dummy Makefile so the clean below works + mkdir $kerneldir/Documentation + touch $kerneldir/Documentation/Makefile + # # Clean and remove files not needed for building modules. # Some distributions go through a lot more trouble to strip out
@@ -262,22 +263,21 @@ kernel_do_install() { oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean make -C $kerneldir _mrproper_scripts find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \; - find $kerneldir/Documentation -name "*.txt" -exec rm '{}' \; # As of Linux kernel version 3.0.1, the clean target removes # arch/powerpc/lib/crtsavres.o which is present in # KBUILD_LDFLAGS_MODULE, making it required to build external modules. if [ ${ARCH} = "powerpc" ]; then - cp arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o + cp -nl arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o fi # Necessary for building modules like compat-wireless. if [ -f include/generated/bounds.h ]; then - cp include/generated/bounds.h $kerneldir/include/generated/bounds.h + cp -nl include/generated/bounds.h $kerneldir/include/generated/bounds.h fi if [ -d arch/${ARCH}/include/generated ]; then mkdir -p $kerneldir/arch/${ARCH}/include/generated/ - cp -fR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/ + cp -fnlR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/ fi # Remove the following binaries which cause strip or arch QA errors
Quoted 2 lines · Richard Purdie
Whilst the comment says we can't do this, its incorrect, install operates on WORKDIR and its hard for different parts of WORKDIR to be on different
Is it hard? Or is it not possible / not supported?
My understanding was it was possible and therefor we couldn't hardlink. I'm thrilled at the prospect of being wrong there though!
Quoted 9 lines
filesystems. Hardlinking instead of copying is therefore a nice performance gain.
Also, completely skip the Documentation directory (adding a dummy
Makefileto keep make happy) and tweak the other cp commands to link since if we don't we'd get "this is the same file" type errors from cp for some kernel versions.For
do_installon linux-yocto, this takes it from 227s -> 84s.
This, and the others, are fantastic improvements, thanks RP!
--
Darren
Quoted 62 lines
Signed-off-by Richard Purdie <[email protected]>---
meta/classes/kernel.bbclass +9
-9@@ -236,20 +236,21 @@ kernel_do_install() { # dir. This ensures the original Makefiles are used and not the # redirecting Makefiles in the build directory. # - # work and sysroots can be on different partitions, so we can't rely on - # hardlinking, unfortunately. - # - find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./.*" -print0 | cpio --null -pdu $kerneldir + find . -depth -not -name "*.cmd" -not -name "*.o" -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir cp .config $kerneldir if [ "${S}" != "${B}" ]; then pwd="$PWD" cd "${S}" - find . -depth -not -path "./.*" -print0 | cpio --null -pdu $kerneldir + find . -depth -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir cd "$pwd" fi install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE} install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION} + # Dummy Makefile so the clean below works + mkdir $kerneldir/Documentation + touch $kerneldir/Documentation/Makefile + # # Clean and remove files not needed for building modules. # Some distributions go through a lot more trouble to strip out@@ -262,22 +263,21 @@ kernel_do_install() { oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean make -C $kerneldir _mrproper_scripts find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \; - find $kerneldir/Documentation -name "*.txt" -exec rm '{}' \; # As of Linux kernel version 3.0.1, the clean target removes # arch/powerpc/lib/crtsavres.o which is present in # KBUILD_LDFLAGS_MODULE, making it required to build external modules. if [ ${ARCH} = "powerpc" ]; then - cp arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o + cp -nl arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o fi # Necessary for building modules like compat-wireless. if [ -f include/generated/bounds.h ]; then - cp include/generated/bounds.h $kerneldir/include/generated/bounds.h + cp -nl include/generated/bounds.h $kerneldir/include/generated/bounds.h fi if [ -d arch/${ARCH}/include/generated ]; then mkdir -p $kerneldir/arch/${ARCH}/include/generated/ - cp -fR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/ + cp -fnlR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/ fi # Remove the following binaries which cause strip or arch QA errors
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
Quoted 5 lines · Darren
Quoted 2 lines · Richard Purdie
Whilst the comment says we can't do this, its incorrect, install operates on WORKDIR and its hard for different parts of WORKDIR to be on different
Is it hard? Or is it not possible / not supported?
Not really possible unless you try very hard. Its not supported. We don't even support WORKDIR and sysroot on different disks really.
Quoted 2 lines
My understanding was it was possible and therefor we couldn't hardlink. I'm thrilled at the prospect of being wrong there though!
I'm happy enough to decide we don't support it (and we never have).
Quoted 11 lines
Quoted 9 lines
filesystems. Hardlinking instead of copying is therefore a nice performance gain.
Also, completely skip the Documentation directory (adding a dummy
Makefileto keep make happy) and tweak the other cp commands to link since if we don't we'd get "this is the same file" type errors from cp for some kernel versions.For
do_installon linux-yocto, this takes it from 227s -> 84s.This, and the others, are fantastic improvements, thanks RP!
I'm rather happy to find something we can easily fix :)
Cheers,
Richard
Quoted 29 lines · Richard Purdie
Quoted 5 lines · Darren
Quoted 2 lines · Richard Purdie
Whilst the comment says we can't do this, its incorrect, install operates on WORKDIR and its hard for different parts of WORKDIR to be on different
Is it hard? Or is it not possible / not supported?
Not really possible unless you try very hard. Its not supported. We don't even support WORKDIR and sysroot on different disks really.
Quoted 2 lines
My understanding was it was possible and therefor we couldn't hardlink. I'm thrilled at the prospect of being wrong there though!
I'm happy enough to decide we don't support it (and we never have).
Quoted 11 lines
Quoted 9 lines
filesystems. Hardlinking instead of copying is therefore a nice performance gain.
Also, completely skip the Documentation directory (adding a dummy
Makefileto keep make happy) and tweak the other cp commands to link since if we don't we'd get "this is the same file" type errors from cp for some kernel versions.For
do_installon linux-yocto, this takes it from 227s -> 84s.This, and the others, are fantastic improvements, thanks RP!
I'm rather happy to find something we can easily fix :)
Quoted 8 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
On my old quad core:
Before: 131.01 seconds
After: 102.29 seconds
Thanks
Andrea
Quoted 40 lines · Andrea Adami
Quoted 29 lines · Richard Purdie
Quoted 5 lines · Darren
Quoted 2 lines · Richard Purdie
Whilst the comment says we can't do this, its incorrect, install operates on WORKDIR and its hard for different parts of WORKDIR to be on different
Is it hard? Or is it not possible / not supported?
Not really possible unless you try very hard. Its not supported. We don't even support WORKDIR and sysroot on different disks really.
Quoted 2 lines
My understanding was it was possible and therefor we couldn't hardlink. I'm thrilled at the prospect of being wrong there though!
I'm happy enough to decide we don't support it (and we never have).
Quoted 11 lines
Quoted 9 lines
filesystems. Hardlinking instead of copying is therefore a nice performance gain.
Also, completely skip the Documentation directory (adding a dummy
Makefileto keep make happy) and tweak the other cp commands to link since if we don't we'd get "this is the same file" type errors from cp for some kernel versions.For
do_installon linux-yocto, this takes it from 227s -> 84s.This, and the others, are fantastic improvements, thanks RP!
I'm rather happy to find something we can easily fix :)
Quoted 8 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 5 lines
On my old quad core:
Before: 131.01 seconds
After: 102.29 seconds
Somehow with both patches for hardlink and after some -ccleanall I'm now down to 84-85 secs.
I expect even more from the /lib/oe/path patch which is atm broken on my setup.
Many Thanks !
Andrea