Skip to contentSkip to conversation

kernel: Use hardlinks for do_populate_sysroot for speed

PATCH In review Richard Purdie 5 participants 13 messages Nov 8, 2013 – Nov 11, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kernel: Use hardlinks for do_populate_sysroot for speed

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

---

meta/classes/kernel.bbclass +2 -2
index 0b6360f..e793ccb 100644
@@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () {
 	fi
 }
 
-sysroot_stage_all_append() {
-	sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}
+python sysroot_stage_all () {
+    oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
 }
 
 kernel_do_configure() {
#1 lore raw
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Bruce

Quoted 20 lines

---

meta/classes/kernel.bbclass +2 -2
index 0b6360f..e793ccb 100644
@@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () {
  	fi
  }
 
-sysroot_stage_all_append() {
-	sysroot_stage_dir ${D}${KERNEL_SRC_PATH} ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}
+python sysroot_stage_all () {
+    oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"), d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
  }
 
  kernel_do_configure() {
#2 lore raw
Quoted 10 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

Quoted 7 lines
Quoted 1 line

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

+1
Looks really promising. Trying them out now in my Yocto dora branch. Btw, would it make sense to group these into a patch set and then number them appropriately?
I realized this when I was trying to apply them since I had to take some care not mixing them up.

Thanks.
Hans

Quoted 30 lines

Bruce

Quoted 19 lines

---

meta/classes/kernel.bbclass +2 -2
index 0b6360f..e793ccb 100644
@@ -308,8 +308,8 @@ kernelscripts_sstate_postinst () {
        fi
  }
 
-sysroot_stage_all_append() {
-       sysroot_stage_dir ${D}${KERNEL_SRC_PATH}
 ${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}
+python sysroot_stage_all () {
+    oe.path.copyhardlinktree(d.expand("${D}${KERNEL_SRC_PATH}"),
 d.expand("${SYSROOT_DESTDIR}${KERNEL_SRC_PATH}"))
  }
Quoted 3 lines
  kernel_do_configure() {

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw
In reply to Bruce Ashfield
Quoted 9 lines · Bruce Ashfield
Quoted 8 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

Quoted 5 lines
Quoted 1 line

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Cheers,

Richard

#4 lore raw
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Quoted 8 lines
Quoted 3 lines

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

Bruce

Quoted 5 lines

Cheers,

Richard

#5 lore raw
Quoted 17 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
Quoted 14 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Quoted 7 lines
Quoted 4 lines

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

I thought I'd share this for people's interest:

http://dan.rpsys.net/kernelbuildissue.png

Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective.

The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm.

So the final thing to build is the kernel by quite some margin, its holding the rest of the build up.

Hopefully these patches start to improve that a bit!

Cheers,

Richard

#6 lore raw
Quoted 34 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
Quoted 14 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Quoted 7 lines
Quoted 4 lines

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

I thought I'd share this for people's interest:

http://dan.rpsys.net/kernelbuildissue.png

Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective.

Quoted 9 lines

The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm.

So the final thing to build is the kernel by quite some margin, its holding the rest of the build up.

Hopefully these patches start to improve that a bit!

Yup, linux-yocto do_package is the biggest build-time hindrance for me and something we've wanted to look into for a long time. Great to see some progress.

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#7 lore raw
Quoted 54 lines · Darren
Quoted 34 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
Quoted 14 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Quoted 7 lines
Quoted 4 lines

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

I thought I'd share this for people's interest:

http://dan.rpsys.net/kernelbuildissue.png

Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective.

Quoted 9 lines

The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm.

So the final thing to build is the kernel by quite some margin, its holding the rest of the build up.

Hopefully these patches start to improve that a bit!

Yup, linux-yocto do_package is the biggest build-time hindrance for me and something we've wanted to look into for a long time. Great to see some progress.

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Quoted 4 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

On my old quad core:

Before: 95.24 seconds
After: 83.25 seconds

Thanks

Andrea

#8 lore raw
Quoted 60 lines · Andrea Adami
Quoted 54 lines · Darren
Quoted 34 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
Quoted 14 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Quoted 7 lines
Quoted 4 lines

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

I thought I'd share this for people's interest:

http://dan.rpsys.net/kernelbuildissue.png

Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective.

Quoted 9 lines

The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm.

So the final thing to build is the kernel by quite some margin, its holding the rest of the build up.

Hopefully these patches start to improve that a bit!

Yup, linux-yocto do_package is the biggest build-time hindrance for me and something we've wanted to look into for a long time. Great to see some progress.

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Quoted 4 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 4 lines

On my old quad core:

Before: 95.24 seconds
After: 83.25 seconds

We have some "standard" tests we use for comparison purposes as detailed on:

https://wiki.yoctoproject.org/wiki/Performance_Test

Some numbers from master and a test branch containing the recent speedups:

fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:37,12:05.50,1:17:53,4:02.68,0:31.71,0:15.70,0:01.61,26347744,5035816 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:59,12:09.26,1:17:29,4:06.75,0:31.82,0:15.69,0:01.70,26348556,5035876 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:21,12:07.23,1:18:21,4:07.44,0:31.79,0:15.63,0:01.62,26351088,5035908 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:23:16,12:15.33,1:18:00,4:02.50,0:31.87,0:15.74,0:01.63,26348132,5035848 fedora19,rpurdie/timing:9d8c0ef3349936b3bd0bbf485b50cf81f4feaf80,poky-10.0.0.final-373-g9d8c0ef,1:19:42,10:28.53,1:15:30,4:12.24,0:31.30,0:15.69,0:01.66,26137344,5055628 fedora19,rpurdie/timing2:eb2221d6247c12322ac38b9ac6f24c9de5877e53,poky-10.0.0.final-401-geb2221d,1:18:51,10:44.26,1:16:32,4:13.47,0:31.27,0:15.69,0:01.65,26137828,5055916

So we have 1:23m -> 1:19m for overall buildtime and 12.0m -> 10.5m for "bitbake virtual/kernel" from cleansstate. Not too bad :)

Thanks to Stefan for running the tests!

Cheers,

Richard

#9 lore raw
Quoted 81 lines · Richard Purdie
Quoted 60 lines · Andrea Adami
Quoted 54 lines · Darren
Quoted 34 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
Quoted 14 lines · Bruce Ashfield
Quoted 9 lines · Richard Purdie

The kernel tree is large and doesn't need to be copied. Override the default sysroot handling function to use a hardlink copying function in python.

This commit also drops the copying of the /lib directory which just contains the kernel modules. We never use those in the sysroot so there is little point in carrying those around.

For linux-yocto this takes the do_populate_sysroot time 24s -> 14s.

Fantastic. One less thing for me to dig into later. I thought this was already in place, so I'm pleasantly surprised that there was a time savings to be found!

Quoted 7 lines
Quoted 4 lines

I was somewhat surprised too.

We still need to optimise what we install in do_install since that is where significant gains can still be made.

Agreed. I started some changes in that area right after ELC-e, I'll try and get them out sooner rather than later.

I thought I'd share this for people's interest:

http://dan.rpsys.net/kernelbuildissue.png

Its the output from pybootchart of a bitbake core-image-sato from scratch. I've zoomed out to put some bars in particular into perspective.

Quoted 9 lines

The pink colour is linux-yocto:do_install, the cyan is linux-yocto:do_package and the blue is linux-yocto:do_populate_sysroot. The uncoloured bar at the bottom is linux-yocto:do_package_write_rpm.

So the final thing to build is the kernel by quite some margin, its holding the rest of the build up.

Hopefully these patches start to improve that a bit!

Yup, linux-yocto do_package is the biggest build-time hindrance for me and something we've wanted to look into for a long time. Great to see some progress.

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Quoted 4 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 4 lines

On my old quad core:

Before: 95.24 seconds
After: 83.25 seconds

We have some "standard" tests we use for comparison purposes as detailed on:

https://wiki.yoctoproject.org/wiki/Performance_Test

Some numbers from master and a test branch containing the recent speedups:

fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:37,12:05.50,1:17:53,4:02.68,0:31.71,0:15.70,0:01.61,26347744,5035816 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:59,12:09.26,1:17:29,4:06.75,0:31.82,0:15.69,0:01.70,26348556,5035876 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:22:21,12:07.23,1:18:21,4:07.44,0:31.79,0:15.63,0:01.62,26351088,5035908 fedora19,master:d6cc7c8ed76c8b1117cf03c7bd4b0742f98f79b3,poky-10.0.0.final-359-gd6cc7c8,1:23:16,12:15.33,1:18:00,4:02.50,0:31.87,0:15.74,0:01.63,26348132,5035848 fedora19,rpurdie/timing:9d8c0ef3349936b3bd0bbf485b50cf81f4feaf80,poky-10.0.0.final-373-g9d8c0ef,1:19:42,10:28.53,1:15:30,4:12.24,0:31.30,0:15.69,0:01.66,26137344,5055628 fedora19,rpurdie/timing2:eb2221d6247c12322ac38b9ac6f24c9de5877e53,poky-10.0.0.final-401-geb2221d,1:18:51,10:44.26,1:16:32,4:13.47,0:31.27,0:15.69,0:01.65,26137828,5055916

Quoted 5 lines

So we have 1:23m -> 1:19m for overall buildtime and 12.0m -> 10.5m for "bitbake virtual/kernel" from cleansstate. Not too bad :)

Thanks to Stefan for running the tests!

I am having some trouble with these patches :( The new approach is using the '-n' flag to the 'cp' command.
That is not supported on our SuSE11 based system. To be honest, I do not know how portable '-n' is?
Some system have it, others seems to provide '-u' instead. I guess by removing '-n' part of the performance gain is lost?
Would it be possible to test for error from the 'cp' command and if it fails try '-u' instead (and cache the result)? Or maybe even better, make this configurable in local.conf (or the distro) for the build platforms that does not support '-n' but can instead fall-back to using '-u'. From what I can tell most systems support one or the other, but never both.

Thanks.
Hans

Quoted 8 lines

Cheers,

Richard

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#10 lore raw
Quoted 4 lines · Hans Beckérus

I am having some trouble with these patches :( The new approach is using the '-n' flag to the 'cp' command.
That is not supported on our SuSE11 based system. To be honest, I do not know how portable '-n' is?

Quoted 8 lines

Some system have it, others seems to provide '-u' instead. I guess by removing '-n' part of the performance gain is lost?
Would it be possible to test for error from the 'cp' command and if it fails try '-u' instead (and cache the result)? Or maybe even better, make this configurable in local.conf (or the distro) for the build platforms that does not support '-n' but can instead fall-back to using '-u'. From what I can tell most systems support one or the other, but never both.

Hmm, its been in coreutils since 2009 which I guess isn't that long in the scheme of things:
http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/cp.c?id=d01338eb3d30e5634f1b4d4179c229f54eea0b44

Just to double check, is cp on your system provided by coreutils or something else?

Cheers,

Richard

#11 lore raw
In reply to Hans Beckérus
Quoted 4 lines · Hans Beckérus

I am having some trouble with these patches :( The new approach is using the '-n' flag to the 'cp' command.
That is not supported on our SuSE11 based system. To be honest, I do not know how portable '-n' is?

Quoted 8 lines

Some system have it, others seems to provide '-u' instead. I guess by removing '-n' part of the performance gain is lost?
Would it be possible to test for error from the 'cp' command and if it fails try '-u' instead (and cache the result)? Or maybe even better, make this configurable in local.conf (or the distro) for the build platforms that does not support '-n' but can instead fall-back to using '-u'. From what I can tell most systems support one or the other, but never both.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=276b9df3588ecd438c3e2f502d56cd6b9fb24676

Thankfully the above works just as well. I used -n to get rid of an error, then added -l for efficiency. We can get away just with -l.

Cheers,

Richard

#12 lore raw
Quoted 14 lines · Richard Purdie
Quoted 4 lines · Hans Beckérus

I am having some trouble with these patches :( The new approach is using the '-n' flag to the 'cp' command.
That is not supported on our SuSE11 based system. To be honest, I do not know how portable '-n' is?

Quoted 8 lines

Some system have it, others seems to provide '-u' instead. I guess by removing '-n' part of the performance gain is lost?
Would it be possible to test for error from the 'cp' command and if it fails try '-u' instead (and cache the result)? Or maybe even better, make this configurable in local.conf (or the distro) for the build platforms that does not support '-n' but can instead fall-back to using '-u'. From what I can tell most systems support one or the other, but never both.

Quoted 5 lines

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=276b9df3588ecd438c3e2f502d56cd6b9fb24676

Thankfully the above works just as well. I used -n to get rid of an error, then added -l for efficiency. We can get away just with -l.

Great! Will try it asap.
Btw, cp was from coreutils back in 2008.

cp (GNU coreutils) 6.12
Copyright (C) 2008 Free Software Foundation, Inc.

Thanks.
Hans

Quoted 5 lines

Cheers,

Richard