Skip to contentSkip to conversation

linux-yocto task performance numbers

DISCUSSION Richard Purdie 4 participants 7 messages Feb 22, 2015 – Feb 23, 2015

lore
Share & tools
lore
#0 lore raw

As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):

do_fetch:                 Elapsed time: 0.04 seconds 
do_unpack:                Elapsed time: 1.61 seconds 
do_kernel_checkout:       Elapsed time: 4.39 seconds 
do_validate_branches:     Elapsed time: 0.47 seconds 
do_patch:                 Elapsed time: 57.77 seconds 
do_kernel_configme:       Elapsed time: 44.92 seconds 
do_kernel_configcheck:    Elapsed time: 8.94 seconds 
do_configure:             Elapsed time: 0.60 seconds 
do_compile:               Elapsed time: 72.95 seconds 
do_compile_kernelmodules: Elapsed time: 34.84 seconds 
do_populate_lic:          Elapsed time: 0.14 seconds 
do_strip:                 Elapsed time: 0.03 seconds 
do_uboot_mkimage:         Elapsed time: 0.03 seconds 
do_install:               Elapsed time: 1.87 seconds 
do_populate_sysroot:      Elapsed time: 0.16 seconds 
do_shared_workdir:        Elapsed time: 0.05 seconds 
do_sizecheck:             Elapsed time: 0.03 seconds 
do_bundle_initramfs:      Elapsed time: 0.03 seconds 
do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds 
do_deploy:                Elapsed time: 13.17 seconds 
do_package:               Elapsed time: 31.54 seconds 
do_packagedata:           Elapsed time: 0.59 seconds 
do_package_qa:            Elapsed time: 5.30 seconds 
do_package_write_ipk:     Elapsed time: 83.24 seconds 
do_package_write_rpm:     Elapsed time: 44.58 seconds 

The "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.

Some tasks like the compile tasks are understandable and likely minimised by upstream work already.

The other tasks which as consuming a disproportionate amount of time are kernel_configme and patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?

Cheers,

Richard

#1 lore raw
Quoted 36 lines · Richard Purdie

As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):

do_fetch:                 Elapsed time: 0.04 seconds
do_unpack:                Elapsed time: 1.61 seconds
do_kernel_checkout:       Elapsed time: 4.39 seconds
do_validate_branches:     Elapsed time: 0.47 seconds
do_patch:                 Elapsed time: 57.77 seconds
do_kernel_configme:       Elapsed time: 44.92 seconds
do_kernel_configcheck:    Elapsed time: 8.94 seconds
do_configure:             Elapsed time: 0.60 seconds
do_compile:               Elapsed time: 72.95 seconds
do_compile_kernelmodules: Elapsed time: 34.84 seconds
do_populate_lic:          Elapsed time: 0.14 seconds
do_strip:                 Elapsed time: 0.03 seconds
do_uboot_mkimage:         Elapsed time: 0.03 seconds
do_install:               Elapsed time: 1.87 seconds
do_populate_sysroot:      Elapsed time: 0.16 seconds
do_shared_workdir:        Elapsed time: 0.05 seconds
do_sizecheck:             Elapsed time: 0.03 seconds
do_bundle_initramfs:      Elapsed time: 0.03 seconds
do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
do_deploy:                Elapsed time: 13.17 seconds
do_package:               Elapsed time: 31.54 seconds
do_packagedata:           Elapsed time: 0.59 seconds
do_package_qa:            Elapsed time: 5.30 seconds
do_package_write_ipk:     Elapsed time: 83.24 seconds
do_package_write_rpm:     Elapsed time: 44.58 seconds

The "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.

Quoted 7 lines

Some tasks like the compile tasks are understandable and likely minimised by upstream work already.

The other tasks which as consuming a disproportionate amount of time are kernel_configme and patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?

I have some more changes that I didn't get done in M3, since finalizing the 3.19 kernel took all my cycles.

But in general validation takes time, those pre-cursor steps are what allow the kernel configuration audit to be performed, and also for the patches to be pushed with proper attribution (and to confirm that it has been done).

But again, I do have some more changes that bring the time down, but there's a floor to performance without breaking use cases.

Bruce

Quoted 10 lines

Cheers,

Richard

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"
#2 lore raw
In reply to Richard Purdie
Quoted 36 lines · Richard Purdie

As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):

do_fetch:                 Elapsed time: 0.04 seconds
do_unpack:                Elapsed time: 1.61 seconds
do_kernel_checkout:       Elapsed time: 4.39 seconds
do_validate_branches:     Elapsed time: 0.47 seconds
do_patch:                 Elapsed time: 57.77 seconds
do_kernel_configme:       Elapsed time: 44.92 seconds
do_kernel_configcheck:    Elapsed time: 8.94 seconds
do_configure:             Elapsed time: 0.60 seconds
do_compile:               Elapsed time: 72.95 seconds
do_compile_kernelmodules: Elapsed time: 34.84 seconds
do_populate_lic:          Elapsed time: 0.14 seconds
do_strip:                 Elapsed time: 0.03 seconds
do_uboot_mkimage:         Elapsed time: 0.03 seconds
do_install:               Elapsed time: 1.87 seconds
do_populate_sysroot:      Elapsed time: 0.16 seconds
do_shared_workdir:        Elapsed time: 0.05 seconds
do_sizecheck:             Elapsed time: 0.03 seconds
do_bundle_initramfs:      Elapsed time: 0.03 seconds
do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
do_deploy:                Elapsed time: 13.17 seconds
do_package:               Elapsed time: 31.54 seconds
do_packagedata:           Elapsed time: 0.59 seconds
do_package_qa:            Elapsed time: 5.30 seconds
do_package_write_ipk:     Elapsed time: 83.24 seconds
do_package_write_rpm:     Elapsed time: 44.58 seconds

The "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.

Quoted 7 lines

Some tasks like the compile tasks are understandable and likely minimised by upstream work already.

The other tasks which as consuming a disproportionate amount of time are kernel_configme and patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?

Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:

% time bitbake -f -c kernel_configme linux-yocto

real    0m7.378s
user    0m4.356s
sys     0m5.553s

So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:

real    0m26.371s
user    0m12.561s
sys     0m19.534s

For do_patch (and that is before the already in progress changes that I mentioned earlier).

Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.

Cheers,

Bruce

Quoted 5 lines

Cheers,

Richard

#3 lore raw
Quoted 17 lines · Bruce Ashfield

Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:

% time bitbake -f -c kernel_configme linux-yocto

real    0m7.378s
user    0m4.356s
sys     0m5.553s

So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:

real    0m26.371s
user    0m12.561s
sys     0m19.534s
Quoted 7 lines

For do_patch (and that is before the already in progress changes that I mentioned earlier).

Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.

I think I messed up and ran against a slightly different version. The numbers confirmed with 3.19 are:

do_bundle_initramfs: Elapsed time: 0.03 seconds
do_compile: Elapsed time: 78.07 seconds
do_compile_kernelmodules: Elapsed time: 35.23 seconds
do_configure: Elapsed time: 0.60 seconds
do_deploy: Elapsed time: 14.39 seconds
do_fetch: Elapsed time: 0.04 seconds
do_install: Elapsed time: 2.02 seconds
do_kernel_checkout: Elapsed time: 10.67 seconds
do_kernel_configcheck: Elapsed time: 8.03 seconds
do_kernel_configme: Elapsed time: 70.41 seconds
do_kernel_link_vmlinux: Elapsed time: 0.03 seconds
do_package: Elapsed time: 33.48 seconds
do_packagedata: Elapsed time: 2.30 seconds
do_package_qa: Elapsed time: 4.94 seconds
do_package_write_ipk: Elapsed time: 68.72 seconds
do_package_write_rpm: Elapsed time: 44.33 seconds
do_patch: Elapsed time: 56.43 seconds
do_populate_lic: Elapsed time: 0.14 seconds
do_populate_sysroot: Elapsed time: 0.16 seconds
do_shared_workdir: Elapsed time: 0.04 seconds
do_sizecheck: Elapsed time: 0.03 seconds
do_strip: Elapsed time: 0.03 seconds
do_uboot_mkimage: Elapsed time: 0.02 seconds
do_unpack: Elapsed time: 5.06 seconds
do_validate_branches: Elapsed time: 0.69 seconds

To get the numbers:

bitbake linux-yocto -c cleansstate
bitbake linux-yocto

$ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)

So configcheck is better but the other times still seem high.

Cheers,

Richard

#4 lore raw

Also, for reference/interest, the recent kernel populate_sysroot changes took the kernel build time from over 11 mins to 8 mins. With the move to 3.19:

fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:58.74 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:57.59 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.18 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:00.81 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:01.44 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.12 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:29.71 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:37.66 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.41 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:34.24 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.19

You can see 3.19 takes around 30-35s longer to build. These numbers are extracted from our benchmark machine which monitors things like this.

Cheers,

Richard

#5 lore raw
In reply to Richard Purdie
Quoted 55 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield

Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:

% time bitbake -f -c kernel_configme linux-yocto

real    0m7.378s
user    0m4.356s
sys     0m5.553s

So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:

real    0m26.371s
user    0m12.561s
sys     0m19.534s
Quoted 7 lines

For do_patch (and that is before the already in progress changes that I mentioned earlier).

Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.

I think I messed up and ran against a slightly different version. The numbers confirmed with 3.19 are:

do_bundle_initramfs: Elapsed time: 0.03 seconds
do_compile: Elapsed time: 78.07 seconds
do_compile_kernelmodules: Elapsed time: 35.23 seconds
do_configure: Elapsed time: 0.60 seconds
do_deploy: Elapsed time: 14.39 seconds
do_fetch: Elapsed time: 0.04 seconds
do_install: Elapsed time: 2.02 seconds
do_kernel_checkout: Elapsed time: 10.67 seconds
do_kernel_configcheck: Elapsed time: 8.03 seconds
do_kernel_configme: Elapsed time: 70.41 seconds
do_kernel_link_vmlinux: Elapsed time: 0.03 seconds
do_package: Elapsed time: 33.48 seconds
do_packagedata: Elapsed time: 2.30 seconds
do_package_qa: Elapsed time: 4.94 seconds
do_package_write_ipk: Elapsed time: 68.72 seconds
do_package_write_rpm: Elapsed time: 44.33 seconds
do_patch: Elapsed time: 56.43 seconds
do_populate_lic: Elapsed time: 0.14 seconds
do_populate_sysroot: Elapsed time: 0.16 seconds
do_shared_workdir: Elapsed time: 0.04 seconds
do_sizecheck: Elapsed time: 0.03 seconds
do_strip: Elapsed time: 0.03 seconds
do_uboot_mkimage: Elapsed time: 0.02 seconds
do_unpack: Elapsed time: 5.06 seconds
do_validate_branches: Elapsed time: 0.69 seconds

Quoted 7 lines

To get the numbers:

bitbake linux-yocto -c cleansstate
bitbake linux-yocto

$ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)

So configcheck is better but the other times still seem high.

Yep. configme should never take that long, so something different is happening in your qemu build than mine. I'll dig up my other speedup changes and run them against the same machine build and see what I get.

I just did a run for qemumips, and patching took 9 seconds :)

Bruce

Quoted 5 lines

Cheers,

Richard

#6 lore raw
In reply to Richard Purdie
Quoted 36 lines · Richard Purdie

As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):

do_fetch:                 Elapsed time: 0.04 seconds
do_unpack:                Elapsed time: 1.61 seconds
do_kernel_checkout:       Elapsed time: 4.39 seconds
do_validate_branches:     Elapsed time: 0.47 seconds
do_patch:                 Elapsed time: 57.77 seconds
do_kernel_configme:       Elapsed time: 44.92 seconds
do_kernel_configcheck:    Elapsed time: 8.94 seconds
do_configure:             Elapsed time: 0.60 seconds
do_compile:               Elapsed time: 72.95 seconds
do_compile_kernelmodules: Elapsed time: 34.84 seconds
do_populate_lic:          Elapsed time: 0.14 seconds
do_strip:                 Elapsed time: 0.03 seconds
do_uboot_mkimage:         Elapsed time: 0.03 seconds
do_install:               Elapsed time: 1.87 seconds
do_populate_sysroot:      Elapsed time: 0.16 seconds
do_shared_workdir:        Elapsed time: 0.05 seconds
do_sizecheck:             Elapsed time: 0.03 seconds
do_bundle_initramfs:      Elapsed time: 0.03 seconds
do_kernel_link_vmlinux:   Elapsed time: 0.03 seconds
do_deploy:                Elapsed time: 13.17 seconds
do_package:               Elapsed time: 31.54 seconds
do_packagedata:           Elapsed time: 0.59 seconds
do_package_qa:            Elapsed time: 5.30 seconds
do_package_write_ipk:     Elapsed time: 83.24 seconds
do_package_write_rpm:     Elapsed time: 44.58 seconds

The "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.

Quoted 7 lines

Some tasks like the compile tasks are understandable and likely minimised by upstream work already.

The other tasks which as consuming a disproportionate amount of time are kernel_configme and patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?

Agreed, that was may take away from the above.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center