linux-yocto task performance numbers
As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):
do_fetch: Elapsed time: 0.04 seconds do_unpack: Elapsed time: 1.61 seconds do_kernel_checkout: Elapsed time: 4.39 seconds do_validate_branches: Elapsed time: 0.47 seconds do_patch: Elapsed time: 57.77 seconds do_kernel_configme: Elapsed time: 44.92 seconds do_kernel_configcheck: Elapsed time: 8.94 seconds do_configure: Elapsed time: 0.60 seconds do_compile: Elapsed time: 72.95 seconds do_compile_kernelmodules: Elapsed time: 34.84 seconds do_populate_lic: Elapsed time: 0.14 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.03 seconds do_install: Elapsed time: 1.87 seconds do_populate_sysroot: Elapsed time: 0.16 seconds do_shared_workdir: Elapsed time: 0.05 seconds do_sizecheck: Elapsed time: 0.03 seconds do_bundle_initramfs: Elapsed time: 0.03 seconds do_kernel_link_vmlinux: Elapsed time: 0.03 seconds do_deploy: Elapsed time: 13.17 seconds do_package: Elapsed time: 31.54 seconds do_packagedata: Elapsed time: 0.59 seconds do_package_qa: Elapsed time: 5.30 seconds do_package_write_ipk: Elapsed time: 83.24 seconds do_package_write_rpm: Elapsed time: 44.58 seconds
The "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.
Some tasks like the compile tasks are understandable and likely minimised by upstream work already.
The other tasks which as consuming a disproportionate amount of time are kernel_configme and patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?
Cheers,
Richard
Quoted 36 lines · Richard Purdie
As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):
do_fetch: Elapsed time: 0.04 seconds do_unpack: Elapsed time: 1.61 seconds do_kernel_checkout: Elapsed time: 4.39 seconds do_validate_branches: Elapsed time: 0.47 seconds do_patch: Elapsed time: 57.77 seconds do_kernel_configme: Elapsed time: 44.92 seconds do_kernel_configcheck: Elapsed time: 8.94 seconds do_configure: Elapsed time: 0.60 seconds do_compile: Elapsed time: 72.95 seconds do_compile_kernelmodules: Elapsed time: 34.84 seconds do_populate_lic: Elapsed time: 0.14 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.03 seconds do_install: Elapsed time: 1.87 seconds do_populate_sysroot: Elapsed time: 0.16 seconds do_shared_workdir: Elapsed time: 0.05 seconds do_sizecheck: Elapsed time: 0.03 seconds do_bundle_initramfs: Elapsed time: 0.03 seconds do_kernel_link_vmlinux: Elapsed time: 0.03 seconds do_deploy: Elapsed time: 13.17 seconds do_package: Elapsed time: 31.54 seconds do_packagedata: Elapsed time: 0.59 seconds do_package_qa: Elapsed time: 5.30 seconds do_package_write_ipk: Elapsed time: 83.24 seconds do_package_write_rpm: Elapsed time: 44.58 secondsThe "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.
Quoted 7 lines
Some tasks like the compile tasks are understandable and likely minimised by upstream work already.
The other tasks which as consuming a disproportionate amount of time are
kernel_configmeand patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?
I have some more changes that I didn't get done in M3, since finalizing the 3.19 kernel took all my cycles.
But in general validation takes time, those pre-cursor steps are what allow the kernel configuration audit to be performed, and also for the patches to be pushed with proper attribution (and to confirm that it has been done).
But again, I do have some more changes that bring the time down, but there's a floor to performance without breaking use cases.
Bruce
Quoted 10 lines
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 36 lines · Richard Purdie
As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):
do_fetch: Elapsed time: 0.04 seconds do_unpack: Elapsed time: 1.61 seconds do_kernel_checkout: Elapsed time: 4.39 seconds do_validate_branches: Elapsed time: 0.47 seconds do_patch: Elapsed time: 57.77 seconds do_kernel_configme: Elapsed time: 44.92 seconds do_kernel_configcheck: Elapsed time: 8.94 seconds do_configure: Elapsed time: 0.60 seconds do_compile: Elapsed time: 72.95 seconds do_compile_kernelmodules: Elapsed time: 34.84 seconds do_populate_lic: Elapsed time: 0.14 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.03 seconds do_install: Elapsed time: 1.87 seconds do_populate_sysroot: Elapsed time: 0.16 seconds do_shared_workdir: Elapsed time: 0.05 seconds do_sizecheck: Elapsed time: 0.03 seconds do_bundle_initramfs: Elapsed time: 0.03 seconds do_kernel_link_vmlinux: Elapsed time: 0.03 seconds do_deploy: Elapsed time: 13.17 seconds do_package: Elapsed time: 31.54 seconds do_packagedata: Elapsed time: 0.59 seconds do_package_qa: Elapsed time: 5.30 seconds do_package_write_ipk: Elapsed time: 83.24 seconds do_package_write_rpm: Elapsed time: 44.58 secondsThe "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.
Quoted 7 lines
Some tasks like the compile tasks are understandable and likely minimised by upstream work already.
The other tasks which as consuming a disproportionate amount of time are
kernel_configmeand patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?
Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:
% time bitbake -f -c kernel_configme linux-yocto
real 0m7.378s user 0m4.356s sys 0m5.553s
So I'm showing 7 seconds, versus 45 in your run .. and I'm showing:
real 0m26.371s user 0m12.561s sys 0m19.534s
For do_patch (and that is before the already in progress changes that I mentioned earlier).
Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.
Cheers,
Bruce
Quoted 5 lines
Cheers,
Richard
Quoted 17 lines · Bruce Ashfield
Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:
% time bitbake -f -c
kernel_configmelinux-yoctoreal 0m7.378s user 0m4.356s sys 0m5.553sSo I'm showing 7 seconds, versus 45 in your run .. and I'm showing:
real 0m26.371s user 0m12.561s sys 0m19.534s
Quoted 7 lines
For
do_patch(and that is before the already in progress changes that I mentioned earlier).Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.
I think I messed up and ran against a slightly different version. The numbers confirmed with 3.19 are:
do_bundle_initramfs: Elapsed time: 0.03 secondsdo_compile: Elapsed time: 78.07 secondsdo_compile_kernelmodules: Elapsed time: 35.23 secondsdo_configure: Elapsed time: 0.60 secondsdo_deploy: Elapsed time: 14.39 secondsdo_fetch: Elapsed time: 0.04 secondsdo_install: Elapsed time: 2.02 secondsdo_kernel_checkout: Elapsed time: 10.67 secondsdo_kernel_configcheck: Elapsed time: 8.03 secondsdo_kernel_configme: Elapsed time: 70.41 secondsdo_kernel_link_vmlinux: Elapsed time: 0.03 secondsdo_package: Elapsed time: 33.48 secondsdo_packagedata: Elapsed time: 2.30 secondsdo_package_qa: Elapsed time: 4.94 secondsdo_package_write_ipk: Elapsed time: 68.72 secondsdo_package_write_rpm: Elapsed time: 44.33 secondsdo_patch: Elapsed time: 56.43 secondsdo_populate_lic: Elapsed time: 0.14 secondsdo_populate_sysroot: Elapsed time: 0.16 secondsdo_shared_workdir: Elapsed time: 0.04 secondsdo_sizecheck: Elapsed time: 0.03 secondsdo_strip: Elapsed time: 0.03 secondsdo_uboot_mkimage: Elapsed time: 0.02 secondsdo_unpack: Elapsed time: 5.06 secondsdo_validate_branches: Elapsed time: 0.69 seconds
To get the numbers:
bitbake linux-yocto -c cleansstate
bitbake linux-yocto
$ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)
So configcheck is better but the other times still seem high.
Cheers,
Richard
Also, for reference/interest, the recent kernel populate_sysroot changes took the kernel build time from over 11 mins to 8 mins. With the move to 3.19:
fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:58.74 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:57.59 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.18 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:00.81 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 8:01.44 fedora19,master:5db6a27f4847e51c6434a968f81d3ceffcc54952 7:59.12 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:29.71 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:37.66 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.41 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:34.24 fedora19,master:172606338757d0b788e3382bf413102647307e1d 8:36.19
You can see 3.19 takes around 30-35s longer to build. These numbers are extracted from our benchmark machine which monitors things like this.
Cheers,
Richard
Quoted 55 lines · Richard Purdie
Quoted 17 lines · Bruce Ashfield
Was this the 3.19 kernel you used for these stats ? I just ran a quick test (i.e. not with buildstats) with my latest kernel tree (on my crappy I/O laptop) and get the following:
% time bitbake -f -c
kernel_configmelinux-yoctoreal 0m7.378s user 0m4.356s sys 0m5.553sSo I'm showing 7 seconds, versus 45 in your run .. and I'm showing:
real 0m26.371s user 0m12.561s sys 0m19.534sQuoted 7 lines
For
do_patch(and that is before the already in progress changes that I mentioned earlier).Just with those differences alone, sounds like we should log this in bugzilla and put the details of the configuration and build machines in there .. since I'll need to isolate what is causing those extra long runs, and ensure that everything is pushed to the various trees.
I think I messed up and ran against a slightly different version. The numbers confirmed with 3.19 are:
do_bundle_initramfs: Elapsed time: 0.03 secondsdo_compile: Elapsed time: 78.07 secondsdo_compile_kernelmodules: Elapsed time: 35.23 secondsdo_configure: Elapsed time: 0.60 secondsdo_deploy: Elapsed time: 14.39 secondsdo_fetch: Elapsed time: 0.04 secondsdo_install: Elapsed time: 2.02 secondsdo_kernel_checkout: Elapsed time: 10.67 secondsdo_kernel_configcheck: Elapsed time: 8.03 secondsdo_kernel_configme: Elapsed time: 70.41 secondsdo_kernel_link_vmlinux: Elapsed time: 0.03 secondsdo_package: Elapsed time: 33.48 secondsdo_packagedata: Elapsed time: 2.30 secondsdo_package_qa: Elapsed time: 4.94 secondsdo_package_write_ipk: Elapsed time: 68.72 secondsdo_package_write_rpm: Elapsed time: 44.33 secondsdo_patch: Elapsed time: 56.43 secondsdo_populate_lic: Elapsed time: 0.14 secondsdo_populate_sysroot: Elapsed time: 0.16 secondsdo_shared_workdir: Elapsed time: 0.04 secondsdo_sizecheck: Elapsed time: 0.03 secondsdo_strip: Elapsed time: 0.03 secondsdo_uboot_mkimage: Elapsed time: 0.02 secondsdo_unpack: Elapsed time: 5.06 secondsdo_validate_branches: Elapsed time: 0.69 seconds
Quoted 7 lines
To get the numbers:
bitbake linux-yocto -c cleansstate
bitbake linux-yocto$ (cd tmp/buildstats/linux-yocto-qemux86-64/201502230954/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0/; grep El * | sed s/linux-yocto-3.19+gitAUTOINC+8897ef68b3_43b9eced9b-r0// | sed s/^.*:://)So configcheck is better but the other times still seem high.
Yep. configme should never take that long, so something different is happening in your qemu build than mine. I'll dig up my other speedup changes and run them against the same machine build and see what I get.
I just did a run for qemumips, and patching took 9 seconds :)
Bruce
Quoted 5 lines
Cheers,
Richard
Quoted 36 lines · Richard Purdie
As people know, I've been looking at performance a little, one of the benchmarks is how long the kernel takes to build. I dumped out the task performance data from buildstats for a linux-yocto build (nothing else running):
do_fetch: Elapsed time: 0.04 seconds do_unpack: Elapsed time: 1.61 seconds do_kernel_checkout: Elapsed time: 4.39 seconds do_validate_branches: Elapsed time: 0.47 seconds do_patch: Elapsed time: 57.77 seconds do_kernel_configme: Elapsed time: 44.92 seconds do_kernel_configcheck: Elapsed time: 8.94 seconds do_configure: Elapsed time: 0.60 seconds do_compile: Elapsed time: 72.95 seconds do_compile_kernelmodules: Elapsed time: 34.84 seconds do_populate_lic: Elapsed time: 0.14 seconds do_strip: Elapsed time: 0.03 seconds do_uboot_mkimage: Elapsed time: 0.03 seconds do_install: Elapsed time: 1.87 seconds do_populate_sysroot: Elapsed time: 0.16 seconds do_shared_workdir: Elapsed time: 0.05 seconds do_sizecheck: Elapsed time: 0.03 seconds do_bundle_initramfs: Elapsed time: 0.03 seconds do_kernel_link_vmlinux: Elapsed time: 0.03 seconds do_deploy: Elapsed time: 13.17 seconds do_package: Elapsed time: 31.54 seconds do_packagedata: Elapsed time: 0.59 seconds do_package_qa: Elapsed time: 5.30 seconds do_package_write_ipk: Elapsed time: 83.24 seconds do_package_write_rpm: Elapsed time: 44.58 secondsThe "core" was getting blamed for a lot of the build time. As can be seen, the "core" isn't taking that much time now, apart from the fact that ipk packaging seems to be taking twice the time of rpm which needs looking into.
Quoted 7 lines
Some tasks like the compile tasks are understandable and likely minimised by upstream work already.
The other tasks which as consuming a disproportionate amount of time are
kernel_configmeand patch, I believe we need to look into those a little further too. To put it into context, should the kernel compile be at the same order of magnitude as the patch and configure?
Agreed, that was may take away from the above.
Signature · 2 lines
Darren Hart Intel Open Source Technology Center