lib/oe/path: Fix performance issue got copyhardlinktree()
PATCH 1/1 lib/oe/path: Fix performance issue got copyhardlinktree()
With the directory copy was added to avoid race issues, it wasn't noticed that tar was recursing the directories and copying files too. This is completely crazy when we hardlink those files in the next command.
Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto 256s -> 178s for example).
---
meta/lib/oe/path.py +1 -1
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst): if (os.stat(src).st_dev == os.stat(dst).st_dev): # Need to copy directories only with tar first since cp will error if two # writers try and create a directory at the same time - cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst) + cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) if os.path.isdir(src): src = src + "/*"
Quoted 7 lines · Richard Purdie
With the directory copy was added to avoid race issues, it wasn't noticed that tar was recursing the directories and copying files too. This is completely crazy when we hardlink those files in the next command.
Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (
do_packagefor linux-yocto 256s -> 178s for example).
Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)
Cheers,
Bruce
Quoted 18 lines
Signed-off-by Richard Purdie <[email protected]>---
meta/lib/oe/path.py +1
-1@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst): if (os.stat(src).st_dev == os.stat(dst).st_dev): # Need to copy directories only with tar first since cp will error if two # writers try and create a directory at the same time - cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst) + cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) if os.path.isdir(src): src = src + "/*"
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (
do_packagefor linux-yocto
256s -> 178s for example).Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)
Cheers,
Bruce
Quoted 18 lines
Signed-off-by Richard Purdie <[email protected]>---
meta/lib/oe/path.py +0
-0@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst): if (os.stat(src).st_dev == os.stat(dst).st_dev): # Need to copy directories only with tar first since cp willerror if two
# writers try and create a directory at the same timemeta/lib/oe/path.py +1
-3- cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst) + cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) if os.path.isdir(src):
Quoted 8 lines
Quoted 3 lines
src = src + "/*"_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
After this patch do_populate_sysroot fails:
ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Logfile of failure stored in:
/oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363 Log data follows:
| DEBUG: Executing python function sstate_task_prefunc | DEBUG: Python function sstate_task_prefunc finished | DEBUG: Executing python function do_populate_sysroot | DEBUG: Executing python function sysroot_stage_all | DEBUG: Python function sysroot_stage_all finished | DEBUG: Python function do_populate_sysroot finished | DEBUG: Executing python function do_qa_staging | NOTE: QA checking staging | DEBUG: Python function do_qa_staging finished | DEBUG: Executing python function sstate_task_postfunc | DEBUG: Staging files from /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/sysroot-destdir to /oe/oe-core/build/tmp-eglibc/sysroots/collie | DEBUG: Executing shell function kernelscripts_sstate_postinst | NOTE: make scripts | HOSTCC scripts/basic/fixdep | HOSTCC scripts/kconfig/conf.o | SHIPPED scripts/kconfig/zconf.tab.c | SHIPPED scripts/kconfig/zconf.lex.c | SHIPPED scripts/kconfig/zconf.hash.c | HOSTCC scripts/kconfig/zconf.tab.o | HOSTLD scripts/kconfig/conf | scripts/kconfig/conf --silentoldconfig Kconfig | *** | *** Configuration file ".config" not found! | *** | *** Please run some configurator (e.g. "make oldconfig" or | *** "make menuconfig" or "make xconfig"). | *** | make[2]: *** [silentoldconfig] Error 1 | make[1]: *** [silentoldconfig] Error 2 | make: *** No rule to make target `include/config/auto.conf', needed
by `scripts'. Stop.
| ERROR: oe_runmake failed
| WARNING: /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/run.kernelscripts_sstate_postinst.26363:1 exit 1 from
| [ "populate_sysroot" = "populate_sysroot" -o "populate_sysroot" =
"populate_sysroot_setscene" ]
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Task 5 (/oe/oe-core/meta/recipes-kernel/linux/linux-yocto_3.10.bb,do_populate_sysroot) failed with exit code '1'
Cheers
Andrea
Quoted 51 lines · Andrea Adami
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (
do_packagefor linux-yocto
256s -> 178s for example).Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)
Cheers,
Bruce
Quoted 18 lines
Signed-off-by Richard Purdie <[email protected]>---
meta/lib/oe/path.py +0
-0@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst): if (os.stat(src).st_dev == os.stat(dst).st_dev): # Need to copy directories only with tar first since cp willerror if two
# writers try and create a directory at the same timemeta/lib/oe/path.py +1
-3- cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst) + cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) if os.path.isdir(src):Quoted 8 lines
Quoted 3 lines
src = src + "/*"_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 1 line
After this patch
do_populate_sysrootfails:
This should fix it:
Cheers,
Richard
Quoted 54 lines · Richard Purdie
Quoted 51 lines · Andrea Adami
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie
With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (
do_packagefor linux-yocto
256s -> 178s for example).Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)
Cheers,
Bruce
Quoted 18 lines
Signed-off-by Richard Purdie <[email protected]>---
meta/lib/oe/path.py +0
-0@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst): if (os.stat(src).st_dev == os.stat(dst).st_dev): # Need to copy directories only with tar first since cp willerror if two
# writers try and create a directory at the same timemeta/lib/oe/path.py +1
-3- cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst) + cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) check_output(cmd, shell=True, stderr=subprocess.STDOUT) if os.path.isdir(src):Quoted 8 lines
Quoted 3 lines
src = src + "/*"_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-coreQuoted 1 line
After this patch
do_populate_sysrootfails:
Quoted 8 lines
This should fix it:
Cheers,
Richard
Yes, thx for the magic fix.
Now with all the last patches my results are much better:
do_install:
Elapsed time: 83.77 seconds CPU usage: 3.0% (worst before 131s) (avg before:103 seconds CPU usage: 2.9%)
do_populate_sysroot:
Elapsed time: 16.12 seconds CPU usage: 18.5%
(avg before: 95 seconds CPU usage: 6.5%)
Cheers
Andrea