Skip to contentSkip to conversation

lib/oe/path: Fix performance issue got copyhardlinktree()

PATCH In review Richard Purdie 3 participants 5 messages Nov 8, 2013 – Nov 9, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lib/oe/path: Fix performance issue got copyhardlinktree()

With the directory copy was added to avoid race issues, it wasn't noticed that tar was recursing the directories and copying files too. This is completely crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto 256s -> 178s for example).

---

meta/lib/oe/path.py +1 -1
index 1310e38..d0588ba 100644
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two 
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
         check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         if os.path.isdir(src):
             src = src + "/*"
#1 lore raw
Quoted 7 lines · Richard Purdie

With the directory copy was added to avoid race issues, it wasn't noticed that tar was recursing the directories and copying files too. This is completely crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto 256s -> 178s for example).

Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)

Cheers,

Bruce

Quoted 18 lines

---

meta/lib/oe/path.py +1 -1
index 1310e38..d0588ba 100644
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
          # Need to copy directories only with tar first since cp will error if two
          # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
          check_output(cmd, shell=True, stderr=subprocess.STDOUT)
          if os.path.isdir(src):
              src = src + "/*"
#2 lore raw
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie

With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto
256s -> 178s for example).

Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)

Cheers,

Bruce

Quoted 18 lines

---

meta/lib/oe/path.py +0 -0
index 1310e38..d0588ba 100644
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
          # Need to copy directories only with tar first since cp will

error if two

          # writers try and create a directory at the same time
meta/lib/oe/path.py +1 -3
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
          check_output(cmd, shell=True, stderr=subprocess.STDOUT)
          if os.path.isdir(src):
Quoted 8 lines
Quoted 3 lines
              src = src + "/*"

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

After this patch do_populate_sysroot fails:

ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Logfile of failure stored in:
/oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363 Log data follows:

| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_populate_sysroot
| DEBUG: Executing python function sysroot_stage_all
| DEBUG: Python function sysroot_stage_all finished
| DEBUG: Python function do_populate_sysroot finished
| DEBUG: Executing python function do_qa_staging
| NOTE: QA checking staging
| DEBUG: Python function do_qa_staging finished
| DEBUG: Executing python function sstate_task_postfunc
| DEBUG: Staging files from
/oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/sysroot-destdir
to /oe/oe-core/build/tmp-eglibc/sysroots/collie
| DEBUG: Executing shell function kernelscripts_sstate_postinst
| NOTE: make scripts
|   HOSTCC  scripts/basic/fixdep
|   HOSTCC  scripts/kconfig/conf.o
|   SHIPPED scripts/kconfig/zconf.tab.c
|   SHIPPED scripts/kconfig/zconf.lex.c
|   SHIPPED scripts/kconfig/zconf.hash.c
|   HOSTCC  scripts/kconfig/zconf.tab.o
|   HOSTLD  scripts/kconfig/conf
| scripts/kconfig/conf --silentoldconfig Kconfig
| ***
| *** Configuration file ".config" not found!
| ***
| *** Please run some configurator (e.g. "make oldconfig" or
| *** "make menuconfig" or "make xconfig").
| ***
| make[2]: *** [silentoldconfig] Error 1
| make[1]: *** [silentoldconfig] Error 2
| make: *** No rule to make target `include/config/auto.conf', needed

by `scripts'. Stop.
| ERROR: oe_runmake failed
| WARNING: /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/run.kernelscripts_sstate_postinst.26363:1 exit 1 from
| [ "populate_sysroot" = "populate_sysroot" -o "populate_sysroot" =
"populate_sysroot_setscene" ]
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Task 5 (/oe/oe-core/meta/recipes-kernel/linux/linux-yocto_3.10.bb,
do_populate_sysroot) failed with exit code '1'

Cheers

Andrea

#3 lore raw
Quoted 51 lines · Andrea Adami
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie

With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto
256s -> 178s for example).

Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)

Cheers,

Bruce

Quoted 18 lines

---

meta/lib/oe/path.py +0 -0
index 1310e38..d0588ba 100644
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
          # Need to copy directories only with tar first since cp will

error if two

          # writers try and create a directory at the same time
meta/lib/oe/path.py +1 -3
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
          check_output(cmd, shell=True, stderr=subprocess.STDOUT)
          if os.path.isdir(src):
Quoted 8 lines
Quoted 3 lines
              src = src + "/*"

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 1 line

After this patch do_populate_sysroot fails:

This should fix it:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=5237c8e2c1780983c000247ffa0c890b4bb5980a

Cheers,

Richard

#4 lore raw
Quoted 54 lines · Richard Purdie
Quoted 51 lines · Andrea Adami
Quoted 40 lines · Bruce Ashfield
Quoted 11 lines · Richard Purdie

With the directory copy was added to avoid race issues, it wasn't noticed that
tar was recursing the directories and copying files too. This is completely
crazy when we hardlink those files in the next command.

Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto
256s -> 178s for example).

Every second makes a difference in this beast .. when adding up a zillion package runs a week. :)

Cheers,

Bruce

Quoted 18 lines

---

meta/lib/oe/path.py +0 -0
index 1310e38..d0588ba 100644
@@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
          # Need to copy directories only with tar first since cp will

error if two

          # writers try and create a directory at the same time
meta/lib/oe/path.py +1 -3
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
          check_output(cmd, shell=True, stderr=subprocess.STDOUT)
          if os.path.isdir(src):
Quoted 8 lines
Quoted 3 lines
              src = src + "/*"

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 1 line

After this patch do_populate_sysroot fails:

Quoted 8 lines

This should fix it:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=5237c8e2c1780983c000247ffa0c890b4bb5980a

Cheers,

Richard

Yes, thx for the magic fix.
Now with all the last patches my results are much better:

do_install:
Elapsed time: 83.77 seconds CPU usage: 3.0% (worst before 131s) (avg before:103 seconds CPU usage: 2.9%)

do_populate_sysroot:
Elapsed time: 16.12 seconds CPU usage: 18.5%
(avg before: 95 seconds CPU usage: 6.5%)

Cheers

Andrea