Performance improvement in install_sstatepkg

(quoted diff) +1 -0

+	sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" %

(sstateinst, fixmefn, sstate_sed_cmd)

Quoted 2 lines · On Thu, 10 May 2012 18:13:38 -0500 Mark Hatle <[email protected]> wrote:

(quoted diff) +1 -0

+	sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" %

(sstateinst, fixmefn, sstate_sed_cmd)

How confident are we that the file names can never have whitespace in them?

In a pathological case, lots of files to process, the sstate_installpkg performance was very poor. It interated over each file and ran 3 individual sed commands per file. Changing this to keep iterating but running only a single command took about 1/3 time time.

However, when looking at the corresponding sstate_hardcode_path function, it was clear we could optimize this further.

Using the same encoding logic to specify only the minimumal sed operation necessary, and using xargs to avoid the os.system call the install step was able to be performed in 13% of the original time.

Example timing numbers for perl:

3m7s original code
1m20s single sed, but interating
0m26s using xargs and limited sed

Signed-off-by Mark Hatle <[email protected]>

---
 meta/classes/sstate.bbclass |   28 +++++++++++++++++++++-------
 1 files changed, 21 insertions(+), 7 deletions(-)

meta/classes/sstate.bbclass +18 -7

index a8c98e5..ad7d121 100644
@@ -174,18 +174,29 @@ def sstate_installpkg(ss, d):
     bb.build.exec_func('sstate_unpack_package', d)
 
     # Fixup hardcoded paths
+    #
+    # Note: The logic below must match the reverse logic in
+    # sstate_hardcode_path(d)
+
     fixmefn =  sstateinst + "fixmepath"
     if os.path.isfile(fixmefn):
         staging = d.getVar('STAGING_DIR', True)
         staging_target = d.getVar('STAGING_DIR_TARGET', True)
         staging_host = d.getVar('STAGING_DIR_HOST', True)
-        fixmefd = open(fixmefn, "r")
-        fixmefiles = fixmefd.readlines()
-        fixmefd.close()
-        for file in fixmefiles:
-            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % (staging_target, sstateinst + file))
-            os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % (staging_host, sstateinst + file))
-            os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, sstateinst + file))
+
+	if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
+		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % (staging)
+	elif bb.data.inherits_class('cross', d):
+		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; s:FIXMESTAGINGDIR:%s:g'" % (staging_target, staging)
+	else:
+		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % (staging_host)
+
+	# Add sstateinst to each filename in fixmepath, use xargs to efficiently call sed
+	sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, fixmefn, sstate_sed_cmd)
+
+	print "Replacing fixme paths in sstate package: %s" % (sstate_hardcode_cmd)
+	os.system(sstate_hardcode_cmd)
+
         # Need to remove this or we'd copy it into the target directory and may 
         # conflict with another writer
         os.remove(fixmefn)

Significant performance improvement when large numbers of files are involved with install_sstatepkg.

To reproduce the problem:

bitbake perl
bitbake -c clean perl
time bitbake perl

The following changes since commit 910452727fc277c1caec7612b36c37b58d845350:

eglibc: Add patch to fix /var installation location (2012-05-09 21:42:37 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib mhatle/sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate

Mark Hatle (1):
sstate.bbclass: Improve sstate_installpkg performance

  meta/classes/sstate.bbclass |   28 +++++++++++++++++++++-------
  1 files changed, 21 insertions(+), 7 deletions(-)