Further optimizations to the sstate.bbclass

A few minor changes to the generation of the fixmepaths file in the sstate cache file generation can lead to significant performance improvements in recipes that generate large file sets.

The following changes since commit 9cd30beba77497288eeb2545920bc23f2a77cf16:

.gitignore: add /bitbake to the ignore list for external distributions (2012-05-11 17:59:53 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib mhatle/sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate

Mark Hatle (1):
sstate.bbclass: Optimize the generation and install path fixups

 meta/classes/sstate.bbclass |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

1.7.3.4

The fixmepath file that is generated contains a list of all of the files that need their paths fixed. In the previous version the fixmepath was generated to include all of the files that sed may have changed. In the new version, we first grep the files to see if they contain a path that needs to be changed, only then do we perform the sed operation on those files.

This results in a modest performance increate in the creation of the sstate file. The following numbers include the do_package and do_populate_sysroot tasks on the perl recipe.

Before the change:

real    4m23.018s
user    1m57.067s
sys     1m33.327s

After the change:

real    4m13.083s
user    1m54.062s
sys     1m26.064s

However, a more significnt performance gain is felt during the extraction/install of sstate cache files, as the fixmepaths file now has a significantly smaller list of files to modify.

Before the change:

real    0m39.798s
user    0m11.158s
sys     0m12.642s

After the change:

real    0m25.511s
user    0m8.408s
sys     0m5.077s

(All numbers above were recorded with a cold filesystem cache on a machine with 12 GB of ram.)

---
 meta/classes/sstate.bbclass |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

index ad7d121..3fc615d 100644
@@ -321,10 +321,13 @@ def sstate_hardcode_path(d):
 	sstate_builddir = d.getVar('SSTATE_BUILDDIR', True)
 
 	if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
+		sstate_grep_cmd = "grep -l -e '%s'" % (staging)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIR:g'" % (staging)
 	elif bb.data.inherits_class('cross', d):
+		sstate_grep_cmd = "grep -l -e '(%s|%s)'" % (staging_target, staging)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRTARGET:g; s:%s:FIXMESTAGINGDIR:g'" % (staging_target, staging)
 	else:
+		sstate_grep_cmd = "grep -l -e '%s'" % (staging_host)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRHOST:g'" % (staging_host)
 	
 	sstate_scan_cmd = d.getVar('SSTATE_SCAN_CMD', True)

@@ -333,7 +336,9 @@ def sstate_hardcode_path(d):
 	# fixmepath file needs relative paths, drop sstate_builddir prefix
 	sstate_filelist_relative_cmd = "sed -i -e 's:^%s::g' %sfixmepath" % (sstate_builddir, sstate_builddir)
 
-	sstate_hardcode_cmd = "%s | %s | xargs %s" % (sstate_scan_cmd, sstate_filelist_cmd, sstate_sed_cmd)
+	# Limit the fixpaths and sed operations based on the initial grep search
+	# This has the side effect of making sure the vfs cache is hot
+	sstate_hardcode_cmd = "%s | xargs %s | %s | xargs %s" % (sstate_scan_cmd, sstate_grep_cmd, sstate_filelist_cmd, sstate_sed_cmd)
 
 	print "Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)
 	os.system(sstate_hardcode_cmd)

1.7.3.4

Merged into OE-Core

Thanks
Sau!