Skip to contentSkip to conversation

package: Add cachedpath optimisation

PATCH Richard Purdie 2 participants 3 messages Mar 15, 2013 – Mar 18, 2013

lore
Share & tools
lore
#0 lore raw

Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.

This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.

---

meta/classes/base.bbclass +1 -1
index 4d69caf..9d5b0ad 100644
@@ -10,7 +10,7 @@ inherit utility-tasks
 inherit metadata_scm
 inherit logging
 
-OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath"
 OE_IMPORTS[type] = "list"
 
 def oe_import(d):
meta/classes/package.bbclass +8 -5
index 1625ebd..720e2b1 100644
@@ -729,7 +729,7 @@ python split_and_strip_files () {
     baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True))
     if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT', True) != '1') and \
             (d.getVar('INHIBIT_PACKAGE_STRIP', True) != '1'):
-        for root, dirs, files in os.walk(dvar):
+        for root, dirs, files in oe.cachedpath.walk(dvar):
             for f in files:
                 file = os.path.join(root, f)
                 if file.endswith(".ko") and file.find("/lib/modules/") != -1:
@@ -743,7 +743,7 @@ python split_and_strip_files () {
                     continue
 
                 try:
-                    ltarget = oe.path.realpath(file, dvar, False)
+                    ltarget = oe.cachedpath.realpath(file, dvar, False)
                     s = os.lstat(ltarget)
                 except OSError, (err, strerror):
                     if err != errno.ENOENT:
@@ -969,7 +969,7 @@ python populate_packages () {
     os.chdir(workdir)
 
     unshipped = []
-    for root, dirs, files in os.walk(dvar):
+    for root, dirs, files in oe.cachedpath.walk(dvar):
         dir = root[len(dvar):]
         if not dir:
             dir = os.sep
@@ -1003,7 +1003,7 @@ python package_fixsymlinks () {
         for path in pkgfiles[pkg]:
                 rpath = path[len(inst_root):]
                 pkg_files[pkg].append(rpath)
-                rtarget = oe.path.realpath(path, inst_root, True, assume_dir = True)
+                rtarget = oe.cachedpath.realpath(path, inst_root, True, assume_dir = True)
                 if not os.path.lexists(rtarget):
                     dangling_links[pkg].append(os.path.normpath(rtarget[len(inst_root):]))
 
@@ -1775,6 +1775,9 @@ python do_package () {
     # as any change to rpmdeps requires this to be rerun.
     # PACKAGE_BBCLASS_VERSION = "1"
 
+    # Init cachedpath
+    oe.cachedpath.initcache()
+
     ###########################################################################
     # Sanity test the setup
     ###########################################################################
@@ -1835,7 +1838,7 @@ python do_package () {
     pkgdest = d.getVar('PKGDEST', True)
     for pkg in packages:
         pkgfiles[pkg] = []
-        for walkroot, dirs, files in os.walk(pkgdest + "/" + pkg):
+        for walkroot, dirs, files in oe.cachedpath.walk(pkgdest + "/" + pkg):
             for file in files:
                 pkgfiles[pkg].append(walkroot + os.sep + file)
 
meta/lib/oe/cachedpath.py new file +190 -0

190 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · Richard Purdie

Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.

Quoted 7 lines

This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.

Have you considered having the initcache() return a cache object, either with the rest as methods on it, or pass that in? I think it'd be substantially cleaner than messing with globals, particularly given with globals we tend to have issues with cache lifetime and invalidation — if the cache was returned, it would go away when the object gets collected.

Signature · 1 line
Christopher Larson
#2 lore raw
Quoted 26 lines · Chris Larson

On Thu, Mar 14, 2013 at 6:57 PM, Richard Purdie <[email protected]> wrote:
Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which
involves a syscall into the kernel. There is no caching since they could
conceivably have changed on disk. The result is that for something like
the do_package task of the kernel we're spending over two minutes making
868,000 individual stat calls for 23,000 files. This is suboptimal.

This patch adds lib/oe/cachedpath.py which are a set of replacement
functions for these operations which use cached stat data rather than
hitting the kernel each time. It gives a nice performance improvement
halving the build time of the kernel do_package.

Signed-off-by: Richard Purdie
<[email protected]>

Quoted 6 lines

Have you considered having the initcache() return a cache object, either with the rest as methods on it, or pass that in? I think it'd be substantially cleaner than messing with globals, particularly given with globals we tend to have issues with cache lifetime and invalidation — if the cache was returned, it would go away when the object gets collected.

Agreed. Due to the way package.bbclass is structured, we currently need a global there but that is no reason we shouldn't have a sensible class/object structure for the new code. I've sent out a v2 which hopefully does better in that regard.

Cheers,

Richard

Quoted 1 line