package: Add cachedpath optimisation
Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.
This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.
---
meta/classes/base.bbclass +1 -1
@@ -10,7 +10,7 @@ inherit utility-tasks inherit metadata_scm inherit logging -OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb" +OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath" OE_IMPORTS[type] = "list" def oe_import(d):
meta/classes/package.bbclass +8 -5
@@ -729,7 +729,7 @@ python split_and_strip_files () { baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True)) if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT', True) != '1') and \ (d.getVar('INHIBIT_PACKAGE_STRIP', True) != '1'): - for root, dirs, files in os.walk(dvar): + for root, dirs, files in oe.cachedpath.walk(dvar): for f in files: file = os.path.join(root, f) if file.endswith(".ko") and file.find("/lib/modules/") != -1:
@@ -743,7 +743,7 @@ python split_and_strip_files () { continue try: - ltarget = oe.path.realpath(file, dvar, False) + ltarget = oe.cachedpath.realpath(file, dvar, False) s = os.lstat(ltarget) except OSError, (err, strerror): if err != errno.ENOENT:
@@ -969,7 +969,7 @@ python populate_packages () { os.chdir(workdir) unshipped = [] - for root, dirs, files in os.walk(dvar): + for root, dirs, files in oe.cachedpath.walk(dvar): dir = root[len(dvar):] if not dir: dir = os.sep
@@ -1003,7 +1003,7 @@ python package_fixsymlinks () { for path in pkgfiles[pkg]: rpath = path[len(inst_root):] pkg_files[pkg].append(rpath) - rtarget = oe.path.realpath(path, inst_root, True, assume_dir = True) + rtarget = oe.cachedpath.realpath(path, inst_root, True, assume_dir = True) if not os.path.lexists(rtarget): dangling_links[pkg].append(os.path.normpath(rtarget[len(inst_root):]))
@@ -1775,6 +1775,9 @@ python do_package () { # as any change to rpmdeps requires this to be rerun. # PACKAGE_BBCLASS_VERSION = "1" + # Init cachedpath + oe.cachedpath.initcache() + ########################################################################### # Sanity test the setup ###########################################################################
@@ -1835,7 +1838,7 @@ python do_package () { pkgdest = d.getVar('PKGDEST', True) for pkg in packages: pkgfiles[pkg] = [] - for walkroot, dirs, files in os.walk(pkgdest + "/" + pkg): + for walkroot, dirs, files in oe.cachedpath.walk(pkgdest + "/" + pkg): for file in files: pkgfiles[pkg].append(walkroot + os.sep + file)
meta/lib/oe/cachedpath.py new file +190 -0
190 lines not on this page · raw message
Quoted 7 lines · Richard Purdie
Currently, various standard library operations like
os.walk(),os.path.isdir()andos.path.islink()each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like thedo_packagetask of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.
Quoted 7 lines
This patch adds
lib/oe/cachedpath.pywhich are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kerneldo_package.Signed-off-by Richard Purdie <[email protected]>
Have you considered having the initcache() return a cache object, either with the rest as methods on it, or pass that in? I think it'd be substantially cleaner than messing with globals, particularly given with globals we tend to have issues with cache lifetime and invalidation — if the cache was returned, it would go away when the object gets collected.
Signature · 1 line
Christopher Larson
Quoted 26 lines · Chris Larson
On Thu, Mar 14, 2013 at 6:57 PM, Richard Purdie <[email protected]> wrote:
Currently, various standard library operations likeos.walk(),os.path.isdir()andos.path.islink()each call stat or lstat which
involves a syscall into the kernel. There is no caching since they could
conceivably have changed on disk. The result is that for something like
thedo_packagetask of the kernel we're spending over two minutes making
868,000 individual stat calls for 23,000 files. This is suboptimal.This patch adds
lib/oe/cachedpath.pywhich are a set of replacement
functions for these operations which use cached stat data rather than
hitting the kernel each time. It gives a nice performance improvement
halving the build time of the kerneldo_package.Signed-off-by: Richard Purdie
<[email protected]>
Quoted 6 lines
Have you considered having the
initcache()return a cache object, either with the rest as methods on it, or pass that in? I think it'd be substantially cleaner than messing with globals, particularly given with globals we tend to have issues with cache lifetime and invalidation — if the cache was returned, it would go away when the object gets collected.
Agreed. Due to the way package.bbclass is structured, we currently need a global there but that is no reason we shouldn't have a sensible class/object structure for the new code. I've sent out a v2 which hopefully does better in that regard.
Cheers,
Richard