package: Add cachedpath optimisation
PATCH v2 1/1 package: Add cachedpath optimisation
Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.
This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.
---
meta/classes/base.bbclass +1 -1
@@ -10,7 +10,7 @@ inherit utility-tasks inherit metadata_scm inherit logging -OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb" +OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath" OE_IMPORTS[type] = "list" def oe_import(d):
meta/classes/package.bbclass +34 -24
176 lines not on this page · raw message
meta/lib/oe/cachedpath.py new file +236 -0
236 lines not on this page · raw message
Quoted 7 lines · Richard Purdie
Currently, various standard library operations like
os.walk(),os.path.isdir()andos.path.islink()each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like thedo_packagetask of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.
Quoted 7 lines
This patch adds
lib/oe/cachedpath.pywhich are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kerneldo_package.Signed-off-by Richard Purdie <[email protected]>
Looks good, nice job.
Signature · 1 line
Christopher Larson