Skip to contentSkip to conversation

package: Add cachedpath optimisation

PATCH In review Richard Purdie 2 participants 2 messages Mar 18, 2013

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 package: Add cachedpath optimisation

Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.

This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.

---

meta/classes/base.bbclass +1 -1
index 4ec1eda..5fe9a84 100644
@@ -10,7 +10,7 @@ inherit utility-tasks
 inherit metadata_scm
 inherit logging
 
-OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb"
+OE_IMPORTS += "os sys time oe.path oe.utils oe.data oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath"
 OE_IMPORTS[type] = "list"
 
 def oe_import(d):
meta/classes/package.bbclass +34 -24

176 lines not on this page · raw message

meta/lib/oe/cachedpath.py new file +236 -0

236 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · Richard Purdie

Currently, various standard library operations like os.walk(), os.path.isdir() and os.path.islink() each call stat or lstat which involves a syscall into the kernel. There is no caching since they could conceivably have changed on disk. The result is that for something like the do_package task of the kernel we're spending over two minutes making 868,000 individual stat calls for 23,000 files. This is suboptimal.

Quoted 7 lines

This patch adds lib/oe/cachedpath.py which are a set of replacement functions for these operations which use cached stat data rather than hitting the kernel each time. It gives a nice performance improvement halving the build time of the kernel do_package.

Looks good, nice job.

Signature · 1 line
Christopher Larson