Skip to contentSkip to conversation

codeparser cache improvements

PATCH New Richard Purdie 1 participant 1 message Jul 25, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 codeparser cache improvements

It turns out the codeparser cache is the bottleneck I've been observing when running bitbake commands, particularly as it grows. There are some things we can do about this:

* We were processing the cache with "intern()" at save time. Its actually much more memory efficient to do this at creation time.

* Use hashable objects such as frozenset rather than set so that we can compare objects

* De-duplicate the cache objects, link duplicates to the same object saving memory and disk usage and improving speed

* Using custom setstate/getstate to avoid the overhead of object attribute names in the cache file

To make this work, a global cache was needed for the list of set objects as this was the only way I could find to get the data in at setstate object creation time :(.

Parsing shows a modest improvement with these changes, cache load time is significantly better, cache save time is reduced since there is now no need to reprocess the data and cache is much smaller.

We can drop the compress_keys() code and internSet code from the shared cache core since its no longer used and replaced by codeparser specific pieces.

bitbake/lib/bb/cache.py +0 -12
index c7f3b7a..f892d7d 100644
@@ -764,16 +764,6 @@ class MultiProcessCache(object):
 
         self.cachedata = data
 
-    def internSet(self, items):
-        new = set()
-        for i in items:
-            new.add(intern(i))
-        return new
-
-    def compress_keys(self, data):
-        # Override in subclasses if desired
-        return
-
     def create_cachedata(self):
         data = [{}]
         return data
@@ -833,8 +823,6 @@ class MultiProcessCache(object):
             self.merge_data(extradata, data)
             os.unlink(f)
 
-        self.compress_keys(data)
-
         with open(self.cachefile, "wb") as f:
             p = pickle.Pickler(f, -1)
             p.dump([data, self.__class__.CACHE_VERSION])
bitbake/lib/bb/codeparser.py +109 -34

192 lines not on this page · raw message