Skip to contentSkip to conversation

Hob Cache for speed up switching

PATCH In review Ke Liping 2 participants 6 messages Jul 11, 2011 – Jul 21, 2011 Arm STM32

lore
Share & tools
lore

Ke Liping

author
#0 lore raw

cover Hob Cache for speed up switching

From: Liping Ke <[email protected]>

The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.

The following changes since commit f974868cadf74d0cf8380b2c6a4152d2fa4590b5:
Saul Wold (1):
libx11: ensure nativesdk uses correct DEPENDS and XCB flags

are available in the git repository at:

git://git.pokylinux.org/poky-contrib lke/hob_cache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/hob_cache

Liping Ke (2):
Use Pref as the centralized place for storing Hob attributes Cache target dependency tree for Hob

 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |   19 ++++++++++++++++---
 bitbake/lib/bb/ui/crumbs/hobprefs.py        |    3 ++-
 bitbake/lib/bb/ui/crumbs/tasklistmodel.py   |    5 +++++
 bitbake/lib/bb/ui/hob.py                    |   24 +++++++++++++++++++++---
 4 files changed, 44 insertions(+), 7 deletions(-)
author
#1 lore raw

PATCH 1/2 Hob Cache for speed up switching

From: Liping Ke <[email protected]>

This patch is the centralized place for storing Hob attributes. Most of informations in Prefs will be used when caching user-select packages information. Those information will be the cache hash-key which is useful in deciding whether the configuration combination value has been cached or not

---
 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |    6 +++---
 bitbake/lib/bb/ui/crumbs/hobprefs.py        |    3 ++-
 bitbake/lib/bb/ui/hob.py                    |    6 ++++--
 3 files changed, 9 insertions(+), 6 deletions(-)
bitbake/lib/bb/ui/crumbs/hobeventhandler.py +3 -3
index fa79e0c..8c8d9e2 100644
@@ -99,7 +99,7 @@ class HobHandler(gobject.GObject):
             args.extend(argument)
             self.server.runCommand(args)
 
-    def handle_event(self, event, running_build, pbar):
+    def handle_event(self, event, running_build, pbar, hobprefs):
         if not event:
 	    return
 
@@ -161,11 +161,11 @@ class HobHandler(gobject.GObject):
             pbar.set_fraction(1.0)
         return
 
-    def event_handle_idle_func (self, eventHandler, running_build, pbar):
+    def event_handle_idle_func (self, eventHandler, running_build, pbar, hobrefs):
         # Consume as many messages as we can in the time available to us
         event = eventHandler.getEvent()
         while event:
-            self.handle_event(event, running_build, pbar)
+            self.handle_event(event, running_build, pbar, hobrefs)
             event = eventHandler.getEvent()
         return True
 
bitbake/lib/bb/ui/crumbs/hobprefs.py +2 -1
index f186410..05e8170 100644
@@ -143,7 +143,7 @@ class HobPrefs(gtk.Dialog):
             glib.idle_add(self.handler.reload_data)
 
     def __init__(self, configurator, handler, curr_sdk_mach, curr_distro, pclass,
-                 cpu_cnt, pmake, bbthread, image_types):
+                 cpu_cnt, pmake, bbthread, image_types, curr_mach):
         """
         """
         gtk.Dialog.__init__(self, "Preferences", None,
@@ -159,6 +159,7 @@ class HobPrefs(gtk.Dialog):
         self.configurator = configurator
 
         self.curr_sdk_mach = curr_sdk_mach
+        self.curr_mach = curr_mach
         self.curr_distro = curr_distro
         self.curr_package_format = pclass
         self.curr_output_type = None
bitbake/lib/bb/ui/hob.py +4 -2
index 06d936e..76cb294 100644
@@ -160,6 +160,7 @@ class MainWindow (gtk.Window):
         mach = combo.get_active_text()
         if mach != self.curr_mach:
             self.curr_mach = mach
+            self.prefs.curr_mach = mach
             # Flush this straight to the file as MACHINE is changed
             # independently of other 'Preferences'
             self.configurator.setLocalConfVar('MACHINE', mach)
@@ -887,7 +888,7 @@ def main (server, eventHandler):
     pkg, sep, pclass = pclasses[0].rpartition("_")
 
     prefs = HobPrefs(configurator, handler, sdk_mach, distro, pclass, cpu_cnt,
-                     pmake, bbthread, image_types)
+                     pmake, bbthread, image_types, mach)
     layers = LayerEditor(configurator, None)
     window = MainWindow(taskmodel, handler, configurator, prefs, layers, mach)
     prefs.set_parent_window(window)
@@ -918,7 +919,8 @@ def main (server, eventHandler):
                          handler.event_handle_idle_func,
                          eventHandler,
                          window.build,
-                         window.progress)
+                         window.progress,
+                         prefs)
 
     try:
         gtk.main()
Signature · 1 line
1.7.0.4
author
#2 lore raw

PATCH 2/2 Hob Cache for speed up switching

In reply to Ke Liping

From: Liping Ke <[email protected]>

This patch is to cache target dependency tree for Hob (cache impl). Hob User might switch between different configuration combinations (machine, sdk-machine, distro, etc). Each dependency re-calcuation might be time consuming. We use a hashset to cache those result for avoiding recalc. Thus we can speed up UI data loading speed in some cases.

---
 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |   13 +++++++++++++
 bitbake/lib/bb/ui/crumbs/tasklistmodel.py   |    5 +++++
 bitbake/lib/bb/ui/hob.py                    |   18 +++++++++++++++++-
 3 files changed, 35 insertions(+), 1 deletions(-)
bitbake/lib/bb/ui/crumbs/hobeventhandler.py +13 -0
index 8c8d9e2..c6d3bd5 100644
@@ -110,6 +110,19 @@ class HobHandler(gobject.GObject):
             self.emit("data-generated")
             self.generating = False
             if event._model:
+                # Currently, use these three property as the key, in the future
+                # more elements might be needed
+                mach = hobprefs.curr_mach
+                if mach == None:
+                    mach = ""
+                sdk_mach = hobprefs.curr_sdk_mach
+                if sdk_mach == None:
+                    sdk_mach = ""
+                distro = hobprefs.curr_distro
+                if distro == None:
+                    distro = ""
+                hash_key = mach + "," + sdk_mach + "," + distro
+                self.model.targetTreeCache[hash_key] = event._model 
                 self.model.populate(event._model)
         elif isinstance(event, bb.event.ConfigFilesFound):
             var = event._variable
bitbake/lib/bb/ui/crumbs/tasklistmodel.py +5 -0
index 5e979b7..bcd6328 100644
@@ -102,6 +102,11 @@ class TaskListModel(gtk.ListStore):
         self.packages = None
         self.images = None
         self.selected_image = None
+        # This var is used for caching target package dependency data. This
+        # data will be impacted by layers, distros, machine, sdk_machine value
+        # which is now all saved in HobPrefs data. For the hashset cache, it's
+        # obvious that the key will be those properties combination.
+        self.targetTreeCache = {}
         
         gtk.ListStore.__init__ (self,
                                 gobject.TYPE_STRING,
bitbake/lib/bb/ui/hob.py +17 -1
index 76cb294..5f6fdf5 100644
@@ -166,7 +166,23 @@ class MainWindow (gtk.Window):
             self.configurator.setLocalConfVar('MACHINE', mach)
             self.configurator.writeLocalConf()
             handler.set_machine(mach)
-            handler.reload_data()
+            mach = self.prefs.curr_mach
+            if mach == None:
+                mach = ""
+            sdk_mach = self.prefs.curr_sdk_mach
+            if sdk_mach == None:
+                sdk_mach = ""
+            distro = self.prefs.curr_distro
+            if distro == None:
+                distro = ""
+            hash_key = mach + "," + sdk_mach + "," + distro
+            # Only when this target tree has not been cached, need we reload
+            # the target tree data
+            if (self.model.targetTreeCache.has_key(hash_key) == False):
+                handler.reload_data()
+            # Otherwise, simply reload the cache data
+            else:
+                handler.model.populate(self.model.targetTreeCache[hash_key])
 
     def update_machines(self, handler, machines):
         active = 0
Signature · 1 line
1.7.0.4
#3 lore raw
In reply to Ke Liping
Quoted 6 lines · Ke Liping

From: Liping Ke <[email protected]>

The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.

I'm still not certain about merging this change, do you feel it's still worth it once we've merged your change to reduce the tree generation time?

If so we need to revise the second patch to invalidate the cached value correctly. sdk-machine shouldn't affect the available packages but which layers are enabled certainly will.

Cheers,
Joshua

Signature · 3 lines
Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre
author
#4 lore raw

Hi, Josh

Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful. Because when using UI, users tend to switch between all combination of menu frequently.
Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.

Yes. Definitely we need to make sure the hash key was correct.
Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key. Then we rebase the patch and resend the patch. How do you think about it?

Thanks a lot for your help!
criping

Quoted 35 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Joshua Lock
Sent: Wednesday, July 20, 2011 8:42 AM
To: [email protected]
Subject: Re: [yocto] [PATCH 0/2][Image Creator] Hob Cache for speed up switching

Quoted 6 lines · Ke Liping

From: Liping Ke <[email protected]>

The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.

I'm still not certain about merging this change, do you feel it's still worth it once we've merged your change to reduce the tree generation time?

If so we need to revise the second patch to invalidate the cached value correctly. sdk-machine shouldn't affect the available packages but which layers are enabled certainly will.

Cheers,
Joshua
--
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology Centre

_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#5 lore raw
Quoted 10 lines · Liping

Hi, Josh

Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful. Because when using UI, users tend to switch between all combination of menu frequently.
Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.

Yes. Definitely we need to make sure the hash key was correct.
Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key. Then we rebase the patch and resend the patch. How do you think about it?

I agree on reviewing the status later.

Thanks,
Joshua

Signature · 3 lines
Joshua Lock
        Yocto Project "Johannes Factotum"
        Intel Open Source Technology Centre