Hob Cache for speed up switching
cover Hob Cache for speed up switching
From: Liping Ke <[email protected]>
The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.
The following changes since commit f974868cadf74d0cf8380b2c6a4152d2fa4590b5:
Saul Wold (1):
libx11: ensure nativesdk uses correct DEPENDS and XCB flags
are available in the git repository at:
git://git.pokylinux.org/poky-contrib lke/hob_cache
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/hob_cache
Liping Ke (2):
Use Pref as the centralized place for storing Hob attributes Cache target dependency tree for Hob
bitbake/lib/bb/ui/crumbs/hobeventhandler.py | 19 ++++++++++++++++--- bitbake/lib/bb/ui/crumbs/hobprefs.py | 3 ++- bitbake/lib/bb/ui/crumbs/tasklistmodel.py | 5 +++++ bitbake/lib/bb/ui/hob.py | 24 +++++++++++++++++++++--- 4 files changed, 44 insertions(+), 7 deletions(-)
PATCH 1/2 Hob Cache for speed up switching
From: Liping Ke <[email protected]>
This patch is the centralized place for storing Hob attributes. Most of informations in Prefs will be used when caching user-select packages information. Those information will be the cache hash-key which is useful in deciding whether the configuration combination value has been cached or not
--- bitbake/lib/bb/ui/crumbs/hobeventhandler.py | 6 +++--- bitbake/lib/bb/ui/crumbs/hobprefs.py | 3 ++- bitbake/lib/bb/ui/hob.py | 6 ++++-- 3 files changed, 9 insertions(+), 6 deletions(-)
bitbake/lib/bb/ui/crumbs/hobeventhandler.py +3 -3
@@ -99,7 +99,7 @@ class HobHandler(gobject.GObject): args.extend(argument) self.server.runCommand(args) - def handle_event(self, event, running_build, pbar): + def handle_event(self, event, running_build, pbar, hobprefs): if not event: return
@@ -161,11 +161,11 @@ class HobHandler(gobject.GObject): pbar.set_fraction(1.0) return - def event_handle_idle_func (self, eventHandler, running_build, pbar): + def event_handle_idle_func (self, eventHandler, running_build, pbar, hobrefs): # Consume as many messages as we can in the time available to us event = eventHandler.getEvent() while event: - self.handle_event(event, running_build, pbar) + self.handle_event(event, running_build, pbar, hobrefs) event = eventHandler.getEvent() return True
bitbake/lib/bb/ui/crumbs/hobprefs.py +2 -1
@@ -143,7 +143,7 @@ class HobPrefs(gtk.Dialog): glib.idle_add(self.handler.reload_data) def __init__(self, configurator, handler, curr_sdk_mach, curr_distro, pclass, - cpu_cnt, pmake, bbthread, image_types): + cpu_cnt, pmake, bbthread, image_types, curr_mach): """ """ gtk.Dialog.__init__(self, "Preferences", None,
@@ -159,6 +159,7 @@ class HobPrefs(gtk.Dialog): self.configurator = configurator self.curr_sdk_mach = curr_sdk_mach + self.curr_mach = curr_mach self.curr_distro = curr_distro self.curr_package_format = pclass self.curr_output_type = None
bitbake/lib/bb/ui/hob.py +4 -2
@@ -160,6 +160,7 @@ class MainWindow (gtk.Window): mach = combo.get_active_text() if mach != self.curr_mach: self.curr_mach = mach + self.prefs.curr_mach = mach # Flush this straight to the file as MACHINE is changed # independently of other 'Preferences' self.configurator.setLocalConfVar('MACHINE', mach)
@@ -887,7 +888,7 @@ def main (server, eventHandler): pkg, sep, pclass = pclasses[0].rpartition("_") prefs = HobPrefs(configurator, handler, sdk_mach, distro, pclass, cpu_cnt, - pmake, bbthread, image_types) + pmake, bbthread, image_types, mach) layers = LayerEditor(configurator, None) window = MainWindow(taskmodel, handler, configurator, prefs, layers, mach) prefs.set_parent_window(window)
@@ -918,7 +919,8 @@ def main (server, eventHandler): handler.event_handle_idle_func, eventHandler, window.build, - window.progress) + window.progress, + prefs) try: gtk.main()
Signature · 1 line
1.7.0.4
PATCH 2/2 Hob Cache for speed up switching
In reply to Ke LipingFrom: Liping Ke <[email protected]>
This patch is to cache target dependency tree for Hob (cache impl). Hob User might switch between different configuration combinations (machine, sdk-machine, distro, etc). Each dependency re-calcuation might be time consuming. We use a hashset to cache those result for avoiding recalc. Thus we can speed up UI data loading speed in some cases.
--- bitbake/lib/bb/ui/crumbs/hobeventhandler.py | 13 +++++++++++++ bitbake/lib/bb/ui/crumbs/tasklistmodel.py | 5 +++++ bitbake/lib/bb/ui/hob.py | 18 +++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletions(-)
bitbake/lib/bb/ui/crumbs/hobeventhandler.py +13 -0
@@ -110,6 +110,19 @@ class HobHandler(gobject.GObject): self.emit("data-generated") self.generating = False if event._model: + # Currently, use these three property as the key, in the future + # more elements might be needed + mach = hobprefs.curr_mach + if mach == None: + mach = "" + sdk_mach = hobprefs.curr_sdk_mach + if sdk_mach == None: + sdk_mach = "" + distro = hobprefs.curr_distro + if distro == None: + distro = "" + hash_key = mach + "," + sdk_mach + "," + distro + self.model.targetTreeCache[hash_key] = event._model self.model.populate(event._model) elif isinstance(event, bb.event.ConfigFilesFound): var = event._variable
bitbake/lib/bb/ui/crumbs/tasklistmodel.py +5 -0
@@ -102,6 +102,11 @@ class TaskListModel(gtk.ListStore): self.packages = None self.images = None self.selected_image = None + # This var is used for caching target package dependency data. This + # data will be impacted by layers, distros, machine, sdk_machine value + # which is now all saved in HobPrefs data. For the hashset cache, it's + # obvious that the key will be those properties combination. + self.targetTreeCache = {} gtk.ListStore.__init__ (self, gobject.TYPE_STRING,
bitbake/lib/bb/ui/hob.py +17 -1
@@ -166,7 +166,23 @@ class MainWindow (gtk.Window): self.configurator.setLocalConfVar('MACHINE', mach) self.configurator.writeLocalConf() handler.set_machine(mach) - handler.reload_data() + mach = self.prefs.curr_mach + if mach == None: + mach = "" + sdk_mach = self.prefs.curr_sdk_mach + if sdk_mach == None: + sdk_mach = "" + distro = self.prefs.curr_distro + if distro == None: + distro = "" + hash_key = mach + "," + sdk_mach + "," + distro + # Only when this target tree has not been cached, need we reload + # the target tree data + if (self.model.targetTreeCache.has_key(hash_key) == False): + handler.reload_data() + # Otherwise, simply reload the cache data + else: + handler.model.populate(self.model.targetTreeCache[hash_key]) def update_machines(self, handler, machines): active = 0
Signature · 1 line
1.7.0.4
Quoted 6 lines · Ke Liping
From: Liping Ke <[email protected]>
The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.
I'm still not certain about merging this change, do you feel it's still worth it once we've merged your change to reduce the tree generation time?
If so we need to revise the second patch to invalidate the cached value correctly. sdk-machine shouldn't affect the available packages but which layers are enabled certainly will.
Cheers,
Joshua
Signature · 3 lines
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology CentreHi, Josh
Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful. Because when using UI, users tend to switch between all combination of menu frequently.
Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.
Yes. Definitely we need to make sure the hash key was correct.
Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key. Then we rebase the patch and resend the patch. How do you think about it?
Thanks a lot for your help!
criping
Quoted 35 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Joshua Lock
Sent: Wednesday, July 20, 2011 8:42 AM
To: [email protected]
Subject: Re: [yocto] [PATCH 0/2][Image Creator] Hob Cache for speed up switchingQuoted 6 lines · Ke Liping
From: Liping Ke <[email protected]>
The two patches are the implmentation of Hob Cache for accelarating UI switching. If one configuration has been saved before, when user switch back to the same configuration combination, we will use the cache content instead of re-triggering parsing/loading phase.
I'm still not certain about merging this change, do you feel it's still worth it once we've merged your change to reduce the tree generation time?
If so we need to revise the second patch to invalidate the cached value correctly. sdk-machine shouldn't affect the available packages but which layers are enabled certainly will.
Cheers,
Joshua
--
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology Centre_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Quoted 10 lines · Liping
Hi, Josh
Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful. Because when using UI, users tend to switch between all combination of menu frequently.
Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.Yes. Definitely we need to make sure the hash key was correct.
Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key. Then we rebase the patch and resend the patch. How do you think about it?
I agree on reviewing the status later.
Thanks,
Joshua
Signature · 3 lines
Joshua Lock
Yocto Project "Johannes Factotum"
Intel Open Source Technology Centre