hob changes for Yocto 1.1
cover hob changes for Yocto 1.1
This series addresses issues for the looming Yocto 1.1 code freeze. This means that the series contains a 3 genuine patches followed by 3 hide my shame patches to disable features which either a) weren't implmented in time or b) weren't deemed to be stable enough for inclusion in the 1.1 release.
Regards,
Joshua
The following changes since commit 639db8c766cada7180f9447f51303f9b30d7e817:
fetch2/git: Allow to specify the name of the checkout directory (2011-09-02 14:23:43 +0100)
are available in the git repository at:
git://github.com/incandescant/bitbake hob
https://github.com/incandescant/bitbake/tree/hob
Joshua Lock (6):
hob: use both pre and post files for hob configuration
hob: reflect defaultsetup being default distro
hob: add a test to ensure hob is run with the required pre and post
files
ui/crumbs/hobprefs: disable 'build toolchain with headers' ui/crumbs/runningbuild: mask run_buildstats failure
hob: disable removal of packages
lib/bb/ui/crumbs/configurator.py | 94 +++++++++++++++++++++++----------- lib/bb/ui/crumbs/hobeventhandler.py | 11 +++-- lib/bb/ui/crumbs/hobprefs.py | 34 ++++++------ lib/bb/ui/crumbs/runningbuild.py | 6 ++ lib/bb/ui/hob.py | 69 ++++++++++++++++++------- 5 files changed, 142 insertions(+), 72 deletions(-)
Signature · 1 line
1.7.6
PATCH 1/6 hob: use both pre and post files for hob configuration
We need to set various variables *before* parse begins, the simplest way to ensure this is to use a pre configuration file for the relevant configuration entries.
This series adapts hob to use both pre and post files to store its configuration. Any variables which affect initial parse are set in the pre file and all others in the post file.
Unfortunately this requires hob related code to have even more hard-coded data as to what is relevant but this is the simplest way to solve issues with variables and parse order at this time.
Addresses [YOCTO #1281]
--- lib/bb/ui/crumbs/configurator.py | 94 +++++++++++++++++++++++----------- lib/bb/ui/crumbs/hobeventhandler.py | 11 +++-- lib/bb/ui/crumbs/hobprefs.py | 22 ++++---- lib/bb/ui/hob.py | 4 +- 4 files changed, 83 insertions(+), 48 deletions(-)
lib/bb/ui/crumbs/configurator.py +63 -31
@@ -40,12 +40,13 @@ class Configurator(gobject.GObject): def __init__(self): gobject.GObject.__init__(self) - self.local = None self.bblayers = None self.enabled_layers = {} self.loaded_layers = {} self.config = {} self.orig_config = {} + self.preconf = None + self.postconf = None # NOTE: cribbed from the cooker... def _parse(self, f, data, include=False):
@@ -55,18 +56,16 @@ class Configurator(gobject.GObject): parselog.critical("Unable to parse %s: %s" % (f, exc)) sys.exit(1) - def _loadLocalConf(self, path): + def _loadConf(self, path): def getString(var): return bb.data.getVar(var, data, True) or "" - self.local = path - if self.orig_config: del self.orig_config self.orig_config = {} data = bb.data.init() - data = self._parse(self.local, data) + data = self._parse(path, data) # We only need to care about certain variables mach = getString('MACHINE')
@@ -76,6 +75,8 @@ class Configurator(gobject.GObject): if sdkmach and sdkmach != self.config.get('SDKMACHINE', ''): self.config['SDKMACHINE'] = sdkmach distro = getString('DISTRO') + if not distro: + distro = "defaultsetup" if distro and distro != self.config.get('DISTRO', ''): self.config['DISTRO'] = distro bbnum = getString('BB_NUMBER_THREADS')
@@ -109,10 +110,10 @@ class Configurator(gobject.GObject): self.orig_config = copy.deepcopy(self.config) - def setLocalConfVar(self, var, val): + def setConfVar(self, var, val): self.config[var] = val - def getLocalConfVar(self, var): + def getConfVar(self, var): if var in self.config: return self.config[var] else:
@@ -135,9 +136,17 @@ class Configurator(gobject.GObject): self.emit("layers-loaded") def _addConfigFile(self, path): + conffiles = ["local.conf", "hob-pre.conf", "hob-post.conf"] pref, sep, filename = path.rpartition("/") - if filename == "local.conf" or filename == "hob.local.conf": - self._loadLocalConf(path) + + if filename == "hob-pre.conf": + self.preconf = path + + if filename == "hob-post.conf": + self.postconf = path + + if filename in conffiles: + self._loadConf(path) elif filename == "bblayers.conf": self._loadLayerConf(path)
@@ -220,22 +229,8 @@ class Configurator(gobject.GObject): with open(conffile, "w") as new: new.write("".join(contents)) - def writeLocalConf(self): - # Dictionary containing only new or modified variables - changed_values = {} - for var in self.config: - val = self.config[var] - if self.orig_config.get(var, None) != val: - changed_values[var] = val - - if not len(changed_values): - return - - # read the original conf into a list - with open(self.local, 'r') as config: - config_lines = config.readlines() - - new_config_lines = ["\n"] + def updateConf(self, orig_lines, changed_values): + new_config_lines = [] for var in changed_values: # Convenience function for re.subn(). If the pattern matches # return a string which contains an assignment using the same
@@ -254,10 +249,10 @@ class Configurator(gobject.GObject): # Iterate over the local.conf lines and if they are a match # for the pattern comment out the line and append a new line # with the new VAR op "value" entry - for line in config_lines: + for line in orig_lines: new_line, replacements = p.subn(replace_val, line) if replacements: - config_lines[cnt] = "#%s" % line + orig_lines[cnt] = "#%s" % line new_config_lines.append(new_line) replaced = True cnt = cnt + 1
@@ -266,16 +261,53 @@ class Configurator(gobject.GObject): new_config_lines.append("%s = \"%s\"\n" % (var, changed_values[var])) # Add the modified variables - config_lines.extend(new_config_lines) + orig_lines.extend(new_config_lines) + return orig_lines + + def writeConf(self): + pre_vars = ["MACHINE", "SDKMACHINE", "DISTRO", + "INCOMPATIBLE_LICENSE"] + post_vars = ["BB_NUMBER_THREADS", "PARALLEL_MAKE", "PACKAGE_CLASSES", + "IMAGE_FSTYPES", "HOB_BUILD_TOOLCHAIN", + "HOB_BUILD_TOOLCHAIN_HEADERS"] + pre_values = {} + post_values = {} + changed_values = {} + pre_lines = None + post_lines = None - self.writeConfFile(self.local, config_lines) + for var in self.config: + val = self.config[var] + if self.orig_config.get(var, None) != val: + changed_values[var] = val + + if not len(changed_values): + return + + for var in changed_values: + if var in pre_vars: + pre_values[var] = changed_values[var] + elif var in post_vars: + post_values[var] = changed_values[var] + + with open(self.preconf, 'r') as pre: + pre_lines = pre.readlines() + pre_lines = self.updateConf(pre_lines, pre_values) + if len(pre_lines): + self.writeConfFile(self.preconf, pre_lines) + + with open(self.postconf, 'r') as post: + post_lines = post.readlines() + post_lines = self.updateConf(post_lines, post_values) + if len(post_lines): + self.writeConfFile(self.postconf, post_lines) del self.orig_config self.orig_config = copy.deepcopy(self.config) def insertTempBBPath(self, bbpath, bbfiles): # read the original conf into a list - with open(self.local, 'r') as config: + with open(self.postconf, 'r') as config: config_lines = config.readlines() if bbpath:
@@ -283,7 +315,7 @@ class Configurator(gobject.GObject): if bbfiles: config_lines.append("BBFILES := \"${BBFILES} %s\"\n" % bbfiles) - self.writeConfFile(self.local, config_lines) + self.writeConfFile(self.postconf, config_lines) def writeLayerConf(self): # If we've not added/removed new layers don't write
lib/bb/ui/crumbs/hobeventhandler.py +7 -4
@@ -65,7 +65,7 @@ class HobHandler(gobject.GObject): gobject.TYPE_STRING,)), } - (CFG_PATH_LOCAL, CFG_PATH_HOB, CFG_PATH_LAYERS, CFG_FILES_DISTRO, CFG_FILES_MACH, CFG_FILES_SDK, FILES_MATCH_CLASS, GENERATE_TGTS, REPARSE_FILES, BUILD_IMAGE) = range(10) + (CFG_PATH_LOCAL, CFG_PATH_PRE, CFG_PATH_POST, CFG_PATH_LAYERS, CFG_FILES_DISTRO, CFG_FILES_MACH, CFG_FILES_SDK, FILES_MATCH_CLASS, GENERATE_TGTS, REPARSE_FILES, BUILD_IMAGE) = range(11) def __init__(self, taskmodel, server): gobject.GObject.__init__(self)
@@ -90,9 +90,12 @@ class HobHandler(gobject.GObject): self.generating = True if self.current_command == self.CFG_PATH_LOCAL: - self.current_command = self.CFG_PATH_HOB - self.server.runCommand(["findConfigFilePath", "hob.local.conf"]) - elif self.current_command == self.CFG_PATH_HOB: + self.current_command = self.CFG_PATH_PRE + self.server.runCommand(["findConfigFilePath", "hob-pre.conf"]) + elif self.current_command == self.CFG_PATH_PRE: + self.current_command = self.CFG_PATH_POST + self.server.runCommand(["findConfigFilePath", "hob-post.conf"]) + elif self.current_command == self.CFG_PATH_POST: self.current_command = self.CFG_PATH_LAYERS self.server.runCommand(["findConfigFilePath", "bblayers.conf"]) elif self.current_command == self.CFG_PATH_LAYERS:
lib/bb/ui/crumbs/hobprefs.py +11 -11
@@ -38,13 +38,13 @@ class HobPrefs(gtk.Dialog): else: self.selected_image_types = handler.remove_image_output_type(ot) - self.configurator.setLocalConfVar('IMAGE_FSTYPES', "%s" % " ".join(self.selected_image_types).lstrip(" ")) + self.configurator.setConfVar('IMAGE_FSTYPES', "%s" % " ".join(self.selected_image_types).lstrip(" ")) def sdk_machine_combo_changed_cb(self, combo, handler): sdk_mach = combo.get_active_text() if sdk_mach != self.curr_sdk_mach: self.curr_sdk_mach = sdk_mach - self.configurator.setLocalConfVar('SDKMACHINE', sdk_mach) + self.configurator.setConfVar('SDKMACHINE', sdk_mach) handler.set_sdk_machine(sdk_mach) def update_sdk_machines(self, handler, sdk_machines):
@@ -67,7 +67,7 @@ class HobPrefs(gtk.Dialog): distro = combo.get_active_text() if distro != self.curr_distro: self.curr_distro = distro - self.configurator.setLocalConfVar('DISTRO', distro) + self.configurator.setConfVar('DISTRO', distro) handler.set_distro(distro) self.reload_required = True
@@ -91,7 +91,7 @@ class HobPrefs(gtk.Dialog): package_format = combo.get_active_text() if package_format != self.curr_package_format: self.curr_package_format = package_format - self.configurator.setLocalConfVar('PACKAGE_CLASSES', 'package_%s' % package_format) + self.configurator.setConfVar('PACKAGE_CLASSES', 'package_%s' % package_format) handler.set_package_format(package_format) self.reload_required = True
@@ -113,7 +113,7 @@ class HobPrefs(gtk.Dialog): def include_gplv3_cb(self, toggle): excluded = toggle.get_active() - orig_incompatible = self.configurator.getLocalConfVar('INCOMPATIBLE_LICENSE') + orig_incompatible = self.configurator.getConfVar('INCOMPATIBLE_LICENSE') new_incompatible = "" if excluded: if not orig_incompatible:
@@ -125,18 +125,18 @@ class HobPrefs(gtk.Dialog): if new_incompatible != orig_incompatible: self.handler.set_incompatible_license(new_incompatible) - self.configurator.setLocalConfVar('INCOMPATIBLE_LICENSE', new_incompatible) + self.configurator.setConfVar('INCOMPATIBLE_LICENSE', new_incompatible) self.reload_required = True def change_bb_threads_cb(self, spinner): val = spinner.get_value_as_int() self.handler.set_bbthreads(val) - self.configurator.setLocalConfVar('BB_NUMBER_THREADS', val) + self.configurator.setConfVar('BB_NUMBER_THREADS', val) def change_make_threads_cb(self, spinner): val = spinner.get_value_as_int() self.handler.set_pmake(val) - self.configurator.setLocalConfVar('PARALLEL_MAKE', "-j %s" % val) + self.configurator.setConfVar('PARALLEL_MAKE', "-j %s" % val) def toggle_toolchain_cb(self, check): enabled = check.get_active()
@@ -144,7 +144,7 @@ class HobPrefs(gtk.Dialog): if enabled: toolchain = '1' self.handler.toggle_toolchain(enabled) - self.configurator.setLocalConfVar('HOB_BUILD_TOOLCHAIN', toolchain) + self.configurator.setConfVar('HOB_BUILD_TOOLCHAIN', toolchain) def toggle_headers_cb(self, check): enabled = check.get_active()
@@ -152,13 +152,13 @@ class HobPrefs(gtk.Dialog): if enabled: headers = '1' self.handler.toggle_toolchain_headers(enabled) - self.configurator.setLocalConfVar('HOB_BUILD_TOOLCHAIN_HEADERS', headers) + self.configurator.setConfVar('HOB_BUILD_TOOLCHAIN_HEADERS', headers) def set_parent_window(self, parent): self.set_transient_for(parent) def write_changes(self): - self.configurator.writeLocalConf() + self.configurator.writeConf() def prefs_response_cb(self, dialog, response): if self.reload_required:
lib/bb/ui/hob.py +2 -2
@@ -192,8 +192,8 @@ class MainWindow (gtk.Window): self.curr_mach = mach # Flush this straight to the file as MACHINE is changed # independently of other 'Preferences' - self.configurator.setLocalConfVar('MACHINE', mach) - self.configurator.writeLocalConf() + self.configurator.setConfVar('MACHINE', mach) + self.configurator.writeConf() handler.set_machine(mach) handler.reload_data()
Signature · 1 line
1.7.6
PATCH 2/6 hob: reflect defaultsetup being default distro
In reply to Joshua LockIf no value is set for DISTRO the defaultsetup policy is used, reflect this in the UI by having defaultsetup selected in the Distribution combo when no other DISTRO is set.
--- lib/bb/ui/hob.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
lib/bb/ui/hob.py +2 -0
@@ -992,6 +992,8 @@ def main (server, eventHandler): if not sdk_mach: sdk_mach = server.runCommand(["getVariable", "SDK_ARCH"]) distro = server.runCommand(["getVariable", "DISTRO"]) + if not distro: + distro = "defaultsetup" bbthread = server.runCommand(["getVariable", "BB_NUMBER_THREADS"]) if not bbthread: bbthread = 1
Signature · 1 line
1.7.6
PATCH 3/6 hob: add a test to ensure hob is run with the required pre and post files
In reply to Joshua Lockhob requires pre and post configuration files to store configuration values in, whilst this should (and will) be fixed long-term for so long as we require these files we should alert the user should they run without them.
Fixes [YOCTO #1383]
--- lib/bb/ui/hob.py | 21 +++++++++++++++++++++ 1 files changed, 21 insertions(+), 0 deletions(-)
lib/bb/ui/hob.py +21 -0
@@ -982,6 +982,27 @@ class MainWindow (gtk.Window): def main (server, eventHandler): gobject.threads_init() + # NOTE: For now we require that the user run with pre and post files to + # read and store configuration set in the GUI. + # We hope to adjust this long term as tracked in Yocto Bugzilla #1441 + # http://bugzilla.pokylinux.org/show_bug.cgi?id=1441 + reqfiles = 0 + dep_files = server.runCommand(["getVariable", "__depends"]) or set() + dep_files.union(server.runCommand(["getVariable", "__base_depends"]) or set()) + for f in dep_files: + if f[0].endswith("hob-pre.conf"): + reqfiles = reqfiles + 1 + elif f[0].endswith("hob-post.conf"): + reqfiles = reqfiles + 1 + if reqfiles == 2: + break + if reqfiles < 2: + print("""The hob UI requires a pre file named hob-pre.conf and a post +file named hob-post.conf to store and read its configuration from. Please run +hob with these files, i.e.\n +\bitbake -u hob -r conf/hob-pre.conf -R conf/hob-post.conf""") + return + taskmodel = TaskListModel() configurator = Configurator() handler = HobHandler(taskmodel, server)
Signature · 1 line
1.7.6
PATCH 4/6 ui/crumbs/hobprefs: disable 'build toolchain with headers'
In reply to Joshua LockThe generic meta-toolchain-sdk we are currently building when this option is enabled is likely unsuitable for the majority of images built with hob. Remove this option from the Preferences UI until such a time as we can correctly implement this feature to include the library headers for the selected packages.
Addresses [YOCTO #1302]
--- lib/bb/ui/crumbs/hobprefs.py | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
lib/bb/ui/crumbs/hobprefs.py +6 -6
@@ -326,9 +326,9 @@ class HobPrefs(gtk.Dialog): self.sdk_machine_combo.set_tooltip_text("Select the host architecture of the external machine") self.sdk_machine_combo.show() hbox.pack_start(self.sdk_machine_combo, expand=False, fill=False, padding=6) - headerscheck = gtk.CheckButton("Include development headers with toolchain") - headerscheck.show() - headerscheck.set_active(self.build_toolchain_headers) - headerscheck.connect("toggled", self.toggle_headers_cb) - hbox.pack_start(headerscheck, expand=False, fill=False, padding=6) - self.connect("response", self.prefs_response_cb) + # headerscheck = gtk.CheckButton("Include development headers with toolchain") + # headerscheck.show() + # headerscheck.set_active(self.build_toolchain_headers) + # headerscheck.connect("toggled", self.toggle_headers_cb) + # hbox.pack_start(headerscheck, expand=False, fill=False, padding=6) + # self.connect("response", self.prefs_response_cb)
Signature · 1 line
1.7.6
PATCH 5/6 ui/crumbs/runningbuild: mask run_buildstats failure
In reply to Joshua LockThe buildstats handler causes an exception with: "'NoneType' object has no attribute 'startswith'" early a build via hob, leaving a glaring red row which means nothing to the user.
Mask this error until such a time as we have opportunity to correctly diagnose and fix the root problem.
Workaround fix for [YOCTO #1433]
--- lib/bb/ui/crumbs/runningbuild.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
lib/bb/ui/crumbs/runningbuild.py +6 -0
@@ -91,6 +91,12 @@ class RunningBuild (gobject.GObject): parent = self.tasks_to_iter[(package, task)] if(isinstance(event, logging.LogRecord)): + # FIXME: this is a hack! More info in Yocto #1433 + # http://bugzilla.pokylinux.org/show_bug.cgi?id=1433, temporarily + # mask the error message as it's not informative for the user. + if event.msg.startswith("Execution of event handler 'run_buildstats' failed"): + return + if (event.levelno < logging.INFO or event.msg.startswith("Running task")): return # don't add these to the list
Signature · 1 line
1.7.6
PATCH 6/6 hob: disable removal of packages
In reply to Joshua LockIt's felt that the stability of package deselection is not sufficient for the upcoming release and thus package removal should be disabled.
I'd actually like to see this patch, or its effects, reverted as soon as the release bits have been frozen so that this issue can continue to be worked on.
--- lib/bb/ui/hob.py | 42 ++++++++++++++++++++++++------------------ 1 files changed, 24 insertions(+), 18 deletions(-)
lib/bb/ui/hob.py +24 -18
@@ -544,14 +544,17 @@ class MainWindow (gtk.Window): """ # Whether the item is currently included inc = self.model[opath][self.model.COL_INC] + # FIXME: due to inpredictability of the removal of packages we are + # temporarily disabling this feature # If the item is already included, mark it for removal then # the sweep_up() method finds affected items and marks them # appropriately - if inc: - self.model.mark(opath) - self.model.sweep_up() - # If the item isn't included, mark it for inclusion - else: + # if inc: + # self.model.mark(opath) + # self.model.sweep_up() + # # If the item isn't included, mark it for inclusion + # else: + if not inc: self.model.include_item(item_path=opath, binb="User Selected", image_contents=image)
@@ -563,19 +566,22 @@ class MainWindow (gtk.Window): inc = model[path][self.model.COL_INC] # Warn user before removing included packages if inc: - pn = model[path][self.model.COL_NAME] - revdeps = self.model.find_reverse_depends(pn) - if len(revdeps): - lbl = "<b>Remove %s?</b>\n\nThis action cannot be undone and all packages which depend on this will be removed\nPackages which depend on %s include %s." % (pn, pn, ", ".join(revdeps).rstrip(",")) - else: - lbl = "<b>Remove %s?</b>\n\nThis action cannot be undone." % pn - dialog = CrumbsDialog(self, lbl, gtk.STOCK_DIALOG_WARNING) - dialog.add_button(gtk.STOCK_CANCEL, gtk.RESPONSE_CANCEL) - dialog.add_button("Remove", gtk.RESPONSE_OK) - response = dialog.run() - dialog.destroy() - if response == gtk.RESPONSE_CANCEL: - return + # FIXME: due to inpredictability of the removal of packages we are + # temporarily disabling this feature + return + # pn = model[path][self.model.COL_NAME] + # revdeps = self.model.find_reverse_depends(pn) + # if len(revdeps): + # lbl = "<b>Remove %s?</b>\n\nThis action cannot be undone and all packages which depend on this will be removed\nPackages which depend on %s include %s." % (pn, pn, ", ".join(revdeps).rstrip(",")) + # else: + # lbl = "<b>Remove %s?</b>\n\nThis action cannot be undone." % pn + # dialog = CrumbsDialog(self, lbl, gtk.STOCK_DIALOG_WARNING) + # dialog.add_button(gtk.STOCK_CANCEL, gtk.RESPONSE_CANCEL) + # dialog.add_button("Remove", gtk.RESPONSE_OK) + # response = dialog.run() + # dialog.destroy() + # if response == gtk.RESPONSE_CANCEL: + # return self.set_busy_cursor() # Convert path to path in original model
Signature · 1 line
1.7.6
Richard Purdie
AppliedQuoted 17 lines · Joshua Lock
This series addresses issues for the looming Yocto 1.1 code freeze. This means that the series contains a 3 genuine patches followed by 3 hide my shame patches to disable features which either a) weren't implmented in time or b) weren't deemed to be stable enough for inclusion in the 1.1 release.
Regards,
JoshuaThe following changes since commit
639db8c766cada7180f9447f51303f9b30d7e817:fetch2/git: Allow to specify the name of the checkout directory (2011-09-02 14:23:43 +0100)
are available in the git repository at:
git://github.com/incandescant/bitbakehob
https://github.com/incandescant/bitbake/tree/hob
Quoted 8 lines
Joshua Lock (6):
hob: use both pre and post files for hob configuration
hob: reflect defaultsetup being default distro
hob: add a test to ensure hob is run with the required pre and post
files
ui/crumbs/hobprefs: disable 'build toolchain with headers' ui/crumbs/runningbuild: maskrun_buildstatsfailure
hob: disable removal of packages
Merged to master, thanks.
Richard