Add support for python structured logging
cover Add support for python structured logging
Adds support for users to provide configuration for Python structured logging in bitbake. This gives a high degree of flexibility to the way that logs are capture and displayed, so users can generate very targeted logging if desired. The initial use case for this is to maintain the high level of hash equivalence logging on the Yocto autobuilders, while reducing the noise for most other users.
I've added documentation updates for the new BB_LOGCONFIG variable, but it probably needs some grooming.
Joshua Watt (15):
bitbake: lib/bb/msg.py: Convert default domains to a dictionary
bitbake: knotty: Handle logging messages with specific logger
bitbake: lib/bb/msg.py: Use log level instead of debug count
bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
bitbake: knotty: Add commented logging_tree code
bitbake: lib/bb/msg.py: Add filter utilities
bitbake: lib/bb/msg.py: Remove unused filters
bitbake: lib/bb/msg.py: Add helper to set logging config
bitbake: knotty: Remove dependency on format variable
bitbake: knotty: Setup logs with config helper
bitbake: worker: Remove unnecessary addDefaultLogFilter
bitbake: Log hash equivalence with a different logger
bitbake: Add autobuilder logging configuration
bitbake: Lower hash equivalence logging
bitbake: doc: Add documentation for BB_LOGCONFIG
bitbake/bin/bitbake-worker | 3 +- bitbake/contrib/autobuilderlog.json | 27 +++ .../bitbake-user-manual-execution.xml | 97 ++++++++++ .../bitbake-user-manual-ref-variables.xml | 11 ++ bitbake/lib/bb/__init__.py | 7 +- bitbake/lib/bb/msg.py | 172 +++++++++++++---- bitbake/lib/bb/runqueue.py | 13 +- bitbake/lib/bb/siggen.py | 17 +- bitbake/lib/bb/tinfoil.py | 4 +- bitbake/lib/bb/ui/knotty.py | 173 ++++++++++++------ 10 files changed, 419 insertions(+), 105 deletions(-) create mode 100644 bitbake/contrib/autobuilderlog.json
Signature · 1 line
2.17.1
PATCH 1/15 bitbake: lib/bb/msg.py: Convert default domains to a dictionary
Converts the default domain variable to a dictionary where the keys are the logging domains and the values are the logging level (instead of the debug count). This makes it easier to deal with the logging domains and the awkward conversion from a list to a dictionary only needs to be done once when logging is initialized. Finally, other code has been written that already assumes this variable is a dictionary, see:
f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")
--- bitbake/lib/bb/__init__.py | 5 +++-- bitbake/lib/bb/msg.py | 17 +++++++---------- 2 files changed, 10 insertions(+), 12 deletions(-)
bitbake/lib/bb/__init__.py +3 -2
@@ -43,12 +43,13 @@ class BBLogger(Logger): Logger.__init__(self, name) def bbdebug(self, level, msg, *args, **kwargs): + loglevel = logging.DEBUG - level + 1 if not bb.event.worker_pid: - if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]): + if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]): return if level > (bb.msg.loggerDefaultDebugLevel): return - return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs) + return self.log(loglevel, msg, *args, **kwargs) def plain(self, msg, *args, **kwargs): return self.log(logging.INFO + 1, msg, *args, **kwargs)
bitbake/lib/bb/msg.py +7 -10
@@ -138,7 +138,7 @@ class BBLogFilterStdOut(BBLogFilter): loggerDefaultDebugLevel = 0 loggerDefaultVerbose = False loggerVerboseLogs = False -loggerDefaultDomains = [] +loggerDefaultDomains = {} def init_msgconfig(verbose, debug, debug_domains=None): """
@@ -148,15 +148,16 @@ def init_msgconfig(verbose, debug, debug_domains=None): bb.msg.loggerDefaultVerbose = verbose if verbose: bb.msg.loggerVerboseLogs = True + + bb.msg.loggerDefaultDomains = {} if debug_domains: - bb.msg.loggerDefaultDomains = debug_domains - else: - bb.msg.loggerDefaultDomains = [] + for (domainarg, iterator) in groupby(debug_domains): + dlevel = len(tuple(iterator)) + bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 def constructLogOptions(): debug = loggerDefaultDebugLevel verbose = loggerDefaultVerbose - domains = loggerDefaultDomains if debug: level = BBLogFormatter.DEBUG - debug + 1
@@ -165,11 +166,7 @@ def constructLogOptions(): else: level = BBLogFormatter.NOTE - debug_domains = {} - for (domainarg, iterator) in groupby(domains): - dlevel = len(tuple(iterator)) - debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 - return level, debug_domains + return level, loggerDefaultDomains def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): level, debug_domains = constructLogOptions()
Signature · 1 line
2.17.1
PATCH 2/15 bitbake: knotty: Handle logging messages with specific logger
In reply to Joshua WattHandles the log messages from the bitbake server with the specific logger that the event originated from. This allows hierarchical logging configurations to work as expected.
--- bitbake/lib/bb/ui/knotty.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/bb/ui/knotty.py +1 -1
@@ -522,7 +522,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn'): event.msg = event.fn + ': ' + event.msg - logger.handle(event) + logging.getLogger(event.name).handle(event) continue if isinstance(event, bb.build.TaskFailedSilent):
Signature · 1 line
2.17.1
PATCH 3/15 bitbake: lib/bb/msg.py: Use log level instead of debug count
In reply to Joshua WattPasses around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used
--- bitbake/bin/bitbake-worker | 2 +- bitbake/lib/bb/__init__.py | 2 +- bitbake/lib/bb/msg.py | 22 +++++++++------------- bitbake/lib/bb/runqueue.py | 2 +- 4 files changed, 12 insertions(+), 16 deletions(-)
bitbake/bin/bitbake-worker +1 -1
@@ -414,7 +414,7 @@ class BitbakeWorker(object): def handle_workerdata(self, data): self.workerdata = pickle.loads(data) - bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"] + bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"] bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"] bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"] bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
bitbake/lib/bb/__init__.py +1 -1
@@ -47,7 +47,7 @@ class BBLogger(Logger): if not bb.event.worker_pid: if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]): return - if level > (bb.msg.loggerDefaultDebugLevel): + if loglevel > bb.msg.loggerDefaultLogLevel: return return self.log(loglevel, msg, *args, **kwargs)
bitbake/lib/bb/msg.py +9 -13
@@ -135,7 +135,7 @@ class BBLogFilterStdOut(BBLogFilter): # Message control functions # -loggerDefaultDebugLevel = 0 +loggerDefaultLogLevel = BBLogFormatter.NOTE loggerDefaultVerbose = False loggerVerboseLogs = False loggerDefaultDomains = {}
@@ -144,11 +144,17 @@ def init_msgconfig(verbose, debug, debug_domains=None): """ Set default verbosity and debug levels config the logger """ - bb.msg.loggerDefaultDebugLevel = debug bb.msg.loggerDefaultVerbose = verbose if verbose: bb.msg.loggerVerboseLogs = True + if debug: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.DEBUG - debug + 1 + elif verbose: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.VERBOSE + else: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.NOTE + bb.msg.loggerDefaultDomains = {} if debug_domains: for (domainarg, iterator) in groupby(debug_domains):
@@ -156,17 +162,7 @@ def init_msgconfig(verbose, debug, debug_domains=None): bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 def constructLogOptions(): - debug = loggerDefaultDebugLevel - verbose = loggerDefaultVerbose - - if debug: - level = BBLogFormatter.DEBUG - debug + 1 - elif verbose: - level = BBLogFormatter.VERBOSE - else: - level = BBLogFormatter.NOTE - - return level, loggerDefaultDomains + return loggerDefaultLogLevel, loggerDefaultDomains def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): level, debug_domains = constructLogOptions()
bitbake/lib/bb/runqueue.py +1 -1
@@ -1254,7 +1254,7 @@ class RunQueue: "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs, "fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" : bb.parse.siggen.get_taskdata(), - "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel, + "logdefaultlevel" : bb.msg.loggerDefaultLogLevel, "logdefaultverbose" : bb.msg.loggerDefaultVerbose, "logdefaultverboselogs" : bb.msg.loggerVerboseLogs, "logdefaultdomain" : bb.msg.loggerDefaultDomains,
Signature · 1 line
2.17.1
PATCH 4/15 bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
In reply to Joshua WattAdds a __repr__ function for BBLogFormatter. This allows it to get a human readable string when printed using the logging_tree module
--- bitbake/lib/bb/msg.py | 3 +++ 1 file changed, 3 insertions(+)
bitbake/lib/bb/msg.py +3 -0
@@ -98,6 +98,9 @@ class BBLogFormatter(logging.Formatter): def enable_color(self): self.color_enabled = True + def __repr__(self): + return "%s fmt='%s' color=%s" % (self.__class__.__name__, self._fmt, "True" if self.color_enabled else "False") + class BBLogFilter(object): def __init__(self, handler, level, debug_domains): self.stdlevel = level
Signature · 1 line
2.17.1
PATCH 5/15 bitbake: knotty: Add commented logging_tree code
In reply to Joshua WattAdds a comment section that can be easily uncommented to enable dumping the logging tree. This module is extremely useful for debugging issued with logging configuration
--- bitbake/lib/bb/ui/knotty.py | 5 +++++ 1 file changed, 5 insertions(+)
bitbake/lib/bb/ui/knotty.py +5 -0
@@ -430,6 +430,11 @@ def main(server, eventHandler, params, tf = TerminalFilter): llevel, debug_domains = bb.msg.constructLogOptions() server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list]) + # The logging_tree module is *extremely* helpful in debugging logging + # domains. Uncomment here to dump the logging tree when bitbake starts + #import logging_tree + #logging_tree.printout() + universe = False if not params.observe_only: params.updateFromServer(server)
Signature · 1 line
2.17.1
PATCH 6/15 bitbake: lib/bb/msg.py: Add filter utilities
In reply to Joshua WattAdds generic filter utilities that can be used to filter when the log level is greater than or equal to a defined value, or below a defined value, as well as a generic function to translate a string to a logging level (or bitbake logging level)
--- bitbake/lib/bb/msg.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
bitbake/lib/bb/msg.py +35 -0
@@ -135,6 +135,28 @@ class BBLogFilterStdOut(BBLogFilter): return True return False +class LogFilterGEQLevel(logging.Filter): + def __init__(self, level): + self.strlevel = str(level) + self.level = stringToLevel(level) + + def __repr__(self): + return "%s level >= %s (%d)" % (self.__class__.__name__, self.strlevel, self.level) + + def filter(self, record): + return (record.levelno >= self.level) + +class LogFilterLTLevel(logging.Filter): + def __init__(self, level): + self.strlevel = str(level) + self.level = stringToLevel(level) + + def __repr__(self): + return "%s level < %s (%d)" % (self.__class__.__name__, self.strlevel, self.level) + + def filter(self, record): + return (record.levelno < self.level) + # Message control functions #
@@ -175,6 +197,19 @@ def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): cls(handler, level, debug_domains) +def stringToLevel(level): + try: + return int(level) + except ValueError: + pass + + try: + return getattr(logging, level) + except AttributeError: + pass + + return getattr(BBLogFormatter, level) + # # Message handling functions #
Signature · 1 line
2.17.1
PATCH 7/15 bitbake: lib/bb/msg.py: Remove unused filters
In reply to Joshua WattNow that the filter are described using the python logging structure, these classes are no longer needed.
--- bitbake/lib/bb/msg.py | 16 ---------------- 1 file changed, 16 deletions(-)
bitbake/lib/bb/msg.py +0 -16
@@ -119,22 +119,6 @@ class BBLogFilter(object): return True return False -class BBLogFilterStdErr(BBLogFilter): - def filter(self, record): - if not BBLogFilter.filter(self, record): - return False - if record.levelno >= logging.ERROR: - return True - return False - -class BBLogFilterStdOut(BBLogFilter): - def filter(self, record): - if not BBLogFilter.filter(self, record): - return False - if record.levelno < logging.ERROR: - return True - return False - class LogFilterGEQLevel(logging.Filter): def __init__(self, level): self.strlevel = str(level)
Signature · 1 line
2.17.1
PATCH 8/15 bitbake: lib/bb/msg.py: Add helper to set logging config
In reply to Joshua WattAdds a helper function to setup the structure logging information in bitbake. The helper function takes a default configuration and an optional path to a user config file that can be merged into the default.
--- bitbake/lib/bb/msg.py | 93 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+)
bitbake/lib/bb/msg.py +93 -0
@@ -13,6 +13,7 @@ Message handling infrastructure for bitbake import sys import copy import logging +import logging.config from itertools import groupby import bb import bb.event
@@ -227,3 +228,95 @@ def has_console_handler(logger): if handler.stream in [sys.stderr, sys.stdout]: return True return False + +def setLoggingConfig(defaultconfig, userconfigfile=None): + logconfig = copy.deepcopy(defaultconfig) + + if userconfigfile: + with open(userconfigfile, 'r') as f: + if userconfigfile.endswith('.yml') or userconfigfile.endswith('.yaml'): + import yaml + userconfig = yaml.load(f) + elif userconfigfile.endswith('.json') or userconfigfile.endswith('.cfg'): + import json + userconfig = json.load(f) + else: + raise BaseException("Unrecognized file format: %s" % userconfigfile) + + if userconfig.get('bitbake_merge', True): + # Merge config with the default config + if userconfig.get('version') != logconfig['version']: + raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version'))) + + # Set some defaults to make merging easier + userconfig.setdefault("loggers", {}) + + # If a handler, formatter, or filter is defined in the user + # config, it will replace an existing one in the default config + for k in ("handlers", "formatters", "filters"): + logconfig.setdefault(k, {}).update(userconfig.get(k, {})) + + seen_loggers = set() + for name, l in logconfig["loggers"].items(): + # If the merge option is set, merge the handlers and + # filters. Otherwise, if it is False, this logger won't get + # add to the set of seen loggers and will replace the + # existing one + if l.get('bitbake_merge', True): + ulogger = userconfig["loggers"].setdefault(name, {}) + ulogger.setdefault("handlers", []) + ulogger.setdefault("filters", []) + + # Merge lists + l.setdefault("handlers", []).extend(ulogger["handlers"]) + l.setdefault("filters", []).extend(ulogger["filters"]) + + # Replace other properties if present + if "level" in ulogger: + l["level"] = ulogger["level"] + + if "propagate" in ulogger: + l["propagate"] = ulogger["propagate"] + + seen_loggers.add(name) + + # Add all loggers present in the user config, but not any that + # have already been processed + for name in set(userconfig["loggers"].keys()) - seen_loggers: + logconfig["loggers"][name] = userconfig["loggers"][name] + + else: + # Replace the entire default config + logconfig = userconfig + + # Convert all level parameters to integers in case users want to use the + # bitbake defined level names + for h in logconfig["handlers"].values(): + if "level" in h: + h["level"] = bb.msg.stringToLevel(h["level"]) + + for l in logconfig["loggers"].values(): + if "level" in l: + l["level"] = bb.msg.stringToLevel(l["level"]) + + logging.config.dictConfig(logconfig) + + # The user may have specified logging domains they want at a higher debug + # level than the standard. + for name, l in logconfig["loggers"].items(): + if not name.startswith("BitBake."): + continue + + if not "level" in l: + continue + + curlevel = bb.msg.loggerDefaultDomains.get(name) + # Note: level parameter should already be a int because of conversion + # above + newlevel = int(l["level"]) + if curlevel is None or newlevel < curlevel: + bb.msg.loggerDefaultDomains[name] = newlevel + + # TODO: I don't think that setting the global log level should be necessary + #if newlevel < bb.msg.loggerDefaultLogLevel: + # bb.msg.loggerDefaultLogLevel = newlevel
Signature · 1 line
2.17.1
PATCH 9/15 bitbake: knotty: Remove dependency on format variable
In reply to Joshua WattPassing around the log formatter variable was unnecessary since the log levels of interest can be accesses as class members of bb.msg.BBLogFormatter. Switching to do this will make using the structured python logging much easier, since it can be difficult to extract out the formatter for a specific handler.
--- bitbake/lib/bb/tinfoil.py | 4 +--- bitbake/lib/bb/ui/knotty.py | 32 ++++++++++++++++++++------------ 2 files changed, 21 insertions(+), 15 deletions(-)
bitbake/lib/bb/tinfoil.py +1 -3
@@ -735,11 +735,9 @@ class Tinfoil: console = handler elif handler.stream == sys.stderr: errconsole = handler - format_str = "%(levelname)s: %(message)s" - format = bb.msg.BBLogFormatter(format_str) helper.shutdown = 0 parseprogress = None - termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet) + termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, quiet=self.quiet) try: while True: try:
bitbake/lib/bb/ui/knotty.py +20 -12
@@ -109,12 +109,11 @@ def pluralise(singular, plural, qty): class InteractConsoleLogFilter(logging.Filter): - def __init__(self, tf, format): + def __init__(self, tf): self.tf = tf - self.format = format def filter(self, record): - if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): + if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): return False self.tf.clearFooter() return True
@@ -150,7 +149,7 @@ class TerminalFilter(object): cr = (25, 80) return cr - def __init__(self, main, helper, console, errconsole, format, quiet): + def __init__(self, main, helper, console, errconsole, quiet): self.main = main self.helper = helper self.cuu = None
@@ -180,7 +179,16 @@ class TerminalFilter(object): termios.tcsetattr(fd, termios.TCSADRAIN, new) curses.setupterm() if curses.tigetnum("colors") > 2: - format.enable_color() + if console: + try: + console.formatter.enable_color() + except AttributeError: + pass + if errconsole: + try: + errconsole.formatter.enable_color() + except AttributeError: + pass self.ed = curses.tigetstr("ed") if self.ed: self.cuu = curses.tigetstr("cuu")
@@ -197,9 +205,9 @@ class TerminalFilter(object): bb.note("Unable to use interactive mode for this terminal, using fallback") return if console: - console.addFilter(InteractConsoleLogFilter(self, format)) + console.addFilter(InteractConsoleLogFilter(self)) if errconsole: - errconsole.addFilter(InteractConsoleLogFilter(self, format)) + errconsole.addFilter(InteractConsoleLogFilter(self)) self.main_progress = None
@@ -469,7 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): printinterval = 5000 lastprint = time.time() - termfilter = tf(main, helper, console, errconsole, format, params.options.quiet) + termfilter = tf(main, helper, console, errconsole, params.options.quiet) atexit.register(termfilter.finish) while True:
@@ -508,21 +516,21 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, logging.LogRecord): lastprint = time.time() printinterval = 5000 - if event.levelno >= format.ERROR: + if event.levelno >= bb.msg.BBLogFormatter.ERROR: errors = errors + 1 return_value = 1 - elif event.levelno == format.WARNING: + elif event.levelno == bb.msg.BBLogFormatter.WARNING: warnings = warnings + 1 if event.taskpid != 0: # For "normal" logging conditions, don't show note logs from tasks # but do show them if the user has changed the default log level to # include verbose/debug messages - if event.levelno <= format.NOTE and (event.levelno < llevel or (event.levelno == format.NOTE and llevel != format.VERBOSE)): + if event.levelno <= bb.msg.BBLogFormatter.NOTE and (event.levelno < llevel or (event.levelno == bb.msg.BBLogFormatter.NOTE and llevel != bb.msg.BBLogFormatter.VERBOSE)): continue # Prefix task messages with recipe/task - if event.taskpid in helper.pidmap and event.levelno != format.PLAIN: + if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN: taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]] event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn'):
Signature · 1 line
2.17.1
PATCH 10/15 bitbake: knotty: Setup logs with config helper
In reply to Joshua WattSets up logging in knotty to use python's structured logging config and the bb.msg.setLoggingConfig() helper to setup logging. This allows the user to specify additional logging mechanism in a config file with BB_LOGCONFIG
--- bitbake/lib/bb/ui/knotty.py | 134 ++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 38 deletions(-)
bitbake/lib/bb/ui/knotty.py +96 -38
@@ -370,7 +370,11 @@ def _log_settings_from_server(server, observe_only): if error: logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error) raise BaseException(error) - return includelogs, loglines, consolelogfile + logconfigfile, error = server.runCommand([cmd, "BB_LOGCONFIG"]) + if error: + logger.error("Unable to get the value of BB_LOGCONFIG variable: %s" % error) + raise BaseException(error) + return includelogs, loglines, consolelogfile, logconfigfile _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.LogRecord", "bb.build.TaskFailed", "bb.build.TaskBase", "bb.event.ParseStarted",
@@ -387,7 +391,87 @@ def main(server, eventHandler, params, tf = TerminalFilter): if not params.observe_only: params.updateToServer(server, os.environ.copy()) - includelogs, loglines, consolelogfile = _log_settings_from_server(server, params.observe_only) + includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only) + + loglevel, _ = bb.msg.constructLogOptions() + + if params.options.quiet == 0: + console_loglevel = loglevel + elif params.options.quiet > 2: + console_loglevel = bb.msg.BBLogFormatter.ERROR + else: + console_loglevel = bb.msg.BBLogFormatter.WARNING + + logconfig = { + "version": 1, + "handlers": { + "BitBake.console": { + "class": "logging.StreamHandler", + "formatter": "BitBake.consoleFormatter", + "level": console_loglevel, + "stream": "ext://sys.stdout", + "filters": ["BitBake.stdoutFilter"], + }, + "BitBake.errconsole": { + "class": "logging.StreamHandler", + "formatter": "BitBake.consoleFormatter", + "level": loglevel, + "stream": "ext://sys.stderr", + "filters": ["BitBake.stderrFilter"], + }, + }, + "formatters": { + # This format instance will get color output enabled by the + # terminal + "BitBake.consoleFormatter" : { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + }, + # The file log requires a separate instance so that it doesn't get + # color enabled + "BitBake.logfileFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + "filters": { + "BitBake.stdoutFilter": { + "()": "bb.msg.LogFilterLTLevel", + "level": "ERROR" + }, + "BitBake.stderrFilter": { + "()": "bb.msg.LogFilterGEQLevel", + "level": "ERROR" + } + }, + "loggers": { + "BitBake": { + "level": loglevel, + "handlers": ["BitBake.console", "BitBake.errconsole"], + } + }, + "disable_existing_loggers": False + } + + # Enable the console log file if enabled + if consolelogfile and not params.options.show_environment and not params.options.show_versions: + logconfig["handlers"]["BitBake.consolelog"] ={ + "class": "logging.FileHandler", + "formatter": "BitBake.logfileFormatter", + "level": "INFO", + "filename": consolelogfile, + } + logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog") + + bb.utils.mkdirhier(os.path.dirname(consolelogfile)) + loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') + bb.utils.remove(loglink) + try: + os.symlink(os.path.basename(consolelogfile), loglink) + except OSError: + pass + + bb.msg.setLoggingConfig(logconfig, logconfigfile) if sys.stdin.isatty() and sys.stdout.isatty(): log_exec_tty = True
@@ -396,23 +480,16 @@ def main(server, eventHandler, params, tf = TerminalFilter): helper = uihelper.BBUIHelper() - console = logging.StreamHandler(sys.stdout) - errconsole = logging.StreamHandler(sys.stderr) - format_str = "%(levelname)s: %(message)s" - format = bb.msg.BBLogFormatter(format_str) - if params.options.quiet == 0: - forcelevel = None - elif params.options.quiet > 2: - forcelevel = bb.msg.BBLogFormatter.ERROR - else: - forcelevel = bb.msg.BBLogFormatter.WARNING - bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel) - bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr) - console.setFormatter(format) - errconsole.setFormatter(format) - if not bb.msg.has_console_handler(logger): - logger.addHandler(console) - logger.addHandler(errconsole) + # Look for the specially designated handlers which need to be passed to the + # terminal handler + console = None + errconsole = None + for h in logger.handlers: + name = getattr(h, '_name', None) + if name == 'BitBake.console': + console = h + elif name == 'BitBake.errconsole': + errconsole = h bb.utils.set_process_name("KnottyUI")
@@ -420,21 +497,6 @@ def main(server, eventHandler, params, tf = TerminalFilter): server.terminateServer() return - consolelog = None - if consolelogfile and not params.options.show_environment and not params.options.show_versions: - bb.utils.mkdirhier(os.path.dirname(consolelogfile)) - conlogformat = bb.msg.BBLogFormatter(format_str) - consolelog = logging.FileHandler(consolelogfile) - bb.msg.addDefaultlogFilter(consolelog) - consolelog.setFormatter(conlogformat) - logger.addHandler(consolelog) - loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') - bb.utils.remove(loglink) - try: - os.symlink(os.path.basename(consolelogfile), loglink) - except OSError: - pass - llevel, debug_domains = bb.msg.constructLogOptions() server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
@@ -761,8 +823,4 @@ def main(server, eventHandler, params, tf = TerminalFilter): if e.errno == errno.EPIPE: pass - if consolelog: - logger.removeHandler(consolelog) - consolelog.close() - return return_value
Signature · 1 line
2.17.1
PATCH 11/15 bitbake: worker: Remove unnecessary addDefaultLogFilter
In reply to Joshua WattAdding the default log filter here is unnecessary because there are no defined logging domains when it is called, which means it does no actual filtering.
--- bitbake/bin/bitbake-worker | 1 - 1 file changed, 1 deletion(-)
bitbake/bin/bitbake-worker +0 -1
@@ -65,7 +65,6 @@ if 0: format_str = "%(levelname)s: %(message)s" conlogformat = bb.msg.BBLogFormatter(format_str) consolelog = logging.FileHandler(logfilename) - bb.msg.addDefaultlogFilter(consolelog) consolelog.setFormatter(conlogformat) logger.addHandler(consolelog)
Signature · 1 line
2.17.1
PATCH 12/15 bitbake: Log hash equivalence with a different logger
In reply to Joshua WattSwitches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging.
--- bitbake/lib/bb/runqueue.py | 11 ++++++----- bitbake/lib/bb/siggen.py | 17 +++++++++-------- 2 files changed, 15 insertions(+), 13 deletions(-)
bitbake/lib/bb/runqueue.py +6 -5
@@ -27,6 +27,7 @@ import pprint bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") +hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv") __find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
@@ -2263,7 +2264,7 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - logger.info("Task %s unihash changed to %s" % (tid, unihash)) + hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash)) self.rqdata.runtaskentries[tid].unihash = unihash bb.parse.siggen.set_unihash(tid, unihash) toprocess.add(tid)
@@ -2308,7 +2309,7 @@ class RunQueueExecute: elif tid in self.scenequeue_covered or tid in self.sq_live: # Already ran this setscene task or it running. Report the new taskhash bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) - logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) remapped = True if not remapped:
@@ -2327,7 +2328,7 @@ class RunQueueExecute: for mc in self.rq.fakeworker: self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>") - logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed))) + hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed))) for tid in changed: if tid not in self.rqdata.runq_setscene_tids:
@@ -2346,7 +2347,7 @@ class RunQueueExecute: # Check no tasks this covers are running for dep in self.sqdata.sq_covered_tasks[tid]: if dep in self.runq_running and dep not in self.runq_complete: - logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid)) + hashequiv_logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid)) valid = False break if not valid:
@@ -2409,7 +2410,7 @@ class RunQueueExecute: for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: - logger.info("Setscene task %s became valid" % tid) + hashequiv_logger.info("Setscene task %s became valid" % tid) if harddepfail: self.sq_task_failoutright(tid)
bitbake/lib/bb/siggen.py +9 -8
@@ -16,6 +16,7 @@ from bb import runqueue import hashserv logger = logging.getLogger('BitBake.SigGen') +hashequiv_logger = logging.getLogger('BitBake.SigGen.HashEquiv') def init(d): siggens = [obj for obj in globals().values()
@@ -502,9 +503,9 @@ class SignatureGeneratorUniHashMixIn(object): # A unique hash equal to the taskhash is not very interesting, # so it is reported it at debug level 2. If they differ, that # is much more interesting, so it is reported at debug level 1 - bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server)) + hashequiv_logger.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server)) else: - bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server)) + hashequiv_logger.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
@@ -578,12 +579,12 @@ class SignatureGeneratorUniHashMixIn(object): new_unihash = data['unihash'] if new_unihash != unihash: - bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) + hashequiv_logger.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d) self.set_unihash(tid, new_unihash) d.setVar('BB_UNIHASH', new_unihash) else: - bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) + hashequiv_logger.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) finally:
@@ -606,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object): method = method + self.extramethod[tid] data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) - bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None: bb.warn("Server unable to handle unihash report")
@@ -615,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object): finalunihash = data['unihash'] if finalunihash == current_unihash: - bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash)) elif finalunihash == wanted_unihash: - bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) self.set_unihash(tid, finalunihash) return True else: # TODO: What to do here? - bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) + hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
Signature · 1 line
2.17.1
PATCH 13/15 bitbake: Add autobuilder logging configuration
In reply to Joshua WattAdds a configuration file that the autobuilder can use to capture interesting logging domains above the ones that show up for normal users on stdout/stderr.
--- bitbake/contrib/autobuilderlog.json | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 bitbake/contrib/autobuilderlog.json
bitbake/contrib/autobuilderlog.json new file +27 -0
@@ -0,0 +1,27 @@ +{ + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "autobuilder.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s" + } + }, + "loggers": { + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + } + } +}
Signature · 1 line
2.17.1
PATCH 14/15 bitbake: Lower hash equivalence logging
In reply to Joshua WattLowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json)
[YOCTO #13813]
--- bitbake/lib/bb/runqueue.py | 6 +++--- bitbake/lib/bb/siggen.py | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-)
bitbake/lib/bb/runqueue.py +3 -3
@@ -2264,7 +2264,7 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash)) + hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash)) self.rqdata.runtaskentries[tid].unihash = unihash bb.parse.siggen.set_unihash(tid, unihash) toprocess.add(tid)
@@ -2309,7 +2309,7 @@ class RunQueueExecute: elif tid in self.scenequeue_covered or tid in self.sq_live: # Already ran this setscene task or it running. Report the new taskhash bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) - hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + hashequiv_logger.verbose("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) remapped = True if not remapped:
@@ -2410,7 +2410,7 @@ class RunQueueExecute: for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: - hashequiv_logger.info("Setscene task %s became valid" % tid) + hashequiv_logger.verbose("Setscene task %s became valid" % tid) if harddepfail: self.sq_task_failoutright(tid)
bitbake/lib/bb/siggen.py +4 -4
@@ -607,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object): method = method + self.extramethod[tid] data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) - hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + hashequiv_logger.verbose('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None: bb.warn("Server unable to handle unihash report")
@@ -616,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object): finalunihash = data['unihash'] if finalunihash == current_unihash: - hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash)) elif finalunihash == wanted_unihash: - hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + hashequiv_logger.verbose('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) self.set_unihash(tid, finalunihash) return True else: # TODO: What to do here? - hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) + hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
Signature · 1 line
2.17.1
PATCH 15/15 bitbake-user-manual: Add documentation for BB_LOGCONFIG
In reply to Joshua WattAdds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging.
--- .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++ .../bitbake-user-manual-ref-variables.xml | 11 +++ 2 files changed, 108 insertions(+)
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +97 -0
@@ -929,4 +929,101 @@ section. </para> </section> + + <section id="logging"> + <title>Logging</title> + <para> + In addition to the standard command line option to control how + verbose builds are when execute, bitbake also supports user defined + configuration of the + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink> + facilities through the + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link> + variable. This variable defines a json or yaml + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink> + that will be intelligently merged into the default configuration. + The logging configuration is merged using the following rules: + <itemizedlist> + <listitem><para> + The user defined configuration will completely replace the default + configuration if top level key + <filename>bitbake_merge</filename> is set to the value + <filename>False</filename>. In this case, all other rules + are ignored. + </para></listitem> + <listitem><para> + The user configuration must have a top level + <filename>version</filename> which must match the value of + the default configuration. + </para></listitem> + <listitem><para> + Any keys defined in the <filename>handlers</filename>, + <filename>formatters</filename>, or <filename>filters</filename>, + will be merged into the same section in the default + configuration, with the user specified keys taking + replacing a default one if there is a conflict. In + practice, this means that if both the default configuration + and user configuration specify a handler named + <filename>myhandler</filename>, the user defined one will + replace the default. To prevent the user from inadvertently + replacing a default handler, formatter, or filter, all of + the default ones are named with a prefix of + "<filename>BitBake.</filename>" + </para></listitem> + <listitem><para> + If a logger is defined by the user with the key + <filename>bitbake_merge</filename> set to + <filename>False</filename>, that logger will be completely + replaced by user configuration. In this case, no other + rules will apply to that logger. + </listitem></para> + <listitem><para> + All user defined <filename>filter</filename> and + <filename>handlers</filename> properties for a given logger + will be merged with corresponding properties from the + default logger. For example, if the user configuration adds + a filter called <filename>myFilter</filename> to the + <filename>BitBake.SigGen</filename>, and the default + configuration adds a filter called + <filename>BitBake.defaultFilter</filename>, both filters + will be applied to the logger + </listitem></para> + </itemizedlist> + </para> + + <para> + As an example, consider the following user logging configuration + file which logs all Hash Equivalence related messages of VERBOSE or + higher to a file called <filename>hashequiv.log</filename> + <literallayout class='monospaced'> + { + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "hashequiv.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s" + } + }, + "loggers": { + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + } + } + } + </literallayout> + </para> + </section> </chapter>
bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +11 -0
@@ -539,6 +539,17 @@ </glossdef> </glossentry> + <glossentry id='var-bb-BB_LOGCONFIG'><glossterm>BB_LOGCONFIG</glossterm> + <glossdef> + <para> + Specifies the name of a config file that contains the user + logging configuration. See + <link linkend="logging">Logging</link> for additional + information + </para> + </glossdef> + </glossentry> + <glossentry id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm> <glossdef> <para>
Signature · 1 line
2.17.1
Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
Quoted 89 lines · Joshua Watt
Adds documentation describing how to use the
BB_LOGCONFIGvariable to enable custom logging.Signed-off-by Joshua Watt <[email protected]>---
.../bitbake-user-manual-execution.xml| 97 +++++++++++++++++++.../bitbake-user-manual-ref-variables.xml| 11 +++2 files changed, 108 insertions(+)bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +71
-0@@ -929,4 +929,101 @@ section. </para> </section> + + <section id="logging"> + <title>Logging</title> + <para> + In addition to the standard command line option to control how + verbose builds are when execute, bitbake also supports user defined + configuration of the + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink> + facilities through the + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link> + variable. This variable defines a json or yaml + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink> + that will be intelligently merged into the default configuration. + The logging configuration is merged using the following rules: + <itemizedlist> + <listitem><para> + The user defined configuration will completely replace the default + configuration if top level key + <filename>bitbake_merge</filename> is set to the value + <filename>False</filename>. In this case, all other rules + are ignored. + </para></listitem> + <listitem><para> + The user configuration must have a top level + <filename>version</filename> which must match the value of + the default configuration. + </para></listitem> + <listitem><para> + Any keys defined in the <filename>handlers</filename>, + <filename>formatters</filename>, or <filename>filters</filename>, + will be merged into the same section in the default + configuration, with the user specified keys taking + replacing a default one if there is a conflict. In + practice, this means that if both the default configuration + and user configuration specify a handler named + <filename>myhandler</filename>, the user defined one will + replace the default. To prevent the user from inadvertently + replacing a default handler, formatter, or filter, all of + the default ones are named with a prefix of + "<filename>BitBake.</filename>" + </para></listitem> + <listitem><para> + If a logger is defined by the user with the key + <filename>bitbake_merge</filename> set to + <filename>False</filename>, that logger will be completely + replaced by user configuration. In this case, no other + rules will apply to that logger. + </listitem></para> + <listitem><para> + All user defined <filename>filter</filename> and + <filename>handlers</filename> properties for a given logger + will be merged with corresponding properties from the + default logger. For example, if the user configuration adds + a filter called <filename>myFilter</filename> to the + <filename>BitBake.SigGen</filename>, and the default + configuration adds a filter called + <filename>BitBake.defaultFilter</filename>, both filters + will be applied to the logger + </listitem></para> + </itemizedlist> + </para> + + <para> + As an example, consider the following user logging configuration + file which logs all Hash Equivalence related messages of VERBOSE or + higher to a file called <filename>hashequiv.log</filename> + <literallayout class='monospaced'> + { + "version": 1, + "handlers": { + "autobuilderlog": {
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml line 1012
Quoted 10 lines
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +10
-0+ "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "hashequiv.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s"
What's the syntax for including a timestamp in the log message? Or is that implicit?
Rich
Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
Quoted 95 lines · Rich Persaud
Quoted 93 lines · Joshua Watt
Adds documentation describing how to use the
BB_LOGCONFIGvariable to enable custom logging.Signed-off-by Joshua Watt <[email protected]>---
.../bitbake-user-manual-execution.xml| 97 +++++++++++++++++++.../bitbake-user-manual-ref-variables.xml| 11 +++2 files changed, 108 insertions(+)bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +75
-0@@ -929,4 +929,101 @@ section. </para> </section> + + <section id="logging"> + <title>Logging</title> + <para> + In addition to the standard command line option to control how + verbose builds are when execute, bitbake also supports user defined + configuration of the + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink> + facilities through the + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link> + variable. This variable defines a json or yaml + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink> + that will be intelligently merged into the default configuration. + The logging configuration is merged using the following rules: + <itemizedlist> + <listitem><para> + The user defined configuration will completely replace the default + configuration if top level key + <filename>bitbake_merge</filename> is set to the value + <filename>False</filename>. In this case, all other rules + are ignored. + </para></listitem> + <listitem><para> + The user configuration must have a top level + <filename>version</filename> which must match the value of + the default configuration. + </para></listitem> + <listitem><para> + Any keys defined in the <filename>handlers</filename>, + <filename>formatters</filename>, or <filename>filters</filename>, + will be merged into the same section in the default + configuration, with the user specified keys taking + replacing a default one if there is a conflict. In + practice, this means that if both the default configuration + and user configuration specify a handler named + <filename>myhandler</filename>, the user defined one will + replace the default. To prevent the user from inadvertently + replacing a default handler, formatter, or filter, all of + the default ones are named with a prefix of + "<filename>BitBake.</filename>" + </para></listitem> + <listitem><para> + If a logger is defined by the user with the key + <filename>bitbake_merge</filename> set to + <filename>False</filename>, that logger will be completely + replaced by user configuration. In this case, no other + rules will apply to that logger. + </listitem></para> + <listitem><para> + All user defined <filename>filter</filename> and + <filename>handlers</filename> properties for a given logger + will be merged with corresponding properties from the + default logger. For example, if the user configuration adds + a filter called <filename>myFilter</filename> to the + <filename>BitBake.SigGen</filename>, and the default + configuration adds a filter called + <filename>BitBake.defaultFilter</filename>, both filters + will be applied to the logger + </listitem></para> + </itemizedlist> + </para> + + <para> + As an example, consider the following user logging configuration + file which logs all Hash Equivalence related messages of VERBOSE or + higher to a file called <filename>hashequiv.log</filename> + <literallayout class='monospaced'> + { + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "hashequiv.log",
Quoted 8 lines
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml line 1012
Quoted 6 lines
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +6
-0+ "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s"What's the syntax for including a timestamp in the log message? Or is that implicit?
It's not implicit, but you can add it easily with %(asctime)s. There are also several other options:
https://docs.python.org/3/library/logging.html#logrecord-attributes
Quoted 2 lines
Rich