Skip to contentSkip to conversation

Add support for python structured logging

PATCH In review Joshua Watt 2 participants 18 messages Mar 9, 2020 – Mar 13, 2020

lore
Share & tools
lore
#0 lore raw

cover Add support for python structured logging

Adds support for users to provide configuration for Python structured logging in bitbake. This gives a high degree of flexibility to the way that logs are capture and displayed, so users can generate very targeted logging if desired. The initial use case for this is to maintain the high level of hash equivalence logging on the Yocto autobuilders, while reducing the noise for most other users.

I've added documentation updates for the new BB_LOGCONFIG variable, but it probably needs some grooming.

Joshua Watt (15):
bitbake: lib/bb/msg.py: Convert default domains to a dictionary
bitbake: knotty: Handle logging messages with specific logger
bitbake: lib/bb/msg.py: Use log level instead of debug count
bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
bitbake: knotty: Add commented logging_tree code
bitbake: lib/bb/msg.py: Add filter utilities
bitbake: lib/bb/msg.py: Remove unused filters
bitbake: lib/bb/msg.py: Add helper to set logging config
bitbake: knotty: Remove dependency on format variable
bitbake: knotty: Setup logs with config helper
bitbake: worker: Remove unnecessary addDefaultLogFilter
bitbake: Log hash equivalence with a different logger
bitbake: Add autobuilder logging configuration
bitbake: Lower hash equivalence logging
bitbake: doc: Add documentation for BB_LOGCONFIG

 bitbake/bin/bitbake-worker                    |   3 +-
 bitbake/contrib/autobuilderlog.json           |  27 +++
 .../bitbake-user-manual-execution.xml         |  97 ++++++++++
 .../bitbake-user-manual-ref-variables.xml     |  11 ++
 bitbake/lib/bb/__init__.py                    |   7 +-
 bitbake/lib/bb/msg.py                         | 172 +++++++++++++----
 bitbake/lib/bb/runqueue.py                    |  13 +-
 bitbake/lib/bb/siggen.py                      |  17 +-
 bitbake/lib/bb/tinfoil.py                     |   4 +-
 bitbake/lib/bb/ui/knotty.py                   | 173 ++++++++++++------
 10 files changed, 419 insertions(+), 105 deletions(-)
 create mode 100644 bitbake/contrib/autobuilderlog.json
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/15 bitbake: lib/bb/msg.py: Convert default domains to a dictionary

Converts the default domain variable to a dictionary where the keys are the logging domains and the values are the logging level (instead of the debug count). This makes it easier to deal with the logging domains and the awkward conversion from a list to a dictionary only needs to be done once when logging is initialized. Finally, other code has been written that already assumes this variable is a dictionary, see:

f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")

---
 bitbake/lib/bb/__init__.py |  5 +++--
 bitbake/lib/bb/msg.py      | 17 +++++++----------
 2 files changed, 10 insertions(+), 12 deletions(-)
bitbake/lib/bb/__init__.py +3 -2
index 88641e280b..acd4af13a8 100644
@@ -43,12 +43,13 @@ class BBLogger(Logger):
         Logger.__init__(self, name)
 
     def bbdebug(self, level, msg, *args, **kwargs):
+        loglevel = logging.DEBUG - level + 1
         if not bb.event.worker_pid:
-            if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]):
+            if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
                 return
             if level > (bb.msg.loggerDefaultDebugLevel):
                 return
-        return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
+        return self.log(loglevel, msg, *args, **kwargs)
 
     def plain(self, msg, *args, **kwargs):
         return self.log(logging.INFO + 1, msg, *args, **kwargs)
bitbake/lib/bb/msg.py +7 -10
index 33c0e2fa19..d1b0e929d4 100644
@@ -138,7 +138,7 @@ class BBLogFilterStdOut(BBLogFilter):
 loggerDefaultDebugLevel = 0
 loggerDefaultVerbose = False
 loggerVerboseLogs = False
-loggerDefaultDomains = []
+loggerDefaultDomains = {}
 
 def init_msgconfig(verbose, debug, debug_domains=None):
     """
@@ -148,15 +148,16 @@ def init_msgconfig(verbose, debug, debug_domains=None):
     bb.msg.loggerDefaultVerbose = verbose
     if verbose:
         bb.msg.loggerVerboseLogs = True
+
+    bb.msg.loggerDefaultDomains = {}
     if debug_domains:
-        bb.msg.loggerDefaultDomains = debug_domains
-    else:
-        bb.msg.loggerDefaultDomains = []
+        for (domainarg, iterator) in groupby(debug_domains):
+            dlevel = len(tuple(iterator))
+            bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
 
 def constructLogOptions():
     debug = loggerDefaultDebugLevel
     verbose = loggerDefaultVerbose
-    domains = loggerDefaultDomains
 
     if debug:
         level = BBLogFormatter.DEBUG - debug + 1
@@ -165,11 +166,7 @@ def constructLogOptions():
     else:
         level = BBLogFormatter.NOTE
 
-    debug_domains = {}
-    for (domainarg, iterator) in groupby(domains):
-        dlevel = len(tuple(iterator))
-        debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
-    return level, debug_domains
+    return level, loggerDefaultDomains
 
 def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
     level, debug_domains = constructLogOptions()
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/15 bitbake: knotty: Handle logging messages with specific logger

In reply to Joshua Watt

Handles the log messages from the bitbake server with the specific logger that the event originated from. This allows hierarchical logging configurations to work as expected.

---
 bitbake/lib/bb/ui/knotty.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/bb/ui/knotty.py +1 -1
index cbb289b05f..a6a92b920f 100644
@@ -522,7 +522,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                         event.msg = taskinfo['title'] + ': ' + event.msg
                 if hasattr(event, 'fn'):
                     event.msg = event.fn + ': ' + event.msg
-                logger.handle(event)
+                logging.getLogger(event.name).handle(event)
                 continue
 
             if isinstance(event, bb.build.TaskFailedSilent):
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/15 bitbake: lib/bb/msg.py: Use log level instead of debug count

In reply to Joshua Watt

Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used

---
 bitbake/bin/bitbake-worker |  2 +-
 bitbake/lib/bb/__init__.py |  2 +-
 bitbake/lib/bb/msg.py      | 22 +++++++++-------------
 bitbake/lib/bb/runqueue.py |  2 +-
 4 files changed, 12 insertions(+), 16 deletions(-)
bitbake/bin/bitbake-worker +1 -1
index 1e641e81c2..fc28a34d8a 100755
@@ -414,7 +414,7 @@ class BitbakeWorker(object):
 
     def handle_workerdata(self, data):
         self.workerdata = pickle.loads(data)
-        bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"]
+        bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"]
         bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"]
         bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"]
         bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
bitbake/lib/bb/__init__.py +1 -1
index acd4af13a8..dc5e91e29a 100644
@@ -47,7 +47,7 @@ class BBLogger(Logger):
         if not bb.event.worker_pid:
             if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
                 return
-            if level > (bb.msg.loggerDefaultDebugLevel):
+            if loglevel > bb.msg.loggerDefaultLogLevel:
                 return
         return self.log(loglevel, msg, *args, **kwargs)
 
bitbake/lib/bb/msg.py +9 -13
index d1b0e929d4..ea6a9543f7 100644
@@ -135,7 +135,7 @@ class BBLogFilterStdOut(BBLogFilter):
 # Message control functions
 #
 
-loggerDefaultDebugLevel = 0
+loggerDefaultLogLevel = BBLogFormatter.NOTE
 loggerDefaultVerbose = False
 loggerVerboseLogs = False
 loggerDefaultDomains = {}
@@ -144,11 +144,17 @@ def init_msgconfig(verbose, debug, debug_domains=None):
     """
     Set default verbosity and debug levels config the logger
     """
-    bb.msg.loggerDefaultDebugLevel = debug
     bb.msg.loggerDefaultVerbose = verbose
     if verbose:
         bb.msg.loggerVerboseLogs = True
 
+    if debug:
+        bb.msg.loggerDefaultLogLevel = BBLogFormatter.DEBUG - debug + 1
+    elif verbose:
+        bb.msg.loggerDefaultLogLevel = BBLogFormatter.VERBOSE
+    else:
+        bb.msg.loggerDefaultLogLevel = BBLogFormatter.NOTE
+
     bb.msg.loggerDefaultDomains = {}
     if debug_domains:
         for (domainarg, iterator) in groupby(debug_domains):
@@ -156,17 +162,7 @@ def init_msgconfig(verbose, debug, debug_domains=None):
             bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
 
 def constructLogOptions():
-    debug = loggerDefaultDebugLevel
-    verbose = loggerDefaultVerbose
-
-    if debug:
-        level = BBLogFormatter.DEBUG - debug + 1
-    elif verbose:
-        level = BBLogFormatter.VERBOSE
-    else:
-        level = BBLogFormatter.NOTE
-
-    return level, loggerDefaultDomains
+    return loggerDefaultLogLevel, loggerDefaultDomains
 
 def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
     level, debug_domains = constructLogOptions()
bitbake/lib/bb/runqueue.py +1 -1
index 32966b4f7c..4106fa4bc4 100644
@@ -1254,7 +1254,7 @@ class RunQueue:
             "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
             "fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv,
             "sigdata" : bb.parse.siggen.get_taskdata(),
-            "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
+            "logdefaultlevel" : bb.msg.loggerDefaultLogLevel,
             "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
             "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
             "logdefaultdomain" : bb.msg.loggerDefaultDomains,
Signature · 1 line
2.17.1
#4 lore raw

PATCH 4/15 bitbake: lib/bb/msg.py: Add repr for BBLogFormatter

In reply to Joshua Watt

Adds a __repr__ function for BBLogFormatter. This allows it to get a human readable string when printed using the logging_tree module

---
 bitbake/lib/bb/msg.py | 3 +++
 1 file changed, 3 insertions(+)
bitbake/lib/bb/msg.py +3 -0
index ea6a9543f7..c70fd80806 100644
@@ -98,6 +98,9 @@ class BBLogFormatter(logging.Formatter):
     def enable_color(self):
         self.color_enabled = True
 
+    def __repr__(self):
+        return "%s fmt='%s' color=%s" % (self.__class__.__name__, self._fmt, "True" if self.color_enabled else "False")
+
 class BBLogFilter(object):
     def __init__(self, handler, level, debug_domains):
         self.stdlevel = level
Signature · 1 line
2.17.1
#5 lore raw

PATCH 5/15 bitbake: knotty: Add commented logging_tree code

In reply to Joshua Watt

Adds a comment section that can be easily uncommented to enable dumping the logging tree. This module is extremely useful for debugging issued with logging configuration

---
 bitbake/lib/bb/ui/knotty.py | 5 +++++
 1 file changed, 5 insertions(+)
bitbake/lib/bb/ui/knotty.py +5 -0
index a6a92b920f..aac12cd479 100644
@@ -430,6 +430,11 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     llevel, debug_domains = bb.msg.constructLogOptions()
     server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
 
+    # The logging_tree module is *extremely* helpful in debugging logging
+    # domains. Uncomment here to dump the logging tree when bitbake starts
+    #import logging_tree
+    #logging_tree.printout()
+
     universe = False
     if not params.observe_only:
         params.updateFromServer(server)
Signature · 1 line
2.17.1
#6 lore raw

PATCH 6/15 bitbake: lib/bb/msg.py: Add filter utilities

In reply to Joshua Watt

Adds generic filter utilities that can be used to filter when the log level is greater than or equal to a defined value, or below a defined value, as well as a generic function to translate a string to a logging level (or bitbake logging level)

---
 bitbake/lib/bb/msg.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
bitbake/lib/bb/msg.py +35 -0
index c70fd80806..8561826a6e 100644
@@ -135,6 +135,28 @@ class BBLogFilterStdOut(BBLogFilter):
             return True
         return False
 
+class LogFilterGEQLevel(logging.Filter):
+    def __init__(self, level):
+        self.strlevel = str(level)
+        self.level = stringToLevel(level)
+
+    def __repr__(self):
+        return "%s level >= %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+    def filter(self, record):
+        return (record.levelno >= self.level)
+
+class LogFilterLTLevel(logging.Filter):
+    def __init__(self, level):
+        self.strlevel = str(level)
+        self.level = stringToLevel(level)
+
+    def __repr__(self):
+        return "%s level < %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+    def filter(self, record):
+        return (record.levelno < self.level)
+
 # Message control functions
 #
 
@@ -175,6 +197,19 @@ def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
 
     cls(handler, level, debug_domains)
 
+def stringToLevel(level):
+    try:
+        return int(level)
+    except ValueError:
+        pass
+
+    try:
+        return getattr(logging, level)
+    except AttributeError:
+        pass
+
+    return getattr(BBLogFormatter, level)
+
 #
 # Message handling functions
 #
Signature · 1 line
2.17.1
#7 lore raw

PATCH 7/15 bitbake: lib/bb/msg.py: Remove unused filters

In reply to Joshua Watt

Now that the filter are described using the python logging structure, these classes are no longer needed.

---
 bitbake/lib/bb/msg.py | 16 ----------------
 1 file changed, 16 deletions(-)
bitbake/lib/bb/msg.py +0 -16
index 8561826a6e..cab079e333 100644
@@ -119,22 +119,6 @@ class BBLogFilter(object):
             return True
         return False
 
-class BBLogFilterStdErr(BBLogFilter):
-    def filter(self, record):
-        if not BBLogFilter.filter(self, record):
-            return False
-        if record.levelno >= logging.ERROR:
-            return True
-        return False
-
-class BBLogFilterStdOut(BBLogFilter):
-    def filter(self, record):
-        if not BBLogFilter.filter(self, record):
-            return False
-        if record.levelno < logging.ERROR:
-            return True
-        return False
-
 class LogFilterGEQLevel(logging.Filter):
     def __init__(self, level):
         self.strlevel = str(level)
Signature · 1 line
2.17.1
#8 lore raw

PATCH 8/15 bitbake: lib/bb/msg.py: Add helper to set logging config

In reply to Joshua Watt

Adds a helper function to setup the structure logging information in bitbake. The helper function takes a default configuration and an optional path to a user config file that can be merged into the default.

---
 bitbake/lib/bb/msg.py | 93 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
bitbake/lib/bb/msg.py +93 -0
index cab079e333..6259af037f 100644
@@ -13,6 +13,7 @@ Message handling infrastructure for bitbake
 import sys
 import copy
 import logging
+import logging.config
 from itertools import groupby
 import bb
 import bb.event
@@ -227,3 +228,95 @@ def has_console_handler(logger):
             if handler.stream in [sys.stderr, sys.stdout]:
                 return True
     return False
+
+def setLoggingConfig(defaultconfig, userconfigfile=None):
+    logconfig = copy.deepcopy(defaultconfig)
+
+    if userconfigfile:
+        with open(userconfigfile, 'r') as f:
+            if userconfigfile.endswith('.yml') or userconfigfile.endswith('.yaml'):
+                import yaml
+                userconfig = yaml.load(f)
+            elif userconfigfile.endswith('.json') or userconfigfile.endswith('.cfg'):
+                import json
+                userconfig = json.load(f)
+            else:
+                raise BaseException("Unrecognized file format: %s" % userconfigfile)
+
+            if userconfig.get('bitbake_merge', True):
+                # Merge config with the default config
+                if userconfig.get('version') != logconfig['version']:
+                    raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version')))
+
+                # Set some defaults to make merging easier
+                userconfig.setdefault("loggers", {})
+
+                # If a handler, formatter, or filter is defined in the user
+                # config, it will replace an existing one in the default config
+                for k in ("handlers", "formatters", "filters"):
+                    logconfig.setdefault(k, {}).update(userconfig.get(k, {}))
+
+                seen_loggers = set()
+                for name, l in logconfig["loggers"].items():
+                    # If the merge option is set, merge the handlers and
+                    # filters. Otherwise, if it is False, this logger won't get
+                    # add to the set of seen loggers and will replace the
+                    # existing one
+                    if l.get('bitbake_merge', True):
+                        ulogger = userconfig["loggers"].setdefault(name, {})
+                        ulogger.setdefault("handlers", [])
+                        ulogger.setdefault("filters", [])
+
+                        # Merge lists
+                        l.setdefault("handlers", []).extend(ulogger["handlers"])
+                        l.setdefault("filters", []).extend(ulogger["filters"])
+
+                        # Replace other properties if present
+                        if "level" in ulogger:
+                            l["level"] = ulogger["level"]
+
+                        if "propagate" in ulogger:
+                            l["propagate"] = ulogger["propagate"]
+
+                        seen_loggers.add(name)
+
+                # Add all loggers present in the user config, but not any that
+                # have already been processed
+                for name in set(userconfig["loggers"].keys()) - seen_loggers:
+                    logconfig["loggers"][name] = userconfig["loggers"][name]
+
+            else:
+                # Replace the entire default config
+                logconfig = userconfig
+
+    # Convert all level parameters to integers in case users want to use the
+    # bitbake defined level names
+    for h in logconfig["handlers"].values():
+        if "level" in h:
+            h["level"] = bb.msg.stringToLevel(h["level"])
+
+    for l in logconfig["loggers"].values():
+        if "level" in l:
+            l["level"] = bb.msg.stringToLevel(l["level"])
+
+    logging.config.dictConfig(logconfig)
+
+    # The user may have specified logging domains they want at a higher debug
+    # level than the standard.
+    for name, l in logconfig["loggers"].items():
+        if not name.startswith("BitBake."):
+            continue
+
+        if not "level" in l:
+            continue
+
+        curlevel = bb.msg.loggerDefaultDomains.get(name)
+        # Note: level parameter should already be a int because of conversion
+        # above
+        newlevel = int(l["level"])
+        if curlevel is None or newlevel < curlevel:
+            bb.msg.loggerDefaultDomains[name] = newlevel
+
+        # TODO: I don't think that setting the global log level should be necessary
+        #if newlevel < bb.msg.loggerDefaultLogLevel:
+        #    bb.msg.loggerDefaultLogLevel = newlevel
Signature · 1 line
2.17.1
#9 lore raw

PATCH 9/15 bitbake: knotty: Remove dependency on format variable

In reply to Joshua Watt

Passing around the log formatter variable was unnecessary since the log levels of interest can be accesses as class members of bb.msg.BBLogFormatter. Switching to do this will make using the structured python logging much easier, since it can be difficult to extract out the formatter for a specific handler.

---
 bitbake/lib/bb/tinfoil.py   |  4 +---
 bitbake/lib/bb/ui/knotty.py | 32 ++++++++++++++++++++------------
 2 files changed, 21 insertions(+), 15 deletions(-)
bitbake/lib/bb/tinfoil.py +1 -3
index 9560eb5b49..5c5be456e2 100644
@@ -735,11 +735,9 @@ class Tinfoil:
                         console = handler
                     elif handler.stream == sys.stderr:
                         errconsole = handler
-            format_str = "%(levelname)s: %(message)s"
-            format = bb.msg.BBLogFormatter(format_str)
             helper.shutdown = 0
             parseprogress = None
-            termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet)
+            termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, quiet=self.quiet)
             try:
                 while True:
                     try:
bitbake/lib/bb/ui/knotty.py +20 -12
index aac12cd479..d5dce7172a 100644
@@ -109,12 +109,11 @@ def pluralise(singular, plural, qty):
 
 
 class InteractConsoleLogFilter(logging.Filter):
-    def __init__(self, tf, format):
+    def __init__(self, tf):
         self.tf = tf
-        self.format = format
 
     def filter(self, record):
-        if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+        if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
             return False
         self.tf.clearFooter()
         return True
@@ -150,7 +149,7 @@ class TerminalFilter(object):
                 cr = (25, 80)
         return cr
 
-    def __init__(self, main, helper, console, errconsole, format, quiet):
+    def __init__(self, main, helper, console, errconsole, quiet):
         self.main = main
         self.helper = helper
         self.cuu = None
@@ -180,7 +179,16 @@ class TerminalFilter(object):
             termios.tcsetattr(fd, termios.TCSADRAIN, new)
             curses.setupterm()
             if curses.tigetnum("colors") > 2:
-                format.enable_color()
+                if console:
+                    try:
+                        console.formatter.enable_color()
+                    except AttributeError:
+                        pass
+                if errconsole:
+                    try:
+                        errconsole.formatter.enable_color()
+                    except AttributeError:
+                        pass
             self.ed = curses.tigetstr("ed")
             if self.ed:
                 self.cuu = curses.tigetstr("cuu")
@@ -197,9 +205,9 @@ class TerminalFilter(object):
             bb.note("Unable to use interactive mode for this terminal, using fallback")
             return
         if console:
-            console.addFilter(InteractConsoleLogFilter(self, format))
+            console.addFilter(InteractConsoleLogFilter(self))
         if errconsole:
-            errconsole.addFilter(InteractConsoleLogFilter(self, format))
+            errconsole.addFilter(InteractConsoleLogFilter(self))
 
         self.main_progress = None
 
@@ -469,7 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     printinterval = 5000
     lastprint = time.time()
 
-    termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
+    termfilter = tf(main, helper, console, errconsole, params.options.quiet)
     atexit.register(termfilter.finish)
 
     while True:
@@ -508,21 +516,21 @@ def main(server, eventHandler, params, tf = TerminalFilter):
             if isinstance(event, logging.LogRecord):
                 lastprint = time.time()
                 printinterval = 5000
-                if event.levelno >= format.ERROR:
+                if event.levelno >= bb.msg.BBLogFormatter.ERROR:
                     errors = errors + 1
                     return_value = 1
-                elif event.levelno == format.WARNING:
+                elif event.levelno == bb.msg.BBLogFormatter.WARNING:
                     warnings = warnings + 1
 
                 if event.taskpid != 0:
                     # For "normal" logging conditions, don't show note logs from tasks
                     # but do show them if the user has changed the default log level to
                     # include verbose/debug messages
-                    if event.levelno <= format.NOTE and (event.levelno < llevel or (event.levelno == format.NOTE and llevel != format.VERBOSE)):
+                    if event.levelno <= bb.msg.BBLogFormatter.NOTE and (event.levelno < llevel or (event.levelno == bb.msg.BBLogFormatter.NOTE and llevel != bb.msg.BBLogFormatter.VERBOSE)):
                         continue
 
                     # Prefix task messages with recipe/task
-                    if event.taskpid in helper.pidmap and event.levelno != format.PLAIN:
+                    if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN:
                         taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]]
                         event.msg = taskinfo['title'] + ': ' + event.msg
                 if hasattr(event, 'fn'):
Signature · 1 line
2.17.1
#10 lore raw

PATCH 10/15 bitbake: knotty: Setup logs with config helper

In reply to Joshua Watt

Sets up logging in knotty to use python's structured logging config and the bb.msg.setLoggingConfig() helper to setup logging. This allows the user to specify additional logging mechanism in a config file with BB_LOGCONFIG

---
 bitbake/lib/bb/ui/knotty.py | 134 ++++++++++++++++++++++++++----------
 1 file changed, 96 insertions(+), 38 deletions(-)
bitbake/lib/bb/ui/knotty.py +96 -38
index d5dce7172a..386f27876d 100644
@@ -370,7 +370,11 @@ def _log_settings_from_server(server, observe_only):
     if error:
         logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
         raise BaseException(error)
-    return includelogs, loglines, consolelogfile
+    logconfigfile, error = server.runCommand([cmd, "BB_LOGCONFIG"])
+    if error:
+        logger.error("Unable to get the value of BB_LOGCONFIG variable: %s" % error)
+        raise BaseException(error)
+    return includelogs, loglines, consolelogfile, logconfigfile
 
 _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.LogRecord",
               "bb.build.TaskFailed", "bb.build.TaskBase", "bb.event.ParseStarted",
@@ -387,7 +391,87 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     if not params.observe_only:
         params.updateToServer(server, os.environ.copy())
 
-    includelogs, loglines, consolelogfile = _log_settings_from_server(server, params.observe_only)
+    includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only)
+
+    loglevel, _ = bb.msg.constructLogOptions()
+
+    if params.options.quiet == 0:
+        console_loglevel = loglevel
+    elif params.options.quiet > 2:
+        console_loglevel = bb.msg.BBLogFormatter.ERROR
+    else:
+        console_loglevel = bb.msg.BBLogFormatter.WARNING
+
+    logconfig = {
+        "version": 1,
+        "handlers": {
+            "BitBake.console": {
+                "class": "logging.StreamHandler",
+                "formatter": "BitBake.consoleFormatter",
+                "level": console_loglevel,
+                "stream": "ext://sys.stdout",
+                "filters": ["BitBake.stdoutFilter"],
+            },
+            "BitBake.errconsole": {
+                "class": "logging.StreamHandler",
+                "formatter": "BitBake.consoleFormatter",
+                "level": loglevel,
+                "stream": "ext://sys.stderr",
+                "filters": ["BitBake.stderrFilter"],
+            },
+        },
+        "formatters": {
+            # This format instance will get color output enabled by the
+            # terminal
+            "BitBake.consoleFormatter" : {
+                "()": "bb.msg.BBLogFormatter",
+                "format": "%(levelname)s: %(message)s"
+            },
+            # The file log requires a separate instance so that it doesn't get
+            # color enabled
+            "BitBake.logfileFormatter": {
+                "()": "bb.msg.BBLogFormatter",
+                "format": "%(levelname)s: %(message)s"
+            }
+        },
+        "filters": {
+            "BitBake.stdoutFilter": {
+                "()": "bb.msg.LogFilterLTLevel",
+                "level": "ERROR"
+            },
+            "BitBake.stderrFilter": {
+                "()": "bb.msg.LogFilterGEQLevel",
+                "level": "ERROR"
+            }
+        },
+        "loggers": {
+            "BitBake": {
+                "level": loglevel,
+                "handlers": ["BitBake.console", "BitBake.errconsole"],
+            }
+        },
+        "disable_existing_loggers": False
+    }
+
+    # Enable the console log file if enabled
+    if consolelogfile and not params.options.show_environment and not params.options.show_versions:
+        logconfig["handlers"]["BitBake.consolelog"] ={
+            "class": "logging.FileHandler",
+            "formatter": "BitBake.logfileFormatter",
+            "level": "INFO",
+            "filename": consolelogfile,
+        }
+        logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog")
+
+        bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+        loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+        bb.utils.remove(loglink)
+        try:
+           os.symlink(os.path.basename(consolelogfile), loglink)
+        except OSError:
+           pass
+
+    bb.msg.setLoggingConfig(logconfig, logconfigfile)
 
     if sys.stdin.isatty() and sys.stdout.isatty():
         log_exec_tty = True
@@ -396,23 +480,16 @@ def main(server, eventHandler, params, tf = TerminalFilter):
 
     helper = uihelper.BBUIHelper()
 
-    console = logging.StreamHandler(sys.stdout)
-    errconsole = logging.StreamHandler(sys.stderr)
-    format_str = "%(levelname)s: %(message)s"
-    format = bb.msg.BBLogFormatter(format_str)
-    if params.options.quiet == 0:
-        forcelevel = None
-    elif params.options.quiet > 2:
-        forcelevel = bb.msg.BBLogFormatter.ERROR
-    else:
-        forcelevel = bb.msg.BBLogFormatter.WARNING
-    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel)
-    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
-    console.setFormatter(format)
-    errconsole.setFormatter(format)
-    if not bb.msg.has_console_handler(logger):
-        logger.addHandler(console)
-        logger.addHandler(errconsole)
+    # Look for the specially designated handlers which need to be passed to the
+    # terminal handler
+    console = None
+    errconsole = None
+    for h in logger.handlers:
+        name = getattr(h, '_name', None)
+        if name == 'BitBake.console':
+            console = h
+        elif name == 'BitBake.errconsole':
+            errconsole = h
 
     bb.utils.set_process_name("KnottyUI")
 
@@ -420,21 +497,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
         server.terminateServer()
         return
 
-    consolelog = None
-    if consolelogfile and not params.options.show_environment and not params.options.show_versions:
-        bb.utils.mkdirhier(os.path.dirname(consolelogfile))
-        conlogformat = bb.msg.BBLogFormatter(format_str)
-        consolelog = logging.FileHandler(consolelogfile)
-        bb.msg.addDefaultlogFilter(consolelog)
-        consolelog.setFormatter(conlogformat)
-        logger.addHandler(consolelog)
-        loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
-        bb.utils.remove(loglink)
-        try:
-           os.symlink(os.path.basename(consolelogfile), loglink)
-        except OSError:
-           pass
-
     llevel, debug_domains = bb.msg.constructLogOptions()
     server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
 
@@ -761,8 +823,4 @@ def main(server, eventHandler, params, tf = TerminalFilter):
         if e.errno == errno.EPIPE:
             pass
 
-    if consolelog:
-        logger.removeHandler(consolelog)
-        consolelog.close()
-
     return return_value
Signature · 1 line
2.17.1
#11 lore raw

PATCH 11/15 bitbake: worker: Remove unnecessary addDefaultLogFilter

In reply to Joshua Watt

Adding the default log filter here is unnecessary because there are no defined logging domains when it is called, which means it does no actual filtering.

---
 bitbake/bin/bitbake-worker | 1 -
 1 file changed, 1 deletion(-)
bitbake/bin/bitbake-worker +0 -1
index fc28a34d8a..97cc0fd60f 100755
@@ -65,7 +65,6 @@ if 0:
     format_str = "%(levelname)s: %(message)s"
     conlogformat = bb.msg.BBLogFormatter(format_str)
     consolelog = logging.FileHandler(logfilename)
-    bb.msg.addDefaultlogFilter(consolelog)
     consolelog.setFormatter(conlogformat)
     logger.addHandler(consolelog)
 
Signature · 1 line
2.17.1
#12 lore raw

PATCH 12/15 bitbake: Log hash equivalence with a different logger

In reply to Joshua Watt

Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging.

---
 bitbake/lib/bb/runqueue.py | 11 ++++++-----
 bitbake/lib/bb/siggen.py   | 17 +++++++++--------
 2 files changed, 15 insertions(+), 13 deletions(-)
bitbake/lib/bb/runqueue.py +6 -5
index 4106fa4bc4..e84890b8b7 100644
@@ -27,6 +27,7 @@ import pprint
 
 bblogger = logging.getLogger("BitBake")
 logger = logging.getLogger("BitBake.RunQueue")
+hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv")
 
 __find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
 
@@ -2263,7 +2264,7 @@ class RunQueueExecute:
             self.updated_taskhash_queue.remove((tid, unihash))
 
             if unihash != self.rqdata.runtaskentries[tid].unihash:
-                logger.info("Task %s unihash changed to %s" % (tid, unihash))
+                hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
                 self.rqdata.runtaskentries[tid].unihash = unihash
                 bb.parse.siggen.set_unihash(tid, unihash)
                 toprocess.add(tid)
@@ -2308,7 +2309,7 @@ class RunQueueExecute:
                 elif tid in self.scenequeue_covered or tid in self.sq_live:
                     # Already ran this setscene task or it running. Report the new taskhash
                     bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
-                    logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+                    hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
                     remapped = True
 
                 if not remapped:
@@ -2327,7 +2328,7 @@ class RunQueueExecute:
             for mc in self.rq.fakeworker:
                 self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
 
-            logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
+            hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
 
         for tid in changed:
             if tid not in self.rqdata.runq_setscene_tids:
@@ -2346,7 +2347,7 @@ class RunQueueExecute:
             # Check no tasks this covers are running
             for dep in self.sqdata.sq_covered_tasks[tid]:
                 if dep in self.runq_running and dep not in self.runq_complete:
-                    logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
+                    hashequiv_logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
                     valid = False
                     break
             if not valid:
@@ -2409,7 +2410,7 @@ class RunQueueExecute:
 
         for (tid, harddepfail, origvalid) in update_tasks:
             if tid in self.sqdata.valid and not origvalid:
-                logger.info("Setscene task %s became valid" % tid)
+                hashequiv_logger.info("Setscene task %s became valid" % tid)
             if harddepfail:
                 self.sq_task_failoutright(tid)
 
bitbake/lib/bb/siggen.py +9 -8
index c2d0c736cf..0357b544b5 100644
@@ -16,6 +16,7 @@ from bb import runqueue
 import hashserv
 
 logger = logging.getLogger('BitBake.SigGen')
+hashequiv_logger = logging.getLogger('BitBake.SigGen.HashEquiv')
 
 def init(d):
     siggens = [obj for obj in globals().values()
@@ -502,9 +503,9 @@ class SignatureGeneratorUniHashMixIn(object):
                 # A unique hash equal to the taskhash is not very interesting,
                 # so it is reported it at debug level 2. If they differ, that
                 # is much more interesting, so it is reported at debug level 1
-                bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
+                hashequiv_logger.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
             else:
-                bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
+                hashequiv_logger.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
         except hashserv.client.HashConnectionError as e:
             bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
 
@@ -578,12 +579,12 @@ class SignatureGeneratorUniHashMixIn(object):
                 new_unihash = data['unihash']
 
                 if new_unihash != unihash:
-                    bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
+                    hashequiv_logger.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
                     bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
                     self.set_unihash(tid, new_unihash)
                     d.setVar('BB_UNIHASH', new_unihash)
                 else:
-                    bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
+                    hashequiv_logger.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
             except hashserv.client.HashConnectionError as e:
                 bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
         finally:
@@ -606,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
                 method = method + self.extramethod[tid]
 
             data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
-            bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+            hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
 
             if data is None:
                 bb.warn("Server unable to handle unihash report")
@@ -615,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
             finalunihash = data['unihash']
 
             if finalunihash == current_unihash:
-                bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+                hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
             elif finalunihash == wanted_unihash:
-                bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+                hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
                 self.set_unihash(tid, finalunihash)
                 return True
             else:
                 # TODO: What to do here?
-                bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+                hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
 
         except hashserv.client.HashConnectionError as e:
             bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
Signature · 1 line
2.17.1
#13 lore raw

PATCH 13/15 bitbake: Add autobuilder logging configuration

In reply to Joshua Watt

Adds a configuration file that the autobuilder can use to capture interesting logging domains above the ones that show up for normal users on stdout/stderr.

---
 bitbake/contrib/autobuilderlog.json | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 bitbake/contrib/autobuilderlog.json
bitbake/contrib/autobuilderlog.json new file +27 -0
new file mode 100644
index 0000000000..103a1141f6
@@ -0,0 +1,27 @@
+{
+    "version": 1,
+    "handlers": {
+        "autobuilderlog": {
+            "class": "logging.FileHandler",
+            "formatter": "logfileFormatter",
+            "level": "DEBUG",
+            "filename": "autobuilder.log",
+            "mode": "w"
+        }
+    },
+    "formatters": {
+            "logfileFormatter": {
+                "format": "%(name)s: %(levelname)s: %(message)s"
+            }
+    },
+    "loggers": {
+        "BitBake.SigGen.HashEquiv": {
+            "level": "VERBOSE",
+            "handlers": ["autobuilderlog"]
+        },
+        "BitBake.RunQueue.HashEquiv": {
+            "level": "VERBOSE",
+            "handlers": ["autobuilderlog"]
+        }
+    }
+}
Signature · 1 line
2.17.1
#14 lore raw

PATCH 14/15 bitbake: Lower hash equivalence logging

In reply to Joshua Watt

Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json)

[YOCTO #13813]

---
 bitbake/lib/bb/runqueue.py | 6 +++---
 bitbake/lib/bb/siggen.py   | 8 ++++----
 2 files changed, 7 insertions(+), 7 deletions(-)
bitbake/lib/bb/runqueue.py +3 -3
index e84890b8b7..cef9b0fbbf 100644
@@ -2264,7 +2264,7 @@ class RunQueueExecute:
             self.updated_taskhash_queue.remove((tid, unihash))
 
             if unihash != self.rqdata.runtaskentries[tid].unihash:
-                hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
+                hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash))
                 self.rqdata.runtaskentries[tid].unihash = unihash
                 bb.parse.siggen.set_unihash(tid, unihash)
                 toprocess.add(tid)
@@ -2309,7 +2309,7 @@ class RunQueueExecute:
                 elif tid in self.scenequeue_covered or tid in self.sq_live:
                     # Already ran this setscene task or it running. Report the new taskhash
                     bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
-                    hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+                    hashequiv_logger.verbose("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
                     remapped = True
 
                 if not remapped:
@@ -2410,7 +2410,7 @@ class RunQueueExecute:
 
         for (tid, harddepfail, origvalid) in update_tasks:
             if tid in self.sqdata.valid and not origvalid:
-                hashequiv_logger.info("Setscene task %s became valid" % tid)
+                hashequiv_logger.verbose("Setscene task %s became valid" % tid)
             if harddepfail:
                 self.sq_task_failoutright(tid)
 
bitbake/lib/bb/siggen.py +4 -4
index 0357b544b5..8bfc45235d 100644
@@ -607,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
                 method = method + self.extramethod[tid]
 
             data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
-            hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+            hashequiv_logger.verbose('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
 
             if data is None:
                 bb.warn("Server unable to handle unihash report")
@@ -616,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
             finalunihash = data['unihash']
 
             if finalunihash == current_unihash:
-                hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+                hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash))
             elif finalunihash == wanted_unihash:
-                hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+                hashequiv_logger.verbose('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
                 self.set_unihash(tid, finalunihash)
                 return True
             else:
                 # TODO: What to do here?
-                hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+                hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
 
         except hashserv.client.HashConnectionError as e:
             bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
Signature · 1 line
2.17.1
#15 lore raw

PATCH 15/15 bitbake-user-manual: Add documentation for BB_LOGCONFIG

In reply to Joshua Watt

Adds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging.

---
 .../bitbake-user-manual-execution.xml         | 97 +++++++++++++++++++
 .../bitbake-user-manual-ref-variables.xml     | 11 +++
 2 files changed, 108 insertions(+)
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +97 -0
index 8f606676b4..3b31f748cc 100644
@@ -929,4 +929,101 @@
             section.
         </para>
     </section>
+
+    <section id="logging">
+        <title>Logging</title>
+        <para>
+            In addition to the standard command line option to control how
+            verbose builds are when execute, bitbake also supports user defined
+            configuration of the
+            <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+            facilities through the
+            <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+            variable. This variable defines a json or yaml
+            <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+            that will be intelligently merged into the default configuration.
+            The logging configuration is merged using the following rules:
+            <itemizedlist>
+                <listitem><para>
+                    The user defined configuration will completely replace the default
+                    configuration if top level key
+                    <filename>bitbake_merge</filename> is set to the value
+                    <filename>False</filename>. In this case, all other rules
+                    are ignored.
+                </para></listitem>
+                <listitem><para>
+                    The user configuration must have a top level
+                    <filename>version</filename> which must match the value of
+                    the default configuration.
+                </para></listitem>
+                <listitem><para>
+                    Any keys defined in the <filename>handlers</filename>,
+                    <filename>formatters</filename>, or <filename>filters</filename>,
+                    will be merged into the same section in the default
+                    configuration, with the user specified keys taking
+                    replacing a default one if there is a conflict. In
+                    practice, this means that if both the default configuration
+                    and user configuration specify a handler named
+                    <filename>myhandler</filename>, the user defined one will
+                    replace the default. To prevent the user from inadvertently
+                    replacing a default handler, formatter, or filter, all of
+                    the default ones are named with a prefix of
+                    "<filename>BitBake.</filename>"
+                </para></listitem>
+                <listitem><para>
+                    If a logger is defined by the user with the key
+                    <filename>bitbake_merge</filename> set to
+                    <filename>False</filename>, that logger will be completely
+                    replaced by user configuration. In this case, no other
+                    rules will apply to that logger.
+                </listitem></para>
+                <listitem><para>
+                    All user defined <filename>filter</filename> and
+                    <filename>handlers</filename> properties for a given logger
+                    will be merged with corresponding properties from the
+                    default logger. For example, if the user configuration adds
+                    a filter called <filename>myFilter</filename> to the
+                    <filename>BitBake.SigGen</filename>, and the default
+                    configuration adds a filter called
+                    <filename>BitBake.defaultFilter</filename>, both filters
+                    will be applied to the logger
+                </listitem></para>
+            </itemizedlist>
+        </para>
+
+        <para>
+            As an example, consider the following user logging configuration
+            file which logs all Hash Equivalence related messages of VERBOSE or
+            higher to a file called <filename>hashequiv.log</filename>
+            <literallayout class='monospaced'>
+    {
+        "version": 1,
+        "handlers": {
+            "autobuilderlog": {
+                "class": "logging.FileHandler",
+                "formatter": "logfileFormatter",
+                "level": "DEBUG",
+                "filename": "hashequiv.log",
+                "mode": "w"
+            }
+        },
+        "formatters": {
+                "logfileFormatter": {
+                    "format": "%(name)s: %(levelname)s: %(message)s"
+                }
+        },
+        "loggers": {
+            "BitBake.SigGen.HashEquiv": {
+                "level": "VERBOSE",
+                "handlers": ["autobuilderlog"]
+            },
+            "BitBake.RunQueue.HashEquiv": {
+                "level": "VERBOSE",
+                "handlers": ["autobuilderlog"]
+            }
+        }
+    }
+            </literallayout>
+        </para>
+    </section>
 </chapter>
bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +11 -0
index bae01d90c0..c4bd1f2584 100644
@@ -539,6 +539,17 @@
             </glossdef>
         </glossentry>
 
+        <glossentry id='var-bb-BB_LOGCONFIG'><glossterm>BB_LOGCONFIG</glossterm>
+            <glossdef>
+                <para>
+                    Specifies the name of a config file that contains the user
+                    logging configuration. See
+                    <link linkend="logging">Logging</link> for additional
+                    information
+                </para>
+            </glossdef>
+        </glossentry>
+
         <glossentry id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
             <glossdef>
                 <para>
Signature · 1 line
2.17.1
#16 lore raw

Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG

Quoted 89 lines · Joshua Watt

Adds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging.

---
.../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
.../bitbake-user-manual-ref-variables.xml | 11 +++

2 files changed, 108 insertions(+)
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +71 -0
index 8f606676b4..3b31f748cc 100644
@@ -929,4 +929,101 @@
            section.
        </para>
    </section>
+
+    <section id="logging">
+        <title>Logging</title>
+        <para>
+            In addition to the standard command line option to control how
+            verbose builds are when execute, bitbake also supports user defined
+            configuration of the
+            <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+            facilities through the
+            <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+            variable. This variable defines a json or yaml
+            <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+            that will be intelligently merged into the default configuration.
+            The logging configuration is merged using the following rules:
+            <itemizedlist>
+                <listitem><para>
+                    The user defined configuration will completely replace the default
+                    configuration if top level key
+                    <filename>bitbake_merge</filename> is set to the value
+                    <filename>False</filename>. In this case, all other rules
+                    are ignored.
+                </para></listitem>
+                <listitem><para>
+                    The user configuration must have a top level
+                    <filename>version</filename> which must match the value of
+                    the default configuration.
+                </para></listitem>
+                <listitem><para>
+                    Any keys defined in the <filename>handlers</filename>,
+                    <filename>formatters</filename>, or <filename>filters</filename>,
+                    will be merged into the same section in the default
+                    configuration, with the user specified keys taking
+                    replacing a default one if there is a conflict. In
+                    practice, this means that if both the default configuration
+                    and user configuration specify a handler named
+                    <filename>myhandler</filename>, the user defined one will
+                    replace the default. To prevent the user from inadvertently
+                    replacing a default handler, formatter, or filter, all of
+                    the default ones are named with a prefix of
+                    "<filename>BitBake.</filename>"
+                </para></listitem>
+                <listitem><para>
+                    If a logger is defined by the user with the key
+                    <filename>bitbake_merge</filename> set to
+                    <filename>False</filename>, that logger will be completely
+                    replaced by user configuration. In this case, no other
+                    rules will apply to that logger.
+                </listitem></para>
+                <listitem><para>
+                    All user defined <filename>filter</filename> and
+                    <filename>handlers</filename> properties for a given logger
+                    will be merged with corresponding properties from the
+                    default logger. For example, if the user configuration adds
+                    a filter called <filename>myFilter</filename> to the
+                    <filename>BitBake.SigGen</filename>, and the default
+                    configuration adds a filter called
+                    <filename>BitBake.defaultFilter</filename>, both filters
+                    will be applied to the logger
+                </listitem></para>
+            </itemizedlist>
+        </para>
+
+        <para>
+            As an example, consider the following user logging configuration
+            file which logs all Hash Equivalence related messages of VERBOSE or
+            higher to a file called <filename>hashequiv.log</filename>
+            <literallayout class='monospaced'>
+    {
+        "version": 1,
+        "handlers": {
+            "autobuilderlog": {

bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml line 1012

Quoted 10 lines
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +10 -0
+                "class": "logging.FileHandler",
+                "formatter": "logfileFormatter",
+                "level": "DEBUG",
+                "filename": "hashequiv.log",
+                "mode": "w"
+            }
+        },
+        "formatters": {
+                "logfileFormatter": {
+                    "format": "%(name)s: %(levelname)s: %(message)s"

What's the syntax for including a timestamp in the log message? Or is that implicit?

Rich

#17 lore raw

Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG

Quoted 95 lines · Rich Persaud
Quoted 93 lines · Joshua Watt

Adds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging.

---
.../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
.../bitbake-user-manual-ref-variables.xml | 11 +++

2 files changed, 108 insertions(+)
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +75 -0
index 8f606676b4..3b31f748cc 100644
@@ -929,4 +929,101 @@
            section.
        </para>
    </section>
+
+    <section id="logging">
+        <title>Logging</title>
+        <para>
+            In addition to the standard command line option to control how
+            verbose builds are when execute, bitbake also supports user defined
+            configuration of the
+            <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+            facilities through the
+            <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+            variable. This variable defines a json or yaml
+            <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+            that will be intelligently merged into the default configuration.
+            The logging configuration is merged using the following rules:
+            <itemizedlist>
+                <listitem><para>
+                    The user defined configuration will completely replace the default
+                    configuration if top level key
+                    <filename>bitbake_merge</filename> is set to the value
+                    <filename>False</filename>. In this case, all other rules
+                    are ignored.
+                </para></listitem>
+                <listitem><para>
+                    The user configuration must have a top level
+                    <filename>version</filename> which must match the value of
+                    the default configuration.
+                </para></listitem>
+                <listitem><para>
+                    Any keys defined in the <filename>handlers</filename>,
+                    <filename>formatters</filename>, or <filename>filters</filename>,
+                    will be merged into the same section in the default
+                    configuration, with the user specified keys taking
+                    replacing a default one if there is a conflict. In
+                    practice, this means that if both the default configuration
+                    and user configuration specify a handler named
+                    <filename>myhandler</filename>, the user defined one will
+                    replace the default. To prevent the user from inadvertently
+                    replacing a default handler, formatter, or filter, all of
+                    the default ones are named with a prefix of
+                    "<filename>BitBake.</filename>"
+                </para></listitem>
+                <listitem><para>
+                    If a logger is defined by the user with the key
+                    <filename>bitbake_merge</filename> set to
+                    <filename>False</filename>, that logger will be completely
+                    replaced by user configuration. In this case, no other
+                    rules will apply to that logger.
+                </listitem></para>
+                <listitem><para>
+                    All user defined <filename>filter</filename> and
+                    <filename>handlers</filename> properties for a given logger
+                    will be merged with corresponding properties from the
+                    default logger. For example, if the user configuration adds
+                    a filter called <filename>myFilter</filename> to the
+                    <filename>BitBake.SigGen</filename>, and the default
+                    configuration adds a filter called
+                    <filename>BitBake.defaultFilter</filename>, both filters
+                    will be applied to the logger
+                </listitem></para>
+            </itemizedlist>
+        </para>
+
+        <para>
+            As an example, consider the following user logging configuration
+            file which logs all Hash Equivalence related messages of VERBOSE or
+            higher to a file called <filename>hashequiv.log</filename>
+            <literallayout class='monospaced'>
+    {
+        "version": 1,
+        "handlers": {
+            "autobuilderlog": {
+                "class": "logging.FileHandler",
+                "formatter": "logfileFormatter",
+                "level": "DEBUG",
+                "filename": "hashequiv.log",
Quoted 8 lines

bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml line 1012

Quoted 6 lines
bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +6 -0
+                "mode": "w"
+            }
+        },
+        "formatters": {
+                "logfileFormatter": {
+                    "format": "%(name)s: %(levelname)s: %(message)s"

What's the syntax for including a timestamp in the log message? Or is that implicit?

It's not implicit, but you can add it easily with %(asctime)s. There are also several other options:
https://docs.python.org/3/library/logging.html#logrecord-attributes

Quoted 2 lines

Rich