Skip to contentSkip to conversation

Saving off warnings to a separate file with Python structured logging

PATCH New Joshua Watt 4 participants 6 messages Jan 26, 2023 – Jan 27, 2023

lore
Share & tools
lore
#0 lore raw

It has been asked how to configure bitbake to save off WARNING messages (and higher) to a separate file when running bitbake. With python structured logging, this can be done relatively easily without having to modify any code.

First, create the structured config file (we'll call it warn.json for this example). This file should contain:

{

    "version": 1,

    "formatters": {

        "warnlogFormatter": {

            "()": "bb.msg.BBLogFormatter",

            "format": "%(levelname)s: %(message)s"

        }

    },

    "handlers": {

        "warnlog": {

            "class": "logging.FileHandler",

            "formatter": "warnlogFormatter",

            "level": "WARNING",

            "filename": "warn.log"

        }

    },

    "loggers": {

        "BitBake": {

            "handlers": ["warnlog"]

        }

    },

    "@disable_existing_loggers": false

}

After this, you need to point bitbake at this configuration file by adding to the path to it in BB_LOGCONFIG, for example:

BB_LOGCONFIG = "warn.json"

After doing this, bitbake should write all WARNING and higher messages to the file warn.log.

For more information on Python structured logging, see https://docs.python.org/3/howto/logging-cookbook.html.

Note that bitbake implements several helper classes for structured logging in lib/bb/msg.py

#1 lore raw

Hi Joshua,

This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?

Cc'ing Michael and the docs mailing list on this.

Thanks for the tip!

Cheers,
Quentin

Quoted 71 lines · Joshua Watt

It has been asked how to configure bitbake to save off WARNING messages (and higher) to a separate file when running bitbake. With python structured logging, this can be done relatively easily without having to modify any code.

First, create the structured config file (we'll call it warn.json for this example). This file should contain:

{

    "version": 1,

    "formatters": {

        "warnlogFormatter": {

            "()": "bb.msg.BBLogFormatter",

            "format": "%(levelname)s: %(message)s"

        }

    },

    "handlers": {

        "warnlog": {

            "class": "logging.FileHandler",

            "formatter": "warnlogFormatter",

            "level": "WARNING",

            "filename": "warn.log"

        }

    },

    "loggers": {

        "BitBake": {

            "handlers": ["warnlog"]

        }

    },

    "@disable_existing_loggers": false

}

After this, you need to point bitbake at this configuration file by adding to the path to it in BB_LOGCONFIG, for example:

BB_LOGCONFIG = "warn.json"

After doing this, bitbake should write all WARNING and higher messages to the file warn.log.

For more information on Python structured logging, see https://urldefense.com/v3/__https://docs.python.org/3/howto/logging-cookbook.html__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTnPSRsiUg$ .

Note that bitbake implements several helper classes for structured logging in lib/bb/msg.py

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14355): https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14355__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTlK7G8NjQ$
Mute This Topic: https://urldefense.com/v3/__https://lists.openembedded.org/mt/96553269/6293953__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTl2pVuArw$
Group Owner: [email protected]
Unsubscribe: https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTne92m1xw$ [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org

Hi Joshua,

This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?

Cc'ing Michael and the docs mailing list on this.

The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)

Cheers,

Richard

#3 lore raw
Quoted 11 lines · Richard Purdie
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org

Hi Joshua,

This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?

Cc'ing Michael and the docs mailing list on this.

The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)

Actually, there is already documentation about BB_LOGCONFIG in https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging. I'll propose an update taking Joshua's guidelines into account. Let me know if you had something else in mind, and if you see obsolete details on this page.

The provided JSON file still seems to work, but it serves a different purpose.

Cheers
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw
Quoted 13 lines · Michael Opdenacker
Quoted 11 lines · Richard Purdie
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org

Hi Joshua,

This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?

Cc'ing Michael and the docs mailing list on this.

The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)

Quoted 8 lines

Actually, there is already documentation about BB_LOGCONFIG in
https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging.
I'll propose an update taking Joshua's guidelines into account. Let me know if you had something else in mind, and if you see obsolete details on this page.

The provided JSON file still seems to work, but it serves a different purpose.

Thanks, I'd forgotten we'd added that example so that makes sense. I think we should add the warnings one as well, the link to the python docs also seems useful and perhaps making it clearer how to use BB_LOGCONFIG?

Cheers,

Richard

#5 lore raw

PATCH 1/1 bitbake-user-manual: show how use BB_LOGCONFIG to log warnings

From: Michael Opdenacker <[email protected]>

---
 .../bitbake-user-manual-execution.rst         | 45 +++++++++++++++++--
 1 file changed, 41 insertions(+), 4 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-execution.rst +41 -4
index f6ebf7ba..4fa3ca4b 100644
@@ -656,7 +656,7 @@ builds are when execute, bitbake also supports user defined
 configuration of the `Python
 logging <https://docs.python.org/3/library/logging.html>`__ facilities
 through the :term:`BB_LOGCONFIG` variable. This
-variable defines a json or yaml `logging
+variable defines a JSON or YAML `logging
 configuration <https://docs.python.org/3/library/logging.config.html>`__
 that will be intelligently merged into the default configuration. The
 logging configuration is merged using the following rules:
@@ -690,9 +690,9 @@ logging configuration is merged using the following rules:
    adds a filter called ``BitBake.defaultFilter``, both filters will be
    applied to the logger
 
-As an example, consider the following user logging configuration file
-which logs all Hash Equivalence related messages of VERBOSE or higher to
-a file called ``hashequiv.log`` ::
+As a first example, you can create a ``hashequiv.json`` user logging
+configuration file to log all Hash Equivalence related messages of ``VERBOSE``
+or higher priority to a file called ``hashequiv.log``::
 
    {
        "version": 1,
@@ -721,3 +721,40 @@ a file called ``hashequiv.log`` ::
            }
        }
    }
+
+Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``::
+
+   BB_LOGCONFIG = "hashequiv.json"
+
+Another example is this ``warn.json`` file to log all ``WARNING`` and
+higher priority messages to a ``warn.log`` file::
+
+  {
+      "version": 1,
+      "formatters": {
+          "warnlogFormatter": {
+              "()": "bb.msg.BBLogFormatter",
+              "format": "%(levelname)s: %(message)s"
+          }
+      },
+
+      "handlers": {
+          "warnlog": {
+              "class": "logging.FileHandler",
+              "formatter": "warnlogFormatter",
+              "level": "WARNING",
+              "filename": "warn.log"
+          }
+      },
+
+      "loggers": {
+          "BitBake": {
+              "handlers": ["warnlog"]
+          }
+      },
+
+      "@disable_existing_loggers": false
+  }
+
+Note that BitBake's helper classes for structured logging are implemented in
+``lib/bb/msg.py``.
Signature · 1 line
2.37.2