Saving off warnings to a separate file with Python structured logging
It has been asked how to configure bitbake to save off WARNING messages (and higher) to a separate file when running bitbake. With python structured logging, this can be done relatively easily without having to modify any code.
First, create the structured config file (we'll call it warn.json for this example). This file should contain:
{
"version": 1,
"formatters": {
"warnlogFormatter": {
"()": "bb.msg.BBLogFormatter",
"format": "%(levelname)s: %(message)s"
}
},
"handlers": {
"warnlog": {
"class": "logging.FileHandler",
"formatter": "warnlogFormatter",
"level": "WARNING",
"filename": "warn.log"
}
},
"loggers": {
"BitBake": {
"handlers": ["warnlog"]
}
},
"@disable_existing_loggers": false
}
After this, you need to point bitbake at this configuration file by adding to the path to it in BB_LOGCONFIG, for example:
BB_LOGCONFIG = "warn.json"
After doing this, bitbake should write all WARNING and higher messages to the file warn.log.
For more information on Python structured logging, see https://docs.python.org/3/howto/logging-cookbook.html.
Note that bitbake implements several helper classes for structured logging in lib/bb/msg.py
Hi Joshua,
This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?
Cc'ing Michael and the docs mailing list on this.
Thanks for the tip!
Cheers,
Quentin
Quoted 71 lines · Joshua Watt
It has been asked how to configure bitbake to save off WARNING messages (and higher) to a separate file when running bitbake. With python structured logging, this can be done relatively easily without having to modify any code.
First, create the structured config file (we'll call it warn.json for this example). This file should contain:
{
"version": 1,
"formatters": {
"warnlogFormatter": {
"()": "bb.msg.BBLogFormatter",
"format": "%(levelname)s: %(message)s"
}
},
"handlers": {
"warnlog": {
"class": "logging.FileHandler",
"formatter": "warnlogFormatter",
"level": "WARNING",
"filename": "warn.log"
}
},
"loggers": {
"BitBake": {
"handlers": ["warnlog"]
}
},
"@disable_existing_loggers": false
}
After this, you need to point bitbake at this configuration file by adding to the path to it in
BB_LOGCONFIG, for example:
BB_LOGCONFIG= "warn.json"After doing this, bitbake should write all WARNING and higher messages to the file warn.log.
For more information on Python structured logging, see https://urldefense.com/v3/__https://docs.python.org/3/howto/logging-cookbook.html__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTnPSRsiUg$ .
Note that bitbake implements several helper classes for structured logging in
lib/bb/msg.py-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14355): https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14355__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTlK7G8NjQ$
Mute This Topic: https://urldefense.com/v3/__https://lists.openembedded.org/mt/96553269/6293953__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTl2pVuArw$
Group Owner: [email protected]
Unsubscribe: https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTne92m1xw$ [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org
Hi Joshua,
This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?
Cc'ing Michael and the docs mailing list on this.
The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)
Cheers,
Richard
Quoted 11 lines · Richard Purdie
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org
Hi Joshua,
This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?
Cc'ing Michael and the docs mailing list on this.
The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)
Actually, there is already documentation about BB_LOGCONFIG in https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging. I'll propose an update taking Joshua's guidelines into account. Let me know if you had something else in mind, and if you see obsolete details on this page.
The provided JSON file still seems to work, but it serves a different purpose.
Cheers
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 13 lines · Michael Opdenacker
Quoted 11 lines · Richard Purdie
Quoted 6 lines · Quentin Schulz via lists.yoctoproject.org
Hi Joshua,
This seems to be too valuable information to be lost in the mailing list archives. Any chance of having this somewhere in the bitbake/yocto docs?
Cc'ing Michael and the docs mailing list on this.
The context was that we discussed this in triage yesterday and said making this public was the first step but then we'd document it in the manuals. Michael was on that call so knows about it! :)
Quoted 8 lines
Actually, there is already documentation about
BB_LOGCONFIGin
https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging.
I'll propose an update taking Joshua's guidelines into account. Let me know if you had something else in mind, and if you see obsolete details on this page.The provided JSON file still seems to work, but it serves a different purpose.
Thanks, I'd forgotten we'd added that example so that makes sense. I think we should add the warnings one as well, the link to the python docs also seems useful and perhaps making it clearer how to use BB_LOGCONFIG?
Cheers,
Richard
PATCH 1/1 bitbake-user-manual: show how use BB_LOGCONFIG to log warnings
From: Michael Opdenacker <[email protected]>
--- .../bitbake-user-manual-execution.rst | 45 +++++++++++++++++-- 1 file changed, 41 insertions(+), 4 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-execution.rst +41 -4
@@ -656,7 +656,7 @@ builds are when execute, bitbake also supports user defined configuration of the `Python logging <https://docs.python.org/3/library/logging.html>`__ facilities through the :term:`BB_LOGCONFIG` variable. This -variable defines a json or yaml `logging +variable defines a JSON or YAML `logging configuration <https://docs.python.org/3/library/logging.config.html>`__ that will be intelligently merged into the default configuration. The logging configuration is merged using the following rules:
@@ -690,9 +690,9 @@ logging configuration is merged using the following rules: adds a filter called ``BitBake.defaultFilter``, both filters will be applied to the logger -As an example, consider the following user logging configuration file -which logs all Hash Equivalence related messages of VERBOSE or higher to -a file called ``hashequiv.log`` :: +As a first example, you can create a ``hashequiv.json`` user logging +configuration file to log all Hash Equivalence related messages of ``VERBOSE`` +or higher priority to a file called ``hashequiv.log``:: { "version": 1,
@@ -721,3 +721,40 @@ a file called ``hashequiv.log`` :: } } } + +Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``:: + + BB_LOGCONFIG = "hashequiv.json" + +Another example is this ``warn.json`` file to log all ``WARNING`` and +higher priority messages to a ``warn.log`` file:: + + { + "version": 1, + "formatters": { + "warnlogFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + + "handlers": { + "warnlog": { + "class": "logging.FileHandler", + "formatter": "warnlogFormatter", + "level": "WARNING", + "filename": "warn.log" + } + }, + + "loggers": { + "BitBake": { + "handlers": ["warnlog"] + } + }, + + "@disable_existing_loggers": false + } + +Note that BitBake's helper classes for structured logging are implemented in +``lib/bb/msg.py``.
Signature · 1 line
2.37.2