Skip to contentSkip to conversation

Improve logger setup in devtool/recipetool/scriptutils

PATCH In review Christopher Larson 3 participants 7 messages Mar 4, 2019 – Mar 5, 2019

lore
Share & tools
lore
#0 lore raw

cover Improve logger setup in devtool/recipetool/scriptutils

From: Christopher Larson <[email protected]>

This improves logger setup in devtool and recipetool to send ERRORs to stderr, the way bitbake itself does, and also sends WARNINGs there, as those are essentially out of band as well. This makes it easier to script with devtool and recipetool, as any *real* subcommand output can be more easily captured without having to manually filter out the cruft.

The following changes since commit b11725db2d5549dc45d8ae36fbf94a5c8e342d69:

xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41 +0000)

are available in the Git repository at:

[email protected]:kergoth/openembedded-core scriptutils-log-improvements

for you to fetch changes up to c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:

oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00 +0000)

----------------------------------------------------------------
Christopher Larson (4):
oe.scriptutils: enable color in a more flexible way oe.scriptutils: add logger_setup_filters recipetool, devtool: set up the logging filters oe.scriptutils: also send WARNING to stderr

 scripts/devtool            |  2 ++
 scripts/lib/scriptutils.py | 30 ++++++++++++++++++++++++------
 scripts/recipetool         |  2 ++
 3 files changed, 28 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/4 oe.scriptutils: enable color in a more flexible way

From: Christopher Larson <[email protected]>

Rather than recreating handlers and forcing them, iterate over the handlers and enable color on ones we can handle. This makes it easier to handle color properly when we introduce the bb.msg default log filters.

---
 scripts/lib/scriptutils.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
scripts/lib/scriptutils.py +7 -6
index 3c60c3a1e6..0633c7066e 100644
@@ -39,12 +39,12 @@ def logger_create(name, stream=None):
 
 def logger_setup_color(logger, color='auto'):
     from bb.msg import BBLogFormatter
-    console = logging.StreamHandler(sys.stdout)
-    formatter = BBLogFormatter("%(levelname)s: %(message)s")
-    console.setFormatter(formatter)
-    logger.handlers = [console]
-    if color == 'always' or (color=='auto' and console.stream.isatty()):
-        formatter.enable_color()
+
+    for handler in logger.handlers:
+        if (isinstance(handler, logging.StreamHandler) and
+            isinstance(handler.formatter, BBLogFormatter)):
+            if color == 'always' or (color == 'auto' and handler.stream.isatty()):
+                handler.formatter.enable_color()
 
 
 def load_plugins(logger, plugins, pluginpath):
@@ -69,6 +69,7 @@ def load_plugins(logger, plugins, pluginpath):
                 plugin.plugin_init(plugins)
             plugins.append(plugin)
 
+
 def git_convert_standalone_clone(repodir):
     """If specified directory is a git repository, ensure it's a standalone clone"""
     import bb.process
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/4 oe.scriptutils: add logger_setup_filters

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

This function sets up the default bb.msg formatter and log filters, which by default sends ERROR messages to stderr rather than stdout.

---
 scripts/lib/scriptutils.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
scripts/lib/scriptutils.py +15 -0
index 0633c7066e..c361d0741d 100644
@@ -29,6 +29,7 @@ import tempfile
 import importlib
 from importlib import machinery
 
+
 def logger_create(name, stream=None):
     logger = logging.getLogger(name)
     loggerhandler = logging.StreamHandler(stream=stream)
@@ -37,6 +38,20 @@ def logger_create(name, stream=None):
     logger.setLevel(logging.INFO)
     return logger
 
+
+def logger_setup_filters(logger):
+    import bb.msg
+
+    console = logging.StreamHandler(sys.stdout)
+    errconsole = logging.StreamHandler(sys.stderr)
+    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
+    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
+    format_str = "%(levelname)s: %(message)s"
+    console.setFormatter(bb.msg.BBLogFormatter(format_str))
+    errconsole.setFormatter(bb.msg.BBLogFormatter(format_str))
+    logger.handlers = [console, errconsole]
+
+
 def logger_setup_color(logger, color='auto'):
     from bb.msg import BBLogFormatter
 
Signature · 1 line
2.20.1
#3 lore raw

PATCH 3/4 recipetool, devtool: set up the logging filters

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

We want errors going to stderr, the way they do in bitbake itself.

---
 scripts/devtool    | 2 ++
 scripts/recipetool | 2 ++
 2 files changed, 4 insertions(+)
scripts/devtool +2 -0
index 0e578c0de3..de1847eef3 100755
@@ -282,11 +282,13 @@ def main():
         logger.debug('Using standard bitbake path %s' % bitbakepath)
         scriptpath.add_oe_lib_path()
 
+    scriptutils.logger_setup_filters(logger)
     scriptutils.logger_setup_color(logger, global_args.color)
 
     if global_args.bbpath is None:
         try:
             tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
+            tinfoil.logger.handlers = logger.handlers
             try:
                 global_args.bbpath = tinfoil.config_data.getVar('BBPATH')
             finally:
scripts/recipetool +2 -0
index 3a3c9b7445..bac9c72107 100755
@@ -75,9 +75,11 @@ def main():
     logger.debug('Found bitbake path: %s' % bitbakepath)
     scriptpath.add_oe_lib_path()
 
+    scriptutils.logger_setup_filters(logger)
     scriptutils.logger_setup_color(logger, global_args.color)
 
     tinfoil = tinfoil_init(False)
+    tinfoil.logger.handlers = logger.handlers
     try:
         for path in (tinfoil.config_data.getVar('BBPATH').split(':')
                      + [scripts_path]):
Signature · 1 line
2.20.1
#4 lore raw

PATCH 4/4 oe.scriptutils: also send WARNING to stderr

In reply to Christopher Larson

From: Christopher Larson <[email protected]>

stderr isn't just for errors, but anything out of band. It's highly unlikely that warning messages are anything but out of band, which makes scripting our tools more troublesome, since the script calling it has to filter out the warnings from stdout to get the real output.

---
 scripts/lib/scriptutils.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/lib/scriptutils.py +4 -2
index c361d0741d..d1a50207c5 100644
@@ -43,9 +43,11 @@ def logger_setup_filters(logger):
     import bb.msg
 
     console = logging.StreamHandler(sys.stdout)
+    bb.msg.addDefaultlogFilter(console)
     errconsole = logging.StreamHandler(sys.stderr)
-    bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut)
-    bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
+    bb.msg.addDefaultlogFilter(errconsole)
+    console.addFilter(lambda r: r.levelno < logging.WARNING)
+    errconsole.addFilter(lambda r: r.levelno >= logging.WARNING)
     format_str = "%(levelname)s: %(message)s"
     console.setFormatter(bb.msg.BBLogFormatter(format_str))
     errconsole.setFormatter(bb.msg.BBLogFormatter(format_str))
Signature · 1 line
2.20.1
Acked-by on the v1 series
#5 lore raw
In reply to Christopher Larson
Quoted 3 lines · NZDT Christopher Larson

From: Christopher Larson <[email protected]>

This improves logger setup in devtool and recipetool to send ERRORs to

stderr,

Quoted 15 lines

the way bitbake itself does, and also sends WARNINGs there, as those are essentially out of band as well. This makes it easier to script with devtool and recipetool, as any *real* subcommand output can be more easily captured without having to manually filter out the cruft.

The following changes since commit b11725db2d5549dc45d8ae36fbf94a5c8e342d69:

xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41 +0000)

are available in the Git repository at:

[email protected]:kergoth/openembedded-core scriptutils-log-improvements

for you to fetch changes up to c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:

Quoted 9 lines

oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00 +0000)

----------------------------------------------------------------
Christopher Larson (4):
oe.scriptutils: enable color in a more flexible way oe.scriptutils: add logger_setup_filters recipetool, devtool: set up the logging filters oe.scriptutils: also send WARNING to stderr

All of these look good to me.

Cheers
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw
Quoted 39 lines · Paul Eggleton
Quoted 4 lines · NZDT Christopher Larson

From: Christopher Larson <[email protected]>

This improves logger setup in devtool and recipetool to send ERRORs to

stderr,

Quoted 25 lines

the way bitbake itself does, and also sends WARNINGs there, as those are
essentially out of band as well. This makes it easier to script with devtool
and recipetool, as any *real* subcommand output can be more easily captured
without having to manually filter out the cruft.

The following changes since commit
b11725db2d5549dc45d8ae36fbf94a5c8e342d69:

xserver-xorg: Fix build errors with clang (2019-03-04 14:26:41
+0000)

are available in the Git repository at:

[email protected]:kergoth/openembedded-core scriptutils-log-
improvements

for you to fetch changes up to
c4f37568236d2e3d8c40d57399d0b275ad5b7c8b:

oe.scriptutils: also send WARNING to stderr (2019-03-04 21:31:00
+0000)

Quoted 7 lines

----------------------------------------------------------------
Christopher Larson (4):
oe.scriptutils: enable color in a more flexible way oe.scriptutils: add logger_setup_filters recipetool, devtool: set up the logging filters oe.scriptutils: also send WARNING to stderr

Quoted 3 lines

All of these look good to me.

Unfortunately there were a couple of issues on the autobuilder in the devtool selftests with these applied:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/297

Cheers,

Richard