Skip to contentSkip to conversation

Shell message fixes (BitBake side)

PATCH New Paul Eggleton 1 participant 3 messages Jul 14, 2015

lore
Share & tools
lore

Paul Eggleton

author
#0 lore raw

cover Shell message fixes (BitBake side)

BitBake side of some fixes for regressions caused by the recent shell message changes.

The following changes since commit 5c5f8da509f6bbc1fad263462142519ef3d54a35:

tests/data: Add new data tests (2015-07-12 22:50:27 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-shell-logging-fixes
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-shell-logging-fixes

Paul Eggleton (2):
lib/bb: provide mechanism to bypass UI log suppression lib/bb/process: check output of select() before reading extrafiles

 lib/bb/__init__.py |  8 ++++----
 lib/bb/build.py    |  7 ++++++-
 lib/bb/process.py  | 21 +++++++++++----------
 3 files changed, 21 insertions(+), 15 deletions(-)
Signature · 1 line
2.1.0
author
#1 lore raw

PATCH 1/2 lib/bb: provide mechanism to bypass UI log suppression

The recent change to connect through the shell logging functions had an unexpected side-effect - bb.error() and bb.fatal() cause a flag to be set internally such that BitBake's UI will not print the full task log on failure; unfortunately we have in places within the OpenEmbedded metadata called these shell logging functions under error situations where we still want to see the full log (i.e., the message we're sending doesn't include the full error). Thus, provide a mechanism to fatally exit with an error but unset the flag, utilising the built-in python logging functionality that allows extra values to be passed in the log record.

Signed-off-by Paul Eggleton <[email protected]>
---
 lib/bb/__init__.py | 8 ++++----
 lib/bb/build.py    | 7 ++++++-
 2 files changed, 10 insertions(+), 5 deletions(-)
lib/bb/__init__.py +4 -4
index 1a30fa1..1f7946e 100644
@@ -94,11 +94,11 @@ def note(*args):
 def warn(*args):
     logger.warn(''.join(args))
 
-def error(*args):
-    logger.error(''.join(args))
+def error(*args, **kwargs):
+    logger.error(''.join(args), extra=kwargs)
 
-def fatal(*args):
-    logger.critical(''.join(args))
+def fatal(*args, **kwargs):
+    logger.critical(''.join(args), extra=kwargs)
     raise BBHandledException()
 
 def deprecated(func, name=None, advice=""):
lib/bb/build.py +6 -1
index cce01fe..3439964 100644
@@ -350,6 +350,8 @@ exit $?
                 # The caller will call exit themselves, so bb.error() is
                 # what we want here rather than bb.fatal()
                 bb.error(value)
+            elif cmd == 'bbfatal_log':
+                bb.error(value, forcelog=True)
             elif cmd == 'bbdebug':
                 splitval = value.split(' ', 1)
                 level = int(splitval[0])

@@ -446,7 +448,10 @@ def _exec_task(fn, task, d, quieterr):
             self.triggered = False
             logging.Handler.__init__(self, logging.ERROR)
         def emit(self, record):
-            self.triggered = True
+            if getattr(record, 'forcelog', False):
+                self.triggered = False
+            else:
+                self.triggered = True
 
     # Handle logfiles
     si = open('/dev/null', 'r')
Signature · 1 line
2.1.0
author
#2 lore raw

PATCH 2/2 lib/bb/process: check output of select() before reading extrafiles

In reply to Paul Eggleton

We're calling select() to find the fds that have data available, so we really ought to check the return to see if the extra file is in there before trying to read from it. This is part of the fix for the performance regression that this code introduced (5-10 minutes extra in a reasonable size OE build); the rest is down to an issue in the way that pseudo currently handles FIFOs and will need to be addressed there,
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993

Solution suggested by Richard Purdie <[email protected]>

Signed-off-by Paul Eggleton <[email protected]>
---
 lib/bb/process.py | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)
lib/bb/process.py +11 -10
index d95a03d..7c79785 100644
@@ -83,15 +83,16 @@ def _logged_communicate(pipe, log, input, extrafiles):
         bb.utils.nonblockingfd(fobj.fileno())
         rin.append(fobj)
 
-    def readextras():
+    def readextras(selected):
         for fobj, func in extrafiles:
-            try:
-                data = fobj.read()
-            except IOError as err:
-                if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
-                    data = None
-            if data is not None:
-                func(data)
+            if fobj in selected:
+                try:
+                    data = fobj.read()
+                except IOError as err:
+                    if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
+                        data = None
+                if data is not None:
+                    func(data)
 
     try:
         while pipe.poll() is None:

@@ -114,12 +115,12 @@ def _logged_communicate(pipe, log, input, extrafiles):
                     errdata.append(data)
                     log.write(data)
 
-            readextras()
+            readextras(r)
 
     finally:    
         log.flush()
 
-    readextras()
+    readextras([fobj for fobj, _ in extrafiles])
 
     if pipe.stdout is not None:
         pipe.stdout.close()
Signature · 1 line
2.1.0