Skip to contentSkip to conversation

knotty.py: Extend knotty to support "real time" log tail for tasks

PATCH New Mark Hatle 1 participant 7 messages May 29, 2013

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

PATCH 1/7 knotty.py: Extend knotty to support "real time" log tail for tasks

From: Jason Wessel <[email protected]>

This patch provides the capability to knotty to have a higher log level where the individual tasks logs can be multiplexed to stdio. This is highly desirable for several different use cases.

1) You can visually spot "hot spots" in building parallel tasks

2) The relative order an sequence of events can be determined in difficult to diagnose inter-related library problems.

3) It provides a nice way to see what all is actually going on and getting logged by various tasks such as a kernel compile where you might want to actually look at the output in a build over build scenario.

4) There are most certainly other aspects where this is useful.

The primary control is to set the BB_RT_LOGLEVEL = "2" in your local.conf. Additional capabilities will follow in successive patches. When parallel tasks are enabled, the logs will be multiplexed and prefixed with the PID of the task. In the non-parallel case there is no PID prefix as everything is linear.

---
 lib/bb/build.py                  |   4 ++
 lib/bb/ui/crumbs/multilogtail.py | 135 +++++++++++++++++++++++++++++++++++++++
 lib/bb/ui/knotty.py              |  44 +++++++++++++
 3 files changed, 183 insertions(+)
 create mode 100644 lib/bb/ui/crumbs/multilogtail.py
lib/bb/build.py +4 -0
index bfc176d..725eb02 100644
@@ -89,6 +89,10 @@ class TaskBase(event.Event):
 class TaskStarted(TaskBase):
     """Task execution started"""
 
+    def __init__(self, t, d):
+        self.logfile = d.getVar("BB_LOGFILE", True) or "";
+        super(TaskStarted, self).__init__(t, d)
+
 class TaskSucceeded(TaskBase):
     """Task execution completed"""
 
lib/bb/ui/crumbs/multilogtail.py new file +135 -0

135 lines not on this page · raw message

lib/bb/ui/knotty.py +44 -0

85 lines not on this page · raw message

Signature · 1 line
1.8.1.2.545.g2f19ada
author
#1 lore raw

PATCH 2/7 knotty.py: Add the ability to dynamically select loglevel from stdin

From: Jason Wessel <[email protected]>

If stdin is a controlling terminal, make it possible to dynamically select a log level with the keys 1 and 2.

1 = normal bitbake knotty logs
2 = realtime multiple tail of task logs

---
 lib/bb/ui/knotty.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
lib/bb/ui/knotty.py +35 -0
index 34ca678..293572f 100644
@@ -218,6 +218,35 @@ class TerminalFilter(object):
             fd = sys.stdin.fileno()
             self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
 
+class StdinMgr:
+    def __init__(self):
+        import termios
+        self.termios = termios
+        self.stdinbackup = None
+        self.fd = None
+        if sys.stdin.isatty():
+            self.fd = sys.stdin.fileno()
+            self.stdinbackup = self.termios.tcgetattr(self.fd)
+            new = self.termios.tcgetattr(self.fd)
+            new[3] = new[3] & ~self.termios.ICANON & ~self.termios.ECHO
+            self.termios.tcsetattr(self.fd, self.termios.TCSANOW, new)
+
+    def poll(self):
+        if not self.stdinbackup:
+            return False
+        return select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], [])
+
+    def restore(self):
+        if self.stdinbackup:
+            self.termios.tcsetattr(self.fd, self.termios.TCSADRAIN,
+                                   self.stdinbackup)
+            self.stdinbackup = None
+        # Force echo back on "just in case" something went haywire with an exception
+        if sys.stdin.isatty():
+            new = self.termios.tcgetattr(self.fd)
+            new[3] = new[3] | self.termios.ECHO
+            self.termios.tcsetattr(self.fd, self.termios.TCSANOW, new)
+
 class RtLogLevel:
     def __init__(self, mlt):
         self.displaytail = False
@@ -317,11 +346,15 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     taskfailures = []
 
     termfilter = tf(main, helper, console, format)
+    stdin_mgr = StdinMgr()
 
     while True:
         try:
             termfilter.updateFooter()
             event = eventHandler.waitEvent(0.25)
+            if stdin_mgr.poll():
+                keyinput = sys.stdin.read(1)
+                rtloglevel.setLevel(keyinput, True)
             # Always try printing any accumulated log files first
             rtloglevel.displayLogs()
             if event is None:
@@ -512,6 +545,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
             if ioerror.args[0] == 4:
                 pass
         except KeyboardInterrupt:
+            stdin_mgr.restore()
             termfilter.clearFooter()
             if main.shutdown == 1:
                 print("\nSecond Keyboard Interrupt, stopping...\n")
@@ -527,6 +561,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
             main.shutdown = main.shutdown + 1
             pass
 
+    stdin_mgr.restore()
     summary = ""
     if taskfailures:
         summary += pluralise("\nSummary: %s task failed:",
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#2 lore raw

PATCH 3/7 msg.py, knotty.py: Allow dynamic toggle of the debug log level

In reply to Mark Hatle

From: Jason Wessel <[email protected]>

If stdin is a controlling tty, make it possible to have log levels 1-4 where you can press a key to dynamically change the log level, while bitbake is running to see what is going on.

1 = default log level
2 = multiple log tail of bitbake tasks
3 = bit bake debug messages
4 = bitbake debug messages + multiple lot tail of bitbake tasks

In order to make this possible a slight modification to the msg logfilter was needed to make it possible to get an object so as to change the filter log level.

---
 lib/bb/msg.py       |  9 ++++++++-
 lib/bb/ui/knotty.py | 21 ++++++++++++++++++---
 2 files changed, 26 insertions(+), 4 deletions(-)
lib/bb/msg.py +8 -1
index e70daee..bb4833d 100644
@@ -119,6 +119,13 @@ class BBLogFilter(object):
         handler.setLevel(loglevel)
         handler.addFilter(self)
 
+    def setFiltLevel(self, handler, level):
+        self.stdlevel = level
+        handler.setLevel(level)
+
+    def getFiltLevel(self):
+        return self.stdlevel
+
     def filter(self, record):
         if record.levelno >= self.stdlevel:
             return True
@@ -164,7 +171,7 @@ def addDefaultlogFilter(handler):
         dlevel = len(tuple(iterator))
         debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
 
-    BBLogFilter(handler, level, debug_domains)
+    return BBLogFilter(handler, level, debug_domains)
 
 #
 # Message handling functions
lib/bb/ui/knotty.py +18 -3
index 293572f..4f59e96 100644
@@ -248,8 +248,11 @@ class StdinMgr:
             self.termios.tcsetattr(self.fd, self.termios.TCSANOW, new)
 
 class RtLogLevel:
-    def __init__(self, mlt):
+    def __init__(self, handler, logfilter, mlt):
         self.displaytail = False
+        self.handler = handler
+        self.logfilter = logfilter
+        self.defaultLevel = logfilter.getFiltLevel()
         self.mlt = mlt
 
     def displayLogs(self):
@@ -260,10 +263,22 @@ class RtLogLevel:
         if input == "1":
             if verbose:
                 print "NOTE: Turning off real time log tail"
+            self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
             self.displaytail = False
         elif input == "2":
             if verbose:
                 print "NOTE: Turning on real time log tail"
+            self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
+            self.displaytail = True
+        elif input == "3":
+            if verbose:
+                print "NOTE: Turning on DEBUG logging"
+            self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
+            self.displaytail = False
+        elif input == "4":
+            if verbose:
+                print "NOTE: Turning on DEBUG logging + real time log tail"
+            self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
             self.displaytail = True
 
 def main(server, eventHandler, params, tf = TerminalFilter):
@@ -299,8 +314,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     console = logging.StreamHandler(sys.stdout)
     format_str = "%(levelname)s: %(message)s"
     format = bb.msg.BBLogFormatter(format_str)
-    bb.msg.addDefaultlogFilter(console)
-    rtloglevel = RtLogLevel(mlt)
+    logfilter = bb.msg.addDefaultlogFilter(console)
+    rtloglevel = RtLogLevel(console, logfilter, mlt)
     bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
     if bb_rt_loglevel and bb_rt_loglevel != "":
         rtloglevel.setLevel(bb_rt_loglevel, False)
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#3 lore raw

PATCH 4/7 knotty.py: update footer on dynamic log level change

In reply to Mark Hatle

From: Jason Wessel <[email protected]>

This patch adds a call to update the task list after changing log modes dynamically such that you can immediately see the remaining tasks.

---
 lib/bb/ui/knotty.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/ui/knotty.py +6 -0
index 4f59e96..5020815 100644
@@ -181,6 +181,10 @@ class TerminalFilter(object):
             sys.stdout.write(self.curses.tparm(self.ed))
         self.footer_present = False
 
+    def updateFooterForce(self):
+        self.footer_present = False
+        self.updateFooter()
+
     def updateFooter(self):
         if not self.cuu:
             return
@@ -370,6 +374,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
             if stdin_mgr.poll():
                 keyinput = sys.stdin.read(1)
                 rtloglevel.setLevel(keyinput, True)
+                termfilter.updateFooterForce()
+
             # Always try printing any accumulated log files first
             rtloglevel.displayLogs()
             if event is None:
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#4 lore raw

PATCH 5/7 knotty.py: Add interactive commands to knotty

In reply to Mark Hatle

From: Jason Wessel <[email protected]>

0 - Linear logging
1 - turn off real time log tail
2 - turn on real time log tail
3 - turn on debug logging
4 - turn on debug logging and real time log tail
t - Display tasks in "top" mode
N - Display all runtime NOTE's that are normally filtered (0 or 1 toggles off) h - display commands

The knotty 2 functionality can be activated in realtime by pressing "t" for top mode or reverted with "0" for normal mode.

---
 lib/bb/ui/knotty.py | 78 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 68 insertions(+), 10 deletions(-)
lib/bb/ui/knotty.py +68 -10
index 5020815..d1ee534 100644
@@ -97,7 +97,7 @@ class InteractConsoleLogFilter(logging.Filter):
         self.format = format
 
     def filter(self, record):
-        if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+        if self.tf.filterOn and record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
             return False
         self.tf.clearFooter()
         return True
@@ -136,6 +136,8 @@ class TerminalFilter(object):
         self.main = main
         self.helper = helper
         self.cuu = None
+        self.topMode = False
+        self.filterOn = False
         self.stdinbackup = None
         self.interactive = sys.stdout.isatty()
         self.footer_present = False
@@ -172,9 +174,31 @@ class TerminalFilter(object):
             self.columns = self.getTerminalColumns()
         except:
             self.cuu = None
+        self.topMode = True
+        self.filterOn = True
         console.addFilter(InteractConsoleLogFilter(self, format))
 
+    def setTopMode(self):
+        if self.topMode:
+            return
+        self.topMode = True
+        self.setFilterOn()
+
+    def setNormalMode(self):
+        if not self.topMode:
+            return
+        self.topMode = False
+        self.setFilterOff()
+
+    def setFilterOn(self):
+        self.filterOn = True
+
+    def setFilterOff(self):
+        self.filterOn = False
+
     def clearFooter(self):
+        if not self.topMode:
+            return
         if self.footer_present:
             lines = self.footer_present
             sys.stdout.write(self.curses.tparm(self.cuu, lines))
@@ -186,7 +210,7 @@ class TerminalFilter(object):
         self.updateFooter()
 
     def updateFooter(self):
-        if not self.cuu:
+        if not self.cuu or not self.topMode:
             return
         activetasks = self.helper.running_tasks
         failedtasks = self.helper.failed_tasks
@@ -252,23 +276,29 @@ class StdinMgr:
             self.termios.tcsetattr(self.fd, self.termios.TCSANOW, new)
 
 class RtLogLevel:
-    def __init__(self, handler, logfilter, mlt):
+    def __init__(self, handler, logfilter, mlt, tf):
         self.displaytail = False
         self.handler = handler
         self.logfilter = logfilter
         self.defaultLevel = logfilter.getFiltLevel()
         self.mlt = mlt
+        self.tf = tf
 
     def displayLogs(self):
         if self.displaytail:
             self.mlt.displayLogs()
 
     def setLevel(self, input, verbose):
-        if input == "1":
+        if input == "1" or input == "0":
             if verbose:
                 print "NOTE: Turning off real time log tail"
             self.logfilter.setFiltLevel(self.handler, self.defaultLevel)
             self.displaytail = False
+            if isinstance(self.tf, TerminalFilter):
+                if input == "0":
+                    self.tf.setNormalMode()
+                else:
+                    self.tf.setTopMode()
         elif input == "2":
             if verbose:
                 print "NOTE: Turning on real time log tail"
@@ -284,6 +314,29 @@ class RtLogLevel:
                 print "NOTE: Turning on DEBUG logging + real time log tail"
             self.logfilter.setFiltLevel(self.handler, logging.DEBUG)
             self.displaytail = True
+        elif input == "t":
+            if verbose:
+                print "NOTE: Activing task \"top\" mode"
+            if isinstance(self.tf, TerminalFilter):
+                self.tf.setTopMode()
+        elif input == "N":
+            if verbose:
+                print "NOTE: Turning on task notes"
+            if isinstance(self.tf, TerminalFilter):
+                self.tf.setFilterOff()
+        elif input == "h" or input == "?":
+            print "============================================="
+            print "Interaction help commands:"
+            print " 0 - Linear logging"
+            print " 1 - turn off real time log tail"
+            print " 2 - turn on real time log tail"
+            print " 3 - turn on debug logging"
+            print " 4 - turn on debug logging and real time log tail"
+            print " t - Display tasks in \"top\" mode"
+            print " N - Display all runtime NOTE's that are normally filtered (0 or 1 toggles off)"
+            print " h - display commands"
+            return False
+        return True
 
 def main(server, eventHandler, params, tf = TerminalFilter):
 
@@ -300,6 +353,10 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     if error:
         logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
         return 1
+    bb_rt_loglevel, error = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
+    if error:
+        logger.error("Unable to get the value of BB_RT_LOGLEVEL variable: %s" % error)
+        return 1
 
     if sys.stdin.isatty() and sys.stdout.isatty():
         log_exec_tty = True
@@ -319,10 +376,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     format_str = "%(levelname)s: %(message)s"
     format = bb.msg.BBLogFormatter(format_str)
     logfilter = bb.msg.addDefaultlogFilter(console)
-    rtloglevel = RtLogLevel(console, logfilter, mlt)
-    bb_rt_loglevel = server.runCommand(["getVariable", "BB_RT_LOGLEVEL"])
-    if bb_rt_loglevel and bb_rt_loglevel != "":
-        rtloglevel.setLevel(bb_rt_loglevel, False)
     console.setFormatter(format)
     logger.addHandler(console)
 
@@ -367,14 +420,19 @@ def main(server, eventHandler, params, tf = TerminalFilter):
     termfilter = tf(main, helper, console, format)
     stdin_mgr = StdinMgr()
 
+    rtloglevel = RtLogLevel(console, logfilter, mlt, termfilter)
+    if bb_rt_loglevel and bb_rt_loglevel != "":
+        for inputkey in bb_rt_loglevel:
+            rtloglevel.setLevel(inputkey, False)
     while True:
         try:
             termfilter.updateFooter()
             event = eventHandler.waitEvent(0.25)
             if stdin_mgr.poll():
                 keyinput = sys.stdin.read(1)
-                rtloglevel.setLevel(keyinput, True)
-                termfilter.updateFooterForce()
+                termfilter.clearFooter()
+                if (rtloglevel.setLevel(keyinput, True)):
+                    termfilter.updateFooterForce()
 
             # Always try printing any accumulated log files first
             rtloglevel.displayLogs()
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#5 lore raw

PATCH 6/7 knotty.py: Add ability to dynamically toggle log location

In reply to Mark Hatle

From: Jason Wessel <[email protected]>

Add the ability to see where all the per task logs were written to disk.

The resulting output looks like where [...] is the full path:

NOTE: LOG: [...]/python-2.7.2-r2.20/temp/log.do_package_setscene.8719
NOTE: LOG: [...]/perl-5.14.2-r7/temp/log.do_package_setscene.8760

---
 lib/bb/ui/knotty.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)
lib/bb/ui/knotty.py +13 -0
index d1ee534..b2fb141 100644
@@ -278,6 +278,7 @@ class StdinMgr:
 class RtLogLevel:
     def __init__(self, handler, logfilter, mlt, tf):
         self.displaytail = False
+        self.displayLogLocations = False
         self.handler = handler
         self.logfilter = logfilter
         self.defaultLevel = logfilter.getFiltLevel()
@@ -324,6 +325,14 @@ class RtLogLevel:
                 print "NOTE: Turning on task notes"
             if isinstance(self.tf, TerminalFilter):
                 self.tf.setFilterOff()
+        elif input == "l":
+            if verbose:
+                print "NOTE: Activating log locations display"
+            self.displayLogLocations = True
+        elif input == "L":
+            if verbose:
+                print "NOTE: Disable log locations display"
+            self.displayLogLocations = False
         elif input == "h" or input == "?":
             print "============================================="
             print "Interaction help commands:"
@@ -332,6 +341,7 @@ class RtLogLevel:
             print " 2 - turn on real time log tail"
             print " 3 - turn on debug logging"
             print " 4 - turn on debug logging and real time log tail"
+            print " l - emit log locations (L to turn off)"
             print " t - Display tasks in \"top\" mode"
             print " N - Display all runtime NOTE's that are normally filtered (0 or 1 toggles off)"
             print " h - display commands"
@@ -442,6 +452,9 @@ def main(server, eventHandler, params, tf = TerminalFilter):
                 continue
             helper.eventHandler(event)
             if isinstance(event, bb.build.TaskStarted):
+                if (rtloglevel.displayLogLocations):
+                    termfilter.clearFooter()
+                    print "NOTE: LOG: %s" % event.logfile
                 mlt.openLog(event.logfile, event.pid)
                 rtloglevel.displayLogs()
 
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#6 lore raw

PATCH 7/7 knotty.py: Handle rare interrupted select call

In reply to Mark Hatle

From: Jason Wessel <[email protected]>

Under a high load and while resizing the terminal window, I received the following traceback:

Traceback (most recent call last):
File "/bitbake/bin//bitbake", line 263, in <module>
ret = main()
File "/bitbake/bin//bitbake", line 252, in main
return server.launchUI(ui_main, server_connection.connection, server_connection.events)
File "/bitbake/lib/bb/server/process.py", line 269, in launchUI
return bb.cooker.server_main(self.cooker, uifunc, *args)
File "/bitbake/lib/bb/cooker.py", line 1407, in server_main
ret = func(*args)
File "/bitbake/lib/bb/ui/knotty.py", line 385, in main
if stdin_mgr.poll():
error: (4, 'Interrupted system call')

Interrupted system calls from a sigwinch are certainly possible and the right thing to do is just restart the system call, or in this case, punt and wait until the next poll cycle.

---
 lib/bb/ui/knotty.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/bb/ui/knotty.py +5 -1
index b2fb141..5cea884 100644
@@ -262,7 +262,11 @@ class StdinMgr:
     def poll(self):
         if not self.stdinbackup:
             return False
-        return select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], [])
+        try:
+            ret = select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], [])
+        except:
+            return False
+        return ret
 
     def restore(self):
         if self.stdinbackup:
Signature · 1 line
1.8.1.2.545.g2f19ada