Skip to contentSkip to conversation

server/process: Handle SIGTERM more gracefully

PATCH In review Richard Purdie 2 participants 2 messages Sep 7, 2015

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 server/process: Handle SIGTERM more gracefully

Currently if you send a SIGTERM to the bitbake UI process, the system basically hangs if tasks are executing. This is because the server process doesn't actually try any kind of shutdown before exiting.

This patch trys executing a stateForceShutdown command first, which is enough to stop any active tasks before the system exits.

I also noticed that terminate can execute multiple times, once at SIGTERM from the handler and once from the real exit. Double execution leads to stack traces and potential hangs (writes to dead pipes), so ensure the code only can run once.

With these fixes, bitbake much more correctly deals with SIGTERM to the UI process.

bitbake/lib/bb/server/process.py +9 -0
index f022b86..5fca350 100644
@@ -114,6 +114,10 @@ class ProcessServer(Process, BaseImplServer):
                 if self.quitout.poll():
                     self.quitout.recv()
                     self.quit = True
+                    try:
+                        self.runCommand(["stateForceShutdown"])
+                    except:
+                        pass
 
                 self.idle_commands(.1, [self.command_channel, self.quitout])
             except Exception:
@@ -123,6 +127,7 @@ class ProcessServer(Process, BaseImplServer):
         bb.event.unregister_UIHhandler(self.event_handle.value)
         self.command_channel.close()
         self.cooker.shutdown(True)
+        self.quitout.close()
 
     def idle_commands(self, delay, fds=None):
         nextsleep = delay
@@ -172,12 +177,16 @@ class BitBakeProcessServerConnection(BitBakeBaseServerConnection):
         self.event_queue = event_queue
         self.connection = ServerCommunicator(self.ui_channel, self.procserver.event_handle, self.procserver)
         self.events = self.event_queue
+        self.terminated = False
 
     def sigterm_terminate(self):
         bb.error("UI received SIGTERM")
         self.terminate()
 
     def terminate(self):
+        if self.terminated:
+            return
+        self.terminated = True
         def flushevents():
             while True:
                 try:
#1 lore raw
Quoted 13 lines · Richard Purdie

Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't actually try any kind of shutdown before exiting.

This patch trys executing a stateForceShutdown command first, which is enough to stop any active tasks before the system exits.

I also noticed that terminate can execute multiple times, once at SIGTERM from the handler and once from the real exit. Double execution leads to stack traces and potential hangs (writes to dead pipes), so ensure the code only can run once.

Quoted 5 lines

With these fixes, bitbake much more correctly deals with SIGTERM to the UI process.

Just as an FYI for consideration in the longer term, you can raise an exception from a signal handler, to handle sigterm the same way as sigint for consistency. Ex.:

https://github.com/kergoth/oe-test-scripts/blob/master/bb_test.py#L12-L17 +
https://github.com/kergoth/oe-test-scripts/blob/master/bb_test.py#L171-L180

Which reminds me, we don't exit with the correct exit code for sigint/sigterm, so the calling shell doesn't know the cause of the exit. See also http://www.cons.org/cracauer/sigint.html -- fantastic article there, highly recommended.

https://docs.python.org/3/library/signal.html#example has an example of raising an exception from another signal handler, so it seems to be kosher.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics