kill-bb: Add it for killing abnormal bitbake processes

The following changes since commit fc634c41e4b3fbaf29dc0104ae6b15757e77f60a:

Apache-2.0-with-LLVM-exception: Add new license file (2019-07-31 23:02:56 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/kill-bb
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/kill-bb

Robert Yang (1):
kill-bb: Add it for killing abnormal bitbake processes

 scripts/kill-bb | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100755 scripts/kill-bb

2.7.4

There might be processes left after Ctr-C, e.g.:

$ rm -f tmp/cache/default-glibc/qemux86/x86_64/
$ bitbake -p

Press 'Ctrl-C' multiple times during parsing, then bitbake processes may not exit, and the worse is that we can't start bitbake again, we can't always reproduce this, but sometime. We can only use "ps ux" to find the processes and kill them one by one. This tool can kill all of them easily.

---
 scripts/kill-bb | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100755 scripts/kill-bb

new file mode 100755
index 0000000..0875b2c
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+#
+# Kill bitbake processes if the process' cwd == cwd
+#
+# Copyright (c) 2019 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import re
+
+pids = [pid for pid in os.listdir('/proc') if pid.isdigit()]
+
+bb_pids = []
+cwd =os.getcwd()
+for pid in pids:
+    cmdline_path = os.path.join('/proc', pid, 'cmdline')
+    pid_cwd = os.path.join('/proc', pid, 'cwd')
+    if os.path.exists(cmdline_path) and os.path.exists(pid_cwd):
+        pid_cwd = os.readlink(pid_cwd)
+        with open(cmdline_path, 'r') as f:
+            cmdline = f.read()
+            # Kill the bitbake process if its cwd == cwd
+            if re.match('python3.*/bitbake/bin/bitbake', cmdline) and pid_cwd == cwd:
+                bb_pids.append(pid)
+
+if not bb_pids:
+    print('No bitbake processes found in current working dir')
+    sys.exit(0)
+
+for pid in bb_pids:
+    print('Killing %s' % pid)
+    os.kill(int(pid), 9)

2.7.4

I've noticed this, and also noticed that it got a lot worse recently.

But let's fix bitbake instead of adding tools to work around it?

Ross

I run builds in lxc containers for this and host contamination reasons.

Several build tools can also escape the bitbake environment and keep running in the build machine if bitbake itself ends or gets killed, so the problems are not only with bitbake.

-Mikko

Heh. As someone who spends a lot of time trying to debug this, I must admit I could use such a script so I'm torn on this one!

Cheers,

Richard

Hi RP and Ross,

It seems that I have figured out the root cause, I can reproduce the problem nearly 100% when parsing:

$ kill-bb; rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p

Press *one* Ctrl-C when the parsing process is at 50%, then I can reproduce the problem:

Keyboard Interrupt, closing down...

Timeout while waiting for a reply from the bitbake server

It hangs at process.join(), according to:

https://docs.python.org/3.7/library/multiprocessing.html

See the section "Joining processes that use queues", it is because the result_queue is not empty, here is a draft patch to fix the problem.

index b4851e1..c11cfec 100644
@@ -2062,6 +2062,14 @@ class CookerParser(object):
             for process in self.processes:
                 self.parser_quit.put(None)
 
+        # Cleanup the queue before call process.join(), otherwise there might be
+        # deadlocks.
+        while True:
+            try:
+               self.result_queue.get(timeout=0.25)
+            except queue.Empty:
+                break
+
         for process in self.processes:
             if force:
                 process.join(.1)

With this patch, I can't reproduce the problem any more, we may also need cleanup parser_quit in theory, but I'm not sure since I can't reproduce the problem anymore.

Now the output is:
Parsing recipes: 49% |#####################################################

                                                  | ETA:  0:00:06

Keyboard Interrupt, closing down...

Parsing recipes: 100% |############################################################################################################| Time: 0:00:08 Parsing of 2804 .bb files complete (0 cached, 1428 parsed). 1987 targets, 1618 skipped, 0 masked, 0 errors.
Execution was interrupted, returning a non-zero exit code.

I will send out the patch after more testing.

This patch can fix the *One* KeyboardInterrupt, there are other problems with two KeyboardInterrupt (traceback), I will try to fix that.

// Robert