Skip to contentSkip to conversation

bitbake: cooker: Kill alive process before join it

PATCH In review Robert Yang 2 participants 4 messages Aug 22, 2019 – Aug 23, 2019

lore
Share & tools
lore
#1 lore raw

PATCH 1/1 bitbake: cooker: Kill alive process before join it

In reply to Robert Yang

Fixed:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

It may hang in 10 mins, there are two problems:
* There might be deadlocks when call process.join() if the queue is not NULL, so we need cleanup the queue before join() it, but:
* The self.result_queue.get(timeout=0.25) may hang if the queue._wlock is hold by SomeOtherProcess, the queue has the following info when it hangs:
'_wlock': <Lock(owner=SomeOtherProcess)>, so that we may can't clean up the queue.

We can kill alvie process before join it to fix the problems.

---
 bitbake/lib/bb/cooker.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
bitbake/lib/bb/cooker.py +4 -8
index 0607fcc..e42bcba 100644
@@ -2082,19 +2082,15 @@ class CookerParser(object):
             for process in self.processes:
                 self.parser_quit.put(None)
 
-        # Cleanup the queue before call process.join(), otherwise there might be
-        # deadlocks.
-        while True:
-            try:
-               self.result_queue.get(timeout=0.25)
-            except queue.Empty:
-                break
-
         for process in self.processes:
             if force:
                 process.join(.1)
                 process.terminate()
             else:
+                # Kill the alive process firstly before join() it to avoid
+                # deadlocks
+                if process.is_alive():
+                    os.kill(process.pid, 9)
                 process.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
Signature · 1 line
2.7.4
#0 lore raw

cover bitbake: cooker: Kill alive process before join it

The following changes since commit 58ab85d3e005476ab5bcc288ad421c0c5d1e0582:

systemd: add PACKAGECONFIG for gnu-efi (2019-08-21 21:52:59 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib rbt/cooker
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/cooker

Robert Yang (1):
bitbake: cooker: Kill alive process before join it

 bitbake/lib/bb/cooker.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
Signature · 1 line
2.7.4
#2 lore raw
In reply to Robert Yang

Looks like you’re partially duplicating what the ‘force’ logic does already, but for the non-force case as well, yes? process.terminate() sounds a lot like sending a SIGTERM, though admittedly I haven’t read the code to Process recently. Perhaps if this is truly necessary, the logic can be simplified.

Quoted 55 lines · On Aug 22, 2019, 12:10 AM -0700, Robert Yang <[email protected]>, wrote:

Fixed:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

It may hang in 10 mins, there are two problems:
* There might be deadlocks when call process.join() if the queue is not NULL, so we need cleanup the queue before join() it, but:
* The self.result_queue.get(timeout=0.25) may hang if the queue._wlock is hold by SomeOtherProcess, the queue has the following info when it hangs:
'_wlock': <Lock(owner=SomeOtherProcess)>, so that we may can't clean up the queue.

We can kill alvie process before join it to fix the problems.

---
bitbake/lib/bb/cooker.py | 12 ++++--------

1 file changed, 4 insertions(+), 8 deletions(-)
bitbake/lib/bb/cooker.py +0 -0
index 0607fcc..e42bcba 100644
@@ -2082,19 +2082,15 @@ class CookerParser(object):

for process in self.processes:
self.parser_quit.put(None)

- # Cleanup the queue before call process.join(), otherwise there might be
- # deadlocks.
- while True:
- try:
- self.result_queue.get(timeout=0.25)
- except queue.Empty:
- break
-
for process in self.processes:
if force:
process.join(.1)
process.terminate()
else:
+ # Kill the alive process firstly before join() it to avoid
+ # deadlocks
+ if process.is_alive():
+ os.kill(process.pid, 9)
process.join()

sync = threading.Thread(target=self.bb_cache.sync) --
2.7.4

--
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel

#3 lore raw

Hi Larson,

Quoted 1 line · Christopher Larson

Looks like you’re partially duplicating what the ‘force’ logic does already, but for the non-force case as well, yes? process.terminate() sounds a lot like sending a SIGTERM, though admittedly I haven’t read the code to Process recently. Perhaps if this is truly necessary, the logic can be simplified.

The process.join(.1) may also hang if the result_queue is not NULL, but AFAIK , this only happens when KeyboardInterrupt or Failed to parse a recipe, so I think that 'force' shutdown's process.jion(.1) won't hang, and I didn't see it hangs in the "force" code, we may have another way to make it better, check result_queue._wlock, but the _wlock is an internal variable, so I gave it up and just use os.kill() to avoid the hang problem completely, but there might be side effects that I didn't know?

// Robert

Quoted 56 lines · On Aug 22, 2019, 12:10 AM -0700, Robert Yang <[email protected]>, wrote:
Quoted 55 lines

Fixed:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

It may hang in 10 mins, there are two problems:
* There might be deadlocks when call process.join() if the queue is not NULL, so we need cleanup the queue before join() it, but:
* The self.result_queue.get(timeout=0.25) may hang if the queue._wlock is hold by SomeOtherProcess, the queue has the following info when it hangs:
'_wlock': <Lock(owner=SomeOtherProcess)>, so that we may can't clean up the queue.

We can kill alvie process before join it to fix the problems.

---
bitbake/lib/bb/cooker.py | 12 ++++--------

1 file changed, 4 insertions(+), 8 deletions(-)
bitbake/lib/bb/cooker.py +0 -0
index 0607fcc..e42bcba 100644
@@ -2082,19 +2082,15 @@ class CookerParser(object):

for process in self.processes:
self.parser_quit.put(None)

- # Cleanup the queue before call process.join(), otherwise there might be
- # deadlocks.
- while True:
- try:
- self.result_queue.get(timeout=0.25)
- except queue.Empty:
- break
-
for process in self.processes:
if force:
process.join(.1)
process.terminate()
else:
+ # Kill the alive process firstly before join() it to avoid
+ # deadlocks
+ if process.is_alive():
+ os.kill(process.pid, 9)
process.join()

sync = threading.Thread(target=self.bb_cache.sync) --
2.7.4

--
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel