Skip to contentSkip to conversation

Force to teminate the Parser process

PATCH In review Kang Kai 2 participants 4 messages May 23, 2012 – May 24, 2012

lore
Share & tools
lore

Kang Kai

author
#0 lore raw

cover Force to teminate the Parser process

This patch is for Yocto 2142.

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop. It hangs on function BitBakeServerConnection::terminate in file server/process.py:
else:
self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as cpu numbers. When quit the Parser processes they make their internal Queue to call cancel_join_thread() to avoid block but don't work at this time.

So force to terminate the Parser processes.

The following changes since commit 3bf8069100e54153d13319dc32ca089327b6a139:

documentation/yocto-project-qs/yocto-project-qs.xml: added quotes (2012-05-01 21:00:36 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib kangkai/distro
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro

Kang Kai (1):
cooker.py: terminate the Parser processes

 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
Signature · 1 line
1.7.5.4
author
#1 lore raw

PATCH 1/1 cooker.py: terminate the Parser processes

[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop. It hangs on function BitBakeServerConnection::terminate in file server/process.py:
else:
self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as cpu numbers. When quit the Parser processes they make their internal Queue to call cancel_join_thread() to avoid block but don't work at this time.
So force to terminate the Parser processes.

Signed-off-by Kang Kai <[email protected]>
---
 bitbake/lib/bb/cooker.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
bitbake/lib/bb/cooker.py +6 -3
index dea0aad..4a4dc38 100644
@@ -1175,7 +1175,7 @@ class BBCooker:
             return
 
         if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True)
             sys.exit(1)
 
         if self.state != state.parsing:

@@ -1608,10 +1608,13 @@ class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)
 
         for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
         self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
Signature · 1 line
1.7.5.4
#2 lore raw

Re: [PATCH 1/1] cooker.py: terminate the Parser processes

terminate()'ing processes can corrupt any queues those processes have open, and thereby disrupt any pending log messages in the queues heading to the ui as a result. If that's not an issue here, so be it, but be aware of the issues -- we had to remove terminate() from the shutdown in the past due to this issue.

Quoted 57 lines · Kang Kai

[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop. It hangs on function BitBakeServerConnection::terminate in file server/process.py:
   else:
       self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as cpu numbers. When quit the Parser processes they make their internal Queue to call cancel_join_thread() to avoid block but don't work at this time.
So force to terminate the Parser processes.

Signed-off-by Kang Kai <[email protected]>

---
 bitbake/lib/bb/cooker.py |    9 ++++++---

 1 files changed, 6 insertions(+), 3 deletions(-)
bitbake/lib/bb/cooker.py +0 -0
index dea0aad..4a4dc38 100644
@@ -1175,7 +1175,7 @@ class BBCooker:

            return

        if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True) sys.exit(1)

        if self.state != state.parsing:

bitbake/lib/bb/cooker.py +0 -1
@@ -1608,10 +1608,13 @@ class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)

        for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
        self.feeder.join()

        sync = threading.Thread(target=self.bb_cache.sync) --
1.7.5.4

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
author
#3 lore raw

Re: [PATCH 1/1] cooker.py: terminate the Parser processes

Quoted 5 lines · Chris Larson

terminate()'ing processes can corrupt any queues those processes have open, and thereby disrupt any pending log messages in the queues heading to the ui as a result. If that's not an issue here, so be it, but be aware of the issues -- we had to remove terminate() from the shutdown in the past due to this issue.

Hi Chris,

Call terminate() directly seems not be elegant. But I didn't figure out why after the queue belongs to Parse process calls cancel_join_thread(), so ..
Before the terminate() the every Parse process, it is waited for 0.1 second to quit.
If the process will not be blocked, I suppose that 0.1 second is enough for them to quit.

What's your opinion?

Thanks,
Kai

Quoted 61 lines · Kang Kai
Quoted 57 lines

[Yocto 2142]

Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop. It hangs on function BitBakeServerConnection::terminate in file server/process.py:
else:
self.procserver.join()
It is waiting for the children process quit.

In stage of parse recipes BBCooker spawns Parser processes as many as cpu numbers. When quit the Parser processes they make their internal Queue to call cancel_join_thread() to avoid block but don't work at this time.
So force to terminate the Parser processes.

Signed-off-by Kang Kai<[email protected]>
---
  bitbake/lib/bb/cooker.py |    9 ++++++---
  1 files changed, 6 insertions(+), 3 deletions(-)
bitbake/lib/bb/cooker.py +6 -4
index dea0aad..4a4dc38 100644
@@ -1175,7 +1175,7 @@ class BBCooker:
             return
 
         if self.state in (state.shutdown, state.stop):
-            self.parser.shutdown(clean=False)
+            self.parser.shutdown(clean=False, force = True)
             sys.exit(1)
 
         if self.state != state.parsing:

@@ -1608,10 +1608,13 @@ class CookerParser(object):
                 self.parser_quit.put(None)
 
             self.jobs.cancel_join_thread()
-            sys.exit(1)
 
         for process in self.processes:
-            process.join()
+            if force:
+                process.join(.1)
+                process.terminate()
+            else:
+                process.join()
         self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
--

1.7.5.4

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel