bitbake: cookerdata: Avoid double exceptions for bb.fatal()
cover bitbake: cookerdata: Avoid double exceptions for bb.fatal()
The following changes since commit c0dc72bad92e2e4dc0aedb48b969709f821baf73:
coreutils: Fix patch upstream status field (2019-05-08 23:36:21 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/python
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/python
Robert Yang (1):
bitbake: cookerdata: Avoid double exceptions for bb.fatal()
bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.7.4
PATCH 1/1 bitbake: cookerdata: Avoid double exceptions for bb.fatal()
The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:
Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello
Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +3 -1
@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data) - except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e: logger.error(str(e))
Signature · 1 line
2.7.4
Quoted 37 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions, e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +3
-1@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data) - except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
Quoted 1 line
logger.error(str(e))
Hi Robert,
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?
Cheers,
Richard
Quoted 42 lines · Richard Purdie
Quoted 37 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions, e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +3
-1@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data) - except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e:Quoted 1 line
logger.error(str(e))
Hi Robert,
Quoted 5 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?
Hi RP,
I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]
Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.
I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one of result_queue or parser_quit have a deadlock, then causes the shutdown():process.join() hanged.
2) The sub processes, parser_quit and result_queue are running dependently, we can't make sure that cooker.py:Parser():realrun() can get data from parser_quit, so there is a race issue, the Parser's subprocess maybe already exited before parser_quit sends data to it.
I think that we can move part of parse_next()'s code into parse_generator() to fix the deadlock problem, and we may need change the working way of parser_quit. But I'm not sure, I need your suggestions before work on it.
// Robert
Quoted 6 lines
Cheers,
Richard
Quoted 69 lines · Robert Yang
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions, e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/cookerdata.py| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object):if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdataQuoted 8 lines
), self.data)
- except (SyntaxError, bb.BBHandledException):
+ except bb.BBHandledException:
+ raise
+ except SyntaxError:
raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
logger.error(str(e))Hi Robert,
Quoted 5 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?
Hi RP,
I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.
I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one ofresult_queueorparser_quithave a deadlock, then causes the shutdown():process.join() hanged.
Quoted 8 lines
2) The sub processes,
parser_quitandresult_queueare running dependently, we can't make sure that cooker.py:Parser():realrun() can get data fromparser_quit, so there is a race issue, the Parser's subprocess maybe already exited beforeparser_quitsends data to it.I think that we can move part of parse_next()'s code into
parse_generator()to fix the deadlock problem, and we may need change the working way ofparser_quit. But I'm not sure, I need your suggestions before work on it.
Hi RP,
Do you have any suggestions, please?
// Robert
Quoted 9 lines
// Robert
Quoted 6 lines
Cheers,
Richard
Quoted 55 lines · Robert Yang
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions, e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/cookerdata.py| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object):if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdataQuoted 8 lines
), self.data)
- except (SyntaxError, bb.BBHandledException):
+ except bb.BBHandledException:
+ raise
+ except SyntaxError:
raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
logger.error(str(e))Hi Robert,
Quoted 5 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?
Hi RP,
I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:
Quoted 8 lines
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.
BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()
Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
It's not easy to hang any more, but still hangs sometimes, I tried to debug it, but didn't find the root cause, the ui/knotty.py can't get event from server, and goes into a dead loop.
event = eventHandler.waitEvent(0) if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
continue
The main.shutdown is always 0 when it hangs.
// Robert
Quoted 23 lines
I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one ofresult_queueorparser_quithave a deadlock, then causes the shutdown():process.join() hanged.2) The sub processes,
parser_quitandresult_queueare running dependently, we can't make sure that cooker.py:Parser():realrun() can get data fromparser_quit, so there is a race issue, the Parser's subprocess maybe already exited beforeparser_quitsends data to it.I think that we can move part of parse_next()'s code into
parse_generator()to fix the deadlock problem, and we may need change the working way ofparser_quit. But I'm not sure, I need your suggestions before work on it.// Robert
Quoted 6 lines
Cheers,
Richard
Quoted 103 lines · Robert Yang
Quoted 69 lines · Robert Yang
Quoted 47 lines · Richard Purdie
Quoted 36 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions,
e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data)bitbake/lib/bb/cookerdata.py class CookerDataBuilder(object): · line 309
Quoted 7 lines
bitbake/lib/bb/cookerdata.py +3
-1- except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e: logger.error(str(e))Hi Robert,
Quoted 7 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?Hi RP,
I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Quoted 5 lines
Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before callprocess.join()Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; doneIt's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, theui/knotty.pycan't get event from server,
and goes into a dead loop.event =
eventHandler.waitEvent(0)if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter()event =eventHandler.waitEvent(0.25)if event is None:
continue
Quoted 1 line
The main.shutdown is always 0 when it hangs.
In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?
I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?
Thanks for the patches, those were tricky issues to track down and solve!
Cheers,
Richard
Quoted 106 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 103 lines · Robert Yang
Quoted 69 lines · Robert Yang
Quoted 47 lines · Richard Purdie
Quoted 36 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions,
e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data)bitbake/lib/bb/cookerdata.py class CookerDataBuilder(object): · line 309
Quoted 7 lines
bitbake/lib/bb/cookerdata.py +3
-1- except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e: logger.error(str(e))Hi Robert,
Quoted 7 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?Hi RP,
I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Quoted 5 lines
Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before callprocess.join()Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; doneIt's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, theui/knotty.pycan't get event from server,
and goes into a dead loop.event =
eventHandler.waitEvent(0)if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter()event =eventHandler.waitEvent(0.25)if event is None:
continueQuoted 1 line
The main.shutdown is always 0 when it hangs.
Quoted 7 lines
In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?
I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?
After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
* Hangs #1 in cooker.py:
2065 # Cleanup the queue before call process.join(), otherwise there might be 2066 # deadlocks. 2067 while True:
2068 try:
2069 self.result_queue.get(timeout=0.25)
2070 except queue.Empty:
2071 break
It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can call self.result_queue.cancel_join_thread() to fix the problem.
* Hangs #2 in cooker.py:
2073 for process in self.processes:
2074 if force:
2075 process.join(.1)
2076 process.terminate()
2077 else:
2078 process.join()
It hangs at process.join(), I added debug code there, it is because the process is alive when join() it, I think that we can use a while loop to check whether the process is alive or not before join(), and force join() after many tries.
I will do more testing before send the patches, make sure it won't hang in hours.
// Robert
Quoted 10 lines
Thanks for the patches, those were tricky issues to track down and solve!
Cheers,
Richard
Quoted 84 lines · Robert Yang
Quoted 82 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 81 lines · Robert Yang
Quoted 68 lines · Robert Yang
Quoted 46 lines · Richard Purdie
Quoted 34 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions,
e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/cookerdata.py| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object):if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdataQuoted 8 lines
), self.data)
- except (SyntaxError, bb.BBHandledException):
+ except bb.BBHandledException:
+ raise
+ except SyntaxError:
raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
logger.error(str(e))Hi Robert,
Quoted 7 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?Hi RP,
I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Quoted 5 lines
Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before callprocess.join()
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
Sorry, the full commands should be:
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
// Rboert
Quoted 75 lines
Quoted 18 lines
Quoted 16 lines
It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, theui/knotty.pycan't get event from server,
and goes into a dead loop.event =
eventHandler.waitEvent(0)if event is None:
if main.shutdown > 1:
breaktermfilter.updateFooter()event =eventHandler.waitEvent(0.25)if event is None:
continueQuoted 1 line
The main.shutdown is always 0 when it hangs.
Quoted 7 lines
In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?
I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?
After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done* Hangs #1 in
cooker.py:2065 # Cleanup the queue before call
process.join(), otherwise there might be 2066 # deadlocks.
2067 while True:
2068 try:
2069self.result_queue.get(timeout=0.25)
2070 except queue.Empty:
2071 breakIt hangs at
self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can callself.result_queue.cancel_join_thread()to fix the problem.* Hangs #2 in
cooker.py:
2073 for process in self.processes:
2074 if force:
2075process.join(.1)
2076process.terminate()
2077 else:
2078process.join()It hangs at
process.join(), I added debug code there, it is because the process is alive whenjoin()it, I think that we can use a while loop to check whether the process is alive or not beforejoin(), and forcejoin()after many tries.I will do more testing before send the patches, make sure it won't hang in hours.
// Robert
Quoted 10 lines
Thanks for the patches, those were tricky issues to track down and solve!
Cheers,
Richard
Quoted 146 lines · Robert Yang
Quoted 105 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 102 lines · Robert Yang
Quoted 68 lines · Robert Yang
Quoted 46 lines · Richard Purdie
Quoted 34 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions,
e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/cookerdata.py| 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +0
-0@@ -301,7 +301,9 @@ class CookerDataBuilder(object):if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdataQuoted 8 lines
), self.data)
- except (SyntaxError, bb.BBHandledException):
+ except bb.BBHandledException:
+ raise
+ except SyntaxError:
raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
logger.error(str(e))Hi Robert,
Quoted 7 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?Hi RP,
I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs]Quoted 5 lines
Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before callprocess.join()Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; doneIt's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, theui/knotty.pycan't get event from server,
and goes into a dead loop.event =
eventHandler.waitEvent(0)if event is None:
if main.shutdown > 1:
breaktermfilter.updateFooter()event =eventHandler.waitEvent(0.25)if event is None:
continueQuoted 1 line
The main.shutdown is always 0 when it hangs.
Quoted 7 lines
In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?
I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?
After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done* Hangs #1 in
cooker.py:2065 # Cleanup the queue before call
process.join(), otherwise there might be 2066 # deadlocks.
2067 while True:
2068 try:
2069self.result_queue.get(timeout=0.25)
2070 except queue.Empty:
2071 breakIt hangs at
self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can callself.result_queue.cancel_join_thread()to fix the problem.* Hangs #2 in
cooker.py:
2073 for process in self.processes:
2074 if force:
2075process.join(.1)
2076process.terminate()
2077 else:
2078process.join()
Quoted 5 lines
It hangs at
process.join(), I added debug code there, it is because the process is alive whenjoin()it, I think that we can use a while loop to check whether the process is alive or not beforejoin(), and forcejoin()after many tries.I will do more testing before send the patches, make sure it won't hang in hours.
After a lot tries, the only reliable way to avoid hanging is os.kill(), I've sent the patch for it, and added reason in the commit message.
// Robert
Quoted 13 lines
// Robert
Quoted 10 lines
Thanks for the patches, those were tricky issues to track down and solve!
Cheers,
Richard
Hi RP,
Quoted 42 lines · Richard Purdie
Quoted 37 lines · Robert Yang
The
bb.fatal()raisesBBHandledException()which causes double exceptions, e.g.:Add 'HOSTTOOLS += "hello"' to
conf/local.conf:$ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: helloUse "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.
[YOCTO #13267]
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)bitbake/lib/bb/cookerdata.py +3
-1@@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data) - except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e:Quoted 1 line
logger.error(str(e))
Hi Robert,
Quoted 5 lines
This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?
I think that I've found the root cause, it is because no code handles BBHandledException during server is starting, so we will *always* get (double) exceptions when BBHandledException comes during server is starting, such as call bb.fatal() during parsing configurations files. I will send a patch to fix it.
// Robert
Quoted 6 lines
Cheers,
Richard