Skip to contentSkip to conversation

bitbake: cookerdata: Avoid double exceptions for bb.fatal()

PATCH In review Robert Yang 2 participants 11 messages May 9, 2019 – Aug 22, 2019

lore
Share & tools
lore
#0 lore raw

cover bitbake: cookerdata: Avoid double exceptions for bb.fatal()

The following changes since commit c0dc72bad92e2e4dc0aedb48b969709f821baf73:

coreutils: Fix patch upstream status field (2019-05-08 23:36:21 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib rbt/python
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/python

Robert Yang (1):
bitbake: cookerdata: Avoid double exceptions for bb.fatal()

 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/1 bitbake: cookerdata: Avoid double exceptions for bb.fatal()

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +3 -1
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
             if multiconfig:
                 bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
             logger.error(str(e))
Signature · 1 line
2.7.4
#2 lore raw
Quoted 37 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +3 -1
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
             if multiconfig:
                 bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
 ), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
Quoted 1 line

logger.error(str(e))

Hi Robert,

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Cheers,

Richard

#3 lore raw
Quoted 42 lines · Richard Purdie
Quoted 37 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
   hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
  bitbake/lib/bb/cookerdata.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +3 -1
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
              if multiconfig:
                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
 ), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
              raise bb.BBHandledException
          except bb.data_smart.ExpansionError as e:
Quoted 1 line

logger.error(str(e))

Hi Robert,

Quoted 5 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one of result_queue or parser_quit have a deadlock, then causes the shutdown():process.join() hanged.

2) The sub processes, parser_quit and result_queue are running dependently, we can't make sure that cooker.py:Parser():realrun() can get data from parser_quit, so there is a race issue, the Parser's subprocess maybe already exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator() to fix the deadlock problem, and we may need change the working way of parser_quit. But I'm not sure, I need your suggestions before work on it.

// Robert

Quoted 6 lines

Cheers,

Richard

#4 lore raw
Quoted 69 lines · Robert Yang
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
   hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
  bitbake/lib/bb/cookerdata.py | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):

              if multiconfig:
                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata

Quoted 8 lines

), self.data)
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
              raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
              logger.error(str(e))

Hi Robert,

Quoted 5 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one of result_queue or parser_quit have a deadlock, then causes the shutdown():process.join() hanged.

Quoted 8 lines

2) The sub processes, parser_quit and result_queue are running dependently, we can't make sure that cooker.py:Parser():realrun() can get data from parser_quit, so there is a race issue, the Parser's subprocess maybe already exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator() to fix the deadlock problem, and we may need change the working way of parser_quit. But I'm not sure, I need your suggestions before work on it.

Hi RP,

Do you have any suggestions, please?

// Robert

Quoted 9 lines

// Robert

Quoted 6 lines

Cheers,

Richard

#5 lore raw
In reply to Robert Yang
Quoted 55 lines · Robert Yang
Quoted 41 lines · Richard Purdie
Quoted 29 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
   hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
  bitbake/lib/bb/cookerdata.py | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):

              if multiconfig:
                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata

Quoted 8 lines

), self.data)
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
              raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
              logger.error(str(e))

Hi Robert,

Quoted 5 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.:

Quoted 8 lines
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

It's not easy to hang any more, but still hangs sometimes, I tried to debug it, but didn't find the root cause, the ui/knotty.py can't get event from server, and goes into a dead loop.

event = eventHandler.waitEvent(0) if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
continue

The main.shutdown is always 0 when it hangs.

// Robert

Quoted 23 lines

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes one of result_queue or parser_quit have a deadlock, then causes the shutdown():process.join() hanged.

2) The sub processes, parser_quit and result_queue are running dependently, we can't make sure that cooker.py:Parser():realrun() can get data from parser_quit, so there is a race issue, the Parser's subprocess maybe already exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator() to fix the deadlock problem, and we may need change the working way of parser_quit. But I'm not sure, I need your suggestions before work on it.

// Robert

Quoted 6 lines

Cheers,

Richard

#6 lore raw
Quoted 103 lines · Robert Yang
Quoted 69 lines · Robert Yang
Quoted 47 lines · Richard Purdie
Quoted 36 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception
occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS)
appear to be 
unavailable in PATH, please install them in order to proceed:
   hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

---
  bitbake/lib/bb/cookerdata.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
              if multiconfig:
 

bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data)

bitbake/lib/bb/cookerdata.py class CookerDataBuilder(object): · line 309

Quoted 7 lines
bitbake/lib/bb/cookerdata.py +3 -1
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
              raise bb.BBHandledException
          except bb.data_smart.ExpansionError as e:
              logger.error(str(e))

Hi Robert,

Quoted 7 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
'helloworld'
[hangs]
Quoted 5 lines

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
done

It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, the ui/knotty.py can't get event from server,
and goes into a dead loop.

event = eventHandler.waitEvent(0) if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
continue

Quoted 1 line

The main.shutdown is always 0 when it hangs.

In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?

I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?

Thanks for the patches, those were tricky issues to track down and solve!

Cheers,

Richard

#7 lore raw
Quoted 106 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 103 lines · Robert Yang
Quoted 69 lines · Robert Yang
Quoted 47 lines · Richard Purdie
Quoted 36 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception
occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS)
appear to be
unavailable in PATH, please install them in order to proceed:
    hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

---
   bitbake/lib/bb/cookerdata.py | 4 +++-
   1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
               if multiconfig:
 

bb.event.fire(bb.event.MultiConfigParsed(self.mcdata ), self.data)

bitbake/lib/bb/cookerdata.py class CookerDataBuilder(object): · line 309

Quoted 7 lines
bitbake/lib/bb/cookerdata.py +3 -1
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
               raise bb.BBHandledException
           except bb.data_smart.ExpansionError as e:
               logger.error(str(e))

Hi Robert,

Quoted 7 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
'helloworld'
[hangs]
Quoted 5 lines

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
done

It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, the ui/knotty.py can't get event from server,
and goes into a dead loop.

event = eventHandler.waitEvent(0) if event is None:
if main.shutdown > 1:
break
termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
continue

Quoted 1 line

The main.shutdown is always 0 when it hangs.

Quoted 7 lines

In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?

I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?

After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

* Hangs #1 in cooker.py:

2065         # Cleanup the queue before call process.join(), otherwise there might be
2066         # deadlocks.
2067         while True:

2068 try:
2069 self.result_queue.get(timeout=0.25)
2070 except queue.Empty:
2071 break

It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can call self.result_queue.cancel_join_thread() to fix the problem.

* Hangs #2 in cooker.py:
2073 for process in self.processes:
2074 if force:
2075 process.join(.1)
2076 process.terminate()
2077 else:
2078 process.join()

It hangs at process.join(), I added debug code there, it is because the process is alive when join() it, I think that we can use a while loop to check whether the process is alive or not before join(), and force join() after many tries.

I will do more testing before send the patches, make sure it won't hang in hours.

// Robert

Quoted 10 lines

Thanks for the patches, those were tricky issues to track down and solve!

Cheers,

Richard

#8 lore raw
Quoted 84 lines · Robert Yang
Quoted 82 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 81 lines · Robert Yang
Quoted 68 lines · Robert Yang
Quoted 46 lines · Richard Purdie
Quoted 34 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception
occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS)
appear to be
unavailable in PATH, please install them in order to proceed:
    hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

---
   bitbake/lib/bb/cookerdata.py | 4 +++-

   1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):

               if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata

Quoted 8 lines

), self.data)
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
               raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
               logger.error(str(e))

Hi Robert,

Quoted 7 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
'helloworld'
[hangs]
Quoted 5 lines

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
done

Sorry, the full commands should be:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

// Rboert

Quoted 75 lines
Quoted 18 lines
Quoted 16 lines

It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, the ui/knotty.py can't get event from server,
and goes into a dead loop.

              event = eventHandler.waitEvent(0) if event is None:
                  if main.shutdown > 1:
                      break termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
                      continue

Quoted 1 line

The main.shutdown is always 0 when it hangs.

Quoted 7 lines

In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?

I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?

After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

* Hangs #1 in cooker.py:

2065         # Cleanup the queue before call process.join(), otherwise there might be 2066         # deadlocks.
2067         while True:
2068             try:
2069                self.result_queue.get(timeout=0.25)
2070             except queue.Empty:
2071                 break

It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can call self.result_queue.cancel_join_thread() to fix the problem.

* Hangs #2 in cooker.py:
2073         for process in self.processes:
2074             if force:
2075                 process.join(.1)
2076                 process.terminate()
2077             else:
2078                 process.join()

It hangs at process.join(), I added debug code there, it is because the process is alive when join() it, I think that we can use a while loop to check whether the process is alive or not before join(), and force join() after many tries.

I will do more testing before send the patches, make sure it won't hang in hours.

// Robert

Quoted 10 lines

Thanks for the patches, those were tricky issues to track down and solve!

Cheers,

Richard

#9 lore raw
In reply to Robert Yang
Quoted 146 lines · Robert Yang
Quoted 105 lines · On 8/16/19 7:03 AM, [email protected] wrote:
Quoted 102 lines · Robert Yang
Quoted 68 lines · Robert Yang
Quoted 46 lines · Richard Purdie
Quoted 34 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception
occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS)
appear to be
unavailable in PATH, please install them in order to proceed:
    hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

---
   bitbake/lib/bb/cookerdata.py | 4 +++-

   1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +0 -0
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):

               if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata

Quoted 8 lines

), self.data)
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
               raise bb.BBHandledException except bb.data_smart.ExpansionError as e:
               logger.error(str(e))

Hi Robert,

Quoted 7 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
'helloworld'
[hangs]
Quoted 5 lines

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
done

It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, the ui/knotty.py can't get event from server,
and goes into a dead loop.

              event = eventHandler.waitEvent(0) if event is None:
                  if main.shutdown > 1:
                      break termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None:
                      continue

Quoted 1 line

The main.shutdown is always 0 when it hangs.

Quoted 7 lines

In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected?

I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events?

After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes:

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

* Hangs #1 in cooker.py:

2065         # Cleanup the queue before call process.join(), otherwise there might be 2066         # deadlocks.
2067         while True:
2068             try:
2069                self.result_queue.get(timeout=0.25)
2070             except queue.Empty:
2071                 break

It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's multiprocessing, and we can call self.result_queue.cancel_join_thread() to fix the problem.

* Hangs #2 in cooker.py:
2073         for process in self.processes:
2074             if force:
2075                 process.join(.1)
2076                 process.terminate()
2077             else:
2078                 process.join()

Quoted 5 lines

It hangs at process.join(), I added debug code there, it is because the process is alive when join() it, I think that we can use a while loop to check whether the process is alive or not before join(), and force join() after many tries.

I will do more testing before send the patches, make sure it won't hang in hours.

After a lot tries, the only reliable way to avoid hanging is os.kill(), I've sent the patch for it, and added reason in the commit message.

// Robert

Quoted 13 lines

// Robert

Quoted 10 lines

Thanks for the patches, those were tricky issues to track down and solve!

Cheers,

Richard

#10 lore raw
In reply to Richard Purdie

Hi RP,

Quoted 42 lines · Richard Purdie
Quoted 37 lines · Robert Yang

The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:

$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
   hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions.

[YOCTO #13267]

---
  bitbake/lib/bb/cookerdata.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
bitbake/lib/bb/cookerdata.py +3 -1
index f8ae410..585edc5 100644
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
              if multiconfig:
                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
 ), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
              raise bb.BBHandledException
          except bb.data_smart.ExpansionError as e:
Quoted 1 line

logger.error(str(e))

Hi Robert,

Quoted 5 lines

This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException?

I think that I've found the root cause, it is because no code handles BBHandledException during server is starting, so we will *always* get (double) exceptions when BBHandledException comes during server is starting, such as call bb.fatal() during parsing configurations files. I will send a patch to fix it.

// Robert

Quoted 6 lines

Cheers,

Richard