memres: fix for --observe-only

Sometimes, we may see the errors:

$ bitbake --observe-only
ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
0x7fbd2e0a8438>
ERROR: Unknown event: <bb.event.RecipeTaskPreProcess object at
0x7fdc6b7e7b00>

Signed-off-by Robert Yang <[email protected]>

---
 bitbake/lib/bb/ui/knotty.py | 2 ++

 1 file changed, 2 insertions(+)

bitbake/lib/bb/ui/knotty.py +0 -0

index 11afb3e7441..71ec168fa6c 100644
@@ -667,11 +667,13 @@ def main(server, eventHandler, params, tf =

TerminalFilter):
                                   bb.event.MultiConfigParsed, bb.event.RecipeParsed, bb.event.RecipePreFinalise, +                                  bb.event.RecipeTaskPreProcess, bb.runqueue.runQueueEvent,

                                   bb.event.OperationStarted, bb.event.OperationCompleted, bb.event.OperationProgress, bb.event.DiskFull, +                                  bb.event.MonitorDiskEvent, bb.event.HeartbeatEvent, bb.build.TaskProgress)):
                 continue

Quoted 28 lines · Robert Yang

Sometimes, we may see the errors:

$ bitbake --observe-only
ERROR: Unknown event: <bb.event.MonitorDiskEvent object at
0x7fbd2e0a8438>
ERROR: Unknown event: <bb.event.RecipeTaskPreProcess object at
0x7fdc6b7e7b00>

Signed-off-by Robert Yang <[email protected]>

---
 bitbake/lib/bb/ui/knotty.py | 2 ++
 1 file changed, 2 insertions(+)

bitbake/lib/bb/ui/knotty.py +0 -0

index 11afb3e7441..71ec168fa6c 100644
@@ -667,11 +667,13 @@ def main(server, eventHandler, params, tf =

TerminalFilter):
bb.event.MultiConfigParsed, bb.event.RecipeParsed, bb.event.RecipePreFinalise,
+ bb.event.RecipeTaskPreProcess,
bb.runqueue.runQueueEvent, bb.event.OperationStarted, bb.event.OperationCompleted, bb.event.OperationProgress, bb.event.DiskFull,

Quoted 4 lines

+ bb.event.MonitorDiskEvent,
bb.event.HeartbeatEvent, bb.build.TaskProgress)):
continue

Do you know why we don't either always see these or always don't see them? I'm a bit worried there may be a deeper issue lurking here. Are those events part of the event mask being set?

For reference, I've been looking at the server abstraction in bitbake and am close to rewriting a large part of bb.server.* and bb.main with a view to simplifying the code structure and making things easier to understand.

I've noticed I see some new events with my change, equally I think its an event mask issue with my new code...

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-cont
rib/commit/?h=rpurdie/wip- rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of debug. When finished I should be able to delete server/__init__.py and server/xmlrpc.py.

Cheers,

Richard

Quoted 6 lines · Richard Purdie

Do you know why we don't either always see these or always don't see
them? I'm a bit worried there may be a deeper issue lurking here. Are
those events part of the event mask being set?

I digged this today, and find the reason, finally. It is harder than I had thought. Take MonitorDiskEvent as an example, the reproducer is:

In terminal 1:

$ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
-k

In Terminal 2:

$ bitbake --observe-only

Note, the time and order is very important, otherwise we can't reproduce
the problem, we must run "bitbake target" in terminal 1 firstly, and then
run "bitbake --observe-only" in terminal 2 after "cache loaded" before runqueue is ready, in another words, run "bitbake --observe- only"
when "Initialising tasks" is displaying in terminal 1, no early or late,
then we will see the error in terminal 2 (And only in terminal 2):

ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f5c3776f940>

The "bitbake world" (first) does everything well, but "bitbake -- observe-only"
(second) doesn't becuase of the following code in lib/bb/main.py:

         try:
             return ui_module.main(server_connection.connection, server_connection.events,
                                   configParams)

The second run of "ui_module.main()" sends server_connection.events (bb.event.MonitorDiskEvent) which is prepared by first run, and knotty.py
doesn't know how to handle it, and we will get the error. So we can't get the error if we run second bitbake earlier (the event is not ready)
or later (the event had been handled by first bitbake).

I think that bb.event.RecipeTaskPreProcess is similar, add them to ignore list should be fine, and I will add these explanations to commit message.

Quoted 4 lines

For reference, I've been looking at the server abstraction in bitbake and am close to rewriting a large part of bb.server.* and bb.main with a view to simplifying the code structure and making things easier to understand.

That's great, something var name is not easy for me to understand, for example, observe_only vs observer_only. And I think that we need a simple way to restart the server, something like bitbake --restart.

Quoted 6 lines

I've noticed I see some new events with my change, equally I think its
an event mask issue with my new code...

Quoted 9 lines

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
cont
rib/commit/?h=rpurdie/wip- rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of
debug. When finished I should be able to delete server/__init__.py and
server/xmlrpc.py.

I will hold on my patches until you've finished the rewriting, and please let me know if anything I can do.

Quoted 49 lines · Robert Yang

Quoted 6 lines · Richard Purdie

Do you know why we don't either always see these or always don't see
them? I'm a bit worried there may be a deeper issue lurking here. Are
those events part of the event mask being set?

I digged this today, and find the reason, finally. It is harder than I had thought. Take MonitorDiskEvent as an example, the reproducer is:

In terminal 1:

$ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
-k

In Terminal 2:

$ bitbake --observe-only

Note, the time and order is very important, otherwise we can't reproduce
the problem, we must run "bitbake target" in terminal 1 firstly, and then
run "bitbake --observe-only" in terminal 2 after "cache loaded" before runqueue is ready, in another words, run "bitbake --observe- only"
when "Initialising tasks" is displaying in terminal 1, no early or late,
then we will see the error in terminal 2 (And only in terminal 2):

ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f5c3776f940>

The "bitbake world" (first) does everything well, but "bitbake -- observe-only"
(second) doesn't becuase of the following code in lib/bb/main.py:

try:
return ui_module.main(server_connection.connection,
server_connection.events,
configParams)

The second run of "ui_module.main()" sends server_connection.events (bb.event.MonitorDiskEvent) which is prepared by first run, and knotty.py
doesn't know how to handle it, and we will get the error. So we can't get the error if we run second bitbake earlier (the event is not ready)
or later (the event had been handled by first bitbake).

Quoted 3 lines

I think that bb.event.RecipeTaskPreProcess is similar, add them to ignore list should be fine, and I will add these explanations to commit message.

I'm not sure I agree here, I think you're right that there is a race issue however I'd rather solve any race issue than workaround it. Whilst you need to add these events today, there are likely other events which might appear tomorrow.

Quoted 24 lines

Quoted 5 lines

For reference, I've been looking at the server abstraction in bitbake and am close to rewriting a large part of bb.server.* and bb.main with a view to simplifying the code structure and making things easier to understand.

That's great, something var name is not easy for me to understand, for example, observe_only vs observer_only. And I think that we need a simple way to restart the server, something like bitbake --restart.

Quoted 6 lines

I've noticed I see some new events with my change, equally I think its
an event mask issue with my new code...

Quoted 9 lines

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
cont
rib/commit/?h=rpurdie/wip- rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of
debug. When finished I should be able to delete server/__init__.py and
server/xmlrpc.py.

Quoted 2 lines

I will hold on my patches until you've finished the rewriting, and please let me know if anything I can do.

The changes are now in master-next. I plan to split the patch up and have been revising it a bit as I find issues. It would be great if you could apply this locally and see if the issues you've found are still present after this change and help with stressing the changes a bit, see what issues we have?

One thing I know is probably broken at the moment is observe-only unfortunately. I am tempted to merge the code changes I have as this gives us a platform to build on which is much more understandable than the existing codebase even if there are issues like that.

Cheers,

Richard

Quoted 50 lines · Richard Purdie

Quoted 49 lines · Robert Yang

Quoted 6 lines · Richard Purdie

Do you know why we don't either always see these or always don't see
them? I'm a bit worried there may be a deeper issue lurking here. Are
those events part of the event mask being set?

I digged this today, and find the reason, finally. It is harder than I had thought. Take MonitorDiskEvent as an example, the reproducer is:

In terminal 1:

$ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
-k

In Terminal 2:

$ bitbake --observe-only

Note, the time and order is very important, otherwise we can't reproduce
the problem, we must run "bitbake target" in terminal 1 firstly, and then
run "bitbake --observe-only" in terminal 2 after "cache loaded" before runqueue is ready, in another words, run "bitbake --observe- only"
when "Initialising tasks" is displaying in terminal 1, no early or late,
then we will see the error in terminal 2 (And only in terminal 2):

ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f5c3776f940>

The "bitbake world" (first) does everything well, but "bitbake -- observe-only"
(second) doesn't becuase of the following code in lib/bb/main.py:

try:
return ui_module.main(server_connection.connection,
server_connection.events,
configParams)

The second run of "ui_module.main()" sends server_connection.events (bb.event.MonitorDiskEvent) which is prepared by first run, and knotty.py
doesn't know how to handle it, and we will get the error. So we can't get the error if we run second bitbake earlier (the event is not ready)
or later (the event had been handled by first bitbake).

Quoted 8 lines

Quoted 3 lines

I think that bb.event.RecipeTaskPreProcess is similar, add them to ignore list should be fine, and I will add these explanations to commit message.

I'm not sure I agree here, I think you're right that there is a race issue however I'd rather solve any race issue than workaround it. Whilst you need to add these events today, there are likely other events which might appear tomorrow.

A rough thoughts on fixing this might be:

The original code:
try:
return ui_module.main(server_connection.connection,
server_connection.events, configParams)

The problem is server_connection.events contains more events than ui_module.main() can handle, we can filter them out: (fake code)

filtered_events = []

for event in server_connection.events:
if event in ui_module._evt_list:
filtered_events.append(event)

And use filtered_events to call ui_module.main():
return ui_module.main(server_connection.connection, filtered_events, configParams)

Then we can totally remove the ignore list in ui like knotty.py:

            # ignore
            if isinstance(event, (bb.event.BuildBase,
                                  bb.event.MetadataEvent,
                                  bb.event.StampUpdate,
                                  bb.event.ConfigParsed,
                                  bb.event.MultiConfigParsed,
                                  bb.event.RecipeParsed,
                                  bb.event.RecipePreFinalise,
                                  bb.runqueue.runQueueEvent,
                                  bb.event.OperationStarted,
                                  bb.event.OperationCompleted,
                                  bb.event.OperationProgress,
                                  bb.event.DiskFull,
                                  bb.event.HeartbeatEvent,
                                  bb.build.TaskProgress)):
                continue

Quoted 33 lines

Quoted 24 lines

Quoted 5 lines

For reference, I've been looking at the server abstraction in bitbake and am close to rewriting a large part of bb.server.* and bb.main with a view to simplifying the code structure and making things easier to understand.

That's great, something var name is not easy for me to understand, for example, observe_only vs observer_only. And I think that we need a simple way to restart the server, something like bitbake --restart.

Quoted 6 lines

I've noticed I see some new events with my change, equally I think its
an event mask issue with my new code...

Quoted 9 lines

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
cont
rib/commit/?h=rpurdie/wip- rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of
debug. When finished I should be able to delete server/__init__.py and
server/xmlrpc.py.

Quoted 2 lines

I will hold on my patches until you've finished the rewriting, and please let me know if anything I can do.

The changes are now in master-next. I plan to split the patch up and have been revising it a bit as I find issues. It would be great if you could apply this locally and see if the issues you've found are still present after this change and help with stressing the changes a bit, see what issues we have?

Quoted 4 lines

One thing I know is probably broken at the moment is observe-only unfortunately. I am tempted to merge the code changes I have as this gives us a platform to build on which is much more understandable than the existing codebase even if there are issues like that.

Sure, I'd like to test them today.

// Robert

Quoted 7 lines

Cheers,

Richard

Hi RP,

I tested master-next + memres today, unfortunately, I couldn't make memres work by now, I fixes part of the problems, but there are still other issues. Here are 3 fixes:

git://git.pokylinux.org/poky-contrib rbt/master-next-memres
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/master-next-memres

Robert Yang (3):
oe-init-build-env-memres: remove "-t xmlrpc" when run bitbake server/process.py: fix self.bitbake_lock.write() bb/main.py: fix infinite loop for --server-only

The main remaing issue is "bitbake --status-only" doesn't work:
* Reproducer:

$ . ../poky/oe-init-build-env-memres .
$ bitbake --status-only
[snip]
OverflowError: getsockaddrarg: port must be 0-65535.

I think that it is because it can't handle BBSERVER or "port == -1" correctly, we may need read port from bitbake.lock or BBSERVER, the similar to "bitbake -m":

* Reproducer:

$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
OverflowError: getsockaddrarg: port must be 0-65535.
WARNING: Could not create socket for localhost:-1 (getsockaddrarg: port must be
0-65535.)

Other problems:
1) $ . ../poky/oe-init-build-env-memres .
Starting bitbake server...
DEBUG: Removed the following variables from the environment: TERMCAP, .. It prints DEBUG messages, maybe because logger is not enabled or set correctly when --server-only

2) Errors when run oe-init-build-env-memres again: (when bitbake.lock exists)

  $ . ../poky/oe-init-build-env-memres .
  $ . ../poky/oe-init-build-env-memres .
  File "/buildarea/lyang1/poky/bitbake/lib/bb/server/xmlrpcclient.py", line 140,
in connectXMLRPC
    s.connect((host, port))
OSError: [Errno 22] Invalid argument

3) See the code of 2), there is a bb.warn() in xmlrpcclient.py:
bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))

But it doesn't print to screen, maybe the similar reason to step 1), and print() works.

// Robert

Quoted 150 lines · Robert Yang

Quoted 50 lines · Richard Purdie

Quoted 49 lines · Robert Yang

Quoted 6 lines · Richard Purdie

Do you know why we don't either always see these or always don't see
them? I'm a bit worried there may be a deeper issue lurking here. Are
those events part of the event mask being set?

I digged this today, and find the reason, finally. It is harder than I had thought. Take MonitorDiskEvent as an example, the reproducer is:

In terminal 1:

$ bitbake -m ; . ../poky/oe-init-build-env-memres . && bitbake world
-k

In Terminal 2:

$ bitbake --observe-only

Note, the time and order is very important, otherwise we can't reproduce
the problem, we must run "bitbake target" in terminal 1 firstly, and then
run "bitbake --observe-only" in terminal 2 after "cache loaded" before runqueue is ready, in another words, run "bitbake --observe- only"
when "Initialising tasks" is displaying in terminal 1, no early or late,
then we will see the error in terminal 2 (And only in terminal 2):

ERROR: Unknown event: <bb.event.MonitorDiskEvent object at 0x7f5c3776f940>

The "bitbake world" (first) does everything well, but "bitbake -- observe-only"
(second) doesn't becuase of the following code in lib/bb/main.py:

try:
return ui_module.main(server_connection.connection,
server_connection.events,
configParams)

The second run of "ui_module.main()" sends server_connection.events (bb.event.MonitorDiskEvent) which is prepared by first run, and knotty.py
doesn't know how to handle it, and we will get the error. So we can't get the error if we run second bitbake earlier (the event is not ready)
or later (the event had been handled by first bitbake).

Quoted 8 lines

Quoted 3 lines

I think that bb.event.RecipeTaskPreProcess is similar, add them to ignore list should be fine, and I will add these explanations to commit message.

I'm not sure I agree here, I think you're right that there is a race issue however I'd rather solve any race issue than workaround it. Whilst you need to add these events today, there are likely other events which might appear tomorrow.

A rough thoughts on fixing this might be:

The original code:
try:
return ui_module.main(server_connection.connection,
server_connection.events, configParams)

The problem is server_connection.events contains more events than ui_module.main() can handle, we can filter them out: (fake code)

filtered_events = []

for event in server_connection.events:
if event in ui_module._evt_list:
filtered_events.append(event)

And use filtered_events to call ui_module.main():
return ui_module.main(server_connection.connection, filtered_events, configParams)

Then we can totally remove the ignore list in ui like knotty.py:

            # ignore
            if isinstance(event, (bb.event.BuildBase,
                                  bb.event.MetadataEvent,
                                  bb.event.StampUpdate,
                                  bb.event.ConfigParsed,
                                  bb.event.MultiConfigParsed,
                                  bb.event.RecipeParsed,
                                  bb.event.RecipePreFinalise,
                                  bb.runqueue.runQueueEvent,
                                  bb.event.OperationStarted,
                                  bb.event.OperationCompleted,
                                  bb.event.OperationProgress,
                                  bb.event.DiskFull,
                                  bb.event.HeartbeatEvent,
                                  bb.build.TaskProgress)):
                continue

Quoted 33 lines

Quoted 24 lines

Quoted 5 lines

For reference, I've been looking at the server abstraction in bitbake and am close to rewriting a large part of bb.server.* and bb.main with a view to simplifying the code structure and making things easier to understand.

That's great, something var name is not easy for me to understand, for example, observe_only vs observer_only. And I think that we need a simple way to restart the server, something like bitbake --restart.

Quoted 6 lines

I've noticed I see some new events with my change, equally I think its
an event mask issue with my new code...

Quoted 9 lines

I pushed my changes onto http://git.yoctoproject.org/cgit.cgi/poky-
cont
rib/commit/?h=rpurdie/wip- rss2&id=7d970e7b9f5499f5fcdb0e73246f106844ecf09b
however I am well aware things don't work properly yet and its full of
debug. When finished I should be able to delete server/__init__.py and
server/xmlrpc.py.

Quoted 2 lines

I will hold on my patches until you've finished the rewriting, and please let me know if anything I can do.

The changes are now in master-next. I plan to split the patch up and have been revising it a bit as I find issues. It would be great if you could apply this locally and see if the issues you've found are still present after this change and help with stressing the changes a bit, see what issues we have?

Quoted 4 lines

One thing I know is probably broken at the moment is observe-only unfortunately. I am tempted to merge the code changes I have as this gives us a platform to build on which is much more understandable than the existing codebase even if there are issues like that.

Sure, I'd like to test them today.

// Robert

Quoted 7 lines

Cheers,

Richard