Skip to contentSkip to conversation

toaster build functionality - preparatory patches

PATCH In review Alex DAMIAN 2 participants 16 messages Jun 3, 2014 – Jun 4, 2014

lore
Share & tools
lore

Alex DAMIAN

author
#0 lore raw

cover toaster build functionality - preparatory patches

From: Alexandru DAMIAN <[email protected]>

Hello,

This is a set of patches that bring punctual fixes, in preparation for deploying Toaster build functionality. They are mainly focused on making bitbake and toaster a bit more resilient on starting and stopping programatically, under the control of a program, and preparing data models for controlling a build.

The patches have been reviewed on the toaster list.

Can you please pull ?

Cheers,
Alex

The following changes since commit 18f9bcbad059608e22fca20309314e1c399acec7:

siggen: Print warning about tainted tasks (2014-06-01 14:26:51 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib adamian/20140503-submission-bb
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/20140503-submission-bb

Alexandru DAMIAN (9):
bitbake: move configuration reading code
xmlrpc: client - remove fatal errors
uievent: retry on handler registration failure
toaster: read database settings from the environment
toaster: startup script noweb mode
toaster: startup script standalone mode
toaster: update toaster to run in managed mode
toaster: add project related models
toasterui: fix django settings environment value

 bin/bitbake                         | 60 +++++++++++++++++++++++++------------
 bin/toaster                         | 58 ++++++++++++++++++++++++++++++-----
 lib/bb/server/xmlrpc.py             | 46 ++++++++++------------------
 lib/bb/ui/buildinfohelper.py        |  2 +-
 lib/bb/ui/toasterui.py              | 16 +++++++---
 lib/bb/ui/uievent.py                | 23 ++++++++++++--
 lib/toaster/orm/models.py           | 19 ++++++++++++
 lib/toaster/toastermain/settings.py | 35 +++++++++++++++++++++-
 8 files changed, 193 insertions(+), 66 deletions(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH 1/9 bitbake: move configuration reading code

From: Alexandru DAMIAN <[email protected]>

The configuration reading code should live in the main bitbake entry point, and the server modules should be supplied with correct configuration instead of attempting to parse from configuration files.

This patch moves the endpoint address reading from XMLRPC to the bitbake main script.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 ++++++++++++++++++++++-
 lib/bb/server/xmlrpc.py | 24 ------------------------
 2 files changed, 22 insertions(+), 25 deletions(-)
bin/bitbake +22 -1
index 86d32cf..fcfe043 100755
@@ -200,6 +200,28 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
             options.servertype = "xmlrpc"
             options.remote_server = os.environ["BBSERVER"]
 
+        # if BBSERVER says to autodetect, let's do that
+        if options.remote_server:
+            [host, port] = options.remote_server.split(":", 2)
+            port = int(port)
+            # use automatic port if port set to -1, means read it from
+            # the bitbake.lock file; this is a bit tricky, but we always expect
+            # to be in the base of the build directory if we need to have a
+            # chance to start the server later, anyway
+            if port == -1:
+                lock_location = "./bitbake.lock"
+                # we try to read the address at all times; if the server is not started,
+                # we'll try to start it after the first connect fails, below
+                try:
+                    lf = open(lock_location, 'r')
+                    remotedef = lf.readline()
+                    [host, port] = remotedef.split(":")
+                    port = int(port)
+                    lf.close()
+                    options.remote_server = remotedef
+                except Exception as e:
+                    sys.exit("Failed to read bitbake.lock (%s), invalid port" % str(e))
+
         return options, targets[1:]
 
 

@@ -309,7 +331,6 @@ def main():
         # we start a stub server that is actually a XMLRPClient that connects to a real server
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
-        server.saveConnectionConfigParams(configParams)
 
     if not configParams.server_only:
         if configParams.status_only:
lib/bb/server/xmlrpc.py +0 -24
index 5dcaa6c..6fc5543 100644
@@ -340,9 +340,6 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
     def saveConnectionDetails(self, remote):
         self.remote = remote
 
-    def saveConnectionConfigParams(self, configParams):
-        self.configParams = configParams
-
     def establishConnection(self, featureset):
         # The format of "remote" must be "server:port"
         try:

@@ -351,27 +348,6 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
         except Exception as e:
             bb.fatal("Failed to read remote definition (%s)" % str(e))
 
-        # use automatic port if port set to -1, meaning read it from
-        # the bitbake.lock file
-        if port == -1:
-            lock_location = "%s/bitbake.lock" % self.configParams.environment.get('BUILDDIR')
-            lock = bb.utils.lockfile(lock_location, False, False)
-            if lock:
-                # This means there is no server running which we can
-                # connect to on the local system.
-                bb.utils.unlockfile(lock)
-                return None
-
-            try:
-                lf = open(lock_location, 'r')
-                remotedef = lf.readline()
-                [host, port] = remotedef.split(":")
-                port = int(port)
-                lf.close()
-                self.remote = remotedef
-            except Exception as e:
-                bb.fatal("Failed to read bitbake.lock (%s)" % str(e))
-
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server
         try:
Signature · 1 line
1.9.1
author
#2 lore raw

PATCH 2/9 xmlrpc: client - remove fatal errors

From: Alexandru DAMIAN <[email protected]>

When we use the XMLRPC client API to connect to a bitbake server, we want to receive errors from the API instead of having the API exiting without warning.

Thus the "bb.fatal" calls have been replaced with "bb.warn" calls, and we re-raise the original exception for handling by the original caller.

The bitbake starting script has been modified to properly test for failures in calling the client API and handle them.

Additional error handling added in the client, as to prevent fatal crashes.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 +++++++++++++----------
 lib/bb/server/xmlrpc.py | 22 +++++++++++++++-------
 lib/bb/ui/uievent.py    |  3 ++-
 3 files changed, 30 insertions(+), 18 deletions(-)
bin/bitbake +13 -10
index fcfe043..ab881c5 100755
@@ -332,26 +332,29 @@ def main():
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
+    def _getServerConnection(server, featureset):
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
+        return server_connection
+
     if not configParams.server_only:
         if configParams.status_only:
-            try:
-                server_connection = server.establishConnection(featureset)
-            except:
-                sys.exit(1)
-            if not server_connection:
-                sys.exit(1)
+            server_connection = _getServerConnection(server, featureset)
             server_connection.terminate()
             sys.exit(0)
 
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection(featureset)
-        if not server_connection:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except:
             if configParams.kill_server:
-                bb.fatal("Server already killed")
+                bb.fatal("Server already offline")
             configParams.bind = configParams.remote_server
             start_server(servermodule, configParams, configuration, featureset)
             bb.event.ui_queue = []
-            server_connection = server.establishConnection(featureset)
+            server_connection = _getServerConnection(server, featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
lib/bb/server/xmlrpc.py +15 -7
index 6fc5543..6ec3452 100644
@@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
 
 def _create_server(host, port, timeout = 60):
     t = BBTransport(timeout)
-    s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t, allow_none=True)
+    s = xmlrpclib.ServerProxy("http://%s:%d/" % (host, port), transport=t, allow_none=True)
     return s, t
 
 class BitBakeServerCommands():

@@ -253,13 +253,18 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
             socktimeout = self.socket.gettimeout() or nextsleep
             socktimeout = min(socktimeout, nextsleep)
             # Mirror what BaseServer handle_request would do
-            fd_sets = select.select(fds, [], [], socktimeout)
-            if fd_sets[0] and self in fd_sets[0]:
-                self._handle_request_noblock()
+            try:
+                fd_sets = select.select(fds, [], [], socktimeout)
+                if fd_sets[0] and self in fd_sets[0]:
+                    self._handle_request_noblock()
+            except IOError:
+                # we ignore interrupted calls
+                pass
 
         # Tell idle functions we're exiting
         for function, data in self._idlefuns.items():
             try:
+                print "Running idle funcs", function, data
                 retval = function(self, data, True)
             except:
                 pass

@@ -346,7 +351,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             [host, port] = self.remote.split(":")
             port = int(port)
         except Exception as e:
-            bb.fatal("Failed to read remote definition (%s)" % str(e))
+            bb.warn("Failed to read remote definition (%s)" % str(e))
+            raise e
 
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server

@@ -356,13 +362,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             ip = s.getsockname()[0]
             s.close()
         except Exception as e:
-            bb.fatal("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            raise e
         try:
             self.serverImpl = XMLRPCProxyServer(host, port)
             self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
             return self.connection.connect()
         except Exception as e:
-            bb.fatal("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            raise e
 
     def endSession(self):
         self.connection.removeClient()
lib/bb/ui/uievent.py +2 -1
index 98658f6..eb760c0 100644
@@ -47,7 +47,8 @@ class BBUIEventQueue:
         self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
 
         if (self.EventHandle == None):
-            bb.fatal("Could not register UI event handler")
+            bb.warn("Could not register UI event handler %s:%d" % (self.host, self.port))
+            raise Exception("Could not register UI event handler")
 
         self.server = server
Signature · 1 line
1.9.1
#3 lore raw

Re: [PATCH 2/9] xmlrpc: client - remove fatal errors

Quoted 89 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

When we use the XMLRPC client API to connect to a bitbake server, we want to receive errors from the API instead of having the API exiting without warning.

Thus the "bb.fatal" calls have been replaced with "bb.warn" calls, and we re-raise the original exception for handling by the original caller.

The bitbake starting script has been modified to properly test for failures in calling the client API and handle them.

Additional error handling added in the client, as to prevent fatal crashes.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 +++++++++++++----------
 lib/bb/server/xmlrpc.py | 22 +++++++++++++++-------
 lib/bb/ui/uievent.py    |  3 ++-
 3 files changed, 30 insertions(+), 18 deletions(-)
bin/bitbake +13 -10
index fcfe043..ab881c5 100755
@@ -332,26 +332,29 @@ def main():
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
+    def _getServerConnection(server, featureset):
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
+        return server_connection
+
     if not configParams.server_only:
         if configParams.status_only:
-            try:
-                server_connection = server.establishConnection(featureset)
-            except:
-                sys.exit(1)
-            if not server_connection:
-                sys.exit(1)
+            server_connection = _getServerConnection(server, featureset)
             server_connection.terminate()
             sys.exit(0)
 
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection(featureset)
-        if not server_connection:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except:
             if configParams.kill_server:
-                bb.fatal("Server already killed")
+                bb.fatal("Server already offline")
             configParams.bind = configParams.remote_server
             start_server(servermodule, configParams, configuration, featureset)
             bb.event.ui_queue = []
-            server_connection = server.establishConnection(featureset)
+            server_connection = _getServerConnection(server, featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
lib/bb/server/xmlrpc.py +3 -4
index 6fc5543..6ec3452 100644
@@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
 
 def _create_server(host, port, timeout = 60):
     t = BBTransport(timeout)
-    s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t, allow_none=True)
+    s = xmlrpclib.ServerProxy("http://%s:%d/" % (host, port), transport=t, allow_none=True)
     return s, t
 
 class BitBakeServerCommands():

@@ -253,13 +253,18 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
             socktimeout = self.socket.gettimeout() or nextsleep
             socktimeout = min(socktimeout, nextsleep)
             # Mirror what BaseServer handle_request would do
-            fd_sets = select.select(fds, [], [], socktimeout)
-            if fd_sets[0] and self in fd_sets[0]:
-                self._handle_request_noblock()
+            try:
+                fd_sets = select.select(fds, [], [], socktimeout)

lib/bb/server/xmlrpc.py class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer): · line 267

Quoted 10 lines
lib/bb/server/xmlrpc.py +6 -0
+                if fd_sets[0] and self in fd_sets[0]:
+                    self._handle_request_noblock()
+            except IOError:
+                # we ignore interrupted calls
+                pass
 
         # Tell idle functions we're exiting
         for function, data in self._idlefuns.items():
             try:
+                print "Running idle funcs", function, data

Did you mean to leave this in?

Cheers,

Richard

Quoted 48 lines
lib/bb/server/xmlrpc.py +6 -3
                 retval = function(self, data, True)
             except:
                 pass

@@ -346,7 +351,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             [host, port] = self.remote.split(":")
             port = int(port)
         except Exception as e:
-            bb.fatal("Failed to read remote definition (%s)" % str(e))
+            bb.warn("Failed to read remote definition (%s)" % str(e))
+            raise e
 
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server

@@ -356,13 +362,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             ip = s.getsockname()[0]
             s.close()
         except Exception as e:
-            bb.fatal("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            raise e
         try:
             self.serverImpl = XMLRPCProxyServer(host, port)
             self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
             return self.connection.connect()
         except Exception as e:
-            bb.fatal("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            raise e
 
     def endSession(self):
         self.connection.removeClient()
lib/bb/ui/uievent.py +2 -1
index 98658f6..eb760c0 100644
@@ -47,7 +47,8 @@ class BBUIEventQueue:
         self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
 
         if (self.EventHandle == None):
-            bb.fatal("Could not register UI event handler")
+            bb.warn("Could not register UI event handler %s:%d" % (self.host, self.port))
+            raise Exception("Could not register UI event handler")
 
         self.server = server
Signature · 1 line
1.9.1
author
#4 lore raw

Re: [PATCH 2/9] xmlrpc: client - remove fatal errors

Yep, something wrong with it ?

The thing is that I'm replacing bb.fatal with bb.warn to get a chance for a remote controlling to intercept errors instead of having the program shutting down.

Alex

Quoted 173 lines · Richard Purdie
Quoted 28 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

When we use the XMLRPC client API to connect to a bitbake server, we want to receive errors from the API instead of having the API exiting without warning.

Thus the "bb.fatal" calls have been replaced with "bb.warn" calls, and we re-raise the original exception for handling by the original caller.

The bitbake starting script has been modified to properly test for failures in calling the client API and handle them.

Additional error handling added in the client, as to prevent fatal crashes.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 +++++++++++++----------
 lib/bb/server/xmlrpc.py | 22 +++++++++++++++-------
 lib/bb/ui/uievent.py    |  3 ++-
 3 files changed, 30 insertions(+), 18 deletions(-)
bin/bitbake +0 -0
index fcfe043..ab881c5 100755

bin/bitbake def main(): · line 332

Quoted 1 line
bin/bitbake +0 -0
@@ -332,26 +332,29 @@ def main():
         server =

servermodule.BitBakeXMLRPCClient(configParams.observe_only)

bin/bitbake def main():

Quoted 7 lines
bin/bitbake +5 -0
         server.saveConnectionDetails(configParams.remote_server)
 
+    def _getServerConnection(server, featureset):
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" %

(configParams.remote_server, str(e)))

bin/bitbake def main():

Quoted 6 lines
bin/bitbake +2 -2
+        return server_connection
+
     if not configParams.server_only:
         if configParams.status_only:
-            try:
-                server_connection =

server.establishConnection(featureset)

Quoted 9 lines
bin/bitbake +1 -4
-            except:
-                sys.exit(1)
-            if not server_connection:
-                sys.exit(1)
+            server_connection = _getServerConnection(server, featureset)
             server_connection.terminate()
             sys.exit(0)
 
         # Setup a connection to the server (cooker)

bin/bitbake def main():

Quoted 10 lines
bin/bitbake +4 -3
-        server_connection = server.establishConnection(featureset)
-        if not server_connection:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except:
             if configParams.kill_server:
-                bb.fatal("Server already killed")
+                bb.fatal("Server already offline")
             configParams.bind = configParams.remote_server
             start_server(servermodule, configParams, configuration,

featureset)

Quoted 11 lines
bin/bitbake +1 -1
             bb.event.ui_queue = []
-            server_connection = server.establishConnection(featureset)
+            server_connection = _getServerConnection(server, featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
lib/bb/server/xmlrpc.py +0 -0
index 6fc5543..6ec3452 100644

lib/bb/server/xmlrpc.py class BBTransport(xmlrpclib.Transport): · line 83

Quoted 4 lines
lib/bb/server/xmlrpc.py +0 -1
@@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
 
 def _create_server(host, port, timeout = 60):
     t = BBTransport(timeout)
-    s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t,

allow_none=True)

lib/bb/server/xmlrpc.py class BBTransport(xmlrpclib.Transport): · line 83

Quoted 1 line
lib/bb/server/xmlrpc.py +1 -0
+    s = xmlrpclib.ServerProxy("http://%s:%d/" % (host, port),

transport=t, allow_none=True)

lib/bb/server/xmlrpc.py class XMLRPCServer(SimpleXMLRPCServer,

Quoted 4 lines
lib/bb/server/xmlrpc.py +0 -0
     return s, t
 
 class BitBakeServerCommands():

@@ -253,13 +253,18 @@ class XMLRPCServer(SimpleXMLRPCServer,

BaseImplServer):

Quoted 8 lines
lib/bb/server/xmlrpc.py +2 -3
             socktimeout = self.socket.gettimeout() or nextsleep
             socktimeout = min(socktimeout, nextsleep)
             # Mirror what BaseServer handle_request would do
-            fd_sets = select.select(fds, [], [], socktimeout)
-            if fd_sets[0] and self in fd_sets[0]:
-                self._handle_request_noblock()
+            try:
+                fd_sets = select.select(fds, [], [], socktimeout)

lib/bb/server/xmlrpc.py class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer): · line 267

Quoted 10 lines
lib/bb/server/xmlrpc.py +6 -0
+                if fd_sets[0] and self in fd_sets[0]:
+                    self._handle_request_noblock()
+            except IOError:
+                # we ignore interrupted calls
+                pass
 
         # Tell idle functions we're exiting
         for function, data in self._idlefuns.items():
             try:
+                print "Running idle funcs", function, data

Did you mean to leave this in?

Cheers,

Richard

Quoted 14 lines
lib/bb/server/xmlrpc.py +2 -1
                 retval = function(self, data, True)
             except:
                 pass

@@ -346,7 +351,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             [host, port] = self.remote.split(":")
             port = int(port)
         except Exception as e:
-            bb.fatal("Failed to read remote definition (%s)" % str(e))
+            bb.warn("Failed to read remote definition (%s)" % str(e))
+            raise e
 
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server

lib/bb/server/xmlrpc.py class BitBakeXMLRPCClient(BitBakeBaseServer): · line 359

Quoted 4 lines
lib/bb/server/xmlrpc.py +0 -1
@@ -356,13 +362,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             ip = s.getsockname()[0]
             s.close()
         except Exception as e:
-            bb.fatal("Could not create socket for %s:%s (%s)" % (host,

port, str(e)))

lib/bb/server/xmlrpc.py class BitBakeXMLRPCClient(BitBakeBaseServer): · line 365

Quoted 1 line
lib/bb/server/xmlrpc.py +1 -0
+            bb.warn("Could not create socket for %s:%s (%s)" % (host,

port, str(e)))

lib/bb/server/xmlrpc.py class BitBakeXMLRPCClient(BitBakeBaseServer):

Quoted 4 lines
lib/bb/server/xmlrpc.py +1 -0
+            raise e
         try:
             self.serverImpl = XMLRPCProxyServer(host, port)
             self.connection =

BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)

lib/bb/server/xmlrpc.py class BitBakeXMLRPCClient(BitBakeBaseServer):

Quoted 3 lines
lib/bb/server/xmlrpc.py +0 -1
             return self.connection.connect()
         except Exception as e:
-            bb.fatal("Could not connect to server at %s:%s (%s)" %

(host, port, str(e)))

lib/bb/server/xmlrpc.py class BitBakeXMLRPCClient(BitBakeBaseServer): · line 372

Quoted 1 line
lib/bb/server/xmlrpc.py +1 -0
+            bb.warn("Could not connect to server at %s:%s (%s)" %

(host, port, str(e)))

lib/bb/ui/uievent.py class BBUIEventQueue: · line 47

Quoted 10 lines
lib/bb/server/xmlrpc.py +1 -0
+            raise e
 
     def endSession(self):
         self.connection.removeClient()
lib/bb/ui/uievent.py +0 -0
index 98658f6..eb760c0 100644
@@ -47,7 +47,8 @@ class BBUIEventQueue:
         self.EventHandle =

self.BBServer.registerEventHandler(self.host, self.port)

lib/bb/ui/uievent.py class BBUIEventQueue:

Quoted 4 lines
lib/bb/ui/uievent.py +1 -1
         if (self.EventHandle == None):
-            bb.fatal("Could not register UI event handler")
+            bb.warn("Could not register UI event handler %s:%d" %

(self.host, self.port))

Quoted 7 lines
lib/bb/ui/uievent.py +1 -1
+            raise Exception("Could not register UI event handler")
 
         self.server = server
 
--

1.9.1

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC
#5 lore raw

Re: [PATCH 2/9] xmlrpc: client - remove fatal errors

Quoted 6 lines · Alexandru

Yep, something wrong with it ?

The thing is that I'm replacing bb.fatal with bb.warn to get a chance for a remote controlling to intercept errors instead of having the program shutting down.

But we don't want a print statement for ever idle call?

Cheers,

Richard

Quoted 210 lines

Alex

On Tue, Jun 3, 2014 at 5:38 PM, Richard Purdie <[email protected]> wrote:
On Tue, 2014-06-03 at 16:26 +0100, Alex DAMIAN wrote:
> From: Alexandru DAMIAN <[email protected]> >
> When we use the XMLRPC client API to connect to a bitbake server,
> we want to receive errors from the API instead of having the > API exiting without warning.
>
> Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
> and we re-raise the original exception for handling by the > original caller.
>
> The bitbake starting script has been modified to properly test
> for failures in calling the client API and handle them. >
> Additional error handling added in the client, as to prevent > fatal crashes.
>
> Signed-off-by: Alexandru DAMIAN <[email protected]> > ---
> bin/bitbake | 23 +++++++++++++----------
> lib/bb/server/xmlrpc.py | 22 +++++++++++++++-------
> lib/bb/ui/uievent.py | 3 ++-
> 3 files changed, 30 insertions(+), 18 deletions(-) >
> diff --git a/bin/bitbake b/bin/bitbake
> index fcfe043..ab881c5 100755
> --- a/bin/bitbake
> +++ b/bin/bitbake
> @@ -332,26 +332,29 @@ def main():
> server =
servermodule.BitBakeXMLRPCClient(configParams.observe_only) >
server.saveConnectionDetails(configParams.remote_server)
>
> + def _getServerConnection(server, featureset):
> + try:
> + server_connection =
server.establishConnection(featureset)
> + except Exception as e:
> + bb.fatal("Could not connect to server %s: %s" %
(configParams.remote_server, str(e)))
> + return server_connection
> +
> if not configParams.server_only:
> if configParams.status_only:
> - try:
> - server_connection =
server.establishConnection(featureset)
> - except:
> - sys.exit(1)
> - if not server_connection:
> - sys.exit(1)
> + server_connection =
_getServerConnection(server, featureset)
> server_connection.terminate()
> sys.exit(0)
>
> # Setup a connection to the server (cooker)
> - server_connection =
server.establishConnection(featureset)
> - if not server_connection:
> + try:
> + server_connection =
server.establishConnection(featureset)
> + except:
> if configParams.kill_server:
> - bb.fatal("Server already killed")
> + bb.fatal("Server already offline")
> configParams.bind = configParams.remote_server
> start_server(servermodule, configParams,
configuration, featureset)
> bb.event.ui_queue = []
> - server_connection =
server.establishConnection(featureset)
> + server_connection =
_getServerConnection(server, featureset)
>
> # Restore the environment in case the UI needs it
> for k in cleanedvars:
> diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
> index 6fc5543..6ec3452 100644
> --- a/lib/bb/server/xmlrpc.py
> +++ b/lib/bb/server/xmlrpc.py
> @@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
>
> def _create_server(host, port, timeout = 60):
> t = BBTransport(timeout)
> - s = xmlrpclib.Server("http://%s:%d/" % (host, port),
transport=t, allow_none=True)
> + s = xmlrpclib.ServerProxy("http://%s:%d/" % (host,
port), transport=t, allow_none=True)
> return s, t
>
> class BitBakeServerCommands():
> @@ -253,13 +253,18 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
> socktimeout = self.socket.gettimeout() or
nextsleep
> socktimeout = min(socktimeout, nextsleep)
> # Mirror what BaseServer handle_request would
do
> - fd_sets = select.select(fds, [], [],
socktimeout)
> - if fd_sets[0] and self in fd_sets[0]:
> - self._handle_request_noblock()
> + try:
> + fd_sets = select.select(fds, [], [],
socktimeout)
> + if fd_sets[0] and self in fd_sets[0]:
> + self._handle_request_noblock()
> + except IOError:
> + # we ignore interrupted calls
> + pass
>
> # Tell idle functions we're exiting
> for function, data in self._idlefuns.items():
> try:
> + print "Running idle funcs", function, data

Did you mean to leave this in?

Cheers,

Richard

> retval = function(self, data, True)
> except:
> pass
> @@ -346,7 +351,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
> [host, port] = self.remote.split(":")
> port = int(port)
> except Exception as e:
> - bb.fatal("Failed to read remote definition (%
s)" % str(e))
> + bb.warn("Failed to read remote definition (%s)"
% str(e))
> + raise e
>
> # We need our IP for the server connection. We get
the IP
> # by trying to connect with the server
> @@ -356,13 +362,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
> ip = s.getsockname()[0]
> s.close()
> except Exception as e:
> - bb.fatal("Could not create socket for %s:%s (%
s)" % (host, port, str(e)))
> + bb.warn("Could not create socket for %s:%s (%
s)" % (host, port, str(e)))
> + raise e
> try:
> self.serverImpl = XMLRPCProxyServer(host, port)
> self.connection =
BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
> return self.connection.connect()
> except Exception as e:
> - bb.fatal("Could not connect to server at %s:%s
(%s)" % (host, port, str(e)))
> + bb.warn("Could not connect to server at %s:%s
(%s)" % (host, port, str(e)))
> + raise e
>
> def endSession(self):
> self.connection.removeClient()
> diff --git a/lib/bb/ui/uievent.py b/lib/bb/ui/uievent.py > index 98658f6..eb760c0 100644
> --- a/lib/bb/ui/uievent.py
> +++ b/lib/bb/ui/uievent.py
> @@ -47,7 +47,8 @@ class BBUIEventQueue:
> self.EventHandle =
self.BBServer.registerEventHandler(self.host, self.port) >
> if (self.EventHandle == None):
> - bb.fatal("Could not register UI event handler")
> + bb.warn("Could not register UI event handler %
s:%d" % (self.host, self.port))
> + raise Exception("Could not register UI event
handler")
>
> self.server = server
>
> --
> 1.9.1
>

Signature · 4 lines
Alex Damian
Yocto Project

SSG / OTC
author
#6 lore raw

Re: [PATCH 2/9] xmlrpc: client - remove fatal errors

Oh, now I get it - my email client didn't highlight that you replied to a specific line -

this is mostly for debugging purposes, to be able to manually monitor the bitbake running by

Quoted 1 line

tail -f bitbake-cookerdaemon.log file

Proabably should be in a different patch, though - let me redo this patch.

Thanks,

Alex

Quoted 227 lines · Richard Purdie
Quoted 6 lines · Alexandru

Yep, something wrong with it ?

The thing is that I'm replacing bb.fatal with bb.warn to get a chance for a remote controlling to intercept errors instead of having the program shutting down.

But we don't want a print statement for ever idle call?

Cheers,

Richard

Quoted 210 lines

Alex

On Tue, Jun 3, 2014 at 5:38 PM, Richard Purdie <[email protected]> wrote:
On Tue, 2014-06-03 at 16:26 +0100, Alex DAMIAN wrote:
> From: Alexandru DAMIAN <[email protected]> >
> When we use the XMLRPC client API to connect to a bitbake server,
> we want to receive errors from the API instead of having the > API exiting without warning.
>
> Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
> and we re-raise the original exception for handling by the > original caller.
>
> The bitbake starting script has been modified to properly test
> for failures in calling the client API and handle them. >
> Additional error handling added in the client, as to prevent > fatal crashes.
>
> Signed-off-by: Alexandru DAMIAN <[email protected]> > ---
> bin/bitbake | 23 +++++++++++++----------
> lib/bb/server/xmlrpc.py | 22 +++++++++++++++-------
> lib/bb/ui/uievent.py | 3 ++-
> 3 files changed, 30 insertions(+), 18 deletions(-) >
> diff --git a/bin/bitbake b/bin/bitbake
> index fcfe043..ab881c5 100755
> --- a/bin/bitbake
> +++ b/bin/bitbake
> @@ -332,26 +332,29 @@ def main():
> server =
servermodule.BitBakeXMLRPCClient(configParams.observe_only) >
server.saveConnectionDetails(configParams.remote_server)
>
> + def _getServerConnection(server, featureset):
> + try:
> + server_connection =
server.establishConnection(featureset)
> + except Exception as e:
> + bb.fatal("Could not connect to server %s: %s" %
(configParams.remote_server, str(e)))
> + return server_connection
> +
> if not configParams.server_only:
> if configParams.status_only:
> - try:
> - server_connection =
server.establishConnection(featureset)
> - except:
> - sys.exit(1)
> - if not server_connection:
> - sys.exit(1)
> + server_connection =
_getServerConnection(server, featureset)
> server_connection.terminate()
> sys.exit(0)
>
> # Setup a connection to the server (cooker)
> - server_connection =
server.establishConnection(featureset)
> - if not server_connection:
> + try:
> + server_connection =
server.establishConnection(featureset)
> + except:
> if configParams.kill_server:
> - bb.fatal("Server already killed")
> + bb.fatal("Server already offline")
> configParams.bind = configParams.remote_server
> start_server(servermodule, configParams,
configuration, featureset)
> bb.event.ui_queue = []
> - server_connection =
server.establishConnection(featureset)
> + server_connection =
_getServerConnection(server, featureset)
>
> # Restore the environment in case the UI needs it
> for k in cleanedvars:
> diff --git a/lib/bb/server/xmlrpc.py b/lib/bb/server/xmlrpc.py
> index 6fc5543..6ec3452 100644
> --- a/lib/bb/server/xmlrpc.py
> +++ b/lib/bb/server/xmlrpc.py
> @@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
>
> def _create_server(host, port, timeout = 60):
> t = BBTransport(timeout)
> - s = xmlrpclib.Server("http://%s:%d/" % (host, port),
transport=t, allow_none=True)
> + s = xmlrpclib.ServerProxy("http://%s:%d/" % (host,
port), transport=t, allow_none=True)
> return s, t
>
> class BitBakeServerCommands():
> @@ -253,13 +253,18 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
> socktimeout = self.socket.gettimeout() or
nextsleep
> socktimeout = min(socktimeout, nextsleep)
> # Mirror what BaseServer handle_request would
do
> - fd_sets = select.select(fds, [], [],
socktimeout)
> - if fd_sets[0] and self in fd_sets[0]:
> - self._handle_request_noblock()
> + try:
> + fd_sets = select.select(fds, [], [],
socktimeout)
> + if fd_sets[0] and self in fd_sets[0]:
> + self._handle_request_noblock()
> + except IOError:
> + # we ignore interrupted calls
> + pass
>
> # Tell idle functions we're exiting
> for function, data in self._idlefuns.items():
> try:
> + print "Running idle funcs", function, data

Did you mean to leave this in?

Cheers,

Richard

> retval = function(self, data, True)
> except:
> pass
> @@ -346,7 +351,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
> [host, port] = self.remote.split(":")
> port = int(port)
> except Exception as e:
> - bb.fatal("Failed to read remote definition (%
s)" % str(e))
> + bb.warn("Failed to read remote definition (%s)"
% str(e))
> + raise e
>
> # We need our IP for the server connection. We get
the IP
> # by trying to connect with the server
> @@ -356,13 +362,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
> ip = s.getsockname()[0]
> s.close()
> except Exception as e:
> - bb.fatal("Could not create socket for %s:%s (%
s)" % (host, port, str(e)))
> + bb.warn("Could not create socket for %s:%s (%
s)" % (host, port, str(e)))
> + raise e
> try:
> self.serverImpl = XMLRPCProxyServer(host, port)
> self.connection =
BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
> return self.connection.connect()
> except Exception as e:
> - bb.fatal("Could not connect to server at %s:%s
(%s)" % (host, port, str(e)))
> + bb.warn("Could not connect to server at %s:%s
(%s)" % (host, port, str(e)))
> + raise e
>
> def endSession(self):
> self.connection.removeClient()
> diff --git a/lib/bb/ui/uievent.py b/lib/bb/ui/uievent.py > index 98658f6..eb760c0 100644
> --- a/lib/bb/ui/uievent.py
> +++ b/lib/bb/ui/uievent.py
> @@ -47,7 +47,8 @@ class BBUIEventQueue:
> self.EventHandle =
self.BBServer.registerEventHandler(self.host, self.port) >
> if (self.EventHandle == None):
> - bb.fatal("Could not register UI event handler")
> + bb.warn("Could not register UI event handler %
s:%d" % (self.host, self.port))
> + raise Exception("Could not register UI event
handler")
>
> self.server = server
>
> --
> 1.9.1
>

--
Alex Damian
Yocto Project

SSG / OTC

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC
author
#7 lore raw

PATCH v2 2/9 xmlrpc: client - remove fatal errors

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

When we use the XMLRPC client API to connect to a bitbake server, we want to receive errors from the API instead of having the API exiting without warning.

Thus the "bb.fatal" calls have been replaced with "bb.warn" calls, and we re-raise the original exception for handling by the original caller.

The bitbake starting script has been modified to properly test for failures in calling the client API and handle them.

Additional error handling added in the client, as to prevent fatal crashes.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 +++++++++++++----------
 lib/bb/server/xmlrpc.py | 21 ++++++++++++++-------
 lib/bb/ui/uievent.py    |  3 ++-
 3 files changed, 29 insertions(+), 18 deletions(-)
bin/bitbake +13 -10
index fcfe043..ab881c5 100755
@@ -332,26 +332,29 @@ def main():
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
+    def _getServerConnection(server, featureset):
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
+        return server_connection
+
     if not configParams.server_only:
         if configParams.status_only:
-            try:
-                server_connection = server.establishConnection(featureset)
-            except:
-                sys.exit(1)
-            if not server_connection:
-                sys.exit(1)
+            server_connection = _getServerConnection(server, featureset)
             server_connection.terminate()
             sys.exit(0)
 
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection(featureset)
-        if not server_connection:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except:
             if configParams.kill_server:
-                bb.fatal("Server already killed")
+                bb.fatal("Server already offline")
             configParams.bind = configParams.remote_server
             start_server(servermodule, configParams, configuration, featureset)
             bb.event.ui_queue = []
-            server_connection = server.establishConnection(featureset)
+            server_connection = _getServerConnection(server, featureset)
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:
lib/bb/server/xmlrpc.py +14 -7
index 6fc5543..d6f4338 100644
@@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
 
 def _create_server(host, port, timeout = 60):
     t = BBTransport(timeout)
-    s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t, allow_none=True)
+    s = xmlrpclib.ServerProxy("http://%s:%d/" % (host, port), transport=t, allow_none=True)
     return s, t
 
 class BitBakeServerCommands():

@@ -253,9 +253,13 @@ class XMLRPCServer(SimpleXMLRPCServer, BaseImplServer):
             socktimeout = self.socket.gettimeout() or nextsleep
             socktimeout = min(socktimeout, nextsleep)
             # Mirror what BaseServer handle_request would do
-            fd_sets = select.select(fds, [], [], socktimeout)
-            if fd_sets[0] and self in fd_sets[0]:
-                self._handle_request_noblock()
+            try:
+                fd_sets = select.select(fds, [], [], socktimeout)
+                if fd_sets[0] and self in fd_sets[0]:
+                    self._handle_request_noblock()
+            except IOError:
+                # we ignore interrupted calls
+                pass
 
         # Tell idle functions we're exiting
         for function, data in self._idlefuns.items():

@@ -346,7 +350,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             [host, port] = self.remote.split(":")
             port = int(port)
         except Exception as e:
-            bb.fatal("Failed to read remote definition (%s)" % str(e))
+            bb.warn("Failed to read remote definition (%s)" % str(e))
+            raise e
 
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server

@@ -356,13 +361,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             ip = s.getsockname()[0]
             s.close()
         except Exception as e:
-            bb.fatal("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not create socket for %s:%s (%s)" % (host, port, str(e)))
+            raise e
         try:
             self.serverImpl = XMLRPCProxyServer(host, port)
             self.connection = BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only, featureset)
             return self.connection.connect()
         except Exception as e:
-            bb.fatal("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            bb.warn("Could not connect to server at %s:%s (%s)" % (host, port, str(e)))
+            raise e
 
     def endSession(self):
         self.connection.removeClient()
lib/bb/ui/uievent.py +2 -1
index 98658f6..eb760c0 100644
@@ -47,7 +47,8 @@ class BBUIEventQueue:
         self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
 
         if (self.EventHandle == None):
-            bb.fatal("Could not register UI event handler")
+            bb.warn("Could not register UI event handler %s:%d" % (self.host, self.port))
+            raise Exception("Could not register UI event handler")
 
         self.server = server
Signature · 1 line
1.9.1
author
#8 lore raw

Re: [PATCH 2/9 v2] xmlrpc: client - remove fatal errors

Hello,

I've refactored the patch to take out the print statement that executed on the last run of the idle functions.

Can you please merge ?

Cheers,
Alex

Quoted 157 lines · Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

When we use the XMLRPC client API to connect to a bitbake server, we want to receive errors from the API instead of having the API exiting without warning.

Thus the "bb.fatal" calls have been replaced with "bb.warn" calls, and we re-raise the original exception for handling by the original caller.

The bitbake starting script has been modified to properly test for failures in calling the client API and handle them.

Additional error handling added in the client, as to prevent fatal crashes.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake             | 23 +++++++++++++----------
 lib/bb/server/xmlrpc.py | 21 ++++++++++++++-------
 lib/bb/ui/uievent.py    |  3 ++-
 3 files changed, 29 insertions(+), 18 deletions(-)
bin/bitbake +13 -10
index fcfe043..ab881c5 100755
@@ -332,26 +332,29 @@ def main():
         server =
 servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
+    def _getServerConnection(server, featureset):
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" %
 (configParams.remote_server, str(e)))
+        return server_connection
+
     if not configParams.server_only:
         if configParams.status_only:
-            try:
-                server_connection = server.establishConnection(featureset)
-            except:
-                sys.exit(1)
-            if not server_connection:
-                sys.exit(1)
+            server_connection = _getServerConnection(server, featureset)
             server_connection.terminate()
             sys.exit(0)
 
         # Setup a connection to the server (cooker)
-        server_connection = server.establishConnection(featureset)
-        if not server_connection:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except:
             if configParams.kill_server:
-                bb.fatal("Server already killed")
+                bb.fatal("Server already offline")
             configParams.bind = configParams.remote_server
             start_server(servermodule, configParams, configuration,
 featureset)
             bb.event.ui_queue = []
-            server_connection = server.establishConnection(featureset)
+            server_connection = _getServerConnection(server, featureset)

# Restore the environment in case the UI needs it for k in cleanedvars:

lib/bb/server/xmlrpc.py +1 -1
index 6fc5543..d6f4338 100644
@@ -80,7 +80,7 @@ class BBTransport(xmlrpclib.Transport):
 
 def _create_server(host, port, timeout = 60):
     t = BBTransport(timeout)
-    s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t,
 allow_none=True)
+    s = xmlrpclib.ServerProxy("http://%s:%d/" % (host, port),
 transport=t, allow_none=True)
     return s, t

class BitBakeServerCommands():

lib/bb/server/xmlrpc.py +0 -0
@@ -253,9 +253,13 @@ class XMLRPCServer(SimpleXMLRPCServer,

BaseImplServer):
socktimeout = self.socket.gettimeout() or nextsleep socktimeout = min(socktimeout, nextsleep) # Mirror what BaseServer handle_request would do

lib/bb/server/xmlrpc.py +11 -6
-            fd_sets = select.select(fds, [], [], socktimeout)
-            if fd_sets[0] and self in fd_sets[0]:
-                self._handle_request_noblock()
+            try:
+                fd_sets = select.select(fds, [], [], socktimeout)
+                if fd_sets[0] and self in fd_sets[0]:
+                    self._handle_request_noblock()
+            except IOError:
+                # we ignore interrupted calls
+                pass
 
         # Tell idle functions we're exiting
         for function, data in self._idlefuns.items():

@@ -346,7 +350,8 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             [host, port] = self.remote.split(":")
             port = int(port)
         except Exception as e:
-            bb.fatal("Failed to read remote definition (%s)" % str(e))
+            bb.warn("Failed to read remote definition (%s)" % str(e))
+            raise e
 
         # We need our IP for the server connection. We get the IP
         # by trying to connect with the server

@@ -356,13 +361,15 @@ class BitBakeXMLRPCClient(BitBakeBaseServer):
             ip = s.getsockname()[0]
             s.close()
         except Exception as e:
-            bb.fatal("Could not create socket for %s:%s (%s)" % (host,
 port, str(e)))
+            bb.warn("Could not create socket for %s:%s (%s)" % (host,
 port, str(e)))
+            raise e
         try:
             self.serverImpl = XMLRPCProxyServer(host, port)
             self.connection =
 BitBakeXMLRPCServerConnection(self.serverImpl, (ip, 0), self.observer_only,
 featureset)
             return self.connection.connect()
         except Exception as e:
-            bb.fatal("Could not connect to server at %s:%s (%s)" % (host,

port, str(e)))
+ bb.warn("Could not connect to server at %s:%s (%s)" % (host,
port, str(e)))

lib/bb/server/xmlrpc.py +1 -0
+            raise e
 
     def endSession(self):
         self.connection.removeClient()
lib/bb/ui/uievent.py +2 -1
index 98658f6..eb760c0 100644
@@ -47,7 +47,8 @@ class BBUIEventQueue:
         self.EventHandle = self.BBServer.registerEventHandler(self.host,
 self.port)
 
         if (self.EventHandle == None):
-            bb.fatal("Could not register UI event handler")
+            bb.warn("Could not register UI event handler %s:%d" %
 (self.host, self.port))
+            raise Exception("Could not register UI event handler")

self.server = server

--
1.9.1

Signature · 3 lines
Alex Damian
Yocto Project
SSG / OTC
author
#9 lore raw

PATCH 3/9 uievent: retry on handler registration failure

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

The registration of a remote UI event handler may fail if the server cooker is currently in some certain states. This may happen, for example, when a remote UI is started very fast after the bitbake server is started, and the server hadn't time to finish initial configuration parsing.

Rather than fail outright, we have the remote UI event retry registration for five time at one-second intervals, in the hope it will succeed.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 lib/bb/ui/uievent.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)
lib/bb/ui/uievent.py +19 -3
index eb760c0..c6b100c 100644
@@ -44,10 +44,26 @@ class BBUIEventQueue:
         server.register_function( self.send_event, "event.sendpickle" )
         server.socket.settimeout(1)
 
-        self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
+        self.EventHandler = None
+        count_tries = 0
 
-        if (self.EventHandle == None):
-            bb.warn("Could not register UI event handler %s:%d" % (self.host, self.port))
+        # the event handler registration may fail here due to cooker being in invalid state
+        # this is a transient situation, and we should retry a couple of times before
+        # giving up
+
+        while self.EventHandler == None and count_tries < 5:
+            self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
+
+            if (self.EventHandle != None):
+                break
+
+            bb.warn("Could not register UI event handler %s:%d, retry" % (self.host, self.port))
+            count_tries += 1
+            import time
+            time.sleep(1)
+
+
+        if self.EventHandle == None:
             raise Exception("Could not register UI event handler")
 
         self.server = server
Signature · 1 line
1.9.1
author
#10 lore raw

PATCH 4/9 toaster: read database settings from the environment

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add the capability to read the database settings for Toaster from the environment. The DATABASE_URL is intepreted and used to override the default settings.

This capability is essential for easy deployment of Toaster in a managed hosted environment, and for creating build environments with custom database settings.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 lib/toaster/toastermain/settings.py | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
lib/toaster/toastermain/settings.py +34 -1
index 645f327..adaa56c 100644
@@ -41,6 +41,39 @@ DATABASES = {
     }
 }
 
+# Reinterpret database settings if we have DATABASE_URL environment variable defined
+import os, re
+
+if 'DATABASE_URL' in os.environ:
+    dburl = os.environ['DATABASE_URL']
+    if dburl.startswith('sqlite3://'):
+        result = re.match('sqlite3://(.*)', dburl)
+        if result is None:
+            raise Exception("ERROR: Could not read sqlite database url: %s" % dburl)
+        DATABASES['default'] = {
+            'ENGINE': 'django.db.backends.sqlite3',
+            'NAME': result.group(1),
+            'USER': '',
+            'PASSWORD': '',
+            'HOST': '',
+            'PORT': '',
+        }
+    elif dburl.startswith('mysql://'):
+        # URL must be in this form: mysql://user:pass@host:port/name
+        result = re.match(r"mysql://([^:]*):([^@]*)@([^:]*):(\d+)/([^/]*)", dburl)
+        if result is None:
+            raise Exception("ERROR: Could not read mysql database url: %s" % dburl)
+        DATABASES['default'] = {
+            'ENGINE': 'django.db.backends.mysql',
+            'NAME': result.group(5),
+            'USER': result.group(1),
+            'PASSWORD': result.group(2),
+            'HOST': result.group(3),
+            'PORT': result.group(4),
+        }
+    else:
+        raise Exception("FIXME: Please implement missing database url schema for url: %s" % dburl)
+
 # Hosts/domain names that are valid for this site; required if DEBUG is False
 # See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
 ALLOWED_HOSTS = []

@@ -51,7 +84,7 @@ ALLOWED_HOSTS = []
 # In a Windows environment this must be set to your system time zone.
 
 # Always use local computer's time zone, find
-import os, hashlib
+import hashlib
 if 'TZ' in os.environ:
     TIME_ZONE = os.environ['TZ']
 else:
Signature · 1 line
1.9.1
author
#11 lore raw

PATCH 5/9 toaster: startup script noweb mode

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We add an option to the startup script, named "noweb" that will start toaster without the embedded web server.

This is useful to start the system for build-only environments, where the web server code is running on a different machine.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/toaster | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bin/toaster +5 -1
index dea69a4..f81e667 100755
@@ -146,11 +146,15 @@ else
 fi
 
 NOTOASTERUI=0
+WEBSERVER=1
 for param in $*; do
     case $param in
     noui )
             NOTOASTERUI=1
     ;;
+    noweb )
+            WEBSERVER=0
+    ;;
     esac
 done
 

@@ -176,7 +180,7 @@ case $CMD in
     start )
         start_success=1
         addtoConfiguration "INHERIT+=\"toaster buildhistory\"" toaster.conf
-        if ! webserverStartAll; then
+        if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
             echo "Failed ${CMD}."
             return 4
         fi
Signature · 1 line
1.9.1
author
#12 lore raw

PATCH 6/9 toaster: startup script standalone mode

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

The toaster starter script was designed to be sourced in a build environment and set up the build recording environment to be used in an interactive mode.

For the standalone web server mode, we modify the toaster script to be run directly from the checked-out sources, without a build environment set up, and run the web server alone. In the standalone mode, the build environemnts and all build activities are controled through the web interface.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/toaster | 52 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 46 insertions(+), 6 deletions(-)
bin/toaster +46 -6
index f81e667..80bda6d 100755
@@ -16,9 +16,15 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
 
-# This script enables toaster event logging and
-# starts bitbake resident server
-# use as:  source toaster [start|stop]
+# This script can be run in two modes.
+
+# When used with "source", from a build directory,
+# it enables toaster event logging and starts the bitbake resident server.
+# use as:  source toaster [start|stop] [noweb] [noui]
+
+# When it is called as a stand-alone script, it starts just the
+# web server, and the building shall be done through the web interface.
+# As script, it will not return to the command prompt. Stop with Ctrl-C.
 
 # Helper function to kill a background toaster development server
 

@@ -30,6 +36,8 @@ function webserverKillAll()
 		while kill -0 $(< ${pidfile}) 2>/dev/null; do
 			kill -SIGTERM -$(< ${pidfile}) 2>/dev/null
 			sleep 1;
+			# Kill processes if they are still running - may happen in interactive shells
+			ps fux | grep "python.*manage.py" | awk '{print $2}' | xargs kill
 		done;
 		rm  ${pidfile}
 		fi

@@ -38,6 +46,12 @@ function webserverKillAll()
 
 function webserverStartAll()
 {
+	# do not start if toastermain points to a valid process
+	if ! cat "${BUILDDIR}/.toastermain.pid" 2>/dev/null | xargs -I{} kill -0 {} ; then
+            retval=1
+            rm "${BUILDDIR}/.toastermain.pid"
+        fi
+
         retval=0
         python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
         python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2

@@ -49,6 +63,8 @@ function webserverStartAll()
             retval=0
             python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=1
         fi
+        python $BBBASEDIR/lib/toaster/manage.py migrate bldcontrol || retval=1
+
         if [ $retval -eq 0 ]; then
             python $BBBASEDIR/lib/toaster/manage.py runserver 0.0.0.0:8000 </dev/null >${BUILDDIR}/toaster_web.log 2>&1 & echo $! >${BUILDDIR}/.toastermain.pid
             sleep 1

@@ -103,19 +119,43 @@ function notify_chldexit() {
 }
 
 
-# We make sure we're running in the current shell and in a good environment
+BBBASEDIR=`dirname ${BASH_SOURCE}`/..
+RUNNING=0
 
 if [ -z "$ZSH_NAME" ] && [ `basename \"$0\"` = `basename \"$BASH_SOURCE\"` ]; then
-    echo "Error: This script needs to be sourced. Please run as 'source toaster [start|stop]'" 1>&2;
+    # We are called as standalone. We refuse to run in a build environment - we need the interactive mode for that.
+    # Start just the web server, point the web browser to the interface, and start any Django services.
+
+    if [ -n "$BUILDDIR" ]; then
+        echo "Error: build/ directory detected. Standalone Toaster will not start in a build environment." 1>&2;
+        return 1;
+    fi
+
+    # Define a fake builddir where only the pid files are actually created. No real builds will take place here.
+    BUILDDIR=/tmp
+    RUNNING=1
+    function trap_ctrlc() {
+        echo "** Stopping system"
+        webserverKillAll
+        RUNNING=0
+    }
+    webserverStartAll || exit 1
+    xdg-open http://0.0.0.0:8000/ >/dev/null 2>&1 &
+    trap trap_ctrlc SIGINT
+    echo "Running. Stop with Ctrl-C"
+    while [ $RUNNING -gt 0 ]; do
+        wait;
+    done
+    echo "**** Exit"
     exit 1
 fi
 
+# We make sure we're running in the current shell and in a good environment
 if [ -z "$BUILDDIR" ] || [ -z `which bitbake` ]; then
     echo "Error: Build environment is not setup or bitbake is not in path." 1>&2;
     return 2
 fi
 
-BBBASEDIR=`dirname ${BASH_SOURCE}`/..
 
 
 # Verify prerequisites
Signature · 1 line
1.9.1
author
#13 lore raw

PATCH 7/9 toaster: update toaster to run in managed mode

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We disable bitbake self start to prevent race condition.

ToasterUI will shutdown the server when the build is done if running in managed mode.

We fix usage of kill server flag in the bitbake binary.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 bin/bitbake            | 34 ++++++++++++++++------------------
 lib/bb/ui/toasterui.py | 16 ++++++++++++----
 2 files changed, 28 insertions(+), 22 deletions(-)
bin/bitbake +16 -18
index ab881c5..f120bac 100755
@@ -301,6 +301,9 @@ def main():
     if configParams.observe_only and (not configParams.remote_server or configParams.bind):
         sys.exit("FATAL: '--observe-only' can only be used by UI clients connecting to a server.\n")
 
+    if configParams.kill_server and not configParams.remote_server:
+        sys.exit("FATAL: '--kill-server' can only be used to terminate a remote server")
+
     if "BBDEBUG" in os.environ:
         level = int(os.environ["BBDEBUG"])
         if level > configuration.debug:

@@ -332,29 +335,14 @@ def main():
         server = servermodule.BitBakeXMLRPCClient(configParams.observe_only)
         server.saveConnectionDetails(configParams.remote_server)
 
-    def _getServerConnection(server, featureset):
-        try:
-            server_connection = server.establishConnection(featureset)
-        except Exception as e:
-            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
-        return server_connection
 
     if not configParams.server_only:
-        if configParams.status_only:
-            server_connection = _getServerConnection(server, featureset)
-            server_connection.terminate()
-            sys.exit(0)
-
-        # Setup a connection to the server (cooker)
         try:
             server_connection = server.establishConnection(featureset)
-        except:
+        except Exception as e:
             if configParams.kill_server:
-                bb.fatal("Server already offline")
-            configParams.bind = configParams.remote_server
-            start_server(servermodule, configParams, configuration, featureset)
-            bb.event.ui_queue = []
-            server_connection = _getServerConnection(server, featureset)
+                sys.exit(0)
+            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
 
         # Restore the environment in case the UI needs it
         for k in cleanedvars:

@@ -362,6 +350,16 @@ def main():
 
         logger.removeHandler(handler)
 
+
+        if configParams.status_only:
+            server_connection.terminate()
+            sys.exit(0)
+
+        if configParams.kill_server:
+            server_connection.connection.terminateServer()
+            bb.event.ui_queue = []
+            sys.exit(0)
+
         try:
             return ui_module.main(server_connection.connection, server_connection.events, configParams)
         finally:
lib/bb/ui/toasterui.py +12 -4
index 9a9fe6f..c1696b2 100644
@@ -226,11 +226,19 @@ def main(server, eventHandler, params ):
 
                 buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
 
+
+                brbe = server.runCommand(["getVariable", "TOASTER_BRBE"])[0]
                 # we start a new build info
-                errors = 0
-                warnings = 0
-                taskfailures = []
-                buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+                if brbe is not None:
+                    print "we are under BuildEnvironment management - after the build, we exit"
+                    server.terminateServer()
+                else:
+                    print "prepared for new build"
+                    errors = 0
+                    warnings = 0
+                    taskfailures = []
+                    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+
                 continue
 
             if isinstance(event, bb.event.MetadataEvent):
Signature · 1 line
1.9.1
author
#14 lore raw

PATCH 8/9 toaster: add project related models

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

We introduce the notion of a project in Toaster as the item that holds the specification for triggering a build: the set of layers used, the set of configuration variable values, and the set of targets to be build.

Builds triggered through Toaster will be associated with a Project, and they will be configured based on the project settings at the moment when the build is ordered.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 lib/toaster/orm/models.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
lib/toaster/orm/models.py +19 -0
index 8aa7126..8d4f21b 100644
@@ -23,6 +23,10 @@ from django.db import models
 from django.db.models import F
 from django.utils.encoding import python_2_unicode_compatible
 
+class Project(models.Model):
+    name = models.CharField(max_length=100)
+    created     = models.DateTimeField(auto_now_add = True)
+    updated     = models.DateTimeField(auto_now = True)
 
 class Build(models.Model):
     SUCCEEDED = 0

@@ -37,6 +41,7 @@ class Build(models.Model):
 
     search_allowed_fields = ['machine', 'cooker_log_path', "target__target", "target__target_image_file__file_name"]
 
+    project = models.ForeignKey(Project)
     machine = models.CharField(max_length=100)
     distro = models.CharField(max_length=100)
     distro_version = models.CharField(max_length=100)

@@ -54,6 +59,9 @@ class Build(models.Model):
         tgts = Target.objects.filter(build_id = self.id).order_by( 'target' );
         return( tgts );
 
+class ProjectTarget(models.Model):
+    project = models.ForeignKey(Project)
+    target = models.CharField(max_length=100)
 
 @python_2_unicode_compatible
 class Target(models.Model):

@@ -324,6 +332,12 @@ class Recipe_Dependency(models.Model):
     dep_type = models.IntegerField(choices=DEPENDS_TYPE)
     objects = Recipe_DependencyManager()
 
+class ProjectLayer(models.Model):
+    project = models.ForeignKey(Project)
+    name = models.CharField(max_length = 100)
+    giturl = models.CharField(max_length = 254)
+    commit = models.CharField(max_length = 254)
+
 class Layer(models.Model):
     name = models.CharField(max_length=100)
     local_path = models.FilePathField(max_length=255)

@@ -338,6 +352,11 @@ class Layer_Version(models.Model):
     priority = models.IntegerField()
 
 
+class ProjectVariable(models.Model):
+    project = models.ForeignKey(Project)
+    name = models.CharField(max_length=100)
+    value = models.TextField(blank = True)
+
 class Variable(models.Model):
     search_allowed_fields = ['variable_name', 'variable_value',
                              'vhistory__file_name', "description"]
Signature · 1 line
1.9.1
author
#15 lore raw

PATCH 9/9 toasterui: fix django settings environment value

In reply to Alex DAMIAN

From: Alexandru DAMIAN <[email protected]>

Previously, the buildinfohelper only set a django settings module environment variable if none were set.

This may lead to problems when the starting the toasterui from an already existing Django environment.

As such, we always override the variable to provide the correct name for the local Django settings module.

Signed-off-by Alexandru DAMIAN <[email protected]>
---
 lib/bb/ui/buildinfohelper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/ui/buildinfohelper.py +1 -1
index 6965570..58df687 100644
@@ -22,7 +22,7 @@ import bb
 import re
 import ast
 
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "toaster.toastermain.settings")
+os.environ["DJANGO_SETTINGS_MODULE"] = "toaster.toastermain.settings"
 
 import toaster.toastermain.settings as toaster_django_settings
 from toaster.orm.models import Build, Task, Recipe, Layer_Version, Layer, Target, LogMessage, HelpText
Signature · 1 line
1.9.1