Skip to contentSkip to conversation

Re-implement prserv on top of asyncrpc

PATCH Superseded Scott Murray 3 participants 10 messages Jul 27, 2021 1 message not in the archive

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover Re-implement prserv on top of asyncrpc

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv, and add a read-only mode to match the hash equivalency server's support.

Changes from v3:
* Scott Murray taking over upstreaming effort from Paul Barker.

* Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

* Patches 2-4 updated by Scott Murray to rebase on top of 3983643 ("bitbake: asyncrpc: Catch early SIGTERM").

* Read-only PR server support patch added to stack to get it into the review process.

Paul Barker (5):
asyncrpc: Wait on writers to close with Python 3.7+
asyncrpc: Ensure that asyncio shutdown is clean
asyncrpc: Handle exceptions
prserv: Replace XML RPC with modern asyncrpc implementation
prserv: Add read-only mode

 bin/bitbake-prserv        |   4 +-
 lib/bb/asyncrpc/client.py |   3 +
 lib/bb/asyncrpc/serv.py   |  34 ++++-
 lib/prserv/db.py          |  65 +++++++--
 lib/prserv/serv.py        | 286 ++++++++++++++++++++------------------
 5 files changed, 239 insertions(+), 153 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v4 1/5 asyncrpc: Wait on writers to close with Python 3.7+

From: Paul Barker <[email protected]>

The close() method of an asyncio stream writer is not synchronous and so may return before the writer has finished closing. In Python 3.7 the wait_closed() method was added which can be used to ensure that a writer has finished closing before continuing. Sadly in Python 3.6 and earlier there is no way to guarantee that a writer has finished closing before moving on.

---
 lib/bb/asyncrpc/client.py | 3 +++
 lib/bb/asyncrpc/serv.py   | 4 ++++
 2 files changed, 7 insertions(+)
lib/bb/asyncrpc/client.py +3 -0
index 3eb4fdde..82b6068a 100644
@@ -7,6 +7,7 @@ import asyncio
 import json
 import os
 import socket
+import sys
 from . import chunkify, DEFAULT_MAX_CHUNK
 
 
@@ -47,6 +48,8 @@ class AsyncClient(object):
 
         if self.writer is not None:
             self.writer.close()
+            if sys.version_info >= (3, 7):
+                await self.writer.wait_closed()
             self.writer = None
 
     async def _send_wrapper(self, proc):
lib/bb/asyncrpc/serv.py +4 -0
index 4084f300..2c219c1e 100644
@@ -75,6 +75,8 @@ class AsyncServerConnection(object):
             self.logger.error(str(e))
         finally:
             self.writer.close()
+            if sys.version_info >= (3, 7):
+                await self.writer.wait_closed()
 
     async def dispatch_message(self, msg):
         for k in self.handlers.keys():
@@ -193,6 +195,8 @@ class AsyncServer(object):
             self.logger.error('Error from client: %s' % str(e), exc_info=True)
             traceback.print_exc()
             writer.close()
+            if sys.version_info >= (3, 7):
+                await writer.wait_closed()
         self.logger.debug('Client disconnected')
 
     def run_loop_forever(self):
Signature · 1 line
2.31.1
#2 lore raw

PATCH v4 2/5 asyncrpc: Ensure that asyncio shutdown is clean

In reply to Scott Murray

From: Paul Barker <[email protected]>

We should ensure that all async tasks are cancelled and then allowed to finish gracefully before closing the asyncio loop.

[updated for asyncrpc SIGTERM handling changes]
---
 lib/bb/asyncrpc/serv.py | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
lib/bb/asyncrpc/serv.py +18 -6
index 2c219c1e..1b7b7a6d 100644
@@ -205,21 +205,33 @@ class AsyncServer(object):
         except KeyboardInterrupt:
             pass
 
-    def signal_handler(self):
+    def handle_signal(self):
         self.logger.debug("Got exit signal")
+        self.loop.create_task(self.shutdown())
+
+    async def shutdown(self):
+        self.logger.debug('Server shutting down')
+
+        # Stop accepting connections
+        self.server.close()
+        await self.server.wait_closed()
+
+        # Cancel all active tasks
+        tasks = [t for t in asyncio.Task.all_tasks(self.loop)
+                 if t is not asyncio.Task.current_task(self.loop)]
+        for task in tasks:
+            task.cancel()
+        await asyncio.gather(*tasks, return_exceptions=True)
         self.loop.stop()
 
     def serve_forever(self):
         asyncio.set_event_loop(self.loop)
         try:
-            self.loop.add_signal_handler(signal.SIGTERM, self.signal_handler)
+            self.loop.add_signal_handler(signal.SIGTERM, self.handle_signal)
             signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGTERM])
+            self.loop.add_signal_handler(signal.SIGINT, self.handle_signal)
 
             self.run_loop_forever()
-            self.server.close()
-
-            self.loop.run_until_complete(self.server.wait_closed())
-            self.logger.debug('Server shutting down')
         finally:
             if self.close_loop:
                 if sys.version_info >= (3, 6):
Signature · 1 line
2.31.1
#3 lore raw

PATCH v4 3/5 asyncrpc: Handle exceptions

In reply to Scott Murray

From: Paul Barker <[email protected]>

If an async task raises an exception it may cause the asyncio loop to hang or may otherwise leave our server in a bad state. To avoid these issues we can add an exception handler to the asyncio loop which attempts to gracefully shut down the server if an unhandled exception occurs.

[updated for asyncrpc SIGTERM handling changes]
---
 lib/bb/asyncrpc/serv.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/asyncrpc/serv.py +6 -0
index 1b7b7a6d..167b30c7 100644
@@ -209,6 +209,11 @@ class AsyncServer(object):
         self.logger.debug("Got exit signal")
         self.loop.create_task(self.shutdown())
 
+    def handle_exception(self, loop, context):
+        msg = context.get("exception", context["message"])
+        self.logger.error("Caught exception: %s" % (msg))
+        loop.create_task(self.shutdown())
+
     async def shutdown(self):
         self.logger.debug('Server shutting down')
 
@@ -227,6 +232,7 @@ class AsyncServer(object):
     def serve_forever(self):
         asyncio.set_event_loop(self.loop)
         try:
+            self.loop.set_exception_handler(self.handle_exception)
             self.loop.add_signal_handler(signal.SIGTERM, self.handle_signal)
             signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGTERM])
             self.loop.add_signal_handler(signal.SIGINT, self.handle_signal)
Signature · 1 line
2.31.1
#4 lore raw

PATCH v4 4/5 prserv: Replace XML RPC with modern asyncrpc implementation

In reply to Scott Murray

From: Paul Barker <[email protected]>

Update the prserv client and server classes to use the modern json and asyncio based RPC system implemented by the asyncrpc module.

[updated for asyncrpc SIGTERM handling changes]
---
 lib/prserv/serv.py | 265 +++++++++++++++++++++++----------------------
 1 file changed, 136 insertions(+), 129 deletions(-)
lib/prserv/serv.py +136 -129
index 5e322bf8..2c572e0f 100644
@@ -4,157 +4,158 @@
 
 import os,sys,logging
 import signal, time
-from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 import socket
 import io
 import sqlite3
-import bb.server.xmlrpcclient
 import prserv
 import prserv.db
 import errno
-import multiprocessing
+import bb.asyncrpc
 
 logger = logging.getLogger("BitBake.PRserv")
 
-class Handler(SimpleXMLRPCRequestHandler):
-    def _dispatch(self,method,params):
-        try:
-            value=self.server.funcs[method](*params)
-        except:
-            import traceback
-            traceback.print_exc()
-            raise
-        return value
-
 PIDPREFIX = "/tmp/PRServer_%s_%s.pid"
 singleton = None
 
+class PRServerClient(bb.asyncrpc.AsyncServerConnection):
+    def __init__(self, reader, writer, table):
+        super().__init__(reader, writer, 'PRSERVICE', logger)
+        self.handlers.update({
+            'get-pr': self.handle_get_pr,
+            'import-one': self.handle_import_one,
+            'export': self.handle_export,
+        })
+        self.table = table
 
-class PRServer(SimpleXMLRPCServer):
-    def __init__(self, dbfile, logfile, interface):
-        ''' constructor '''
-        try:
-            SimpleXMLRPCServer.__init__(self, interface,
-                                        logRequests=False, allow_none=True)
-        except socket.error:
-            ip=socket.gethostbyname(interface[0])
-            port=interface[1]
-            msg="PR Server unable to bind to %s:%s\n" % (ip, port)
-            sys.stderr.write(msg)
-            raise PRServiceConfigError
-
-        self.dbfile=dbfile
-        self.logfile=logfile
-        self.host, self.port = self.socket.getsockname()
-
-        self.register_function(self.getPR, "getPR")
-        self.register_function(self.ping, "ping")
-        self.register_function(self.export, "export")
-        self.register_function(self.importone, "importone")
-        self.register_introspection_functions()
-
-        self.iter_count = 0
-        # 60 iterations between syncs or sync if dirty every ~30 seconds
-        self.iterations_between_sync = 60
-
-    def sigint_handler(self, signum, stack):
-        if self.table:
-            self.table.sync()
-
-    def sigterm_handler(self, signum, stack):
-        if self.table:
-            self.table.sync()
-        raise(SystemExit)
+    def validate_proto_version(self):
+        return (self.proto_version == (1, 0))
 
-    def process_request(self, request, client_address):
-        if request is None:
-            return
+    async def dispatch_message(self, msg):
         try:
-            self.finish_request(request, client_address)
-            self.shutdown_request(request)
-            self.iter_count = (self.iter_count + 1) % self.iterations_between_sync
-            if self.iter_count == 0:
-                self.table.sync_if_dirty()
+            await super().dispatch_message(msg)
         except:
-            self.handle_error(request, client_address)
-            self.shutdown_request(request)
             self.table.sync()
-        self.table.sync_if_dirty()
+            raise
 
-    def serve_forever(self, poll_interval=0.5):
-        signal.signal(signal.SIGINT, self.sigint_handler)
-        signal.signal(signal.SIGTERM, self.sigterm_handler)
+        self.table.sync_if_dirty()
 
-        self.db = prserv.db.PRData(self.dbfile)
-        self.table = self.db["PRMAIN"]
-        return super().serve_forever(poll_interval)
+    async def handle_get_pr(self, request):
+        version = request['version']
+        pkgarch = request['pkgarch']
+        checksum = request['checksum']
 
-    def export(self, version=None, pkgarch=None, checksum=None, colinfo=True):
+        response = None
         try:
-            return self.table.export(version, pkgarch, checksum, colinfo)
+            value = self.table.getValue(version, pkgarch, checksum)
+            response = {'value': value}
+        except prserv.NotFoundError:
+            logger.error("can not find value for (%s, %s)",version, checksum)
         except sqlite3.Error as exc:
             logger.error(str(exc))
-            return None
 
-    def importone(self, version, pkgarch, checksum, value):
-        return self.table.importone(version, pkgarch, checksum, value)
+        self.write_message(response)
 
-    def ping(self):
-        return True
+    async def handle_import_one(self, request):
+        version = request['version']
+        pkgarch = request['pkgarch']
+        checksum = request['checksum']
+        value = request['value']
+
+        value = self.table.importone(version, pkgarch, checksum, value)
+        if value is not None:
+            response = {'value': value}
+        else:
+            response = None
+        self.write_message(response)
 
-    def getinfo(self):
-        return (self.host, self.port)
+    async def handle_export(self, request):
+        version = request['version']
+        pkgarch = request['pkgarch']
+        checksum = request['checksum']
+        colinfo = request['colinfo']
 
-    def getPR(self, version, pkgarch, checksum):
         try:
-            return self.table.getValue(version, pkgarch, checksum)
-        except prserv.NotFoundError:
-            logger.error("can not find value for (%s, %s)",version, checksum)
-            return None
+            (metainfo, datainfo) = self.table.export(version, pkgarch, checksum, colinfo)
         except sqlite3.Error as exc:
             logger.error(str(exc))
-            return None
+            metainfo = datainfo = None
 
-class PRServSingleton(object):
-    def __init__(self, dbfile, logfile, interface):
-        self.dbfile = dbfile
-        self.logfile = logfile
-        self.interface = interface
-        self.host = None
-        self.port = None
+        response = {'metainfo': metainfo, 'datainfo': datainfo}
+        self.write_message(response)
 
-    def start(self):
-        self.prserv = PRServer(self.dbfile, self.logfile, self.interface)
-        self.process = multiprocessing.Process(target=self.prserv.serve_forever)
-        self.process.start()
+class PRServer(bb.asyncrpc.AsyncServer):
+    def __init__(self, dbfile, loop=None):
+        super().__init__(logger, loop)
+        self.dbfile = dbfile
+        self.table = None
 
-        self.host, self.port = self.prserv.getinfo()
+    def accept_client(self, reader, writer):
+        return PRServerClient(reader, writer, self.table)
 
-    def getinfo(self):
-        return (self.host, self.port)
+    def serve_forever(self):
+        self.db = prserv.db.PRData(self.dbfile)
+        self.table = self.db["PRMAIN"]
 
-class PRServerConnection(object):
-    def __init__(self, host, port):
-        if is_local_special(host, port):
-            host, port = singleton.getinfo()
-        self.host = host
-        self.port = port
-        self.connection, self.transport = bb.server.xmlrpcclient._create_server(self.host, self.port)
+        logger.debug("Started PRServer with DBfile: %s, Address: %s, PID: %s" %
+                     (self.dbfile, self.address, str(os.getpid())))
 
-    def getPR(self, version, pkgarch, checksum):
-        return self.connection.getPR(version, pkgarch, checksum)
+        super().serve_forever()
 
-    def ping(self):
-        return self.connection.ping()
+        self.table.sync_if_dirty()
+        self.db.disconnect()
 
-    def export(self,version=None, pkgarch=None, checksum=None, colinfo=True):
-        return self.connection.export(version, pkgarch, checksum, colinfo)
+    def signal_handler(self):
+        super().signal_handler()
+        if self.table:
+            self.table.sync()
 
-    def importone(self, version, pkgarch, checksum, value):
-        return self.connection.importone(version, pkgarch, checksum, value)
+class PRServSingleton(object):
+    def __init__(self, dbfile, logfile, host, port):
+        self.dbfile = dbfile
+        self.logfile = logfile
+        self.host = host
+        self.port = port
 
-    def getinfo(self):
-        return self.host, self.port
+    def start(self):
+        self.prserv = PRServer(self.dbfile)
+        self.prserv.start_tcp_server(self.host, self.port)
+        self.process = self.prserv.serve_as_process()
+
+        if not self.port:
+            self.port = int(self.prserv.address.rsplit(':', 1)[1])
+
+class PRAsyncClient(bb.asyncrpc.AsyncClient):
+    def __init__(self):
+        super().__init__('PRSERVICE', '1.0', logger)
+
+    async def getPR(self, version, pkgarch, checksum):
+        response = await self.send_message(
+            {'get-pr': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum}}
+        )
+        if response:
+            return response['value']
+
+    async def importone(self, version, pkgarch, checksum, value):
+        response = await self.send_message(
+            {'import-one': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum, 'value': value}}
+        )
+        if response:
+            return response['value']
+
+    async def export(self, version, pkgarch, checksum, colinfo):
+        response = await self.send_message(
+            {'export': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum, 'colinfo': colinfo}}
+        )
+        if response:
+            return (response['metainfo'], response['datainfo'])
+
+class PRClient(bb.asyncrpc.Client):
+    def __init__(self):
+        super().__init__()
+        self._add_methods('getPR', 'importone', 'export')
+
+    def _get_async_client(self):
+        return PRAsyncClient()
 
 def run_as_daemon(func, pidfile, logfile):
     """
@@ -240,15 +241,13 @@ def start_daemon(dbfile, host, port, logfile):
                             % pidfile)
         return 1
 
-    server = PRServer(os.path.abspath(dbfile), os.path.abspath(logfile), (ip,port))
-    run_as_daemon(server.serve_forever, pidfile, os.path.abspath(logfile))
+    dbfile = os.path.abspath(dbfile)
+    def daemon_main():
+        server = PRServer(dbfile)
+        server.start_tcp_server(host, port)
+        server.serve_forever()
 
-    # Sometimes, the port (i.e. localhost:0) indicated by the user does not match with
-    # the one the server actually is listening, so at least warn the user about it
-    _,rport = server.getinfo()
-    if port != rport:
-        sys.stdout.write("Server is listening at port %s instead of %s\n"
-                         % (rport,port))
+    run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile))
     return 0
 
 def stop_daemon(host, port):
@@ -302,7 +301,7 @@ def is_running(pid):
     return True
 
 def is_local_special(host, port):
-    if host.strip().upper() == 'localhost'.upper() and (not port):
+    if host.strip().lower() == 'localhost' and not port:
         return True
     else:
         return False
@@ -340,20 +339,19 @@ def auto_start(d):
                auto_shutdown()
         if not singleton:
             bb.utils.mkdirhier(cachedir)
-            singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), ("localhost",0))
+            singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), "localhost", 0)
             singleton.start()
     if singleton:
-        host, port = singleton.getinfo()
+        host = singleton.host
+        port = singleton.port
     else:
         host = host_params[0]
         port = int(host_params[1])
 
     try:
-        connection = PRServerConnection(host,port)
-        connection.ping()
-        realhost, realport = connection.getinfo()
-        return str(realhost) + ":" + str(realport)
-        
+        ping(host, port)
+        return str(host) + ":" + str(port)
+
     except Exception:
         logger.critical("PRservice %s:%d not available" % (host, port))
         raise PRServiceConfigError
@@ -366,8 +364,17 @@ def auto_shutdown():
         singleton = None
 
 def ping(host, port):
-    conn=PRServerConnection(host, port)
+    conn=PRClient()
+    conn.connect_tcp(host, port)
     return conn.ping()
 
 def connect(host, port):
-    return PRServerConnection(host, port)
+    global singleton
+
+    if host.strip().lower() == 'localhost' and not port:
+        host = 'localhost'
+        port = singleton.port
+
+    conn = PRClient()
+    conn.connect_tcp(host, port)
+    return conn
Signature · 1 line
2.31.1
#5 lore raw

PATCH v4 5/5 prserv: Add read-only mode

In reply to Scott Murray

From: Paul Barker <[email protected]>

---
 bin/bitbake-prserv |  4 ++-
 lib/prserv/db.py   | 65 ++++++++++++++++++++++++++++++++++++----------
 lib/prserv/serv.py | 49 ++++++++++++++++++++++------------
 3 files changed, 86 insertions(+), 32 deletions(-)
bin/bitbake-prserv +3 -1
index 1e9b6cbc..bef5ef68 100755
@@ -36,12 +36,14 @@ def main():
                       dest="host", type="string", default=PRHOST_DEFAULT)
     parser.add_option("--port", help="port number(default: 8585)", action="store",
                       dest="port", type="int", default=PRPORT_DEFAULT)
+    parser.add_option("-r", "--read-only", help="open database in read-only mode",
+                      action="store_true")
 
     options, args = parser.parse_args(sys.argv)
     prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
 
     if options.start:
-        ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile))
+        ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only)
     elif options.stop:
         ret=prserv.serv.stop_daemon(options.host, options.port)
     else:
lib/prserv/db.py +51 -14

130 lines not on this page · raw message

lib/prserv/serv.py +32 -17

106 lines not on this page · raw message

Signature · 1 line
2.31.1
#6 lore raw
In reply to Scott Murray
Quoted 16 lines · Scott Murray

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv, and add a read-only mode to match the hash equivalency server's support.

Changes from v3:
  * Scott Murray taking over upstreaming effort from Paul Barker.

  * Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

  * Patches 2-4 updated by Scott Murray to rebase on top of 3983643 ("bitbake: asyncrpc: Catch early SIGTERM").

  * Read-only PR server support patch added to stack to get it into the review process.

Quoted 6 lines

Paul Barker (5):
  asyncrpc: Wait on writers to close with Python 3.7+
  asyncrpc: Ensure that asyncio shutdown is clean
  asyncrpc: Handle exceptions
  prserv: Replace XML RPC with modern asyncrpc implementation
  prserv: Add read-only mode

There is an odd failure on the autobuilder which seems related to this.

https://autobuilder.yoctoproject.org/typhoon/#/builders/75/builds/3746

and looking at the bitbake-cookerdaemon.log:

1958 12:03:42.046562 Running command ['clientComplete']
1958 12:03:42.046659 Command Completed
1958 12:03:42.047003 Processing Client
1958 12:03:42.047061 Disconnecting Client
1958 12:03:42.047215 No timeout, exiting.
1958 12:03:42.147541 Exiting
Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks'

with this last line repeating a 'lot' (log is over 1GB in size).

Cheers,

Richard

#8 lore raw
In reply to a message not in the archive
Quoted 41 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · Scott Murray

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv, and add a read-only mode to match the hash equivalency server's support.

Changes from v3:
  * Scott Murray taking over upstreaming effort from Paul Barker.

  * Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

  * Patches 2-4 updated by Scott Murray to rebase on top of 3983643 ("bitbake: asyncrpc: Catch early SIGTERM").

  * Read-only PR server support patch added to stack to get it into the review process.

Quoted 6 lines

Paul Barker (5):
  asyncrpc: Wait on writers to close with Python 3.7+
  asyncrpc: Ensure that asyncio shutdown is clean
  asyncrpc: Handle exceptions
  prserv: Replace XML RPC with modern asyncrpc implementation
  prserv: Add read-only mode

There is an odd failure on the autobuilder which seems related to this.

https://autobuilder.yoctoproject.org/typhoon/#/builders/75/builds/3746

and looking at the bitbake-cookerdaemon.log:

1958 12:03:42.046562 Running command ['clientComplete']
1958 12:03:42.046659 Command Completed
1958 12:03:42.047003 Processing Client
1958 12:03:42.047061 Disconnecting Client
1958 12:03:42.047215 No timeout, exiting.
1958 12:03:42.147541 Exiting
Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks'

Quoted 1 line

with this last line repeating a 'lot' (log is over 1GB in size).

I should add, this is after the bitbake-layers command used to add the layer during the setup of that build. The later command just times out as the server process remains and is locked up.

Cheers,

Richard

#9 lore raw
Quoted 42 lines · On Tue, 27 Jul 2021 13:28:30 +0100 "Richard Purdie"…
Quoted 41 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · Scott Murray

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv, and add a read-only mode to match the hash equivalency server's support.

Changes from v3:
  * Scott Murray taking over upstreaming effort from Paul Barker.

  * Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

  * Patches 2-4 updated by Scott Murray to rebase on top of 3983643 ("bitbake: asyncrpc: Catch early SIGTERM").

  * Read-only PR server support patch added to stack to get it into the review process.

Quoted 6 lines

Paul Barker (5):
  asyncrpc: Wait on writers to close with Python 3.7+
  asyncrpc: Ensure that asyncio shutdown is clean
  asyncrpc: Handle exceptions
  prserv: Replace XML RPC with modern asyncrpc implementation
  prserv: Add read-only mode

There is an odd failure on the autobuilder which seems related to this.

https://autobuilder.yoctoproject.org/typhoon/#/builders/75/builds/3746

and looking at the bitbake-cookerdaemon.log:

1958 12:03:42.046562 Running command ['clientComplete']
1958 12:03:42.046659 Command Completed
1958 12:03:42.047003 Processing Client
1958 12:03:42.047061 Disconnecting Client
1958 12:03:42.047215 No timeout, exiting.
1958 12:03:42.147541 Exiting
Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks'

Quoted 5 lines
Quoted 1 line

with this last line repeating a 'lot' (log is over 1GB in size).

I should add, this is after the bitbake-layers command used to add the layer during the setup of that build. The later command just times out as the server process remains and is locked up.

What's the Python version there? I've just re-checked the docs for asyncio.Task.all_tasks():

This method is deprecated and will be removed in Python 3.9. Use the asyncio.all_tasks() function instead.

asyncio.all_tasks() was added in Python 3.7. So we need to use asyncio.Task.all_tasks() before 3.7, asyncio.all_tasks() after 3.7.

Signature · 2 lines
Paul Barker
https://pbarker.dev/
#10 lore raw
Quoted 51 lines · Paul Barker
Quoted 42 lines · On Tue, 27 Jul 2021 13:28:30 +0100 "Richard Purdie"…
Quoted 41 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · Scott Murray

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv, and add a read-only mode to match the hash equivalency server's support.

Changes from v3:
  * Scott Murray taking over upstreaming effort from Paul Barker.

  * Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

  * Patches 2-4 updated by Scott Murray to rebase on top of 3983643 ("bitbake: asyncrpc: Catch early SIGTERM").

  * Read-only PR server support patch added to stack to get it into the review process.

Quoted 6 lines

Paul Barker (5):
  asyncrpc: Wait on writers to close with Python 3.7+
  asyncrpc: Ensure that asyncio shutdown is clean
  asyncrpc: Handle exceptions
  prserv: Replace XML RPC with modern asyncrpc implementation
  prserv: Add read-only mode

There is an odd failure on the autobuilder which seems related to this.

https://autobuilder.yoctoproject.org/typhoon/#/builders/75/builds/3746

and looking at the bitbake-cookerdaemon.log:

1958 12:03:42.046562 Running command ['clientComplete']
1958 12:03:42.046659 Command Completed
1958 12:03:42.047003 Processing Client
1958 12:03:42.047061 Disconnecting Client
1958 12:03:42.047215 No timeout, exiting.
1958 12:03:42.147541 Exiting
Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks' Caught exception: type object '_asyncio.Task' has no attribute 'all_tasks'

Quoted 5 lines
Quoted 1 line

with this last line repeating a 'lot' (log is over 1GB in size).

I should add, this is after the bitbake-layers command used to add the layer during the setup of that build. The later command just times out as the server process remains and is locked up.

Quoted 8 lines

What's the Python version there? I've just re-checked the docs for asyncio.Task.all_tasks():

    This method is deprecated and will be removed in Python 3.9. Use the asyncio.all_tasks() function instead.

asyncio.all_tasks() was added in Python 3.7. So we need to use asyncio.Task.all_tasks() before 3.7, asyncio.all_tasks() after 3.7.

That worker used buildtools tarball which would have 3.9 in it.

Cheers,

Richard