Skip to content

Re-implement prserv on top of asyncrpc

Paul Barker bitbake-devel 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest4 patches 1 replyIn review
  2. v24 patches 0 repliesSuperseded
  3. v14 patches 4 repliesSuperseded

Changes in v3 from the cover letter

* Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

* Improved asyncio usage based on a trawl through the docs and various other online resources. The asyncio tutorials from Lynn Root on graceful shutdowns [1] and exception handling [2] were excellent and should probably be required reading for anyone using asyncio in anger.

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4asyncrpc: Wait on writers to close with Python 3.7+lib/bb/asyncrpc/client.py lib/bb/asyncrpc/serv.py+7 -00 replies
2/4asyncrpc: Ensure that asyncio shutdown is cleanlib/bb/asyncrpc/serv.py+21 -121 reply
3/4asyncrpc: Handle exceptionslib/bb/asyncrpc/serv.py+6 -00 replies
4/4prserv: Replace XML RPC with modern asyncrpc implementationlib/prserv/serv.py+136 -1270 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [PATCH v3 0/4] Re-implement prserv on top of asyncrpc

These changes replace the old XML-based RPC system in prserv with the new asyncrpc implementation originally used by hashserv.

Changes from v2:
* Dropped patches which are currently applied to master-next, this series should be applied on top of the current master-next branch.

* Improved asyncio usage based on a trawl through the docs and various other online resources. The asyncio tutorials from Lynn Root on graceful shutdowns [1] and exception handling [2] were excellent and should probably be required reading for anyone using asyncio in anger.

As can be seen in the first patch, Python 3.7+ is required for us to guarantee good behaviour from asyncio stream readers & writers. However, these changes have been written to support Python 3.5+ in line with bitbake's supported Python versions and I've tested this on Python 3.6 (OpenSUSE 15.3).

Hopefully this should resolve the hangs we've seen on the autobuilder or at least bring to light any exceptions which may have been occuring in asyncrpc/prserv.

[1]: https://www.roguelynn.com/words/asyncio-graceful-shutdowns/

[2]: https://www.roguelynn.com/words/asyncio-exception-handling/

Changes from v1:
* Drop obsolete getPR function in PRServer, this was replaced by

    handle_get_pr()

* Add timeouts to catch bitbake hangs seen on the autobuilder. This should turn these hangs into errors which can be more reasonably investigated.

* Add a minor error message fix I spotted.

Paul Barker (4):
asyncrpc: Wait on writers to close with Python 3.7+
asyncrpc: Ensure that asyncio shutdown is clean
asyncrpc: Handle exceptions
prserv: Replace XML RPC with modern asyncrpc implementation

 lib/bb/asyncrpc/client.py |   3 +
 lib/bb/asyncrpc/serv.py   |  41 ++++--
 lib/prserv/serv.py        | 263 ++++++++++++++++++++------------------
 3 files changed, 169 insertions(+), 138 deletions(-)
Signature · 1 line
2.26.2