Call for help for prserv selftests
cover Call for help for prserv selftests
From: Michael Opdenacker <[email protected]>
Hi Joshua and BitBake contributors,
I'm trying to implement the first BitBake selftests for the PR server. I tried to replicate what is done by the hashserv selftests, but I still can't seem to connect to the server I started.
I'm attaching my code (especially the 3rd patch) as a patch series.
I won't give up, but if you have clues about what could be wrong, they would be much appreciated!
Here's the output I'm getting:
mike@xps:~/work/yocto/poky/build$ bitbake-selftest prserv.tests.PRCommonTests
E
Stdout:
SERVER STARTED WITH ADDRESS: 127.0.0.1:33399
CLIENT CONNECTED TO: 127.0.0.1:33399
Error talking to server: [Errno 104] Connection reset by peer
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)======================================================================
ERROR: test_create_pr (prserv.tests.PRCommonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mike/work/yocto/poky/bitbake/lib/prserv/tests.py", line 81, in test_create_pr
result = self.client.test_pr("dummy-1.0-r0", "core2-64", "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a5")
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 201, in wrapper
return self.loop.run_until_complete(downcall(*args, **kwargs))
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/mike/work/yocto/poky/bitbake/lib/prserv/client.py", line 25, in test_pr
response = await self.invoke(
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 164, in invoke
result = await self._send_wrapper(proc)
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 151, in _send_wrapper
raise e
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 138, in _send_wrapper
await self.connect()
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 123, in connect
self.socket = await self._connect_sock()
File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 64, in connect_sock
reader, writer = await asyncio.open_connection(address, port)
File "/usr/lib/python3.10/asyncio/streams.py", line 48, in open_connection
transport, _ = await loop.create_connection(
File "/usr/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
raise exceptions[0]
File "/usr/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
sock = await self._connect_sock( File "/usr/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
return await fut
File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 33399)
Stdout:
SERVER STARTED WITH ADDRESS: 127.0.0.1:33399
CLIENT CONNECTED TO: 127.0.0.1:33399
Error talking to server: [Errno 104] Connection reset by peer Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399) Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399) Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)
----------------------------------------------------------------------
Ran 1 test in 0.015s
FAILED (errors=1)
---
Thanks in advance
Michael.
Michael Opdenacker (3):
prserv: add "upstream" server support
prserv: add "history" argument to "get-pr" request
prserv: start bitbake selftests
bin/bitbake-prserv | 15 ++++- bin/bitbake-selftest | 2 + lib/prserv/__init__.py | 39 +++++++++++++ lib/prserv/client.py | 5 +- lib/prserv/db.py | 127 ++++++++++++++++++++--------------------- lib/prserv/serv.py | 98 ++++++++++++++++++++++++++----- lib/prserv/tests.py | 84 +++++++++++++++++++++++++++ 7 files changed, 289 insertions(+), 81 deletions(-) create mode 100644 bitbake/lib/prserv/tests.py
Signature · 1 line
2.34.1
RFC PATCH 1/3 prserv: add "upstream" server support
From: Michael Opdenacker <[email protected]>
Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y").
Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision.
Use the newly developed functions to simplify the implementation of the get_value() function by merging the code of the earlier _get_value_hist() and _get_value_nohist() functions, removing redundancy between both functions. This also makes the code easier to understand.
--- bin/bitbake-prserv | 15 ++++- lib/prserv/__init__.py | 15 +++++ lib/prserv/client.py | 1 + lib/prserv/db.py | 125 ++++++++++++++++++++--------------------- lib/prserv/serv.py | 95 +++++++++++++++++++++++++++---- 5 files changed, 174 insertions(+), 77 deletions(-)
bin/bitbake-prserv +14 -1
@@ -70,12 +70,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERVER_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() prserv.init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else:
lib/prserv/__init__.py +15 -0
@@ -8,6 +8,7 @@ __version__ = "1.0.0" import os, time import sys, logging +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None)
@@ -18,3 +19,17 @@ def init_logger(logfile, loglevel): class NotFoundError(Exception): pass + +async def create_async_client(addr): + from . import client + + c = client.PRAsyncClient() + + try: + (typ, a) = parse_address(addr) + await c.connect_tcp(*a) + return c + + except Exception as e: + await c.close() + raise e
lib/prserv/client.py +1 -0
@@ -6,6 +6,7 @@ import logging import bb.asyncrpc +from . import create_async_client logger = logging.getLogger("BitBake.PRserv")
lib/prserv/db.py +61 -64
158 lines not on this page · raw message
lib/prserv/serv.py +83 -12
161 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH 2/3 prserv: add "history" argument to "get-pr" request
In reply to [email protected]From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
--- lib/prserv/client.py | 4 ++-- lib/prserv/db.py | 16 ++++++++-------- lib/prserv/serv.py | 5 +++-- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/prserv/client.py +2 -2
@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): super().__init__("PRSERVICE", "1.0", logger) - async def getPR(self, version, pkgarch, checksum): + async def getPR(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} ) if response: return response["value"]
lib/prserv/db.py +8 -8
@@ -163,7 +163,7 @@ class PRTable(object): self.dirty = True - def _get_value(self, version, pkgarch, checksum): + def _get_value(self, version, pkgarch, checksum, history): max_value = self.find_max_value(version, pkgarch)
@@ -177,7 +177,11 @@ class PRTable(object): # version, pkgarch found but not checksum. Create a new value from the maximum one return increase_revision(max_value) - if self.nohist: + if history: + # "history" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + else: # "no-history" mode: only return a value if that's the maximum one for # the version and architecture, otherwise create a new one. # This means that the value cannot decrement.
@@ -185,13 +189,9 @@ class PRTable(object): return value else: return increase_revision(max_value) - else: - # "hist" mode: we found an existing value. We can return it - # whether it's the maximum one or not. - return value - def get_value(self, version, pkgarch, checksum): - value = self._get_value(version, pkgarch, checksum) + def get_value(self, version, pkgarch, checksum, history): + value = self._get_value(version, pkgarch, checksum, history) self.store_value(version, pkgarch, checksum, value) return value
lib/prserv/serv.py +3 -2
@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] if self.upstream_client is None: - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # We have an upstream server.
@@ -105,7 +106,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): # The package is not known upstream, must be a local-only package # Let's compute the PR number using the local-only method - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # The package is known upstream, let's ask the upstream server
Signature · 1 line
2.34.1
Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request
Quoted 6 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:
From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.
It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?
Quoted 97 lines
Signed-off-by Michael Opdenacker <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Joshua Watt <[email protected]>Cc Tim Orling <[email protected]>--- lib/prserv/client.py | 4 ++-- lib/prserv/db.py | 16 ++++++++-------- lib/prserv/serv.py | 5 +++-- 3 files changed, 13 insertions(+), 12 deletions(-)lib/prserv/client.py +2
-2@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): super().__init__("PRSERVICE", "1.0", logger) - async def getPR(self, version, pkgarch, checksum): + async def getPR(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} ) if response: return response["value"]lib/prserv/db.py +8
-8@@ -163,7 +163,7 @@ class PRTable(object): self.dirty = True - def _get_value(self, version, pkgarch, checksum): + def _get_value(self, version, pkgarch, checksum, history): max_value = self.find_max_value(version, pkgarch)@@ -177,7 +177,11 @@ class PRTable(object): # version, pkgarch found but not checksum. Create a new value from the maximum one return increase_revision(max_value) - if self.nohist: + if history: + # "history" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + else: # "no-history" mode: only return a value if that's the maximum one for # the version and architecture, otherwise create a new one. # This means that the value cannot decrement.@@ -185,13 +189,9 @@ class PRTable(object): return value else: return increase_revision(max_value) - else: - # "hist" mode: we found an existing value. We can return it - # whether it's the maximum one or not. - return value - def get_value(self, version, pkgarch, checksum): - value = self._get_value(version, pkgarch, checksum) + def get_value(self, version, pkgarch, checksum, history): + value = self._get_value(version, pkgarch, checksum, history) self.store_value(version, pkgarch, checksum, value) return valuelib/prserv/serv.py +3
-3@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] if self.upstream_client is None: - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # We have an upstream server.@@ -105,7 +106,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): # The package is not known upstream, must be a local-only package # Let's compute the PR number using the local-only method - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # The package is known upstream, let's ask the upstream server --2.34.1
Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request
Hi Joshua
Quoted 6 lines · Joshua Watt
Quoted 5 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:
From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
Quoted 8 lines
Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.
It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?
One version of my series actually removed that feature, but then Richard asked me to keep the code, because of the usefulness of the "history" mode for binary reproducibility. This way, we always generate the same package and revision if we have the same output hash as in a prior build. That's different from the default behavior that increases the revision every time we have a new output hash.
This being said, supporting both modes added substantial complexity to the code, and I wish I had more time to consolidate all this and reduce as much redundancy as possible between modes.
Cheers
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request
Quoted 19 lines · Michael Opdenacker
Hi Joshua
Quoted 6 lines · Joshua Watt
Quoted 5 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:
From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
Quoted 8 lines
Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.
It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?
Quoted 6 lines
One version of my series actually removed that feature, but then Richard asked me to keep the code, because of the usefulness of the "history" mode for binary reproducibility. This way, we always generate the same package and revision if we have the same output hash as in a prior build. That's different from the default behavior that increases the revision every time we have a new output hash.
Ah, that's a different story. API used by end users (but not by bitbake during normal execution) is fine, and a different story than API added just for testing :)
Quoted 13 lines
This being said, supporting both modes added substantial complexity to the code, and I wish I had more time to consolidate all this and reduce as much redundancy as possible between modes.
Cheers
Michael.--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
RFC PATCH 3/3 prserv: start bitbake selftests
In reply to [email protected]From: Michael Opdenacker <[email protected]>
--- bin/bitbake-selftest | 2 + lib/prserv/__init__.py | 24 ++++++++++++ lib/prserv/tests.py | 84 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) create mode 100644 bitbake/lib/prserv/tests.py
bin/bitbake-selftest +2 -0
@@ -15,6 +15,7 @@ import unittest try: import bb import hashserv + import prserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc))
@@ -33,6 +34,7 @@ tests = ["bb.tests.codeparser", "bb.tests.utils", "bb.tests.compression", "hashserv.tests", + "prserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"]
lib/prserv/__init__.py +24 -0
@@ -20,6 +20,30 @@ def init_logger(logfile, loglevel): class NotFoundError(Exception): pass +def create_server(addr, dbpath, upstream=None, read_only=False): + from . import serv + + s = serv.PRServer(dbpath, upstream=upstream, read_only=read_only) + + (typ, a) = parse_address(addr) + s.start_tcp_server(*a) + + return s + +def create_client(addr): + from . import client + + c = client.PRClient() + + try: + (typ, a) = parse_address(addr) + c.connect_tcp(*a) + print("CLIENT CONNECTED TO: %s" %(addr)) + return c + except Exception as e: + c.close() + raise e + async def create_async_client(addr): from . import client
lib/prserv/tests.py new file +84 -0
84 lines not on this page · raw message
Signature · 1 line
2.34.1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [bitbake-devel] [RFC][PATCH 3/3] prserv: start bitbake selftests
Greetings,
I eventually found out why I didn't have any logging...
Quoted 15 lines · Michael Opdenacker via lists.openembedded.org
(quoted diff) +15
-0+class PRTestSetup(object): + + server_index = 0 + client_index = 0 + + def start_server(self, dbpath=None, upstream=None, read_only=False, prefunc=server_prefunc): + + self.server_index += 1 + + if dbpath is None: + dbpath = self.make_dbpath() + + def cleanup_server(server): + if server.process.exitcode is not None: + return
Review
Quoted 10 lines
(quoted diff) +10
-0+ + server.process.terminate() + server.process.join() + + server = create_server(socket.gethostbyname("localhost") + ":0", + dbpath, + upstream=upstream, + read_only=read_only) + + server.serve_as_process(prefunc=prefunc, args=(self.server_index))
I was missing a comma at the end of "args" in the above line. It should have been:
server.serve_as_process(prefunc=prefunc, args=(self.server_index,)
Without the comma, the prefunc function was never called and I got no logging.
Now, I'm studying the server logs :)
Thanks Joshua for the tips!
Cheers
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [bitbake-devel] [RFC][PATCH 3/3] prserv: start bitbake selftests
Quoted 6 lines · Michael Opdenacker via lists.openembedded.org
Quoted 1 line
+
server.serve_as_process(prefunc=prefunc, args=(self.server_index))I was missing a comma at the end of "args" in the above line. It should have been:
server.serve_as_process(prefunc=prefunc, args=(self.server_index,)Without the comma, the prefunc function was never called and I got no logging.
And worse, the asyncrpc server process didn't get started, though the rest of the server was (like creating the database). My first test is now successful and I can implement many more :)
Cheers
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com