Skip to contentSkip to conversation

Call for help for prserv selftests

RFC PATCH In review [email protected] 2 participants 9 messages Apr 16, 2024 – Apr 23, 2024 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

cover Call for help for prserv selftests

From: Michael Opdenacker <[email protected]>

Hi Joshua and BitBake contributors,

I'm trying to implement the first BitBake selftests for the PR server. I tried to replicate what is done by the hashserv selftests, but I still can't seem to connect to the server I started.

I'm attaching my code (especially the 3rd patch) as a patch series.

I won't give up, but if you have clues about what could be wrong, they would be much appreciated!

Here's the output I'm getting:

mike@xps:~/work/yocto/poky/build$ bitbake-selftest prserv.tests.PRCommonTests
E
Stdout:
SERVER STARTED WITH ADDRESS: 127.0.0.1:33399
CLIENT CONNECTED TO: 127.0.0.1:33399
Error talking to server: [Errno 104] Connection reset by peer
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)
Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)

======================================================================
ERROR: test_create_pr (prserv.tests.PRCommonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mike/work/yocto/poky/bitbake/lib/prserv/tests.py", line 81, in test_create_pr

    result = self.client.test_pr("dummy-1.0-r0", "core2-64", "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a5")
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 201, in wrapper
    return self.loop.run_until_complete(downcall(*args, **kwargs))
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/mike/work/yocto/poky/bitbake/lib/prserv/client.py", line 25, in test_pr
    response = await self.invoke(
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 164, in invoke
    result = await self._send_wrapper(proc)
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 151, in _send_wrapper
    raise e
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 138, in _send_wrapper
    await self.connect()
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 123, in connect
    self.socket = await self._connect_sock()
  File "/home/mike/work/yocto/poky/bitbake/lib/bb/asyncrpc/client.py", line 64, in connect_sock
    reader, writer = await asyncio.open_connection(address, port)
  File "/usr/lib/python3.10/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1076, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1060, in create_connection
    sock = await self._connect_sock(

File "/usr/lib/python3.10/asyncio/base_events.py", line 969, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
return await fut
File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 33399)

Stdout:
SERVER STARTED WITH ADDRESS: 127.0.0.1:33399
CLIENT CONNECTED TO: 127.0.0.1:33399
Error talking to server: [Errno 104] Connection reset by peer Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399) Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399) Error talking to server: [Errno 111] Connect call failed ('127.0.0.1', 33399)

----------------------------------------------------------------------
Ran 1 test in 0.015s

FAILED (errors=1)

---

Thanks in advance
Michael.

Michael Opdenacker (3):
prserv: add "upstream" server support
prserv: add "history" argument to "get-pr" request
prserv: start bitbake selftests

 bin/bitbake-prserv     |  15 ++++-
 bin/bitbake-selftest   |   2 +
 lib/prserv/__init__.py |  39 +++++++++++++
 lib/prserv/client.py   |   5 +-
 lib/prserv/db.py       | 127 ++++++++++++++++++++---------------------
 lib/prserv/serv.py     |  98 ++++++++++++++++++++++++++-----
 lib/prserv/tests.py    |  84 +++++++++++++++++++++++++++
 7 files changed, 289 insertions(+), 81 deletions(-)
 create mode 100644 bitbake/lib/prserv/tests.py
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/3 prserv: add "upstream" server support

From: Michael Opdenacker <[email protected]>

Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one.

This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y").

Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format."

Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision.

Use the newly developed functions to simplify the implementation of the get_value() function by merging the code of the earlier _get_value_hist() and _get_value_nohist() functions, removing redundancy between both functions. This also makes the code easier to understand.

Cc Joshua Watt <[email protected]>
Cc Tim Orling <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 bin/bitbake-prserv     |  15 ++++-
 lib/prserv/__init__.py |  15 +++++
 lib/prserv/client.py   |   1 +
 lib/prserv/db.py       | 125 ++++++++++++++++++++---------------------
 lib/prserv/serv.py     |  95 +++++++++++++++++++++++++++----
 5 files changed, 174 insertions(+), 77 deletions(-)
bin/bitbake-prserv +14 -1
index ad0a069401..e39d0fba87 100755
@@ -70,12 +70,25 @@ def main():
         action="store_true",
         help="open database in read-only mode",
     )
+    parser.add_argument(
+        "-u",
+        "--upstream",
+        default=os.environ.get("PRSERVER_UPSTREAM", None),
+        help="Upstream PR service (host:port)",
+    )
 
     args = parser.parse_args()
     prserv.init_logger(os.path.abspath(args.log), args.loglevel)
 
     if args.start:
-        ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only)
+        ret=prserv.serv.start_daemon(
+            args.file,
+            args.host,
+            args.port,
+            os.path.abspath(args.log),
+            args.read_only,
+            args.upstream
+        )
     elif args.stop:
         ret=prserv.serv.stop_daemon(args.host, args.port)
     else:
lib/prserv/__init__.py +15 -0
index 0e0aa34d0e..2ee6a28c04 100644
@@ -8,6 +8,7 @@ __version__ = "1.0.0"
 
 import os, time
 import sys, logging
+from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
 
 def init_logger(logfile, loglevel):
     numeric_level = getattr(logging, loglevel.upper(), None)
@@ -18,3 +19,17 @@ def init_logger(logfile, loglevel):
 
 class NotFoundError(Exception):
     pass
+
+async def create_async_client(addr):
+    from . import client
+
+    c = client.PRAsyncClient()
+
+    try:
+        (typ, a) = parse_address(addr)
+        await c.connect_tcp(*a)
+        return c
+
+    except Exception as e:
+        await c.close()
+        raise e
lib/prserv/client.py +1 -0
index 8471ee3046..89760b6f74 100644
@@ -6,6 +6,7 @@
 
 import logging
 import bb.asyncrpc
+from . import create_async_client
 
 logger = logging.getLogger("BitBake.PRserv")
 
lib/prserv/db.py +61 -64

158 lines not on this page · raw message

lib/prserv/serv.py +83 -12

161 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/3 prserv: add "history" argument to "get-pr" request

In reply to [email protected]

From: Michael Opdenacker <[email protected]>

This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.

Cc Thomas Petazzoni <[email protected]>
Cc Joshua Watt <[email protected]>
Cc Tim Orling <[email protected]>
---
 lib/prserv/client.py |  4 ++--
 lib/prserv/db.py     | 16 ++++++++--------
 lib/prserv/serv.py   |  5 +++--
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/prserv/client.py +2 -2
index 89760b6f74..389172f055 100644
@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
     def __init__(self):
         super().__init__("PRSERVICE", "1.0", logger)
 
-    async def getPR(self, version, pkgarch, checksum):
+    async def getPR(self, version, pkgarch, checksum, history=False):
         response = await self.invoke(
-            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
         )
         if response:
             return response["value"]
lib/prserv/db.py +8 -8
index 8305238f7a..d41b781622 100644
@@ -163,7 +163,7 @@ class PRTable(object):
 
         self.dirty = True
 
-    def _get_value(self, version, pkgarch, checksum):
+    def _get_value(self, version, pkgarch, checksum, history):
 
         max_value = self.find_max_value(version, pkgarch)
 
@@ -177,7 +177,11 @@ class PRTable(object):
             # version, pkgarch found but not checksum. Create a new value from the maximum one
             return increase_revision(max_value)
 
-        if self.nohist:
+        if history:
+            # "history" mode: we found an existing value. We can return it
+            # whether it's the maximum one or not.
+            return value
+        else:
             # "no-history" mode: only return a value if that's the maximum one for
             # the version and architecture, otherwise create a new one.
             # This means that the value cannot decrement.
@@ -185,13 +189,9 @@ class PRTable(object):
                 return value
             else:
                 return increase_revision(max_value)
-        else:
-            # "hist" mode: we found an existing value. We can return it
-            # whether it's the maximum one or not.
-            return value
 
-    def get_value(self, version, pkgarch, checksum):
-        value = self._get_value(version, pkgarch, checksum)
+    def get_value(self, version, pkgarch, checksum, history):
+        value = self._get_value(version, pkgarch, checksum, history)
         self.store_value(version, pkgarch, checksum, value)
         return value
 
lib/prserv/serv.py +3 -2
index 9e07a34445..f692d050b8 100644
@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
         version = request["version"]
         pkgarch = request["pkgarch"]
         checksum = request["checksum"]
+        history = request["history"]
 
         if self.upstream_client is None:
-            value = self.server.table.get_value(version, pkgarch, checksum)
+            value = self.server.table.get_value(version, pkgarch, checksum, history)
             return {"value": value}
 
         # We have an upstream server.
@@ -105,7 +106,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
             # The package is not known upstream, must be a local-only package
             # Let's compute the PR number using the local-only method
 
-            value = self.server.table.get_value(version, pkgarch, checksum)
+            value = self.server.table.get_value(version, pkgarch, checksum, history)
             return {"value": value}
 
         # The package is known upstream, let's ask the upstream server
Signature · 1 line
2.34.1
#3 lore raw

Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request

Quoted 6 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:

From: Michael Opdenacker <[email protected]>

This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.

Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.

It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?

Quoted 97 lines
Cc Thomas Petazzoni <[email protected]>
Cc Joshua Watt <[email protected]>
Cc Tim Orling <[email protected]>
---
 lib/prserv/client.py |  4 ++--
 lib/prserv/db.py     | 16 ++++++++--------
 lib/prserv/serv.py   |  5 +++--
 3 files changed, 13 insertions(+), 12 deletions(-)
lib/prserv/client.py +2 -2
index 89760b6f74..389172f055 100644
@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
     def __init__(self):
         super().__init__("PRSERVICE", "1.0", logger)
 
-    async def getPR(self, version, pkgarch, checksum):
+    async def getPR(self, version, pkgarch, checksum, history=False):
         response = await self.invoke(
-            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
         )
         if response:
             return response["value"]
lib/prserv/db.py +8 -8
index 8305238f7a..d41b781622 100644
@@ -163,7 +163,7 @@ class PRTable(object):
 
         self.dirty = True
 
-    def _get_value(self, version, pkgarch, checksum):
+    def _get_value(self, version, pkgarch, checksum, history):
 
         max_value = self.find_max_value(version, pkgarch)
 
@@ -177,7 +177,11 @@ class PRTable(object):
             # version, pkgarch found but not checksum. Create a new value from the maximum one
             return increase_revision(max_value)
 
-        if self.nohist:
+        if history:
+            # "history" mode: we found an existing value. We can return it
+            # whether it's the maximum one or not.
+            return value
+        else:
             # "no-history" mode: only return a value if that's the maximum one for
             # the version and architecture, otherwise create a new one.
             # This means that the value cannot decrement.
@@ -185,13 +189,9 @@ class PRTable(object):
                 return value
             else:
                 return increase_revision(max_value)
-        else:
-            # "hist" mode: we found an existing value. We can return it
-            # whether it's the maximum one or not.
-            return value
 
-    def get_value(self, version, pkgarch, checksum):
-        value = self._get_value(version, pkgarch, checksum)
+    def get_value(self, version, pkgarch, checksum, history):
+        value = self._get_value(version, pkgarch, checksum, history)
         self.store_value(version, pkgarch, checksum, value)
         return value
 
lib/prserv/serv.py +3 -3
index 9e07a34445..f692d050b8 100644
@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
         version = request["version"]
         pkgarch = request["pkgarch"]
         checksum = request["checksum"]
+        history = request["history"]
 
         if self.upstream_client is None:
-            value = self.server.table.get_value(version, pkgarch, checksum)
+            value = self.server.table.get_value(version, pkgarch, checksum, history)
             return {"value": value}
 
         # We have an upstream server.
@@ -105,7 +106,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
             # The package is not known upstream, must be a local-only package
             # Let's compute the PR number using the local-only method
 
-            value = self.server.table.get_value(version, pkgarch, checksum)
+            value = self.server.table.get_value(version, pkgarch, checksum, history)
             return {"value": value}
 
         # The package is known upstream, let's ask the upstream server
--

2.34.1

#4 lore raw

Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request

Hi Joshua

Quoted 6 lines · Joshua Watt
Quoted 5 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:

From: Michael Opdenacker <[email protected]>

This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.

Quoted 8 lines

Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.

It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?

One version of my series actually removed that feature, but then Richard asked me to keep the code, because of the usefulness of the "history" mode for binary reproducibility. This way, we always generate the same package and revision if we have the same output hash as in a prior build. That's different from the default behavior that increases the revision every time we have a new output hash.

This being said, supporting both modes added substantial complexity to the code, and I wish I had more time to consolidate all this and reduce as much redundancy as possible between modes.

Cheers
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

Re: [RFC][PATCH 2/3] prserv: add "history" argument to "get-pr" request

Quoted 19 lines · Michael Opdenacker

Hi Joshua

Quoted 6 lines · Joshua Watt
Quoted 5 lines · On Tue, Apr 16, 2024 at 12:20 PM <[email protected]> wrote:

From: Michael Opdenacker <[email protected]>

This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.

Quoted 8 lines

Is this useful in normal day-to-day usage? I don't generally like to add functionality just for testing if it can be helped, as the closer you can get to actual end user API calls, the better it covers the end user experience.

It looks like PRData has an (unused) argument to enable history mode? If this is unused do we even actually care about it at all? Maybe it would be better to remove an unused feature than try to test it?

Quoted 6 lines

One version of my series actually removed that feature, but then Richard asked me to keep the code, because of the usefulness of the "history" mode for binary reproducibility. This way, we always generate the same package and revision if we have the same output hash as in a prior build. That's different from the default behavior that increases the revision every time we have a new output hash.

Ah, that's a different story. API used by end users (but not by bitbake during normal execution) is fine, and a different story than API added just for testing :)

Quoted 13 lines

This being said, supporting both modes added substantial complexity to the code, and I wish I had more time to consolidate all this and reduce as much redundancy as possible between modes.

Cheers
Michael.

--
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#6 lore raw

RFC PATCH 3/3 prserv: start bitbake selftests

In reply to [email protected]

From: Michael Opdenacker <[email protected]>

CC Joshua Watt <[email protected]>
CC Richard Purdie <[email protected]>
---
 bin/bitbake-selftest   |  2 +
 lib/prserv/__init__.py | 24 ++++++++++++
 lib/prserv/tests.py    | 84 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+)
 create mode 100644 bitbake/lib/prserv/tests.py
bin/bitbake-selftest +2 -0
index f25f23b1ae..ce901232fe 100755
@@ -15,6 +15,7 @@ import unittest
 try:
     import bb
     import hashserv
+    import prserv
     import layerindexlib
 except RuntimeError as exc:
     sys.exit(str(exc))
@@ -33,6 +34,7 @@ tests = ["bb.tests.codeparser",
          "bb.tests.utils",
          "bb.tests.compression",
          "hashserv.tests",
+         "prserv.tests",
          "layerindexlib.tests.layerindexobj",
          "layerindexlib.tests.restapi",
          "layerindexlib.tests.cooker"]
lib/prserv/__init__.py +24 -0
index 2ee6a28c04..76041f9001 100644
@@ -20,6 +20,30 @@ def init_logger(logfile, loglevel):
 class NotFoundError(Exception):
     pass
 
+def create_server(addr, dbpath, upstream=None, read_only=False):
+    from . import serv
+
+    s = serv.PRServer(dbpath, upstream=upstream, read_only=read_only)
+
+    (typ, a) = parse_address(addr)
+    s.start_tcp_server(*a)
+
+    return s
+
+def create_client(addr):
+    from . import client
+
+    c = client.PRClient()
+
+    try:
+        (typ, a) = parse_address(addr)
+        c.connect_tcp(*a)
+        print("CLIENT CONNECTED TO: %s" %(addr))
+        return c
+    except Exception as e:
+        c.close()
+        raise e
+
 async def create_async_client(addr):
     from . import client
 
lib/prserv/tests.py new file +84 -0

84 lines not on this page · raw message

Signature · 1 line
2.34.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Re: [bitbake-devel] [RFC][PATCH 3/3] prserv: start bitbake selftests

Greetings,

I eventually found out why I didn't have any logging...

Quoted 15 lines · Michael Opdenacker via lists.openembedded.org
(quoted diff) +15 -0
+class PRTestSetup(object):
+
+    server_index = 0
+    client_index = 0
+
+    def start_server(self, dbpath=None, upstream=None, read_only=False, prefunc=server_prefunc):
+
+        self.server_index += 1
+
+        if dbpath is None:
+            dbpath = self.make_dbpath()
+
+        def cleanup_server(server):
+            if server.process.exitcode is not None:
+                return

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+            server.process.terminate()
+            server.process.join()
+
+        server = create_server(socket.gethostbyname("localhost") + ":0",
+                               dbpath,
+                               upstream=upstream,
+                               read_only=read_only)
+
+        server.serve_as_process(prefunc=prefunc, args=(self.server_index))

I was missing a comma at the end of "args" in the above line. It should have been:
server.serve_as_process(prefunc=prefunc, args=(self.server_index,)

Without the comma, the prefunc function was never called and I got no logging.

Now, I'm studying the server logs :)
Thanks Joshua for the tips!
Cheers
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

Re: [bitbake-devel] [RFC][PATCH 3/3] prserv: start bitbake selftests

Quoted 6 lines · Michael Opdenacker via lists.openembedded.org
Quoted 1 line

+ server.serve_as_process(prefunc=prefunc, args=(self.server_index))

I was missing a comma at the end of "args" in the above line. It should have been:
server.serve_as_process(prefunc=prefunc, args=(self.server_index,)

Without the comma, the prefunc function was never called and I got no logging.

And worse, the asyncrpc server process didn't get started, though the rest of the server was (like creating the database). My first test is now successful and I can implement many more :)

Cheers
Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com