prserv: add support for an "upstream" server
cover prserv: add support for an "upstream" server
From: Michael Opdenacker <[email protected]>
This makes it possible to customize an "upstream" distribution by modifying local packages. If the "upstream" package bears revision "x", the local one will have revision "x.y", this having priority over the upstream one.
Multiple levels of upstream servers are supported, so "x.y.z" revisions are possible too.
Take advantage of this work to clean-up and update the prserv code too.
Manual "sanity" tests have been run so far but automated tests (based on Python unittest, as for the hash server tests) are being prepared for the next iteration. They will also to test also supported scenarios.
Code reviews are welcome, especially for more Pythonic ways of implementing some details.
---
Changes in V3:
- Revert the commit removing the so far unused "hist" mode, which we wish to keep for binary reproducibility sake.
- Simplification of get_value() function to take both "hist" and "nohist" modes with the same shared code.
- Add "history" parameter to the "getPR" request, so that the client can ask for the mode of its choice. This will also make it possible to implement tests for both modes.
Note that more requests ("export", "import"...) will also need a "history" parameter, in a future version, after the first tests are implemented.
- Several fixes to bugs either present in V2 or introduced by the V3 code changes.
- Put all the new features at the tip of the branch, to make the cleanup commits easier to merged.
Changes in V2:
- Add this new commit:
prserv: remove unused "hist" mode in the database backend
- Squash commit "prserv: fix read_only test" into commit "prserv: simplify the PRServerClient() interface" (Reported by Richard Purdie)
- Fix the code to support increasing "x.y.z" values, thus supporting several levels of upstream servers.
- db.py: remove duplicate definition of find_max_value() function in db.py
- prserv.py: remove tabs before comments (Python didn't complain)
- db.py: now stores the revision ("value") as TEXT. This way we can store "1.0" without having it transformed to "1" when the default type was INTEGER.
- This allows to fix a regression when the first packages were created with 'r0.1' instead of 'r0.0' initially.
- find_max_value: now returns None instead of '0' when no value is found Before we couldn't tell the difference between a '0'
max value and the absence of such a value.
Michael Opdenacker (12):
prserv: simplify the PRServerClient() interface
prserv: use double quotes by default
bitbake-prserv: replace deprecated optparse by argparse
prserv: use self.logger instead of logger directly
asyncrpc: include parse_address from hashserv
prserv: capitalization and spacing improvements
prserv: add extra requests
prserv: remove redundant exception handler
prserv: correct error message
prserv: remove unnecessary code
prserv: add "upstream" server support
prserv: add "history" argument to "get-pr" request
bin/bitbake-prserv | 99 ++++++++++++----- lib/bb/asyncrpc/client.py | 23 ++++ lib/hashserv/__init__.py | 27 +---- lib/prserv/__init__.py | 21 +++- lib/prserv/client.py | 44 ++++++-- lib/prserv/db.py | 225 ++++++++++++++++++++++---------------- lib/prserv/serv.py | 220 ++++++++++++++++++++++++++----------- 7 files changed, 436 insertions(+), 223 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/12 prserv: simplify the PRServerClient() interface
From: Michael Opdenacker <[email protected]>
serv.py: simplify the PRServerClient() interface by passing the server object instead of multiple arguments, and then retrieving the data through this object.
This replicates what is done for ServerClient() in hashserv/server.py
--- lib/prserv/serv.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
lib/prserv/serv.py +12 -12
@@ -20,16 +20,16 @@ PIDPREFIX = "/tmp/PRServer_%s_%s.pid" singleton = None class PRServerClient(bb.asyncrpc.AsyncServerConnection): - def __init__(self, socket, table, read_only): - super().__init__(socket, 'PRSERVICE', logger) + def __init__(self, socket, server): + super().__init__(socket, 'PRSERVICE', server.logger) + self.server = server + self.handlers.update({ 'get-pr': self.handle_get_pr, 'import-one': self.handle_import_one, 'export': self.handle_export, 'is-readonly': self.handle_is_readonly, }) - self.table = table - self.read_only = read_only def validate_proto_version(self): return (self.proto_version == (1, 0))
@@ -38,10 +38,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): try: return await super().dispatch_message(msg) except: - self.table.sync() + self.server.table.sync() raise else: - self.table.sync_if_dirty() + self.server.table.sync_if_dirty() async def handle_get_pr(self, request): version = request['version']
@@ -50,7 +50,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): response = None try: - value = self.table.getValue(version, pkgarch, checksum) + value = self.server.table.getValue(version, pkgarch, checksum) response = {'value': value} except prserv.NotFoundError: logger.error("can not find value for (%s, %s)",version, checksum)
@@ -61,13 +61,13 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): async def handle_import_one(self, request): response = None - if not self.read_only: + if not self.server.read_only: version = request['version'] pkgarch = request['pkgarch'] checksum = request['checksum'] value = request['value'] - value = self.table.importone(version, pkgarch, checksum, value) + value = self.server.table.importone(version, pkgarch, checksum, value) if value is not None: response = {'value': value}
@@ -80,7 +80,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): colinfo = request['colinfo'] try: - (metainfo, datainfo) = self.table.export(version, pkgarch, checksum, colinfo) + (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo) except sqlite3.Error as exc: logger.error(str(exc)) metainfo = datainfo = None
@@ -88,7 +88,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): return {'metainfo': metainfo, 'datainfo': datainfo} async def handle_is_readonly(self, request): - return {'readonly': self.read_only} + return {'readonly': self.server.read_only} class PRServer(bb.asyncrpc.AsyncServer): def __init__(self, dbfile, read_only=False):
@@ -98,7 +98,7 @@ class PRServer(bb.asyncrpc.AsyncServer): self.read_only = read_only def accept_client(self, socket): - return PRServerClient(socket, self.table, self.read_only) + return PRServerClient(socket, self) def start(self): tasks = super().start()
Signature · 1 line
2.34.1
PATCH 2/12 prserv: use double quotes by default
In reply to [email protected]From: Michael Opdenacker <[email protected]>
To aligh with the hashserv code
--- bin/bitbake-prserv | 4 +-- lib/prserv/__init__.py | 4 +-- lib/prserv/client.py | 20 ++++++------- lib/prserv/db.py | 32 ++++++++++----------- lib/prserv/serv.py | 64 +++++++++++++++++++++--------------------- 5 files changed, 62 insertions(+), 62 deletions(-)
bin/bitbake-prserv +2 -2
@@ -11,14 +11,14 @@ import optparse import warnings warnings.simplefilter("default") -sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib')) +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib")) import prserv import prserv.serv __version__="1.0.0" -PRHOST_DEFAULT='0.0.0.0' +PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 def main():
lib/prserv/__init__.py +2 -2
@@ -12,8 +12,8 @@ import sys,logging def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None) if not isinstance(numeric_level, int): - raise ValueError('Invalid log level: %s' % loglevel) - FORMAT = '%(asctime)-15s %(message)s' + raise ValueError("Invalid log level: %s" % loglevel) + FORMAT = "%(asctime)-15s %(message)s" logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT) class NotFoundError(Exception):
lib/prserv/client.py +10 -10
@@ -11,40 +11,40 @@ logger = logging.getLogger("BitBake.PRserv") class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): - super().__init__('PRSERVICE', '1.0', logger) + super().__init__("PRSERVICE", "1.0", logger) async def getPR(self, version, pkgarch, checksum): response = await self.invoke( - {'get-pr': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} ) if response: - return response['value'] + return response["value"] async def importone(self, version, pkgarch, checksum, value): response = await self.invoke( - {'import-one': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum, 'value': value}} + {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value}} ) if response: - return response['value'] + return response["value"] async def export(self, version, pkgarch, checksum, colinfo): response = await self.invoke( - {'export': {'version': version, 'pkgarch': pkgarch, 'checksum': checksum, 'colinfo': colinfo}} + {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo}} ) if response: - return (response['metainfo'], response['datainfo']) + return (response["metainfo"], response["datainfo"]) async def is_readonly(self): response = await self.invoke( - {'is-readonly': {}} + {"is-readonly": {}} ) if response: - return response['readonly'] + return response["readonly"] class PRClient(bb.asyncrpc.Client): def __init__(self): super().__init__() - self._add_methods('getPR', 'importone', 'export', 'is_readonly') + self._add_methods("getPR", "importone", "export", "is_readonly") def _get_async_client(self): return PRAsyncClient()
lib/prserv/db.py +16 -16
@@ -64,7 +64,7 @@ class PRTable(object): try: return self.conn.execute(*query) except sqlite3.OperationalError as exc: - if 'is locked' in str(exc) and end > time.time(): + if "is locked" in str(exc) and end > time.time(): continue raise exc
@@ -220,18 +220,18 @@ class PRTable(object): metainfo = {} #column info if colinfo: - metainfo['tbl_name'] = self.table - metainfo['core_ver'] = prserv.__version__ - metainfo['col_info'] = [] + metainfo["tbl_name"] = self.table + metainfo["core_ver"] = prserv.__version__ + metainfo["col_info"] = [] data = self._execute("PRAGMA table_info(%s);" % self.table) for row in data: col = {} - col['name'] = row['name'] - col['type'] = row['type'] - col['notnull'] = row['notnull'] - col['dflt_value'] = row['dflt_value'] - col['pk'] = row['pk'] - metainfo['col_info'].append(col) + col["name"] = row["name"] + col["type"] = row["type"] + col["notnull"] = row["notnull"] + col["dflt_value"] = row["dflt_value"] + col["pk"] = row["pk"] + metainfo["col_info"].append(col) #data info datainfo = []
@@ -261,12 +261,12 @@ class PRTable(object): else: data = self._execute(sqlstmt) for row in data: - if row['version']: + if row["version"]: col = {} - col['version'] = row['version'] - col['pkgarch'] = row['pkgarch'] - col['checksum'] = row['checksum'] - col['value'] = row['value'] + col["version"] = row["version"] + col["pkgarch"] = row["pkgarch"] + col["checksum"] = row["checksum"] + col["value"] = row["value"] datainfo.append(col) return (metainfo, datainfo)
@@ -275,7 +275,7 @@ class PRTable(object): for line in self.conn.iterdump(): writeCount = writeCount + len(line) + 1 fd.write(line) - fd.write('\n') + fd.write("\n") return writeCount class PRData(object):
lib/prserv/serv.py +32 -32
159 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 3/12 bitbake-prserv: replace deprecated optparse by argparse
In reply to [email protected]From: Michael Opdenacker <[email protected]>
optparse is deprecated since Python 2.7
Note that this is neither supposed to change the options supported by bitbake-prserv nor the way they are interpreted.
Note that in the "--help" output, long options are now reported for example as "--host HOST" instead of "--host=HOST" but both are equivalent anyway, as they already were with optparse.
--- bin/bitbake-prserv | 82 +++++++++++++++++++++++++++++++--------------- 1 file changed, 55 insertions(+), 27 deletions(-)
bin/bitbake-prserv +55 -27
@@ -7,7 +7,7 @@ import os import sys,logging -import optparse +import argparse import warnings warnings.simplefilter("default")
@@ -16,40 +16,68 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib import prserv import prserv.serv -__version__="1.0.0" +VERSION = "1.1.0" PRHOST_DEFAULT="0.0.0.0" PRPORT_DEFAULT=8585 def main(): - parser = optparse.OptionParser( - version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__), - usage = "%prog < --start | --stop > [options]") + parser = argparse.ArgumentParser( + description="BitBake PR Server. Version=%s" % VERSION, + formatter_class=argparse.RawTextHelpFormatter) - parser.add_option("-f", "--file", help="database filename(default: prserv.sqlite3)", action="store", - dest="dbfile", type="string", default="prserv.sqlite3") - parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store", - dest="logfile", type="string", default="prserv.log") - parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG", - action = "store", type="string", dest="loglevel", default = "INFO") - parser.add_option("--start", help="start daemon", - action="store_true", dest="start") - parser.add_option("--stop", help="stop daemon", - action="store_true", dest="stop") - parser.add_option("--host", help="ip address to bind", action="store", - dest="host", type="string", default=PRHOST_DEFAULT) - parser.add_option("--port", help="port number(default: 8585)", action="store", - dest="port", type="int", default=PRPORT_DEFAULT) - parser.add_option("-r", "--read-only", help="open database in read-only mode", - action="store_true") + parser.add_argument( + "-f", + "--file", + default="prserv.sqlite3", + help="database filename (default: prserv.sqlite3)", + ) + parser.add_argument( + "-l", + "--log", + default="prserv.log", + help="log filename(default: prserv.log)", + ) + parser.add_argument( + "--loglevel", + default="INFO", + help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG", + ) + parser.add_argument( + "--start", + action="store_true", + help="start daemon", + ) + parser.add_argument( + "--stop", + action="store_true", + help="stop daemon", + ) + parser.add_argument( + "--host", + help="ip address to bind", + default=PRHOST_DEFAULT, + ) + parser.add_argument( + "--port", + type=int, + default=PRPORT_DEFAULT, + help="port number (default: 8585)", + ) + parser.add_argument( + "-r", + "--read-only", + action="store_true", + help="open database in read-only mode", + ) - options, args = parser.parse_args(sys.argv) - prserv.init_logger(os.path.abspath(options.logfile),options.loglevel) + args = parser.parse_args() + prserv.init_logger(os.path.abspath(args.log), args.loglevel) - if options.start: - ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) - elif options.stop: - ret=prserv.serv.stop_daemon(options.host, options.port) + if args.start: + ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + elif args.stop: + ret=prserv.serv.stop_daemon(args.host, args.port) else: ret=parser.print_help() return ret
Signature · 1 line
2.34.1
PATCH 4/12 prserv: use self.logger instead of logger directly
In reply to [email protected]From: Michael Opdenacker <[email protected]>
In both the PRServerClient and PRClient objects.
This aligns with what is done in hashserv/server.py and makes it possible to benefit from possible specializations of the logger in the corresponding super classes, instead of using always the global logger.
--- lib/prserv/serv.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
lib/prserv/serv.py +4 -4
@@ -53,9 +53,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): value = self.server.table.getValue(version, pkgarch, checksum) response = {"value": value} except prserv.NotFoundError: - logger.error("can not find value for (%s, %s)",version, checksum) + self.logger.error("can not find value for (%s, %s)",version, checksum) except sqlite3.Error as exc: - logger.error(str(exc)) + self.logger.error(str(exc)) return response
@@ -82,7 +82,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): try: (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo) except sqlite3.Error as exc: - logger.error(str(exc)) + self.logger.error(str(exc)) metainfo = datainfo = None return {"metainfo": metainfo, "datainfo": datainfo}
@@ -105,7 +105,7 @@ class PRServer(bb.asyncrpc.AsyncServer): self.db = prserv.db.PRData(self.dbfile, read_only=self.read_only) self.table = self.db["PRMAIN"] - logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" % + self.logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" % (self.dbfile, self.address, str(os.getpid()))) return tasks
Signature · 1 line
2.34.1
PATCH 5/12 asyncrpc: include parse_address from hashserv
In reply to [email protected]From: Michael Opdenacker <[email protected]>
Moving the code and related definitions from hashserv/__init__.py to asyncrpc/client.py, allowing this function to be used in other asyncrpc clients.
--- lib/bb/asyncrpc/client.py | 23 +++++++++++++++++++++++ lib/hashserv/__init__.py | 27 +-------------------------- 2 files changed, 24 insertions(+), 26 deletions(-)
lib/bb/asyncrpc/client.py +23 -0
@@ -10,11 +10,34 @@ import json import os import socket import sys +import re import contextlib from threading import Thread from .connection import StreamConnection, WebsocketConnection, DEFAULT_MAX_CHUNK from .exceptions import ConnectionClosedError, InvokeError +UNIX_PREFIX = "unix://" +WS_PREFIX = "ws://" +WSS_PREFIX = "wss://" + +ADDR_TYPE_UNIX = 0 +ADDR_TYPE_TCP = 1 +ADDR_TYPE_WS = 2 + +def parse_address(addr): + if addr.startswith(UNIX_PREFIX): + return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX) :],)) + elif addr.startswith(WS_PREFIX) or addr.startswith(WSS_PREFIX): + return (ADDR_TYPE_WS, (addr,)) + else: + m = re.match(r"\[(?P<host>[^\]]*)\]:(?P<port>\d+)$", addr) + if m is not None: + host = m.group("host") + port = m.group("port") + else: + host, port = addr.split(":") + + return (ADDR_TYPE_TCP, (host, int(port))) class AsyncClient(object): def __init__(
lib/hashserv/__init__.py +1 -26
@@ -5,39 +5,14 @@ import asyncio from contextlib import closing -import re import itertools import json from collections import namedtuple from urllib.parse import urlparse - -UNIX_PREFIX = "unix://" -WS_PREFIX = "ws://" -WSS_PREFIX = "wss://" - -ADDR_TYPE_UNIX = 0 -ADDR_TYPE_TCP = 1 -ADDR_TYPE_WS = 2 +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS User = namedtuple("User", ("username", "permissions")) - -def parse_address(addr): - if addr.startswith(UNIX_PREFIX): - return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX) :],)) - elif addr.startswith(WS_PREFIX) or addr.startswith(WSS_PREFIX): - return (ADDR_TYPE_WS, (addr,)) - else: - m = re.match(r"\[(?P<host>[^\]]*)\]:(?P<port>\d+)$", addr) - if m is not None: - host = m.group("host") - port = m.group("port") - else: - host, port = addr.split(":") - - return (ADDR_TYPE_TCP, (host, int(port))) - - def create_server( addr, dbname,
Signature · 1 line
2.34.1
PATCH 6/12 prserv: capitalization and spacing improvements
In reply to [email protected]From: Michael Opdenacker <[email protected]>
Choosing only one style of capitalization
Add extra space after some commas too
Remove idle spaces
--- lib/prserv/__init__.py | 2 +- lib/prserv/db.py | 56 +++++++++++++++++++++--------------------- lib/prserv/serv.py | 10 ++++---- 3 files changed, 34 insertions(+), 34 deletions(-)
lib/prserv/__init__.py +1 -1
@@ -7,7 +7,7 @@ __version__ = "1.0.0" import os, time -import sys,logging +import sys, logging def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None)
lib/prserv/db.py +28 -28
160 lines not on this page · raw message
lib/prserv/serv.py +5 -5
@@ -50,7 +50,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): response = None try: - value = self.server.table.getValue(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum) response = {"value": value} except prserv.NotFoundError: self.logger.error("can not find value for (%s, %s)",version, checksum)
@@ -173,16 +173,16 @@ def run_as_daemon(func, pidfile, logfile): # potentially hanging on a pipe. Handle both cases. si = open("/dev/null", "r") try: - os.dup2(si.fileno(),sys.stdin.fileno()) + os.dup2(si.fileno(), sys.stdin.fileno()) except (AttributeError, io.UnsupportedOperation): sys.stdin = si so = open(logfile, "a+") try: - os.dup2(so.fileno(),sys.stdout.fileno()) + os.dup2(so.fileno(), sys.stdout.fileno()) except (AttributeError, io.UnsupportedOperation): sys.stdout = so try: - os.dup2(so.fileno(),sys.stderr.fileno()) + os.dup2(so.fileno(), sys.stderr.fileno()) except (AttributeError, io.UnsupportedOperation): sys.stderr = so
@@ -253,7 +253,7 @@ def stop_daemon(host, port): portstr = "Wrong port? Other ports listening at %s: %s" % (host, " ".join(ports)) sys.stderr.write("pidfile %s does not exist. Daemon not running? %s\n" - % (pidfile,portstr)) + % (pidfile, portstr)) return 1 try:
Signature · 1 line
2.34.1
PATCH 7/12 prserv: add extra requests
In reply to [email protected]From: Michael Opdenacker <[email protected]>
Useful for connecting a PR server to an upstream one
- "test-package" checks whether the specified package version and arch is known in the database.
- "test-pr" checks a specified output hash is found in the database. Otherwise it returns 'None' instead of a new value.
- "max-package-pr" returns the highest PR number for (version, arch) entries in the database, and None if not found
Add new DB functions supporting the above, plus test_value() which tells whether a given value is available for the specified package and architecture.
--- lib/prserv/client.py | 23 +++++++++++++++++++++- lib/prserv/db.py | 46 ++++++++++++++++++++++++++++++++++++++++++++ lib/prserv/serv.py | 28 +++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 1 deletion(-)
lib/prserv/client.py +22 -1
@@ -20,6 +20,27 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): if response: return response["value"] + async def test_pr(self, version, pkgarch, checksum): + response = await self.invoke( + {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + ) + if response: + return response["value"] + + async def test_package(self, version, pkgarch): + response = await self.invoke( + {"test-package": {"version": version, "pkgarch": pkgarch}} + ) + if response: + return response["value"] + + async def max_package_pr(self, version, pkgarch): + response = await self.invoke( + {"max-package-pr": {"version": version, "pkgarch": pkgarch}} + ) + if response: + return response["value"] + async def importone(self, version, pkgarch, checksum, value): response = await self.invoke( {"import-one": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "value": value}}
@@ -44,7 +65,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): class PRClient(bb.asyncrpc.Client): def __init__(self): super().__init__() - self._add_methods("getPR", "importone", "export", "is_readonly") + self._add_methods("getPR", "test_pr", "test_package", "importone", "export", "is_readonly") def _get_async_client(self): return PRAsyncClient()
lib/prserv/db.py +46 -0
52 lines not on this page · raw message
lib/prserv/serv.py +28 -0
@@ -26,6 +26,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): self.handlers.update({ "get-pr": self.handle_get_pr, + "test-pr": self.handle_test_pr, + "test-package": self.handle_test_package, + "max-package-pr": self.handle_max_package_pr, "import-one": self.handle_import_one, "export": self.handle_export, "is-readonly": self.handle_is_readonly,
@@ -43,6 +46,31 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): else: self.server.table.sync_if_dirty() + async def handle_test_pr(self, request): + '''Finds the PR value corresponding to the request. If not found, returns None and doesn't insert a new value''' + version = request["version"] + pkgarch = request["pkgarch"] + checksum = request["checksum"] + + value = self.server.table.find_value(version, pkgarch, checksum) + return {"value": value} + + async def handle_test_package(self, request): + '''Tells whether there are entries for (version, pkgarch) in the db. Returns True or False''' + version = request["version"] + pkgarch = request["pkgarch"] + + value = self.server.table.test_package(version, pkgarch) + return {"value": value} + + async def handle_max_package_pr(self, request): + '''Finds the greatest PR value for (version, pkgarch) in the db. Returns None if no entry was found''' + version = request["version"] + pkgarch = request["pkgarch"] + + value = self.server.table.find_max_value(version, pkgarch) + return {"value": value} + async def handle_get_pr(self, request): version = request["version"] pkgarch = request["pkgarch"]
Signature · 1 line
2.34.1
Re: [bitbake-devel] [PATCH 07/12] prserv: add extra requests
Quoted 10 lines · Michael Opdenacker via lists.openembedded.org
From: Michael Opdenacker <[email protected]>
Useful for connecting a PR server to an upstream one
- "test-package" checks whether the specified package version and arch is known in the database.
- "test-pr" checks a specified output hash is found in the database. Otherwise it returns 'None' instead of a new value.
Quoted 7 lines
- "max-package-pr" returns the highest PR number for (version, arch) entries in the database, and None if not found
Add new DB functions supporting the above, plus
test_value()which tells whether a given value is available for the specified package and architecture.
I always bounce around in my own code on whether I prefer the "action" or the "object" to be more prominent in my method naming.
i.e. pr_get or get_pr
I tend to land on pr_get() -style most of the time.
That being said, your routines match up wth the existing style so my comment is literally that .. commentary. And I have no concerns with the change.
Quoted 185 lines
Signed-off-by Michael Opdenacker <[email protected]>Cc Joshua Watt <[email protected]>Cc Tim Orling <[email protected]>Cc Thomas Petazzoni <[email protected]>--- lib/prserv/client.py | 23 +++++++++++++++++++++- lib/prserv/db.py | 46 ++++++++++++++++++++++++++++++++++++++++++++ lib/prserv/serv.py | 28 +++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 1 deletion(-)lib/prserv/client.py +21
-0@@ -20,6 +20,27 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): if response: return response["value"] + async def test_pr(self, version, pkgarch, checksum): + response = await self.invoke( + {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + ) + if response: + return response["value"] + + async def test_package(self, version, pkgarch): + response = await self.invoke( + {"test-package": {"version": version, "pkgarch": pkgarch}} + ) + if response: + return response["value"] + + async def max_package_pr(self, version, pkgarch): + response = await self.invoke( + {"max-package-pr": {"version": version, "pkgarch": pkgarch}} + ) + if response: + return response["value"] + async def importone(self, version, pkgarch, checksum, value): response = await self.invoke({"import-one": {"version": version, "pkgarch": pkgarch,"checksum": checksum, "value": value}}
lib/prserv/client.py +1
-1@@ -44,7 +65,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): class PRClient(bb.asyncrpc.Client): def __init__(self): super().__init__() - self._add_methods("getPR", "importone", "export", "is_readonly") + self._add_methods("getPR", "test_pr", "test_package", "importone", "export", "is_readonly") def _get_async_client(self):return
PRAsyncClient()lib/prserv/db.py +2
-0@@ -78,6 +78,52 @@ class PRTable(object): self.sync() self.dirty = False + def test_package(self, version, pkgarch): + """Returns whether the specified package version is found in thedatabase for the specified architecture"""
lib/prserv/db.py +3
-0+ + # Just returns the value if found or None otherwise + data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=?;" % self.table,
lib/prserv/db.py +9
-0+ (version, pkgarch)) + row=data.fetchone() + if row is not None: + return True + else: + return False + + def test_value(self, version, pkgarch, value): + """Returns whether the specified value is found in the databasefor the specified package and architecture"""
lib/prserv/db.py +3
-0+ + # Just returns the value if found or None otherwise + data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? and value=?;" % self.table,
lib/prserv/db.py +9
-0+ (version, pkgarch, value)) + row=data.fetchone() + if row is not None: + return True + else: + return False + + def find_value(self, version, pkgarch, checksum): + """Returns the value for the specified checksum if found or Noneotherwise."""
lib/prserv/db.py +2
-0+ + data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? AND checksum=?;" % self.table,
lib/prserv/db.py +9
-0+ (version, pkgarch, checksum)) + row=data.fetchone() + if row is not None: + return row[0] + else: + return None + + def find_max_value(self, version, pkgarch): + """Returns the greatest value for (version, pkgarch), or None ifnot found. Doesn't create a new value"""
lib/prserv/db.py +2
-0+ + data = self._execute("SELECT max(value) FROM %s where version=?AND pkgarch=?;" % (self.table),
lib/prserv/db.py +7
-0+ (version, pkgarch)) + row = data.fetchone() + if row is not None: + return row[0] + else: + return None + def _get_value_hist(self, version, pkgarch, checksum): data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? AND checksum=?;" % self.table,
(version, pkgarch, checksum))lib/prserv/serv.py +5
-0@@ -26,6 +26,9 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): self.handlers.update({ "get-pr": self.handle_get_pr, + "test-pr": self.handle_test_pr, + "test-package": self.handle_test_package, + "max-package-pr": self.handle_max_package_pr, "import-one": self.handle_import_one, "export": self.handle_export, "is-readonly": self.handle_is_readonly,@@ -43,6 +46,31 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): else: self.server.table.sync_if_dirty() + async def handle_test_pr(self, request): + '''Finds the PR value corresponding to the request. If not found,returns None and doesn't insert a new value'''
lib/prserv/serv.py +9
-0+ version = request["version"] + pkgarch = request["pkgarch"] + checksum = request["checksum"] + + value = self.server.table.find_value(version, pkgarch, checksum) + return {"value": value} + + async def handle_test_package(self, request): + '''Tells whether there are entries for (version, pkgarch) in thedb. Returns True or False'''
lib/prserv/serv.py +8
-0+ version = request["version"] + pkgarch = request["pkgarch"] + + value = self.server.table.test_package(version, pkgarch) + return {"value": value} + + async def handle_max_package_pr(self, request): + '''Finds the greatest PR value for (version, pkgarch) in the db.Returns None if no entry was found'''
lib/prserv/serv.py +6
-1+ version = request["version"] + pkgarch = request["pkgarch"] + + value = self.server.table.find_max_value(version, pkgarch) + return {"value": value} + async def handle_get_pr(self, request): version = request["version"] pkgarch = request["pkgarch"] --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#16081):
https://lists.openembedded.org/g/bitbake-devel/message/16081
Mute This Topic: https://lists.openembedded.org/mt/105479094/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
PATCH 8/12 prserv: remove redundant exception handler
In reply to [email protected]From: Michael Opdenacker <[email protected]>
This exception handler is already present in db.py's get_value() code.
--- lib/prserv/serv.py | 2 -- 1 file changed, 2 deletions(-)
lib/prserv/serv.py +0 -2
@@ -82,8 +82,6 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): response = {"value": value} except prserv.NotFoundError: self.logger.error("can not find value for (%s, %s)",version, checksum) - except sqlite3.Error as exc: - self.logger.error(str(exc)) return response
Signature · 1 line
2.34.1
PATCH 9/12 prserv: correct error message
In reply to [email protected]From: Michael Opdenacker <[email protected]>
according to db.py, prserv.NotFoundError is returned here when adding a new value to the database failed
--- lib/prserv/serv.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/prserv/serv.py +1 -1
@@ -81,7 +81,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): value = self.server.table.get_value(version, pkgarch, checksum) response = {"value": value} except prserv.NotFoundError: - self.logger.error("can not find value for (%s, %s)",version, checksum) + self.logger.error("failure storing value in database for (%s, %s)",version, checksum) return response
Signature · 1 line
2.34.1
PATCH 10/12 prserv: remove unnecessary code
In reply to [email protected]From: Michael Opdenacker <[email protected]>
In db.py, the ifnull() statement guarantees that the SQL request will return a value. It's therefore unnecessary to test the case when no value is found.
--- lib/prserv/db.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
lib/prserv/db.py +1 -5
@@ -172,11 +172,7 @@ class PRTable(object): if self.read_only: data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table), (version, pkgarch)) - row = data.fetchone() - if row is not None: - return row[0] - else: - return 0 + return data.fetchone()[0] try: self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));"
Signature · 1 line
2.34.1
PATCH 11/12 prserv: add "upstream" server support
In reply to [email protected]From: Michael Opdenacker <[email protected]>
Introduce a PRSERVER_UPSTREAM variable that makes the local PR server connect to an "upstream" one.
This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y").
Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format."
Set the comments in the handle_get_pr() function in serv.py for details about the calculation of the local revision.
Use the newly developed functions to simplify the implementation of the get_value() function by merging the code of the earlier _get_value_hist() and _get_value_nohist() functions, removing redundancy between both functions. This also makes the code easier to understand.
--- bin/bitbake-prserv | 15 ++++- lib/prserv/__init__.py | 15 +++++ lib/prserv/client.py | 1 + lib/prserv/db.py | 125 ++++++++++++++++++++--------------------- lib/prserv/serv.py | 95 +++++++++++++++++++++++++++---- 5 files changed, 174 insertions(+), 77 deletions(-)
bin/bitbake-prserv +14 -1
@@ -70,12 +70,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERVER_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() prserv.init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else:
lib/prserv/__init__.py +15 -0
@@ -8,6 +8,7 @@ __version__ = "1.0.0" import os, time import sys, logging +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None)
@@ -18,3 +19,17 @@ def init_logger(logfile, loglevel): class NotFoundError(Exception): pass + +async def create_async_client(addr): + from . import client + + c = client.PRAsyncClient() + + try: + (typ, a) = parse_address(addr) + await c.connect_tcp(*a) + return c + + except Exception as e: + await c.close() + raise e
lib/prserv/client.py +1 -0
@@ -6,6 +6,7 @@ import logging import bb.asyncrpc +from . import create_async_client logger = logging.getLogger("BitBake.PRserv")
lib/prserv/db.py +61 -64
158 lines not on this page · raw message
lib/prserv/serv.py +83 -12
161 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [bitbake-devel] [PATCH 11/12] prserv: add "upstream" server support
Quoted 6 lines · Michael Opdenacker via lists.openembedded.org
From: Michael Opdenacker <[email protected]>
Introduce a
PRSERVER_UPSTREAMvariable that makes the local PR server connect to an "upstream" one.
I assume we'll eventually put something about the new variable in template conf files (commented out), similar to the upstream hasserv ones ?
Quoted 4 lines
This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y").
Update the calculation of the new revisions to support the
Quoted 118 lines
case when prior revisions are not integers, but have an "x.y..." format."
Set the comments in the
handle_get_pr()function inserv.pyfor details about the calculation of the local revision.Use the newly developed functions to simplify the implementation of the
get_value()function by merging the code of the earlier_get_value_hist()and_get_value_nohist()functions, removing redundancy between both functions. This also makes the code easier to understand.Signed-off-by Michael Opdenacker <[email protected]>Cc Joshua Watt <[email protected]>Cc Tim Orling <[email protected]>Cc Thomas Petazzoni <[email protected]>--- bin/bitbake-prserv | 15 ++++- lib/prserv/__init__.py | 15 +++++ lib/prserv/client.py | 1 + lib/prserv/db.py | 125 ++++++++++++++++++++--------------------- lib/prserv/serv.py | 95 +++++++++++++++++++++++++++---- 5 files changed, 174 insertions(+), 77 deletions(-)bin/bitbake-prserv +14
-1@@ -70,12 +70,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERVER_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() prserv.init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port)else:
lib/prserv/__init__.py +15
-0@@ -8,6 +8,7 @@ __version__ = "1.0.0" import os, time import sys, logging +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None)@@ -18,3 +19,17 @@ def init_logger(logfile, loglevel): class NotFoundError(Exception): pass + +async def create_async_client(addr): + from . import client + + c = client.PRAsyncClient() + + try: + (typ, a) = parse_address(addr) + await c.connect_tcp(*a) + return c + + except Exception as e: + await c.close() + raise elib/prserv/client.py +1
-0@@ -6,6 +6,7 @@ import logging import bb.asyncrpc +from . import create_async_client logger = logging.getLogger("BitBake.PRserv")lib/prserv/db.py +2
-0@@ -21,6 +21,20 @@ sqlversion = sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): raise Exception("sqlite3 version 3.3.0 or later is required.") +def increase_revision(ver): + """Take a revision string such as "1" or "1.2.3" or even a number andincrease its last number
lib/prserv/db.py +9
-0+ This fails if the last number is not an integer""" + + fields=str(ver).split('.') + last = fields[-1] + + try: + val = int(last) + except Exception as e: + logger.critical("Unable to increase revision value %s: %s" %
Quoted 3 lines
(ver, e))
+
What's the chain of events that will happen after this log and the return ? The log
message itself isn't going to tell us much, as the package name, etc, aren't available
at this point.
Should we be raising another exception ? Something else ? Such that we can get a better log message from calling code that has more context about what package or what might have lead to the issue ?
Should processing stop and the build fail ? If we don't have a failure, we risk
getting a package with the same version and changes being missed. This could already be the case in other parts of the code, I'm only wondering it while
reading this specific routine. (and I can never remember if logger.cirtical()
aborts execution on it's own).
Quoted 216 lines
lib/prserv/db.py +2
-0+ return ".".join(fields[0:-1] + list(str(val + 1))) + # # "No History" mode - for a given query tuple (version, pkgarch,checksum),
# the returned value will be the largest among all the values of the samelib/prserv/db.py +1
-1@@ -53,7 +67,7 @@ class PRTable(object): (version TEXT NOT NULL, \ pkgarch TEXT NOT NULL, \ checksum TEXT NOT NULL, \ - value INTEGER, \ + value TEXT, \ PRIMARY KEY (version, pkgarch, checksum));" % self.table)def
_execute(self, *query):lib/prserv/db.py +1
-1@@ -119,84 +133,67 @@ class PRTable(object): data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table), (version, pkgarch)) row = data.fetchone() - if row is not None: + # With SELECT max() requests, you have an empty row when thereare no values, therefore the test on row[0]
lib/prserv/db.py +1
-2+ if row is not None and row[0] is not None: return row[0] else: return None - def _get_value_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? AND checksum=?;" % self.table,
lib/prserv/db.py +2
-4- (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] + def find_new_subvalue(self, version, pkgarch, base): + """Take and increase the greatest "<base>.y" value for (version,pkgarch), or return "<base>.1" if not found.
lib/prserv/db.py +3
-0+ This doesn't store a new value.""" + + data = self._execute("SELECT max(value) FROM %s where version=?AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base),
lib/prserv/db.py +3
-0+ (version, pkgarch)) + row = data.fetchone() + # With SELECT max() requests, you have an empty row when thereare no values, therefore the test on row[0]
lib/prserv/db.py +2
-3+ if row is not None and row[0] is not None: + return increase_revision(row[0]) else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM%s where version=? AND pkgarch=?;" % (self.table),
lib/prserv/db.py +1
-8- (version, pkgarch)) - row = data.fetchone() - if row is not None: - return row[0] - else: - return 0 + return base + ".0" - try: - self._execute("INSERT INTO %s VALUES (?, ?, ?, (select
ifnull(max(value)+1, 0)from %s where version=? AND pkgarch=?));"lib/prserv/db.py +4
-5- % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) + def store_value(self, version, pkgarch, checksum, value): + """Store new value in the database""" - self.dirty = True + try: + self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" %(self.table),
lib/prserv/db.py +3
-1+ (version, pkgarch, checksum, value)) + except sqlite3.IntegrityError as exc: + logger.error(str(exc)) - data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? AND checksum=?;" % self.table,
lib/prserv/db.py +1
-9- (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + self.dirty = True - def _get_value_no_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s \ - WHERE version=? AND pkgarch=? AND checksum=?AND \
- value >= (select max(value) from %s where
version=? AND pkgarch=?);"lib/prserv/db.py +0
-2- % (self.table, self.table), - (version, pkgarch, checksum, version,pkgarch))
lib/prserv/db.py +0
-7- row=data.fetchone() - if row is not None: - return row[0] - else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM%s where version=? AND pkgarch=?;" % (self.table),
lib/prserv/db.py +1
-4- (version, pkgarch)) - return data.fetchone()[0] + def _get_value(self, version, pkgarch, checksum): - try: - self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?,(select
ifnull(max(value)+1, 0)from %s where version=? AND pkgarch=?));"lib/prserv/db.py +0
-2- % (self.table, self.table), - (version, pkgarch, checksum, version,pkgarch))
lib/prserv/db.py +4
-5- except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - self.conn.rollback() + max_value = self.find_max_value(version, pkgarch) - self.dirty = True + if max_value is None: + # version, pkgarch completely unknown. Return initial value. + return "0" - data=self._execute("SELECT value FROM %s WHERE version=? ANDpkgarch=? AND checksum=?;" % self.table,
lib/prserv/db.py +4
-6- (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + value = self.find_value(version, pkgarch, checksum) + + if value is None: + # version, pkgarch found but not checksum. Create a new valuefrom the maximum one
lib/prserv/db.py +2
-2+ return increase_revision(max_value) - def get_value(self, version, pkgarch, checksum): if self.nohist: - return self._get_value_no_hist(version, pkgarch, checksum) + # "no-history" mode: only return a value if that's themaximum one for
lib/prserv/db.py +14
-1+ # the version and architecture, otherwise create a new one. + # This means that the value cannot decrement. + if value == max_value: + return value + else: + return increase_revision(max_value) else: - return self._get_value_hist(version, pkgarch, checksum) + # "hist" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + + def get_value(self, version, pkgarch, checksum): + value = self._get_value(version, pkgarch, checksum) + self.store_value(version, pkgarch, checksum, value) + return value def _import_hist(self, version, pkgarch, checksum, value): if self.read_only:lib/prserv/serv.py +1
-0@@ -12,6 +12,7 @@ import sqlite3 import prserv import prserv.db import errno +from . import create_async_client import bb.asyncrpc logger = logging.getLogger("BitBake.PRserv")@@ -76,14 +77,76 @@ classPRServerClient(bb.asyncrpc.AsyncServerConnection):
pkgarch = request["pkgarch"] checksum = request["checksum"]lib/prserv/serv.py +1
-2- response = None - try: + if self.upstream_client is None: value = self.server.table.get_value(version, pkgarch,checksum)
lib/prserv/serv.py +0
-3- response = {"value": value} - except prserv.NotFoundError: - self.logger.error("failure storing value in database for (%s,%s)",version, checksum)
lib/prserv/serv.py +3
-1+ return {"value": value} - return response + # We have an upstream server. + # Check whether the local server already knows the requestedconfiguration
lib/prserv/serv.py +2
-0+ # Here we use find_value(), not get_value(), because we don't want + # to unconditionally add a new generated value to the database.If the configuration
+ # is a new one, the generated value we will add will depend on
what's on the upstream server.lib/prserv/serv.py +13
-0+ + value = self.server.table.find_value(version, pkgarch, checksum) + + if value is not None: + + # The configuration is already known locally. Let's use it. + + return {"value": value} + + # The configuration is a new one for the local server + # Let's ask the upstream server whether it knows it + + known_upstream = await self.upstream_client.test_package(version,
Quoted 3 lines
pkgarch)
+
Remind me .. what happens in the case of timeouts or other issues ? Is there any way to detect it and provide a message ? Right now, it looks like it'll just
fall through to local PR server functionality. Which I don't know is right or wrong,
just that there's no way to really tell it happened.
Quoted 23 lines
lib/prserv/serv.py +3
-0+ if not known_upstream: + + # The package is not known upstream, must be a local-onlypackage
lib/prserv/serv.py +3
-0+ # Let's compute the PR number using the local-only method + + value = self.server.table.get_value(version, pkgarch,checksum)
lib/prserv/serv.py +6
-0+ return {"value": value} + + # The package is known upstream, let's ask the upstream server + # whether it knows our new output hash + + value = await self.upstream_client.test_pr(version, pkgarch,checksum)
lib/prserv/serv.py +4
-0+ + if value is not None: + + # Upstream knows this output hash, let's store it and use ittoo.
lib/prserv/serv.py +3
-0+ + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum,
Quoted 6 lines
value)
+ # If the local server is read only, won't be able to store
the new
+ # value in the database and will have to keep asking the
upstream server
I assume it is in the find_value() call above where we'll short circuit the call
to the upstream server if the hash hasn't changed and we've stored it locally.
Quoted 57 lines
lib/prserv/serv.py +4
-0+ + return {"value": value} + + # The output hash doesn't exist upstream, get the most recentnumber from upstream (x)
lib/prserv/serv.py +3
-0+ # Then, we want to have a new PR value for the local server: x.y + + upstream_max = await self.upstream_client.max_package_pr(version,pkgarch)
+ # Here we know that the package is known upstream, soupstream_maxcan't be None
+ subvalue = self.server.table.find_new_subvalue(version, pkgarch,upstream_max)lib/prserv/serv.py +3
-0+ + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum,subvalue)
lib/prserv/serv.py +6
-0+ + return {"value": subvalue} + + async def process_requests(self): + if self.server.upstream is not None: + self.upstream_client = await
create_async_client(self.server.upstream)lib/prserv/serv.py +8
-0+ else: + self.upstream_client = None + + try: + await super().process_requests() + finally: + if self.upstream_client is not None: + await self.upstream_client.close() async def handle_import_one(self, request): response = None@@ -117,11 +180,12 @@ classPRServerClient(bb.asyncrpc.AsyncServerConnection):
return {"readonly": self.server.read_only}class PRServer(bb.asyncrpc.AsyncServer):
lib/prserv/serv.py +2
-1- def __init__(self, dbfile, read_only=False): + def __init__(self, dbfile, read_only=False, upstream=None): super().__init__(logger) self.dbfile = dbfile self.table = None self.read_only = read_only + self.upstream = upstream def accept_client(self, socket): return PRServerClient(socket, self)@@ -134,6 +198,9 @@ class PRServer(bb.asyncrpc.AsyncServer): self.logger.info("Started PRServer with DBfile: %s, Address: %s,PID: %s" %
(self.dbfile, self.address, str(os.getpid())))lib/prserv/serv.py +2
-0+ if self.upstream is not None: + self.logger.info("And upstream PRServer: %s " %
Quoted 3 lines
(self.upstream))
+
I noticed there were some direct call to the global logger above. Is the mixing of logger and self.logger intentional (based on whether or not the object has a logger .. I assume .. but I wanted to be sure).
Bruce
Quoted 79 lines
return tasks
async def stop(self):
lib/prserv/serv.py +3
-2@@ -147,14 +214,15 @@ class PRServer(bb.asyncrpc.AsyncServer): self.table.sync() class PRServSingleton(object): - def __init__(self, dbfile, logfile, host, port): + def __init__(self, dbfile, logfile, host, port, upstream): self.dbfile = dbfile self.logfile = logfile self.host = host self.port = port + self.upstream = upstream def start(self): - self.prserv = PRServer(self.dbfile) + self.prserv = PRServer(self.dbfile, upstream=self.upstream) self.prserv.start_tcp_server(socket.gethostbyname(self.host), self.port) self.process =
self.prserv.serve_as_process(log_level=logging.WARNING)lib/prserv/serv.py +1
-1@@ -233,7 +301,7 @@ def run_as_daemon(func, pidfile, logfile): os.remove(pidfile) os._exit(0) -def start_daemon(dbfile, host, port, logfile, read_only=False): +def start_daemon(dbfile, host, port, logfile, read_only=False,upstream=None):
ip =socket.gethostbyname(host)
pidfile = PIDPREFIX % (ip, port)
try:lib/prserv/serv.py +0
-0@@ -249,7 +317,7 @@ def start_daemon(dbfile, host, port, logfile,read_only=False):
dbfile =
os.path.abspath(dbfile)
defdaemon_main():lib/prserv/serv.py +5
-2- server = PRServer(dbfile, read_only=read_only) + server = PRServer(dbfile, read_only=read_only, upstream=upstream) server.start_tcp_server(ip, port) server.serve_forever()@@ -336,6 +404,9 @@ def auto_start(d): host = host_params[0].strip().lower() port = int(host_params[1]) + + upstream = d.getVar("PRSERV_UPSTREAM") or None + if is_local_special(host, port): import bb.utils cachedir = (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE"))@@ -350,7 +421,7 @@ def auto_start(d): auto_shutdown() if not singleton: bb.utils.mkdirhier(cachedir) - singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port) + singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port, upstream) singleton.start()if singleton:
host = singleton.host
--
2.34.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#16086):
https://lists.openembedded.org/g/bitbake-devel/message/16086
Mute This Topic: https://lists.openembedded.org/mt/105479101/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [bitbake-devel] [PATCH 11/12] prserv: add "upstream" server support
In reply to [email protected]Quoted 343 lines · Michael Opdenacker via lists.openembedded.org
From: Michael Opdenacker <[email protected]>
Introduce a
PRSERVER_UPSTREAMvariable that makes the local PR server connect to an "upstream" one.This makes it possible to implement local fixes to an upstream package (revision "x", in a way that gives the local update priority (revision "x.y").
Update the calculation of the new revisions to support the case when prior revisions are not integers, but have an "x.y..." format."
Set the comments in the
handle_get_pr()function inserv.pyfor details about the calculation of the local revision.Use the newly developed functions to simplify the implementation of the
get_value()function by merging the code of the earlier_get_value_hist()and_get_value_nohist()functions, removing redundancy between both functions. This also makes the code easier to understand.Signed-off-by Michael Opdenacker <[email protected]>Cc Joshua Watt <[email protected]>Cc Tim Orling <[email protected]>Cc Thomas Petazzoni <[email protected]>--- bin/bitbake-prserv | 15 ++++- lib/prserv/__init__.py | 15 +++++ lib/prserv/client.py | 1 + lib/prserv/db.py | 125 ++++++++++++++++++++--------------------- lib/prserv/serv.py | 95 +++++++++++++++++++++++++++---- 5 files changed, 174 insertions(+), 77 deletions(-)bin/bitbake-prserv +14
-1@@ -70,12 +70,25 @@ def main(): action="store_true", help="open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=os.environ.get("PRSERVER_UPSTREAM", None), + help="Upstream PR service (host:port)", + ) args = parser.parse_args() prserv.init_logger(os.path.abspath(args.log), args.loglevel) if args.start: - ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only) + ret=prserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=prserv.serv.stop_daemon(args.host, args.port) else:lib/prserv/__init__.py +15
-0@@ -8,6 +8,7 @@ __version__ = "1.0.0" import os, time import sys, logging +from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS def init_logger(logfile, loglevel): numeric_level = getattr(logging, loglevel.upper(), None)@@ -18,3 +19,17 @@ def init_logger(logfile, loglevel): class NotFoundError(Exception): pass + +async def create_async_client(addr): + from . import client + + c = client.PRAsyncClient() + + try: + (typ, a) = parse_address(addr) + await c.connect_tcp(*a) + return c + + except Exception as e: + await c.close() + raise elib/prserv/client.py +1
-0@@ -6,6 +6,7 @@ import logging import bb.asyncrpc +from . import create_async_client logger = logging.getLogger("BitBake.PRserv")lib/prserv/db.py +61
-64@@ -21,6 +21,20 @@ sqlversion = sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): raise Exception("sqlite3 version 3.3.0 or later is required.") +def increase_revision(ver): + """Take a revision string such as "1" or "1.2.3" or even a number and increase its last number + This fails if the last number is not an integer""" + + fields=str(ver).split('.') + last = fields[-1] + + try: + val = int(last) + except Exception as e: + logger.critical("Unable to increase revision value %s: %s" % (ver, e)) + + return ".".join(fields[0:-1] + list(str(val + 1))) + # # "No History" mode - for a given query tuple (version, pkgarch, checksum), # the returned value will be the largest among all the values of the same@@ -53,7 +67,7 @@ class PRTable(object): (version TEXT NOT NULL, \ pkgarch TEXT NOT NULL, \ checksum TEXT NOT NULL, \ - value INTEGER, \ + value TEXT, \ PRIMARY KEY (version, pkgarch, checksum));" % self.table) def _execute(self, *query):@@ -119,84 +133,67 @@ class PRTable(object): data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table), (version, pkgarch)) row = data.fetchone() - if row is not None: + # With SELECT max() requests, you have an empty row when there are no values, therefore the test on row[0] + if row is not None and row[0] is not None: return row[0] else: return None - def _get_value_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] + def find_new_subvalue(self, version, pkgarch, base): + """Take and increase the greatest "<base>.y" value for (version, pkgarch), or return "<base>.1" if not found. + This doesn't store a new value.""" + + data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base), + (version, pkgarch)) + row = data.fetchone() + # With SELECT max() requests, you have an empty row when there are no values, therefore the test on row[0] + if row is not None and row[0] is not None: + return increase_revision(row[0]) else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table), - (version, pkgarch)) - row = data.fetchone() - if row is not None: - return row[0] - else: - return 0 + return base + ".0" - try: - self._execute("INSERT INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) + def store_value(self, version, pkgarch, checksum, value): + """Store new value in the database""" - self.dirty = True + try: + self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table), + (version, pkgarch, checksum, value)) + except sqlite3.IntegrityError as exc: + logger.error(str(exc)) - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + self.dirty = True - def _get_value_no_hist(self, version, pkgarch, checksum): - data=self._execute("SELECT value FROM %s \ - WHERE version=? AND pkgarch=? AND checksum=? AND \ - value >= (select max(value) from %s where version=? AND pkgarch=?);" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - row=data.fetchone() - if row is not None: - return row[0] - else: - #no value found, try to insert - if self.read_only: - data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table), - (version, pkgarch)) - return data.fetchone()[0] + def _get_value(self, version, pkgarch, checksum): - try: - self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));" - % (self.table, self.table), - (version, pkgarch, checksum, version, pkgarch)) - except sqlite3.IntegrityError as exc: - logger.error(str(exc)) - self.conn.rollback() + max_value = self.find_max_value(version, pkgarch) - self.dirty = True + if max_value is None: + # version, pkgarch completely unknown. Return initial value. + return "0" - data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table, - (version, pkgarch, checksum)) - row=data.fetchone() - if row is not None: - return row[0] - else: - raise prserv.NotFoundError + value = self.find_value(version, pkgarch, checksum) + + if value is None: + # version, pkgarch found but not checksum. Create a new value from the maximum one + return increase_revision(max_value) - def get_value(self, version, pkgarch, checksum): if self.nohist: - return self._get_value_no_hist(version, pkgarch, checksum) + # "no-history" mode: only return a value if that's the maximum one for + # the version and architecture, otherwise create a new one. + # This means that the value cannot decrement. + if value == max_value: + return value + else: + return increase_revision(max_value) else: - return self._get_value_hist(version, pkgarch, checksum) + # "hist" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + + def get_value(self, version, pkgarch, checksum): + value = self._get_value(version, pkgarch, checksum) + self.store_value(version, pkgarch, checksum, value) + return value def _import_hist(self, version, pkgarch, checksum, value): if self.read_only:lib/prserv/serv.py +47
-6@@ -12,6 +12,7 @@ import sqlite3 import prserv import prserv.db import errno +from . import create_async_client import bb.asyncrpc logger = logging.getLogger("BitBake.PRserv")@@ -76,14 +77,76 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): pkgarch = request["pkgarch"] checksum = request["checksum"] - response = None - try: + if self.upstream_client is None: value = self.server.table.get_value(version, pkgarch, checksum) - response = {"value": value} - except prserv.NotFoundError: - self.logger.error("failure storing value in database for (%s, %s)",version, checksum) + return {"value": value} - return response + # We have an upstream server. + # Check whether the local server already knows the requested configuration + # Here we use find_value(), not get_value(), because we don't want + # to unconditionally add a new generated value to the database. If the configuration + # is a new one, the generated value we will add will depend on what's on the upstream server. + + value = self.server.table.find_value(version, pkgarch, checksum) + + if value is not None: + + # The configuration is already known locally. Let's use it. + + return {"value": value} + + # The configuration is a new one for the local server + # Let's ask the upstream server whether it knows it + + known_upstream = await self.upstream_client.test_package(version, pkgarch) + + if not known_upstream: + + # The package is not known upstream, must be a local-only package + # Let's compute the PR number using the local-only method + + value = self.server.table.get_value(version, pkgarch, checksum) + return {"value": value} + + # The package is known upstream, let's ask the upstream server + # whether it knows our new output hash + + value = await self.upstream_client.test_pr(version, pkgarch, checksum) + + if value is not None: + + # Upstream knows this output hash, let's store it and use it too. + + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum, value) + # If the local server is read only, won't be able to store the new + # value in the database and will have to keep asking the upstream server + + return {"value": value} + + # The output hash doesn't exist upstream, get the most recent number from upstream (x)
lib/prserv/serv.py class PRServerClient(bb.asyncrpc.AsyncServerConnection): · line 137
Quoted 10 lines
lib/prserv/serv.py +10
-0+ # Then, we want to have a new PR value for the local server: x.y + + upstream_max = await self.upstream_client.max_package_pr(version, pkgarch) + # Here we know that the package is known upstream, so upstream_max can't be None + subvalue = self.server.table.find_new_subvalue(version, pkgarch, upstream_max) + + if not self.server.read_only: + self.server.table.store_value(version, pkgarch, checksum, subvalue) + + return {"value": subvalue}
Keep in mind that this code is effectively multithreaded. I can't tell if you'll be having race problems here due to simultaneous accesses, but and sort of workflows where you read something, then update the database based on the read value can have weird race conditions, especially when dealing with many clients at once
Quoted 106 lines
lib/prserv/serv.py +26
-7+ + async def process_requests(self): + if self.server.upstream is not None: + self.upstream_client = await create_async_client(self.server.upstream) + else: + self.upstream_client = None + + try: + await super().process_requests() + finally: + if self.upstream_client is not None: + await self.upstream_client.close() async def handle_import_one(self, request): response = None@@ -117,11 +180,12 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): return {"readonly": self.server.read_only} class PRServer(bb.asyncrpc.AsyncServer): - def __init__(self, dbfile, read_only=False): + def __init__(self, dbfile, read_only=False, upstream=None): super().__init__(logger) self.dbfile = dbfile self.table = None self.read_only = read_only + self.upstream = upstream def accept_client(self, socket): return PRServerClient(socket, self)@@ -134,6 +198,9 @@ class PRServer(bb.asyncrpc.AsyncServer): self.logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" % (self.dbfile, self.address, str(os.getpid()))) + if self.upstream is not None: + self.logger.info("And upstream PRServer: %s " % (self.upstream)) + return tasks async def stop(self):@@ -147,14 +214,15 @@ class PRServer(bb.asyncrpc.AsyncServer): self.table.sync() class PRServSingleton(object): - def __init__(self, dbfile, logfile, host, port): + def __init__(self, dbfile, logfile, host, port, upstream): self.dbfile = dbfile self.logfile = logfile self.host = host self.port = port + self.upstream = upstream def start(self): - self.prserv = PRServer(self.dbfile) + self.prserv = PRServer(self.dbfile, upstream=self.upstream) self.prserv.start_tcp_server(socket.gethostbyname(self.host), self.port) self.process = self.prserv.serve_as_process(log_level=logging.WARNING)@@ -233,7 +301,7 @@ def run_as_daemon(func, pidfile, logfile): os.remove(pidfile) os._exit(0) -def start_daemon(dbfile, host, port, logfile, read_only=False): +def start_daemon(dbfile, host, port, logfile, read_only=False, upstream=None): ip = socket.gethostbyname(host) pidfile = PIDPREFIX % (ip, port) try:@@ -249,7 +317,7 @@ def start_daemon(dbfile, host, port, logfile, read_only=False): dbfile = os.path.abspath(dbfile) def daemon_main(): - server = PRServer(dbfile, read_only=read_only) + server = PRServer(dbfile, read_only=read_only, upstream=upstream) server.start_tcp_server(ip, port) server.serve_forever()@@ -336,6 +404,9 @@ def auto_start(d): host = host_params[0].strip().lower() port = int(host_params[1]) + + upstream = d.getVar("PRSERV_UPSTREAM") or None + if is_local_special(host, port): import bb.utils cachedir = (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE"))@@ -350,7 +421,7 @@ def auto_start(d): auto_shutdown() if not singleton: bb.utils.mkdirhier(cachedir) - singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port) + singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port, upstream) singleton.start() if singleton: host = singleton.host --2.34.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16086): https://lists.openembedded.org/g/bitbake-devel/message/16086
Mute This Topic: https://lists.openembedded.org/mt/105479101/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH 12/12 prserv: add "history" argument to "get-pr" request
In reply to [email protected]From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
--- lib/prserv/client.py | 4 ++-- lib/prserv/db.py | 16 ++++++++-------- lib/prserv/serv.py | 5 +++-- 3 files changed, 13 insertions(+), 12 deletions(-)
lib/prserv/client.py +2 -2
@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): super().__init__("PRSERVICE", "1.0", logger) - async def getPR(self, version, pkgarch, checksum): + async def getPR(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} ) if response: return response["value"]
lib/prserv/db.py +8 -8
@@ -163,7 +163,7 @@ class PRTable(object): self.dirty = True - def _get_value(self, version, pkgarch, checksum): + def _get_value(self, version, pkgarch, checksum, history): max_value = self.find_max_value(version, pkgarch)
@@ -177,7 +177,11 @@ class PRTable(object): # version, pkgarch found but not checksum. Create a new value from the maximum one return increase_revision(max_value) - if self.nohist: + if history: + # "history" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + else: # "no-history" mode: only return a value if that's the maximum one for # the version and architecture, otherwise create a new one. # This means that the value cannot decrement.
@@ -185,13 +189,9 @@ class PRTable(object): return value else: return increase_revision(max_value) - else: - # "hist" mode: we found an existing value. We can return it - # whether it's the maximum one or not. - return value - def get_value(self, version, pkgarch, checksum): - value = self._get_value(version, pkgarch, checksum) + def get_value(self, version, pkgarch, checksum, history): + value = self._get_value(version, pkgarch, checksum, history) self.store_value(version, pkgarch, checksum, value) return value
lib/prserv/serv.py +3 -2
@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] if self.upstream_client is None: - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # We have an upstream server.
@@ -105,7 +106,7 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): # The package is not known upstream, must be a local-only package # Let's compute the PR number using the local-only method - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value} # The package is known upstream, let's ask the upstream server
Signature · 1 line
2.34.1
Re: [bitbake-devel] [PATCH 12/12] prserv: add "history" argument to "get-pr" request
I can't say that I understand the difference between history and no-history from
just reading this series .. but I realize it isn't something being added as part of
this feature addition.
I just wanted to comment that from the name "history" alone, I can't figure out what it is doing and how it impacts PR values.
Bruce
Quoted 124 lines · Michael Opdenacker via lists.openembedded.org
From: Michael Opdenacker <[email protected]>
This makes it possible to test the "history" mode in the future bitbake selftests, to make sure that both "history" and "no history" modes work as expected.
Signed-off-by Michael Opdenacker <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Joshua Watt <[email protected]>Cc Tim Orling <[email protected]>--- lib/prserv/client.py | 4 ++-- lib/prserv/db.py | 16 ++++++++-------- lib/prserv/serv.py | 5 +++-- 3 files changed, 13 insertions(+), 12 deletions(-)lib/prserv/client.py +2
-2@@ -14,9 +14,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient): def __init__(self): super().__init__("PRSERVICE", "1.0", logger) - async def getPR(self, version, pkgarch, checksum): + async def getPR(self, version, pkgarch, checksum, history=False): response = await self.invoke( - {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}} + {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}} )if response:
return response["value"]lib/prserv/db.py +1
-1@@ -163,7 +163,7 @@ class PRTable(object): self.dirty = True - def _get_value(self, version, pkgarch, checksum): + def _get_value(self, version, pkgarch, checksum, history): max_value = self.find_max_value(version, pkgarch)@@ -177,7 +177,11 @@ class PRTable(object): # version, pkgarch found but not checksum. Create a new valuefrom the maximum one
returnincrease_revision(max_value)lib/prserv/db.py +5
-1- if self.nohist: + if history: + # "history" mode: we found an existing value. We can return it + # whether it's the maximum one or not. + return value + else: # "no-history" mode: only return a value if that's themaximum one for
# the version and architecture, otherwise create a new one. # This means that the value cannot decrement.lib/prserv/db.py +2
-6@@ -185,13 +189,9 @@ class PRTable(object): return value else: return increase_revision(max_value) - else: - # "hist" mode: we found an existing value. We can return it - # whether it's the maximum one or not. - return value - def get_value(self, version, pkgarch, checksum): - value = self._get_value(version, pkgarch, checksum) + def get_value(self, version, pkgarch, checksum, history): + value = self._get_value(version, pkgarch, checksum, history) self.store_value(version, pkgarch, checksum, value) return valuelib/prserv/serv.py +2
-1@@ -76,9 +76,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection): version = request["version"] pkgarch = request["pkgarch"] checksum = request["checksum"] + history = request["history"] if self.upstream_client is None: - value = self.server.table.get_value(version, pkgarch, checksum) + value = self.server.table.get_value(version, pkgarch, checksum, history) return {"value": value}# We have an upstream server.
lib/prserv/serv.py +0
-0@@ -105,7 +106,7 @@ classPRServerClient(bb.asyncrpc.AsyncServerConnection):
# The package is not known upstream, must be a local-only
package
# Let's compute the PR number using the local-only method- value = self.server.table.get_value(version, pkgarch,
checksum)
+ value = self.server.table.get_value(version, pkgarch,
checksum, history)
return {"value": value}# The package is known upstream, let's ask the upstream server
--
2.34.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#16087):
https://lists.openembedded.org/g/bitbake-devel/message/16087
Mute This Topic: https://lists.openembedded.org/mt/105479102/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [bitbake-devel] [PATCH 12/12] prserv: add "history" argument to "get-pr" request
Hi Bruce,
Many thanks for your review, much appreciated!
Quoted 6 lines · Bruce Ashfield
I can't say that I understand the difference between history and no-history from just reading this series .. but I realize it isn't something being added as part of this feature addition.
I just wanted to comment that from the name "history" alone, I can't figure out what it is doing and how it impacts PR values.
You'll find details in the code:
https://git.openembedded.org/bitbake/tree/lib/prserv/db.py#n25
Indeed, these modes should be documented. For the moment, only the "No history" mode was used, that's why no document exists yet. Cheers
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 12 lines · Michael Opdenacker via lists.openembedded.org
From: Michael Opdenacker <[email protected]>
This makes it possible to customize an "upstream" distribution by modifying local packages. If the "upstream" package bears revision "x", the local one will have revision "x.y", this having priority over the upstream one.
Multiple levels of upstream servers are supported, so "x.y.z" revisions are possible too.
Take advantage of this work to clean-up and update the prserv code too.
Quoted 9 lines
Manual "sanity" tests have been run so far but automated tests (based on Python unittest, as for the hash server tests) are being prepared for the next iteration. They will also to test also supported scenarios.
Code reviews are welcome, especially for more Pythonic ways of implementing some details.
I went through the code and commented on a few of the larger patches.
I can't say that I'm an expert in the existing PR server and I don't tend to write overly pythonic code .. but I did look through things and can add:
Quoted 90 lines
---
Changes in V3:
- Revert the commit removing the so far unused "hist" mode, which we wish to keep for binary reproducibility sake.
- Simplification of
get_value()function to take both "hist" and "nohist" modes with the same shared code.- Add "history" parameter to the "getPR" request, so that the client can ask for the mode of its choice. This will also make it possible to implement tests for both modes.
Note that more requests ("export", "import"...) will also need a "history" parameter, in a future version, after the first tests are implemented.
- Several fixes to bugs either present in V2 or introduced by the V3 code changes.
- Put all the new features at the tip of the branch, to make the cleanup commits easier to merged.
Changes in V2:
- Add this new commit:
prserv: remove unused "hist" mode in the database backend- Squash commit "prserv: fix
read_onlytest" into commit "prserv: simplify thePRServerClient()interface" (Reported by Richard Purdie)- Fix the code to support increasing "x.y.z" values, thus supporting several levels of upstream servers.
-
db.py: remove duplicate definition offind_max_value()function indb.py-
prserv.py: remove tabs before comments (Python didn't complain)-
db.py: now stores the revision ("value") as TEXT. This way we can store "1.0" without having it transformed to "1" when the default type was INTEGER.- This allows to fix a regression when the first packages were created with 'r0.1' instead of 'r0.0' initially.
-
find_max_value: now returns None instead of '0' when no value is found Before we couldn't tell the difference between a '0'
max value and the absence of such a value.Michael Opdenacker (12):
prserv: simplify thePRServerClient()interface
prserv: use double quotes by default
bitbake-prserv: replace deprecated optparse by argparse
prserv: use self.logger instead of logger directly
asyncrpc: includeparse_addressfrom hashserv
prserv: capitalization and spacing improvements
prserv: add extra requests
prserv: remove redundant exception handler
prserv: correct error message
prserv: remove unnecessary code
prserv: add "upstream" server support
prserv: add "history" argument to "get-pr" requestbin/bitbake-prserv | 99 ++++++++++++----- lib/bb/asyncrpc/client.py | 23 ++++ lib/hashserv/__init__.py | 27 +---- lib/prserv/__init__.py | 21 +++- lib/prserv/client.py | 44 ++++++-- lib/prserv/db.py | 225 ++++++++++++++++++++++---------------- lib/prserv/serv.py | 220 ++++++++++++++++++++++++++----------- 7 files changed, 436 insertions(+), 223 deletions(-)--
2.34.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#16075):
https://lists.openembedded.org/g/bitbake-devel/message/16075
Mute This Topic: https://lists.openembedded.org/mt/105479088/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II