network based PR service
cover network based PR service
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled the poky to use it during the task do_package and do_package_write_xxx. By using the network based PR service and the basichash for BB_SIGNATURE_HANDLER, the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number reflecting the change will be included in the package feed.
The first patch "[PATCH 1/5] Added the PR service." implemented the network based PR service on the server side. It is a XMLRPC server with the sqlite3 as the backend database. The users query an automated value from this service by giving a tuple of (version, checksum). The returned value will be used as a part of the package revision value during the package feed creation process. The following algorihtm is used so this PR service could give multiple build servers new values for the new index tuple and same values for the same index tuple:
IF the index tuple(version, checksum) is found, THEN
return the value.
ELSE
find the max value in the database with the same version. increment it and save it into the database along with the index tuple. return the incremented value.
(Note: If no matching version is found, save and return the value of 0).
ENDIF
To start the network based PR service daemon, run the following command after "sourcing" the environment file oe-init-build-env:
bitbake-prserv --start
See bitbake-prserv --help to see a detailed description of the options.
The remaining 4 patches enable the poky to use the PR service. In order to use it, the user needs to set 2 varialbes of PRSERV_HOST and PRSERV_PORT (which specify the PR service ip address and port, default is "localhost" and "8585" respectively) in the file conf/local.conf under the building directory. Unsetting these 2 variables would disable the poky to use the PR service for backward compatibility.
The following important new variables are introduced in bitbake.conf:
PRFORMAT: The final value goes into the package revision. Default is something like PR.PRAUTO, where PRAUTO is the value returned by the PR service.
PRAUTOINX: The version part of the index tuple for the PR service. Default is PF. The checksum of that tuple is the task checksum of do_package.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: llu/PR-service
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PR-service
Thanks,
Lianhao Lu <[email protected]>
---
Lianhao Lu (5):
Added the PR service.
conf/bitbake.conf: Added variables for PR service. classes/package(prserv).bbclass: Added PR service support. classes/package_xxx.class: Added PR service support. meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT.
bitbake/bin/bitbake-prserv | 53 ++++++++++ bitbake/lib/prserv/__init__.py | 11 ++ bitbake/lib/prserv/db.py | 100 +++++++++++++++++++ bitbake/lib/prserv/serv.py | 198 +++++++++++++++++++++++++++++++++++++ meta-yocto/conf/local.conf.sample | 5 + meta/classes/package.bbclass | 36 +++++-- meta/classes/package_deb.bbclass | 6 +- meta/classes/package_ipk.bbclass | 6 +- meta/classes/package_rpm.bbclass | 4 +- meta/classes/package_tar.bbclass | 4 +- meta/classes/prserv.bbclass | 29 ++++++ meta/conf/bitbake.conf | 14 +++- 12 files changed, 443 insertions(+), 23 deletions(-) create mode 100755 bitbake/bin/bitbake-prserv create mode 100644 bitbake/lib/prserv/__init__.py create mode 100644 bitbake/lib/prserv/db.py create mode 100644 bitbake/lib/prserv/serv.py create mode 100644 meta/classes/prserv.bbclass
PATCH 1/5 Added the PR service.
From: Lianhao Lu <[email protected]>
Added the initial implementation of the server side PR service.
--- bitbake/bin/bitbake-prserv | 53 +++++++++++ bitbake/lib/prserv/__init__.py | 11 ++ bitbake/lib/prserv/db.py | 100 ++++++++++++++++++++ bitbake/lib/prserv/serv.py | 198 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 362 insertions(+), 0 deletions(-) create mode 100755 bitbake/bin/bitbake-prserv create mode 100644 bitbake/lib/prserv/__init__.py create mode 100644 bitbake/lib/prserv/db.py create mode 100644 bitbake/lib/prserv/serv.py
bitbake/bin/bitbake-prserv new file +53 -0
@@ -0,0 +1,53 @@ +#!/usr/bin/env python +import os +import sys,logging +import optparse + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib')) + +import prserv +import prserv.serv + +__version__="1.0.0" + +PRHOST_DEFAULT='' +PRPORT_DEFAULT=8585 + +def main(): + parser = optparse.OptionParser( + version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__), + usage = "%prog [options]") + + parser.add_option("-f", "--file", help="database filename(default prserv.db)", action="store", + dest="dbfile", type="string", default="prserv.db") + parser.add_option("-l", "--log", help="log filename(default prserv.log)", action="store", + dest="logfile", type="string", default="prserv.log") + parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG", + action = "store", type="string", dest="loglevel", default = "WARNING") + parser.add_option("--start", help="start daemon", + action="store_true", dest="start", default="True") + parser.add_option("--stop", help="stop daemon", + action="store_false", dest="start") + parser.add_option("--host", help="ip address to bind", action="store", + dest="host", type="string", default=PRHOST_DEFAULT) + parser.add_option("--port", help="port number(default 8585)", action="store", + dest="port", type="int", default=PRPORT_DEFAULT) + + options, args = parser.parse_args(sys.argv) + + prserv.init_logger(os.path.abspath(options.logfile),options.loglevel) + + if options.start: + prserv.serv.start_daemon(options) + else: + prserv.serv.stop_daemon() + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + traceback.print_exc(5) + sys.exit(ret) +
bitbake/lib/prserv/__init__.py new file +11 -0
@@ -0,0 +1,11 @@ +__version__ = "1.0.0" + +import os, time +import sys,logging + +def init_logger(logfile, loglevel): + numeric_level = getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError('Invalid log level: %s' % loglevel) + logging.basicConfig(level=numeric_level, filename=logfile) +
bitbake/lib/prserv/db.py new file +100 -0
100 lines not on this page · raw message
bitbake/lib/prserv/serv.py new file +198 -0
198 lines not on this page · raw message
Signature · 1 line
1.7.0.4
PATCH 2/5 conf/bitbake.conf: Added variables for PR service.
In reply to Lianhao LuFrom: Lianhao Lu <[email protected]>
1. change BB_SIGNATURE_HANDLER from basic to basichash.
2. Added following variables for PR service:USE_PR_SERV: flag of whether to use the network PR service
PRFORMAT: format of revision to be used in tasks package_write_xxx.
PRAUTOINX: search index for the network PR service
--- meta/conf/bitbake.conf | 14 ++++++++++++-- 1 files changed, 12 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +12 -2
@@ -153,8 +153,13 @@ PR = "${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[2] or 'r0 PF = "${PN}-${EXTENDPE}${PV}-${PR}" EXTENDPE = "${@['','${PE\x7d_'][bb.data.getVar('PE',d,1) > 0]}" EXTENDPEVER = "${@['','${PE\x7d:'][bb.data.getVar('PE',d,1) > 0]}" -EXTENDPV = "${EXTENDPEVER}${PV}-${PR}" +EXTENDPV = "${EXTENDPEVER}${PV}-${PRFORMAT}" P = "${PN}-${PV}" +EXTENDPREXTRA = "${@['.${PREXTRA\x7d',''][bb.data.getVar('PREXTRA',d,1) is None]}" +EXTENDPRAUTO = "${@['.${PRAUTO\x7d',''][bb.data.getVar('PRAUTO',d,1) is None]}" +PRAUTOINX = "${PF}${EXTENDPREXTRA}" +PRFORMAT = "${PR}${EXTENDPREXTRA}${EXTENDPRAUTO}" + # Base package name # Automatically derives "foo" from "foo-native", "foo-cross" or "foo-initial"
@@ -163,6 +168,11 @@ SPECIAL_PKGSUFFIX = "-native -cross -initial -intermediate -nativesdk -crosssdk BPN = "${@base_prune_suffix(bb.data.getVar('PN', d, True), bb.data.getVar('SPECIAL_PKGSUFFIX', d, True).split(), d)}" BP = "${BPN}-${PV}" +# +# network based PR service +# +USE_PR_SERV = "${@[1,0][(bb.data.getVar('PRSERV_HOST',d,1) is None) or (bb.data.getVar('PRSERV_PORT',d,1) is None)]}" + # Package info. SECTION = "base"
@@ -707,7 +717,7 @@ DISTRO[unexport] = "1" TRANSLATED_TARGET_ARCH ??= ${TARGET_ARCH} # Setup our default hash policy -BB_SIGNATURE_HANDLER ?= "basic" +BB_SIGNATURE_HANDLER ?= "basichash" BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)" BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE"
Signature · 1 line
1.7.0.4
Re: [PATCH 2/5] conf/bitbake.conf: Added variables for PR service.
Quoted 20 lines · Lianhao Lu
From: Lianhao Lu <[email protected]>
1. change
BB_SIGNATURE_HANDLERfrom basic to basichash.2. Added following variables for PR service:
USE_PR_SERV: flag of whether to use the network PR service
PRFORMAT: format of revision to be used in taskspackage_write_xxx.
PRAUTOINX: search index for the network PR serviceSigned-off-by Lianhao Lu <[email protected]>--- meta/conf/bitbake.conf | 14 ++++++++++++-- 1 files changed, 12 insertions(+), 2 deletions(-)meta/conf/bitbake.conf +0
-0@@ -153,8 +153,13 @@ PR = "${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[2] or 'r0 PF = "${PN}-${EXTENDPE}${PV}-${PR}"
meta/conf/bitbake.conf PR = "${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[2] or 'r0 · line 162
Quoted 10 lines
meta/conf/bitbake.conf +6
-1EXTENDPE = "${@['','${PE\x7d_'][bb.data.getVar('PE',d,1) > 0]}" EXTENDPEVER = "${@['','${PE\x7d:'][bb.data.getVar('PE',d,1) > 0]}" -EXTENDPV = "${EXTENDPEVER}${PV}-${PR}" +EXTENDPV = "${EXTENDPEVER}${PV}-${PRFORMAT}" P = "${PN}-${PV}" +EXTENDPREXTRA = "${@['.${PREXTRA\x7d',''][bb.data.getVar('PREXTRA',d,1) is None]}" +EXTENDPRAUTO = "${@['.${PRAUTO\x7d',''][bb.data.getVar('PRAUTO',d,1) is None]}" +PRAUTOINX = "${PF}${EXTENDPREXTRA}" +PRFORMAT = "${PR}${EXTENDPREXTRA}${EXTENDPRAUTO}" +
There is also one thing we need to sync up with regarding openembedded and these patches which is the use of PKGV and PKGR in package.bbclass. Those patches are not merged into OE-Core yet but are have a similar intention of allowing customisation to the PV and PR fields. I would like to see if we can use PKGR for the use case we have here as well.
See:
http://git.openembedded.net/cgit.cgi/openembedded/tree/classes/package.bbclass
(I was only reminded of this recently in an IRC conversation with Otavio)
Minor nitpick but new code should replace:
bb.data.getVar('PRAUTO',d,1)
with
d.getVar('PRAUTO', True)
which is functionally equivalent, neater and more pythonic. Its not a major issue but worth highlighting.
Cheers,
Richard
PATCH 3/5 classes/package(prserv).bbclass: Added PR service support.
In reply to Lianhao LuFrom: Lianhao Lu <[email protected]>
1. Added package_get_auto_rev to PACKAGEFUNCS to get the auto incremented value(PRAUTO) from remote PR service.
2. Save PRFORMAT to pkgdata to be used by package_write_xxx.
3. Added supporting functions in prserv.bbclass.
--- meta/classes/package.bbclass | 36 +++++++++++++++++++++++++----------- meta/classes/prserv.bbclass | 29 +++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 11 deletions(-) create mode 100644 meta/classes/prserv.bbclass
meta/classes/package.bbclass +25 -11
@@ -7,34 +7,37 @@ # # There are the following default steps but PACKAGEFUNCS can be extended: # -# a) perform_packagecopy - Copy D into PKGD +# a) package_get_auto_rev - get PRAUTO from remote PR service # -# b) package_do_split_locales - Split out the locale files, updates FILES and PACKAGES +# b) perform_packagecopy - Copy D into PKGD # -# c) split_and_strip_files - split the files into runtime and debug and strip them. +# c) package_do_split_locales - Split out the locale files, updates FILES and PACKAGES +# +# d) split_and_strip_files - split the files into runtime and debug and strip them. # Debug files include debug info split, and associated sources that end up in -dbg packages # -# d) populate_packages - Split the files in PKGD into separate packages in PKGDEST/<pkgname> +# e) populate_packages - Split the files in PKGD into separate packages in PKGDEST/<pkgname> # Also triggers the binary stripping code to put files in -dbg packages. # -# e) package_do_filedeps - Collect perfile run-time dependency metadata +# f) package_do_filedeps - Collect perfile run-time dependency metadata # The data is stores in FILER{PROVIDES,DEPENDS}_file_pkg variables with # a list of affected files in FILER{PROVIDES,DEPENDS}FLIST_pkg # -# f) package_do_shlibs - Look at the shared libraries generated and autotmatically add any +# g) package_do_shlibs - Look at the shared libraries generated and autotmatically add any # depenedencies found. Also stores the package name so anyone else using this library # knows which package to depend on. # -# g) package_do_pkgconfig - Keep track of which packages need and provide which .pc files +# h) package_do_pkgconfig - Keep track of which packages need and provide which .pc files # -# h) read_shlibdeps - Reads the stored shlibs information into the metadata +# i) read_shlibdeps - Reads the stored shlibs information into the metadata # -# i) package_depchains - Adds automatic dependencies to -dbg and -dev packages +# j) package_depchains - Adds automatic dependencies to -dbg and -dev packages # -# j) emit_pkgdata - saves the packaging data into PKGDATA_DIR for use in later +# k) emit_pkgdata - saves the packaging data into PKGDATA_DIR for use in later # packaging steps inherit packagedata +inherit prserv PKGD = "${WORKDIR}/package" PKGDEST = "${WORKDIR}/packages-split"
@@ -324,6 +327,15 @@ def runtime_mapping_rename (varname, d): # Package functions suitable for inclusion in PACKAGEFUNCS # +python package_get_auto_rev() { + if bb.data.getVar('USE_PR_SERV', d, True): + auto_rev=get_auto_rev(d) + if auto_rev is None: + bb.fatal("Can NOT get auto revision from remote PR service") + return + bb.data.setVar('PRAUTO',str(auto_rev),d) +} + python package_do_split_locales() { if (bb.data.getVar('PACKAGE_NO_LOCALE', d, True) == '1'): bb.debug(1, "package requested not splitting locales")
@@ -771,6 +783,7 @@ python emit_pkgdata() { write_if_exists(sf, pkg, 'PN') write_if_exists(sf, pkg, 'PV') write_if_exists(sf, pkg, 'PR') + write_if_exists(sf, pkg, 'PRFORMAT') write_if_exists(sf, pkg, 'DESCRIPTION') write_if_exists(sf, pkg, 'SUMMARY') write_if_exists(sf, pkg, 'RDEPENDS')
@@ -1346,7 +1359,8 @@ python package_depchains() { } PACKAGE_PREPROCESS_FUNCS ?= "" -PACKAGEFUNCS ?= "perform_packagecopy \ +PACKAGEFUNCS ?= "package_get_auto_rev \ + perform_packagecopy \ ${PACKAGE_PREPROCESS_FUNCS} \ package_do_split_locales \ split_and_strip_files \
meta/classes/prserv.bbclass new file +29 -0
@@ -0,0 +1,29 @@ +def make_conn(d): + import prserv.serv + host=bb.data.getVar("PRSERV_HOST",d,True) + port=bb.data.getVar("PRSERV_PORT",d,True) + try: + conn=None + conn=prserv.serv.PRServerConnection(host,int(port)) + bb.data.setVar("__PRSERV_CONN",conn,d) + except Exception, exc: + bb.fatal("Connecting to PR service %s:%s failed: %s" % (host, port, str(exc))) + + return conn + +def get_auto_rev(d): + if not bb.data.getVar('USE_PR_SERV', d, True): + bb.warn("Not using network based PR service") + return None + + conn=bb.data.getVar("__PRSERV_CONN", d, True) + if conn is None: + conn=make_conn(d) + if conn is None: + return None + + version=bb.data.getVar("PF", d, True) + checksum=bb.data.getVar("BB_TASKHASH", d, True) + auto_rev=conn.getPR(version,checksum) + bb.debug(1,"get_auto_rev: version: %s checksum: %s result %d" % (version, checksum, auto_rev)) + return auto_rev
Signature · 1 line
1.7.0.4
Re: [PATCH 3/5] classes/package(prserv).bbclass: Added PR service support.
Quoted 26 lines · Lianhao Lu
From: Lianhao Lu <[email protected]>
1. Added
package_get_auto_revto PACKAGEFUNCS to get the auto incremented value(PRAUTO) from remote PR service.2. Save PRFORMAT to pkgdata to be used by
package_write_xxx.3. Added supporting functions in prserv.bbclass.
Signed-off-by Lianhao Lu <[email protected]>--- meta/classes/package.bbclass | 36 +++++++++++++++++++++++++----------- meta/classes/prserv.bbclass | 29 +++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+), 11 deletions(-) create mode 100644 meta/classes/prserv.bbclassmeta/classes/package.bbclass +2
-0@@ -324,6 +327,15 @@ def runtime_mapping_rename (varname, d): # Package functions suitable for inclusion in PACKAGEFUNCS # +python package_get_auto_rev() { + if bb.data.getVar('USE_PR_SERV', d, True):
meta/classes/package.bbclass def runtime_mapping_rename (varname, d): · line 341
Quoted 10 lines
meta/classes/package.bbclass +7
-0+ auto_rev=get_auto_rev(d) + if auto_rev is None: + bb.fatal("Can NOT get auto revision from remote PR service") + return + bb.data.setVar('PRAUTO',str(auto_rev),d) +} + python package_do_split_locales() { if (bb.data.getVar('PACKAGE_NO_LOCALE', d, True) == '1'): bb.debug(1, "package requested not splitting locales")
This looks a little confusing to me as at first glance I'd have said it was related to SRCREV auto incrementing. Can we add something to to with pr to the function name? Function names like get_auto_rev() and make_conf() also need clearer namespacing to include prserv as a prefix so its clearer where the functions are from and what they do.
Cheers,
Richard
PATCH 4/5 classes/package_xxx.class: Added PR service support.
In reply to Lianhao LuFrom: Lianhao Lu <[email protected]>
Use PRFORMAT instead of PR to build various package feed in tasks of pacakge_write_xxx.
--- meta/classes/package_deb.bbclass | 6 +++--- meta/classes/package_ipk.bbclass | 6 +++--- meta/classes/package_rpm.bbclass | 4 ++-- meta/classes/package_tar.bbclass | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-)
meta/classes/package_deb.bbclass +3 -3
@@ -272,7 +272,7 @@ python do_package_deb () { except ValueError: pass if not g and bb.data.getVar('ALLOW_EMPTY', localdata) != "1": - bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, True), bb.data.getVar('PR', localdata, True))) + bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, True), bb.data.getVar('PRFORMAT', localdata, True))) bb.utils.unlockfile(lf) continue
@@ -290,9 +290,9 @@ python do_package_deb () { fields = [] pe = bb.data.getVar('PE', d, True) if pe and int(pe) > 0: - fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PR']]) + fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PRFORMAT']]) else: - fields.append(["Version: %s-%s\n", ['PV', 'PR']]) + fields.append(["Version: %s-%s\n", ['PV', 'PRFORMAT']]) fields.append(["Description: %s\n", ['DESCRIPTION']]) fields.append(["Section: %s\n", ['SECTION']]) fields.append(["Priority: %s\n", ['PRIORITY']])
meta/classes/package_ipk.bbclass +3 -3
@@ -241,7 +241,7 @@ python do_package_ipk () { except ValueError: pass if not g and bb.data.getVar('ALLOW_EMPTY', localdata) != "1": - bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PR', localdata, 1))) + bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PRFORMAT', localdata, 1))) bb.utils.unlockfile(lf) continue
@@ -256,9 +256,9 @@ python do_package_ipk () { fields = [] pe = bb.data.getVar('PE', d, 1) if pe and int(pe) > 0: - fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PR']]) + fields.append(["Version: %s:%s-%s\n", ['PE', 'PV', 'PRFORMAT']]) else: - fields.append(["Version: %s-%s\n", ['PV', 'PR']]) + fields.append(["Version: %s-%s\n", ['PV', 'PRFORMAT']]) fields.append(["Description: %s\n", ['DESCRIPTION']]) fields.append(["Section: %s\n", ['SECTION']]) fields.append(["Priority: %s\n", ['PRIORITY']])
meta/classes/package_rpm.bbclass +2 -2
@@ -383,7 +383,7 @@ python write_specfile () { srcname = bb.data.getVar('PN', d, True) srcsummary = (bb.data.getVar('SUMMARY', d, True) or bb.data.getVar('DESCRIPTION', d, True) or ".") srcversion = bb.data.getVar('PV', d, True).replace('-', '+') - srcrelease = bb.data.getVar('PR', d, True) + srcrelease = bb.data.getVar('PRFORMAT', d, True) srcepoch = (bb.data.getVar('PE', d, True) or "") srclicense = bb.data.getVar('LICENSE', d, True) srcsection = bb.data.getVar('SECTION', d, True)
@@ -438,7 +438,7 @@ python write_specfile () { splitsummary = (bb.data.getVar('SUMMARY', localdata, True) or bb.data.getVar('DESCRIPTION', localdata, True) or ".") splitversion = (bb.data.getVar('PV', localdata, True) or "").replace('-', '+') - splitrelease = (bb.data.getVar('PR', localdata, True) or "") + splitrelease = (bb.data.getVar('PRFORMAT', localdata, True) or "") splitepoch = (bb.data.getVar('PE', localdata, True) or "") splitlicense = (bb.data.getVar('LICENSE', localdata, True) or "") splitsection = (bb.data.getVar('SECTION', localdata, True) or "")
meta/classes/package_tar.bbclass +2 -2
@@ -3,7 +3,7 @@ inherit package IMAGE_PKGTYPE ?= "tar" python package_tar_fn () { - fn = os.path.join(bb.data.getVar('DEPLOY_DIR_TAR', d), "%s-%s-%s.tar.gz" % (bb.data.getVar('PKG', d), bb.data.getVar('PV', d), bb.data.getVar('PR', d))) + fn = os.path.join(bb.data.getVar('DEPLOY_DIR_TAR', d), "%s-%s-%s.tar.gz" % (bb.data.getVar('PKG', d), bb.data.getVar('PV', d), bb.data.getVar('PRFORMAT', d))) fn = bb.data.expand(fn, d) bb.data.setVar('PKGFN', fn, d) }
@@ -83,7 +83,7 @@ python do_package_tar () { os.chdir(root) from glob import glob if not glob('*'): - bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PR', localdata, 1))) + bb.note("Not creating empty archive for %s-%s-%s" % (pkg, bb.data.getVar('PV', localdata, 1), bb.data.getVar('PRFORMAT', localdata, 1))) continue ret = os.system("tar -czf %s %s" % (tarfn, '.')) if ret != 0:
Signature · 1 line
1.7.0.4
PATCH 5/5 meta-yocto/local.conf.sample: Added PRSERV_HOST and PRSERV_PORT.
In reply to Lianhao LuFrom: Lianhao Lu <[email protected]>
--- meta-yocto/conf/local.conf.sample | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
meta-yocto/conf/local.conf.sample +5 -0
@@ -216,3 +216,8 @@ NO32LIBS = "1" # GNOME, SCREEN, XTERM and KONSOLE #TERMCMD = "${KONSOLE_TERMCMD}" #TERMCMDRUN = "${KONSOLE_TERMCMDRUN}" + + +# The network based PR service host and port +PRSERV_HOST = "localhost" +PRSERV_PORT = "8585"
Signature · 1 line
1.7.0.4
Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
Quoted 8 lines
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled the poky to use it during the task
do_packageanddo_package_write_xxx. By using the network based PR service and the basichash forBB_SIGNATURE_HANDLER, the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number reflecting the change will be included in the package feed.
Does it have a public/private mode? In the angstrom case, we only want developers to 'submit' PR bumps and users only retrieve them.
regards,
Koen
Quoted 11 lines · Koen Kooi
Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
Quoted 8 lines
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled the poky to use it during the task
do_packageanddo_package_write_xxx. By using the network based PR service and the basichash forBB_SIGNATURE_HANDLER, the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number reflecting the change will be included in the package feed.
Quoted 2 lines
Does it have a public/private mode? In the angstrom case, we only want developers to 'submit' PR bumps and users only retrieve them.
It doesn't look like it but it wouldn't be hard to only submit PR changes if some kind of token was present.
The big question would be, if a user tried building something for which PR no record was on the server and didn't have submit privileges, what should the server return and how should the build behave? Hard error?
If its not a hard error, you are likely to run into local reproducibility issues and also conflicts with the server?
Cheers,
Richard
Op 19 mei 2011, om 13:38 heeft Richard Purdie het volgende geschreven:
Quoted 18 lines · Koen Kooi
Quoted 11 lines
Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
Quoted 8 lines
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled the poky to use it during the task
do_packageanddo_package_write_xxx. By using the network based PR service and the basichash forBB_SIGNATURE_HANDLER, the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number reflecting the change will be included in the package feed.Quoted 2 lines
Does it have a public/private mode? In the angstrom case, we only want developers to 'submit' PR bumps and users only retrieve them.
It doesn't look like it but it wouldn't be hard to only submit PR changes if some kind of token was present.
Quoted 6 lines
The big question would be, if a user tried building something for which PR no record was on the server and didn't have submit privileges, what should the server return and how should the build behave? Hard error?
If its not a hard error, you are likely to run into local reproducibility issues and also conflicts with the server?
I would go for a warning on the client side and an error in the log in the server side. If the server people care enough they'll check the log for errors and add the missing info.
I think we need to give this a try and see what improvements we need.
regards,
Koen
Quoted 24 lines · Koen Kooi
Op 19 mei 2011, om 13:38 heeft Richard Purdie het volgende geschreven:
Quoted 22 lines · Koen Kooi
Quoted 11 lines
Op 19 mei 2011, om 12:29 heeft Lianhao Lu het volgende geschreven:
Quoted 8 lines
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled the poky to use it during the task
do_packageanddo_package_write_xxx. By using the network based PR service and the basichash forBB_SIGNATURE_HANDLER, the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number reflecting the change will be included in the package feed.Quoted 2 lines
Does it have a public/private mode? In the angstrom case, we only want developers to 'submit' PR bumps and users only retrieve them.
It doesn't look like it but it wouldn't be hard to only submit PR changes if some kind of token was present.
The big question would be, if a user tried building something for which PR no record was on the server and didn't have submit privileges, what should the server return and how should the build behave? Hard error?
Quoted 6 lines
Quoted 2 lines
If its not a hard error, you are likely to run into local reproducibility issues and also conflicts with the server?
I would go for a warning on the client side and an error in the log in the server side. If the server people care enough they'll check the log for errors and add the missing info.
How is the client meant to continue on when the value it gets from the server is effectively a "don't know" reply? Default to 0?
Thinking about it, its probably fine as long as it defaulted back to "lr" or something to show there was likely a local change that caused it to deviate from what was on the server. It just needs to be clear it doesn't correspond to the upstream server.
Quoted 1 line
I think we need to give this a try and see what improvements we need.
Agreed.
Cheers,
Richard
2011/5/19 Lianhao Lu <[email protected]>
Quoted 99 lines
From: Lianhao Lu <[email protected]>
This series of 5 patches implemented the network based PR service and enabled
the poky to use it during the taskdo_packageanddo_package_write_xxx. By using the network based PR service and the basichash forBB_SIGNATURE_HANDLER,
the poky user may not need to bump the PR manually everytime he/she changes the recipe. The package will get automatically rebuilt and new revision number
reflecting the change will be included in the package feed.The first patch "[PATCH 1/5] Added the PR service." implemented the network based PR service on the server side. It is a XMLRPC server with the sqlite3 as the backend database. The users query an automated value from this service
by giving a tuple of (version, checksum). The returned value will be used as a part of the package revision value during the package feed creation process. The following algorihtm is used so this PR service could give
multiple build servers new values for the new index tuple and same values for
the same index tuple:IF the index
tuple(version, checksum)is found, THEN
return the value.
ELSE
find the max value in the database with the same version. increment it and save it into the database along with the index tuple. return the incremented value.
(Note: If no matching version is found, save and return the value of 0).
ENDIFTo start the network based PR service daemon, run the following command after
"sourcing" the environment file oe-init-build-env:bitbake-prserv --start
See bitbake-prserv --help to see a detailed description of the options.
The remaining 4 patches enable the poky to use the PR service. In order to use
it, the user needs to set 2 varialbes ofPRSERV_HOSTandPRSERV_PORT(which specify the PR service ip address and port, default is "localhost" and "8585"
respectively) in the fileconf/local.confunder the building directory. Unsetting these 2 variables would disable the poky to use the PR service for
backward compatibility.The following important new variables are introduced in bitbake.conf:
PRFORMAT: The final value goes into the package revision. Default is something
like PR.PRAUTO, where PRAUTO is the value returned by the PR service.PRAUTOINX: The version part of the index tuple for the PR service. Default is PF.
The checksum of that tuple is the task checksum ofdo_package.Pull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: llu/PR-service
Browse:
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/PR-serviceThanks,
Lianhao Lu <[email protected]>
---Lianhao Lu (5):
Added the PR service.
conf/bitbake.conf: Added variables for PR service. classes/package(prserv).bbclass: Added PR service support. classes/package_xxx.class: Added PR service support. meta-yocto/local.conf.sample: AddedPRSERV_HOSTandPRSERV_PORT.bitbake/bin/bitbake-prserv | 53 ++++++++++ bitbake/lib/prserv/__init__.py | 11 ++ bitbake/lib/prserv/db.py | 100 +++++++++++++++++++ bitbake/lib/prserv/serv.py | 198 +++++++++++++++++++++++++++++++++++++ meta-yocto/conf/local.conf.sample | 5 + meta/classes/package.bbclass | 36 +++++-- meta/classes/package_deb.bbclass | 6 +- meta/classes/package_ipk.bbclass | 6 +- meta/classes/package_rpm.bbclass | 4 +- meta/classes/package_tar.bbclass | 4 +- meta/classes/prserv.bbclass | 29 ++++++ meta/conf/bitbake.conf | 14 +++- 12 files changed, 443 insertions(+), 23 deletions(-) create mode 100755 bitbake/bin/bitbake-prserv create mode 100644 bitbake/lib/prserv/__init__.py create mode 100644 bitbake/lib/prserv/db.py create mode 100644 bitbake/lib/prserv/serv.py create mode 100644 meta/classes/prserv.bbclass
Quoted 1 line
What should I do to either disable this for some recipes, or use a
different (private) server?
We do not wish to rely on an external server for proprietary recipes (or for recipes for which we made a local change in an overlay).
How is this case handled?
Frans
Quoted 8 lines
Quoted 2 lines
What should I do to either disable this for some recipes, or use a
different (private) server?
We do not wish to rely on an external server for proprietary recipes (or for recipes for which we made a local change in an overlay). How is this case handled?Ok found some of the answers in the patches, but that has raised some more
questions for which I did not see the answer right away.
Would it be possible to have a PR_SERVER per layer?
And are the values cached locally? (e.g. what happens if the PR_SERVER is not availalbe).
I guess it would also be helpful to have some usage documentation. (actually I seem to recall that at some point in time within OE we had the policy that if you introduced a new var you should also update the docs for it; not sure if that was formalized somewhere, but definitely seems a good plan).
Frans
Quoted 5 lines · Frans Meulenbroeks
What should I do to either disable this for some recipes, or use a different (private) server?
We do not wish to rely on an external server for proprietary recipes (or for recipes for which we made a local change in an overlay).
How is this case handled?
This could be handled by doing things like setting:
PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
or
PRSERV_HOST_pn-myprivaterecipe = ""
to disable it.
One thing we lack is a good way to apply changes like this, only if code is within a given layer. Even in that case, it should be possible with anonymous python to look at the location of the current .bb file and then conditionally set these variables as appropriate to the right server.
Certainly this needs to be thought about and documented but I think there are ways to do it.
Cheers,
Richard
2011/5/19 Richard Purdie <[email protected]>
Quoted 24 lines · Frans Meulenbroeks
Quoted 3 lines
What should I do to either disable this for some recipes, or use a different (private) server?
We do not wish to rely on an external server for proprietary recipes (orfor
Quoted 2 lines
recipes for which we made a local change in an overlay). How is this case handled?
This could be handled by doing things like setting:
PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
or
PRSERV_HOST_pn-myprivaterecipe = ""
to disable it.
One thing we lack is a good way to apply changes like this, only if code is within a given layer. Even in that case, it should be possible with anonymous python to look at the location of the current .bb file and then conditionally set these variables as appropriate to the right server.
Quoted 4 lines
Certainly this needs to be thought about and documented but I think there are ways to do it.
Pardon my ignorance, but I do not really understand the complete flow and
way of working.
Anyway if I do something like:
PRSERV_HOST_pn-myprivaterecipe = ""
would I still be able to use PR in my recipe (like I do today)?
Also we do have the issue that it is desired to be able to rebuild without network connectivity (e.g. while temporary offline while travelling). Would that still be possible?
For layers, one solution could be to allow variable overriding on the overlay level. I can imagine there are more uses for that (and I understand this requires changes to the bitbake machinery).
Frans.
Quoted 29 lines · Frans Meulenbroeks
2011/5/19 Richard Purdie <[email protected]>
Quoted 27 lines · Frans Meulenbroeks
Quoted 3 lines
What should I do to either disable this for some recipes, or use a different (private) server?
We do not wish to rely on an external server for proprietary recipes (orfor
Quoted 2 lines
recipes for which we made a local change in an overlay). How is this case handled?
This could be handled by doing things like setting:
PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
or
PRSERV_HOST_pn-myprivaterecipe = ""
to disable it.
One thing we lack is a good way to apply changes like this, only if code is within a given layer. Even in that case, it should be possible with anonymous python to look at the location of the current .bb file and then conditionally set these variables as appropriate to the right server.
Certainly this needs to be thought about and documented but I think there are ways to do it.
Quoted 6 lines
Quoted 1 line
Pardon my ignorance, but I do not really understand the complete flow and
way of working.
Anyway if I do something like:
PRSERV_HOST_pn-myprivaterecipe = ""
would I still be able to use PR in my recipe (like I do today)?
Yes, since the PR server appends to this.
Quoted 3 lines
Also we do have the issue that it is desired to be able to rebuild without network connectivity (e.g. while temporary offline while travelling). Would that still be possible?
You could use a local PR server. Obviously connecting to one central server without any network connectivity isn't going to happen so we have to be realistic about expectations.
To make a perfect rebuild the local PR server would need a dump of the database on the central server. There isn't code for that at the moment and I don't think its the highest priority task out there or the most important use case but its certainly possible for someone to add.
Quoted 3 lines
For layers, one solution could be to allow variable overriding on the overlay level. I can imagine there are more uses for that (and I understand this requires changes to the bitbake machinery).
There is certainly a use case for something like this. The exact implementation and workings needs a lot more thought and discussion though. I believe its at least already possible in anonymous python (and if not, any extensions needed shouldn't be invasive by comparison).
Cheers,
Richard
2011/5/19 Richard Purdie <[email protected]>
Quoted 48 lines · Frans Meulenbroeks
Quoted 6 lines
2011/5/19 Richard Purdie <[email protected]>
Quoted 4 lines · Frans Meulenbroeks
Quoted 3 lines
What should I do to either disable this for some recipes, or use a different (private) server?
We do not wish to rely on an external server for proprietary recipes(or
Quoted 8 lines
Quoted 8 lines
for
Quoted 2 lines
recipes for which we made a local change in an overlay). How is this case handled?
This could be handled by doing things like setting:
PRSERV_HOST_pn-myprivaterecipe = "somelocalhost"
Quoted 7 lines
Quoted 7 lines
or
PRSERV_HOST_pn-myprivaterecipe = ""
to disable it.
One thing we lack is a good way to apply changes like this, only if
code
Quoted 9 lines
Quoted 9 lines
is within a given layer. Even in that case, it should be possible with anonymous python to look at the location of the current .bb file and then conditionally set these variables as appropriate to the right server.
Certainly this needs to be thought about and documented but I think there are ways to do it.
Pardon my ignorance, but I do not really understand the complete flow
and
Quoted 5 lines
way of working.
Anyway if I do something like:
PRSERV_HOST_pn-myprivaterecipe = ""
would I still be able to use PR in my recipe (like I do today)?Yes, since the PR server appends to this.
Quoted 1 line
Also we do have the issue that it is desired to be able to rebuild
without
Quoted 1 line
network connectivity (e.g. while temporary offline while travelling).
Would
Quoted 1 line
that still be possible?
Quoted 9 lines
You could use a local PR server. Obviously connecting to one central server without any network connectivity isn't going to happen so we have to be realistic about expectations.
To make a perfect rebuild the local PR server would need a dump of the database on the central server. There isn't code for that at the moment and I don't think its the highest priority task out there or the most important use case but its certainly possible for someone to add.
I'd say it would already be nice if some caching is being done locally (just like is done with e.g. downloads).
Quoted 4 lines
Quoted 3 lines
For layers, one solution could be to allow variable overriding on the overlay level. I can imagine there are more uses for that (and I
understand
Quoted 8 lines
Quoted 1 line
this requires changes to the bitbake machinery).
There is certainly a use case for something like this. The exact implementation and workings needs a lot more thought and discussion though. I believe its at least already possible in anonymous python (and if not, any extensions needed shouldn't be invasive by comparison).
Hm. you consider this PR change to be non-invasive?
BTW I am not saying it is not good, and I understand the problem that you want to solve, but I feel this could require some more thought wrt the issues I raised before in this thread (and some more documentation and usage info).
Frans.
Quoted 10 lines · Frans Meulenbroeks
2011/5/19 Richard Purdie <[email protected]>
Quoted 8 lines · Frans Meulenbroeks
Quoted 3 lines
2011/5/19 Richard Purdie <[email protected]>
Quoted 1 line · Frans Meulenbroeks
You could use a local PR server. Obviously connecting to one central server without any network connectivity isn't going to happen so we have to be realistic about expectations.
Quoted 8 lines
Quoted 5 lines
To make a perfect rebuild the local PR server would need a dump of the database on the central server. There isn't code for that at the moment and I don't think its the highest priority task out there or the most important use case but its certainly possible for someone to add.
I'd say it would already be nice if some caching is being done locally (just like is done with e.g. downloads).
The difference is you need all the data, not just a given value. The equivalent in downloads is downloading every possible download version in advance just in case so the analogy isn't a good one. In this case it would be possible to do that though as the data is smaller.
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
For layers, one solution could be to allow variable overriding on the overlay level. I can imagine there are more uses for that (and I
understand
Quoted 1 line
this requires changes to the bitbake machinery).
There is certainly a use case for something like this. The exact implementation and workings needs a lot more thought and discussion though. I believe its at least already possible in anonymous python (and if not, any extensions needed shouldn't be invasive by comparison).
Hm. you consider this PR change to be non-invasive?
No, this isn't what I said. "allow variable overriding on the overlay level" is invasive. Adding functionality to allow this kind of thing from anonymous python is by comparison less invasive.
Its also a tangential issue to the PR part although obviously useful in connection with it.
Quoted 4 lines
BTW I am not saying it is not good, and I understand the problem that you want to solve, but I feel this could require some more thought wrt the issues I raised before in this thread (and some more documentation and usage info).
At this point, an implementation which we can look at, experiment with and document is better than none at all :).
Ideally some of these issues would have come up at the design stage but we'll work through this and I think things area heading the right way. Nobody is saying this is 100% complete or solves every problem.
I don't believe this code needs to solve every problem straight off but it does at least need to be extensible to cover them in future by design as far as possible.
Cheers,
Richard
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Frans Meulenbroeks
2011/5/19 Richard Purdie <[email protected]>
Quoted 13 lines · Frans Meulenbroeks
Quoted 3 lines
2011/5/19 Richard Purdie <[email protected]>
Quoted 1 line · Frans Meulenbroeks
You could use a local PR server. Obviously connecting to one central server without any network connectivity isn't going to happen so we have to be realistic about expectations.
To make a perfect rebuild the local PR server would need a dump of the database on the central server. There isn't code for that at the moment and I don't think its the highest priority task out there or the most important use case but its certainly possible for someone to add.
Quoted 7 lines
Quoted 2 lines
I'd say it would already be nice if some caching is being done locally (just like is done with e.g. downloads).
The difference is you need all the data, not just a given value. The equivalent in downloads is downloading every possible download version in advance just in case so the analogy isn't a good one. In this case it would be possible to do that though as the data is smaller.
I've been thinking about this. There are really three use-cases that I see.
The first (difficult one) is what is implemented:
Q: I have a group of developers all building software, I want everyone's packages to get the same version string if the package is the same.
This answers that pretty well, a global server can be enabled for a group of developers to work with. Of course the read/write question is "interesting". One possible solution for the read-only user is to allow for a default version to be specified. At a former company, when the users ran their own builds we simply returned "custom" as the build version string.
Second case, raised by Koen:
Q: I'm building a distribution that I want to share with others, how can they work with it -- yet make their own changes?
In this case I think it makes sense for the distribution vendor to export their database of checksums and version values. Then the end user can use this to seed their PR server. (Would be nice if you could look at multiple servers, starting with an authoritative and working down from there until its found.. assuming R/O means "not found". If it's not found then writing to a specific location would make sense to me.)
Third case...
Q: I'm building the software myself and simply want to track versions.
This one to me is likely a common use case. Pretty similar to the first group case, but I don't need or want a "network" service.. just a way to manage things on my local build machines. So either run it as a local service, or figure out a way to directly access a database as part of build environment...
[Fourth case]
Q: I don't care about versioning beyond what the recipe contains.
This is the existing case, and certainly needs to continue working....
Quoted 27 lines
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
For layers, one solution could be to allow variable overriding on the overlay level. I can imagine there are more uses for that (and I
understand
Quoted 1 line
this requires changes to the bitbake machinery).
There is certainly a use case for something like this. The exact implementation and workings needs a lot more thought and discussion though. I believe its at least already possible in anonymous python (and if not, any extensions needed shouldn't be invasive by comparison).
Hm. you consider this PR change to be non-invasive?
No, this isn't what I said. "allow variable overriding on the overlay level" is invasive. Adding functionality to allow this kind of thing from anonymous python is by comparison less invasive.
Its also a tangential issue to the PR part although obviously useful in connection with it.
Quoted 4 lines
BTW I am not saying it is not good, and I understand the problem that you want to solve, but I feel this could require some more thought wrt the issues I raised before in this thread (and some more documentation and usage info).
At this point, an implementation which we can look at, experiment with and document is better than none at all :).
Quoted 7 lines
Ideally some of these issues would have come up at the design stage but we'll work through this and I think things area heading the right way. Nobody is saying this is 100% complete or solves every problem.
I don't believe this code needs to solve every problem straight off but it does at least need to be extensible to cover them in future by design as far as possible.
Ya, I agree. The problem is fairly well know.. We need a way to track build changes and share this tracking between builds. I think this is a reasonable solution, but we certainly need to find and resolve the various issues that people will have with it.
I'm certainly in favor of adding this into oe-core, but having it disabled by default -- for now. Then those interested can experiment and work on solutions to the problems they are facing. (The distribution problem I think will be the trickiest to resolve.)
--Mark
Quoted 9 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
This patch series looks really good to me in general. I know I've given feedback about some pieces that need tweaking but overall its good and I'm pleased with this. There are a few corner cases people have raised and some tweaks we need to make but nothing that can't be built from this foundation! :)
Quoted 4 lines · Lianhao Lu
To start the network based PR service daemon, run the following command after "sourcing" the environment file oe-init-build-env:
bitbake-prserv --start
The one other area I'm going to worry about is the "build out the box" use case. Nowhere in the quickstart guide do we mention to the user that they need to run this. I'd also argue that we shouldn't require the user to run this at all either and we should be able to autolaunch and use it in the localhost usecase. Most users shouldn't care, it should just work. Those who need to use it can either preconfigure it for their users, or will know how to setup and start the server themselves.
A case in point is an autobuilder running 4 different builds. Ideally, in the default case, each automated build would setup and just use its own server on a non-conflicting free port on localhost.
I'm not 100% sure how we make this happen and I can give it some thought but it is the final piece of this puzzle we need to solve.
Cheers,
Richard