Allow PR Service and hash equiv together
v7 cover Allow PR Service and hash equiv together
v7:
Update packagedata_translate_pr_autoinc function to handle when there is no pkgdata to process.
Exception: bb.process.ExecutionError: Execution of '/mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440' failed with exit code 123:
sed: no input files
WARNING: /mnt/b/yoe/master/build/tmp/work/cortexa7t2hf-neon-vfpv4-yoe-linux-gnueabi/libtool-cross/2.4.6-r0/temp/run.packagedata_translate_pr_autoinc.2499440:151 exit 123 from 'xargs sed -e 's,@PRSERV_PV_AUTOINC@,AUTOINC,g' -e 's,@EXTENDPRAUTO@,.0,g' -i'
Uses the --no-run-if-empty options for xargs.
v6:
Refactor do_packagedata to use existing copy routine, and then sed "inline" to translate EXTENDPRAUTO and AUTOINC parts.
v5 (not sent to mailing list):
Refactor do_packagedata and create a custom copy routine. The routine also uses sed to translate EXTENDPRAUTO and AUTOINC. This also introduces changes to the new package_convert_pr_autoinc to always translate PRSERV_PV_AUTOINC and EXTENDPRAUTO to @PRSERV_PV_AUTOINC@ and @EXTENDPRAUTO@. So all users of the do_package components will see this translated version.
Remove the commit that moved package_get_auto_pr into the packagedata.bbclass.
v4 (not sent to mailing list):
rename package_convert_autoinc to package_convert_pr_autoinc.
Revert the creation of 'exclude_pkgdata_vars', and all of the custom processing of the excluded variables.
(prior patch 1 and 2 were merged, so no longer part of this.
v3 (not sent to mailing list):
Address all patch comments from the list, except for the 'sed' refactor items.
v2:
Most comments have been addressed to create a v2 version. I've refactored the commits to make a few things more clear, basically moving code around and fixing minor issues BEFORE the big patch.
Before there were two patches that together implemented the PR Serv/Hash work. This has been combined into a single patch and the oe_nohash stuff has simply been removed as no longer applicable.
The only comment that was NOT addressed in this was the suggestion to sed the pkgdata files in do_packagedata. I'm hesitent to do this as sed with ${...} in them has proven to be fragile for me in the past. If we decide that is necessary, I'd suggest we start with this set and then sed with ${...} in them has proven to be fragile for me in the past. If we decide that is necessary, I'd suggest we start with this set and then make that change after this proves to work (or not).
v1:
Before PR service didn't work reliably with hash equivalency. Generally you ended up with results that may not be reproducible, even if you started with the same PR service database and hash equivalency database.
Overtime, intermediate PR values would be created that would cause thing to get out of sync in the case of certain rebuilds or other corner cases.
The set refactors the PR service to work along side the new hash equiv system. It moves the PR and AUTOINC lookup to AFTER the do_package task is complete. This allows us to use the do_package unihash for lookup.
Additionally this fixed a small issue with the kernel, where the PR value could get incremented twice. The fix is an artifact of the other changes that cause us to only run the PR service work once per recipe.
This has been tested with the following workflow, which covers one of the critical corner cases for me:
configure local.conf with:
BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"
PRSERV_HOST ??= "localhost:0"
INHERIT += "reproducible_build"
INHERIT += "buildhistory"
bitbake glibc linux-yocto
# Modify meta/recipes-core/glibc/glibc_2.32.bb, add a comment
# to the do_patch_append(). This will taint the hash of this # function.
bitbake glibc linux-yocto
# The system should have detected the output was the same, and
# no proceed past do_package in glibc. The kernel should not # have built at all.
# Store/mv the tmp and sstate-cache from that build elsewhere
# repeat the run
bitbake glibc linux-yocto
# Compare the results of tmp/deploy/<package>/* between last
# and current run. # # The contents should be the same (filenames specifically). # # Also the kernel should be r0.0, not r0.1.
Note: if the hash equivalency database or PR server database (located in the cache directory) is removed, the values may not be the same as the previous run.
Additionally while testing the various package_*.bbclass files, it was noted that package_tar.bbclass was not working the same way as the others. This was correct as a standalone patch.
Mark Hatle (3):
buildhistory.bbclass: Rework to use read_subpackage_metadata kernel.bbclass: Move away from calling package_get_auto_pr package.bbclass: hash equivalency and pr service
meta/classes/buildhistory.bbclass | 49 ++++++++++++-------------- meta/classes/kernel.bbclass | 5 ++- meta/classes/package.bbclass | 58 ++++++++++++++++++++++++++----- meta/conf/bitbake.conf | 1 + 4 files changed, 77 insertions(+), 36 deletions(-)
Signature · 1 line
2.17.1
PATCH v7 1/3 buildhistory.bbclass: Rework to use read_subpackage_metadata
Using this mechanism ensures that we have a single point to implement the loading of the package and subpackage meta data. This also then allows the buildhistory class to use the regular datastore vs it's own custom arrays for processing history items.
--- meta/classes/buildhistory.bbclass | 49 ++++++++++++++----------------- 1 file changed, 22 insertions(+), 27 deletions(-)
meta/classes/buildhistory.bbclass +22 -27
@@ -258,20 +258,15 @@ python buildhistory_emit_pkghistory() { rcpinfo.config = sortlist(oe.utils.squashspaces(d.getVar('PACKAGECONFIG') or "")) write_recipehistory(rcpinfo, d) - pkgdest = d.getVar('PKGDEST') + bb.build.exec_func("read_subpackage_metadata", d) + for pkg in packagelist: - pkgdata = {} - with open(os.path.join(pkgdata_dir, 'runtime', pkg)) as f: - for line in f.readlines(): - item = line.rstrip('\n').split(': ', 1) - key = item[0] - if key.endswith('_' + pkg): - key = key[:-len(pkg)-1] - pkgdata[key] = item[1].encode('latin-1').decode('unicode_escape') - - pkge = pkgdata.get('PKGE', '0') - pkgv = pkgdata['PKGV'] - pkgr = pkgdata['PKGR'] + localdata = d.createCopy() + localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg) + + pkge = localdata.getVar("PKGE") or '0' + pkgv = localdata.getVar("PKGV") + pkgr = localdata.getVar("PKGR") # # Find out what the last version was # Make sure the version did not decrease
@@ -288,31 +283,31 @@ python buildhistory_emit_pkghistory() { pkginfo = PackageInfo(pkg) # Apparently the version can be different on a per-package basis (see Python) - pkginfo.pe = pkgdata.get('PE', '0') - pkginfo.pv = pkgdata['PV'] - pkginfo.pr = pkgdata['PR'] - pkginfo.pkg = pkgdata['PKG'] + pkginfo.pe = localdata.getVar("PE") or '0' + pkginfo.pv = localdata.getVar("PV") + pkginfo.pr = localdata.getVar("PR") + pkginfo.pkg = localdata.getVar("PKG") pkginfo.pkge = pkge pkginfo.pkgv = pkgv pkginfo.pkgr = pkgr - pkginfo.rprovides = sortpkglist(oe.utils.squashspaces(pkgdata.get('RPROVIDES', ""))) - pkginfo.rdepends = sortpkglist(oe.utils.squashspaces(pkgdata.get('RDEPENDS', ""))) - pkginfo.rrecommends = sortpkglist(oe.utils.squashspaces(pkgdata.get('RRECOMMENDS', ""))) - pkginfo.rsuggests = sortpkglist(oe.utils.squashspaces(pkgdata.get('RSUGGESTS', ""))) - pkginfo.rreplaces = sortpkglist(oe.utils.squashspaces(pkgdata.get('RREPLACES', ""))) - pkginfo.rconflicts = sortpkglist(oe.utils.squashspaces(pkgdata.get('RCONFLICTS', ""))) - pkginfo.files = oe.utils.squashspaces(pkgdata.get('FILES', "")) + pkginfo.rprovides = sortpkglist(oe.utils.squashspaces(localdata.getVar("RPROVIDES") or "")) + pkginfo.rdepends = sortpkglist(oe.utils.squashspaces(localdata.getVar("RDEPENDS") or "")) + pkginfo.rrecommends = sortpkglist(oe.utils.squashspaces(localdata.getVar("RRECOMMENDS") or "")) + pkginfo.rsuggests = sortpkglist(oe.utils.squashspaces(localdata.getVar("RSUGGESTS") or "")) + pkginfo.replaces = sortpkglist(oe.utils.squashspaces(localdata.getVar("RREPLACES") or "")) + pkginfo.rconflicts = sortpkglist(oe.utils.squashspaces(localdata.getVar("RCONFLICTS") or "")) + pkginfo.files = oe.utils.squashspaces(localdata.getVar("FILES") or "") for filevar in pkginfo.filevars: - pkginfo.filevars[filevar] = pkgdata.get(filevar, "") + pkginfo.filevars[filevar] = localdata.getVar(filevar) or "" # Gather information about packaged files - val = pkgdata.get('FILES_INFO', '') + val = localdata.getVar('FILES_INFO') or '' dictval = json.loads(val) filelist = list(dictval.keys()) filelist.sort() pkginfo.filelist = " ".join([shlex.quote(x) for x in filelist]) - pkginfo.size = int(pkgdata['PKGSIZE']) + pkginfo.size = int(localdata.getVar('PKGSIZE') or '0') write_pkghistory(pkginfo, d)
Signature · 1 line
2.17.1
PATCH v7 2/3 kernel.bbclass: Move away from calling package_get_auto_pr
In reply to Mark Hatle...instead we call read_subpackage_metadata.
Calling package_get_auto_pr *should* result in the same PKGV AUTOINC replacement. However, it will also end up changing PKGR differently then do_package as the BB_TASKHASH used will be for the wrong task.
Generally this won't cause any real-world issue, but it could cause problems.
Moving to read_subpackage_metadata ensures that the values used in do_package will be read in and used for kernel deployment.
--- meta/classes/kernel.bbclass | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +4 -1
@@ -748,7 +748,10 @@ kernel_do_deploy() { done fi } -do_deploy[prefuncs] += "package_get_auto_pr" + +# We deploy to filenames that include PKGV and PKGR, read the saved data to +# ensure we get the right values for both +do_deploy[prefuncs] += "read_subpackage_metadata" addtask deploy after do_populate_sysroot do_packagedata
Signature · 1 line
2.17.1
PATCH v7 3/3 package.bbclass: hash equivalency and pr service
In reply to Mark HatleWhen the PR service is enabled a number of small changes may happen to variables. In the do_package step a call to package_get_auto_pr will end up setting PRAUTO and modifying PKGV (if AUTOINC is there).
PRAUTO is then used by EXTENDPRAUTO, which is then used to generate PKGR.
Since this behavior typically happens BEFORE the BB_UNIHASH is calculated for do_package, we need a way to defer the expansion until after we have the unihash value.
Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded to placeholder values is the easiest way to deal with this. All other variables are expanded as expected.
In the next task, typically do_packagedata, we will then use the UNIHASH from the do_package to get the PR (AUTOPR) as well as generate the AUTOINC replacement value (now PRSERV_PV_AUTOINC).
The do_packagedata then translates the placeholders to the final values when copying the data from pkgdata to pkgdata-pdata-input.
--- meta/classes/package.bbclass | 58 +++++++++++++++++++++++++++++++----- meta/conf/bitbake.conf | 1 + 2 files changed, 51 insertions(+), 8 deletions(-)
meta/classes/package.bbclass +50 -8
@@ -7,7 +7,7 @@ # # There are the following default steps but PACKAGEFUNCS can be extended: # -# a) package_get_auto_pr - get PRAUTO from remote PR service +# a) package_convert_pr_autoinc - convert AUTOINC in PKGV to ${PRSERV_PV_AUTOINC} # # b) perform_packagecopy - Copy D into PKGD #
@@ -664,12 +664,20 @@ def runtime_mapping_rename (varname, pkg, d): #bb.note("%s after: %s" % (varname, d.getVar(varname))) # -# Package functions suitable for inclusion in PACKAGEFUNCS +# Used by do_packagedata (and possibly other routines post do_package) # +package_get_auto_pr[vardepsexclude] = "BB_TASKDEPDATA" python package_get_auto_pr() { import oe.prservice - import re + + def get_do_package_hash(pn): + if d.getVar("BB_RUNTASK") != "do_package": + taskdepdata = d.getVar("BB_TASKDEPDATA", False) + for dep in taskdepdata: + if taskdepdata[dep][1] == "do_package" and taskdepdata[dep][0] == pn: + return taskdepdata[dep][6] + return None # Support per recipe PRSERV_HOST pn = d.getVar('PN')
@@ -681,15 +689,22 @@ python package_get_auto_pr() { # PR Server not active, handle AUTOINC if not d.getVar('PRSERV_HOST'): - if 'AUTOINC' in pkgv: - d.setVar("PKGV", pkgv.replace("AUTOINC", "0")) + d.setVar("PRSERV_PV_AUTOINC", "0") return auto_pr = None pv = d.getVar("PV") version = d.getVar("PRAUTOINX") pkgarch = d.getVar("PACKAGE_ARCH") - checksum = d.getVar("BB_TASKHASH") + checksum = get_do_package_hash(pn) + + # If do_package isn't in the dependencies, we can't get the checksum... + if not checksum: + bb.warn('Task %s requested do_package unihash, but it was not available.' % d.getVar('BB_RUNTASK')) + #taskdepdata = d.getVar("BB_TASKDEPDATA", False) + #for dep in taskdepdata: + # bb.warn('%s:%s = %s' % (taskdepdata[dep][0], taskdepdata[dep][1], taskdepdata[dep][6])) + return if d.getVar('PRSERV_LOCKDOWN'): auto_pr = d.getVar('PRAUTO_' + version + '_' + pkgarch) or d.getVar('PRAUTO_' + version) or None
@@ -707,7 +722,7 @@ python package_get_auto_pr() { srcpv = bb.fetch2.get_srcrev(d) base_ver = "AUTOINC-%s" % version[:version.find(srcpv)] value = conn.getPR(base_ver, pkgarch, srcpv) - d.setVar("PKGV", pkgv.replace("AUTOINC", str(value))) + d.setVar("PRSERV_PV_AUTOINC", str(value)) auto_pr = conn.getPR(version, pkgarch, checksum) except Exception as e:
@@ -717,6 +732,22 @@ python package_get_auto_pr() { d.setVar('PRAUTO',str(auto_pr)) } +# +# Package functions suitable for inclusion in PACKAGEFUNCS +# + +python package_convert_pr_autoinc() { + pkgv = d.getVar("PKGV") + + # Adjust pkgv as necessary... + if 'AUTOINC' in pkgv: + d.setVar("PKGV", pkgv.replace("AUTOINC", "${PRSERV_PV_AUTOINC}")) + + # Change PRSERV_PV_AUTOINC and EXTENDPRAUTO usage to special values + d.setVar('PRSERV_PV_AUTOINC', '@PRSERV_PV_AUTOINC@') + d.setVar('EXTENDPRAUTO', '@EXTENDPRAUTO@') +} + LOCALEBASEPN ??= "${PN}" python package_do_split_locales() {
@@ -2335,7 +2366,7 @@ python do_package () { package_qa_handle_error("var-undefined", msg, d) return - bb.build.exec_func("package_get_auto_pr", d) + bb.build.exec_func("package_convert_pr_autoinc", d) ########################################################################### # Optimisations
@@ -2407,9 +2438,20 @@ addtask do_package_setscene # Copy from PKGDESTWORK to tempdirectory as tempdirectory can be cleaned at both # do_package_setscene and do_packagedata_setscene leading to races python do_packagedata () { + bb.build.exec_func("package_get_auto_pr", d) + src = d.expand("${PKGDESTWORK}") dest = d.expand("${WORKDIR}/pkgdata-pdata-input") oe.path.copyhardlinktree(src, dest) + + bb.build.exec_func("packagedata_translate_pr_autoinc", d) +} + +# Translate the EXTENDPRAUTO and AUTOINC to the final values +packagedata_translate_pr_autoinc() { + find ${WORKDIR}/pkgdata-pdata-input -type f | xargs --no-run-if-empty \ + sed -e 's,@PRSERV_PV_AUTOINC@,${PRSERV_PV_AUTOINC},g' \ + -e 's,@EXTENDPRAUTO@,${EXTENDPRAUTO},g' -i } addtask packagedata before do_build after do_package
meta/conf/bitbake.conf +1 -0
@@ -208,6 +208,7 @@ PF = "${PN}-${EXTENDPE}${PV}-${PR}" EXTENDPE = "${@['','${PE}_'][int(d.getVar('PE') or 0) > 0]}" P = "${PN}-${PV}" +PRSERV_PV_AUTOINC = "AUTOINC" PRAUTO = "" EXTENDPRAUTO = "${@['.${PRAUTO}', ''][not d.getVar('PRAUTO')]}" PRAUTOINX = "${PF}"
Signature · 1 line
2.17.1
Re: [OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service
Quoted 24 lines · Mark Hatle
When the PR service is enabled a number of small changes may happen to variables. In the
do_packagestep a call topackage_get_auto_prwill end up setting PRAUTO and modifying PKGV (if AUTOINC is there).PRAUTO is then used by EXTENDPRAUTO, which is then used to generate PKGR.
Since this behavior typically happens BEFORE the
BB_UNIHASHis calculated fordo_package, we need a way to defer the expansion until after we have the unihash value.Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded to placeholder values is the easiest way to deal with this. All other
variables are expanded as expected.In the next task, typically
do_packagedata, we will then use the UNIHASH from thedo_packageto get the PR (AUTOPR) as well as generate the AUTOINC replacement value (nowPRSERV_PV_AUTOINC).The
do_packagedatathen translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.
Quoted 6 lines
Signed-off-by Mark Hatle <[email protected]>--- meta/classes/package.bbclass | 58 +++++++++++++++++++++++++++++++--- -- meta/conf/bitbake.conf | 1 + 2 files changed, 51 insertions(+), 8 deletions(-)
This looked ok in testing apart from:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1289
(and similar)
which should reproduce with:
oe-selftest -r prservice.BitbakePrTests
basically the PRServ tests need updating to match the code changes.
Cheers,
Richard
Re: [OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service
Quoted 34 lines · Richard Purdie
Quoted 24 lines · Mark Hatle
When the PR service is enabled a number of small changes may happen to variables. In the
do_packagestep a call topackage_get_auto_prwill end up setting PRAUTO and modifying PKGV (if AUTOINC is there).PRAUTO is then used by EXTENDPRAUTO, which is then used to generate PKGR.
Since this behavior typically happens BEFORE the
BB_UNIHASHis calculated fordo_package, we need a way to defer the expansion until after we have the unihash value.Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded to placeholder values is the easiest way to deal with this. All other
variables are expanded as expected.In the next task, typically
do_packagedata, we will then use the UNIHASH from thedo_packageto get the PR (AUTOPR) as well as generate the AUTOINC replacement value (nowPRSERV_PV_AUTOINC).The
do_packagedatathen translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.Quoted 6 lines
Signed-off-by Mark Hatle <[email protected]>--- meta/classes/package.bbclass | 58 +++++++++++++++++++++++++++++++--- -- meta/conf/bitbake.conf | 1 + 2 files changed, 51 insertions(+), 8 deletions(-)This looked ok in testing apart from:
Quoted 8 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1289
(and similar)which should reproduce with:
oe-selftest -r prservice.BitbakePrTests
basically the PRServ tests need updating to match the code changes.
I see the error in the logs, but I'm not able to reproduce this. No errors are reported on my system. I added some additional debugging and I'm seeing that it's reading from
.../tmp-glibc/pkgdata/qemux86-64/runtime/m4
This is the correct one, as it should have the complete value of PKGR as it was the one transferred by do_packagedata.
Is there a standard way to keep the temp files around so that I can see what the values actually are?
--Mark
Quoted 7 lines
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [master][PATCH v7 3/3] package.bbclass: hash equivalency and pr service
Just an FYI.. To reproduce the failure you HAVE to use the 'poky' distro. I had switched to 'nodistro' because the system complained about "SANITY_TESTED_DISTROS"..
Working on this now...
--Mark
Quoted 69 lines · Mark Hatle
Quoted 34 lines · Richard Purdie
Quoted 24 lines · Mark Hatle
When the PR service is enabled a number of small changes may happen to variables. In the
do_packagestep a call topackage_get_auto_prwill end up setting PRAUTO and modifying PKGV (if AUTOINC is there).PRAUTO is then used by EXTENDPRAUTO, which is then used to generate PKGR.
Since this behavior typically happens BEFORE the
BB_UNIHASHis calculated fordo_package, we need a way to defer the expansion until after we have the unihash value.Writing out the pkgdata files w/o AUTOPR and PKGV (AUTOINC) expanded to placeholder values is the easiest way to deal with this. All other
variables are expanded as expected.In the next task, typically
do_packagedata, we will then use the UNIHASH from thedo_packageto get the PR (AUTOPR) as well as generate the AUTOINC replacement value (nowPRSERV_PV_AUTOINC).The
do_packagedatathen translates the placeholders to the final values
when copying the data from pkgdata to pkgdata-pdata-input.Quoted 6 lines
Signed-off-by Mark Hatle <[email protected]>--- meta/classes/package.bbclass | 58 +++++++++++++++++++++++++++++++--- -- meta/conf/bitbake.conf | 1 + 2 files changed, 51 insertions(+), 8 deletions(-)This looked ok in testing apart from:
Quoted 8 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1289
(and similar)which should reproduce with:
oe-selftest -r prservice.BitbakePrTests
basically the PRServ tests need updating to match the code changes.
I see the error in the logs, but I'm not able to reproduce this. No errors are reported on my system. I added some additional debugging and I'm seeing that it's reading from
.../tmp-glibc/pkgdata/qemux86-64/runtime/m4
This is the correct one, as it should have the complete value of PKGR as it was the one transferred by
do_packagedata.Is there a standard way to keep the temp files around so that I can see what the values actually are?
--Mark
Quoted 9 lines
Cheers,
Richard