packagefeed-stability: add class to help reduce package feed churn
v2 cover packagefeed-stability: add class to help reduce package feed churn
* V2
- Make ipk and deb work, now all of rpm, ipk and deb work well.
- World rebuild testing:
1) bitbake world
2) Edit base_do_configure()
3) bitbake world
There are 667 recipes *rebuilt*:
a) For rpm: 61 recipes' PR have been bumped (less than 10%)
b) For ipk and deb: 55 recipes' PR have been bumped (less than 10%)
The result is better than I expected. I've check these packages, some of the PR bump are caused by pseudo's permission issues. For the others, they are caused by dbg, ptest of elf different.
* V1
- This work is based on Paul's packagefeed-stability, thanks Paul. It is aimed for reducing PR bumps for binary packages when use PRServer. It works well for rpm:
1) Afer bumped rpm-native's PR and rebuild world, less than 20 recipes' Packages bumped PR, they are because of pseudo, I had sent email before:
http://lists.openembedded.org/pipermail/openembedded-core/2016-July/123511.html
2) When elf files are rebuilt, it will bump the PR.
- Sent build-compare' patches to upstream.
// Robert
The following changes since commit da7a2c7b00b40a8759dbe9f4ab6df3e337e3d6b6:
useradd-staticids: use map() instead of imap() (2016-07-12 23:11:57 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/feed
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/feed
Paul Eggleton (1):
packagefeed-stability: add class to help reduce package feed churn
Robert Yang (8):
rpm: make --nosignature work
build-compare: remove space at head
build-compare: make pkg-diff.sh 75% faster
build-compare: fix checking for named pipe and others
build-compare: improve deb and ipk checking package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES packagefeed-stability.bbclass: minor fixes packagefeed-stability.bbclass: copy all packages of a recipe
meta/classes/package_deb.bbclass | 2 + meta/classes/package_ipk.bbclass | 2 + meta/classes/packagefeed-stability.bbclass | 239 ++++++++++++++ .../build-compare/build-compare_git.bb | 6 + ...001-Add-support-for-deb-and-ipk-packaging.patch | 4 +- ...functions.sh-improve-deb-and-ipk-checking.patch | 117 +++++++ .../files/functions.sh-remove-space-at-head.patch | 41 +++ ...nctions.sh-run-rpm-once-to-make-it-faster.patch | 361 +++++++++++++++++++++ .../pkg-diff.sh-check-for-fifo-named-pipe.patch | 35 ++ ...-check_single_file-return-at-once-when-sa.patch | 37 +++ ...diff.sh-remove-space-in-the-end-for-ftype.patch | 32 ++ ...0001-system.h-query.c-support-nosignature.patch | 64 ++++ meta/recipes-devtools/rpm/rpm_5.4.16.bb | 1 + 13 files changed, 939 insertions(+), 2 deletions(-) create mode 100644 meta/classes/packagefeed-stability.bbclass create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch create mode 100644 meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
Signature · 1 line
2.9.0
PATCH v2 1/9 rpm: make --nosignature work
OE-core uses rpm's --nosignature, but it never worked:self._invoke_smart('config --set rpm-check-signatures=false')
Now fix it with:
* Define SUPPORT_NOSIGNATURES to 1 in system.h
* !QVA_ISSET(qva->qva_flags, SIGNATURE) -> QVA_ISSET(qva->qva_flags, SIGNATURE), otherwise, when use --nosignature would read database and verify signature, this is not expected.
This can fix some race issues, for example, when more than one process are querying rpm file with "rpm -qp --nosignature", they may hang up because of race issues (the processes are trying to get RW/RD lock on the database, but they shouldn't read the database at all since -qp and --nosignature are used).
--- ...0001-system.h-query.c-support-nosignature.patch | 64 ++++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_5.4.16.bb | 1 + 2 files changed, 65 insertions(+) create mode 100644 meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch
meta/recipes-devtools/rpm/rpm/0001-system.h-query.c-support-nosignature.patch new file +64 -0
@@ -0,0 +1,64 @@ +From d8fcc3cbeec4defb5babde5bd09ea294dd3fd08b Mon Sep 17 00:00:00 2001 +From: Robert Yang <[email protected]> +Date: Sun, 3 Jul 2016 19:01:59 -0700 +Subject: [PATCH] system.h/query.c: support nosignature + +* Define SUPPORT_NOSIGNATURES to 1 in system.h +* !QVA_ISSET(qva->qva_flags, SIGNATURE) -> QVA_ISSET(qva->qva_flags, SIGNATURE), + otherwise, when use --nosignature would read database and verify + signature, this is not expected. + +Upstream-Status: Submitting [Sent email to [email protected]] + +Signed-off-by: Robert Yang <[email protected]> +--- + lib/query.c | 6 +++--- + system.h | 4 ++-- + 2 files changed, 5 insertions(+), 5 deletions(-) + +diff --git a/lib/query.c b/lib/query.c +index 50a7453..b761d76 100644 +--- a/lib/query.c ++++ b/lib/query.c +@@ -954,19 +954,19 @@ JBJDEBUG((stderr, "--> %s(%p,%p,%p)\n", __FUNCTION__, ts, qva, argv)); + vsflags = (rpmVSFlags) rpmExpandNumeric("%{?_vsflags_query}"); + vsflags = (rpmVSFlags) 0; /* XXX FIXME: ignore default disablers. */ + #if defined(SUPPORT_NOSIGNATURES) +- if (!QVA_ISSET(qva->qva_flags, DIGEST)) { ++ if (QVA_ISSET(qva->qva_flags, DIGEST)) { + VSF_SET(vsflags, NOSHA1HEADER); + VSF_SET(vsflags, NOMD5HEADER); + VSF_SET(vsflags, NOSHA1); + VSF_SET(vsflags, NOMD5); + } +- if (!QVA_ISSET(qva->qva_flags, SIGNATURE)) { ++ if (QVA_ISSET(qva->qva_flags, SIGNATURE)) { + VSF_SET(vsflags, NODSAHEADER); + VSF_SET(vsflags, NORSAHEADER); + VSF_SET(vsflags, NODSA); + VSF_SET(vsflags, NORSA); + } +- if (!QVA_ISSET(qva->qva_flags, HDRCHK)) { ++ if (QVA_ISSET(qva->qva_flags, HDRCHK)) { + VSF_SET(vsflags, NOHDRCHK); + } + VSF_CLR(vsflags, NEEDPAYLOAD); /* XXX needed? */ +diff --git a/system.h b/system.h +index 2ff8906..ad4619a 100644 +--- a/system.h ++++ b/system.h +@@ -787,9 +787,9 @@ static inline const char *rcsid(const char *p) { \ + #define SUPPORT_I18NSTRING_TYPE 1 + + /** +- * Eliminate signature/digest disablers. ++ * Signature/digest disablers. + */ +-#undef SUPPORT_NOSIGNATURES ++#define SUPPORT_NOSIGNATURES 1 + + /** + * Permit ar(1) payloads. Disabled while rpmio/iosm.c is under development. +-- +2.9.0 +
meta/recipes-devtools/rpm/rpm_5.4.16.bb +1 -0
@@ -117,6 +117,7 @@ SRC_URI += " \ file://rpmqv.c-check-_gpg_passphrase-before-ask-for-input.patch \ file://0001-Disable-__sync_add_and_fetch_8-on-nios2.patch \ file://gcc6-stdlib.patch \ + file://0001-system.h-query.c-support-nosignature.patch \ " # OE specific changes
Signature · 1 line
2.9.0
PATCH v2 2/9 build-compare: remove space at head
In reply to Robert YangThe command like:
rpm -qp --nodigest --nosignature --qf '<foo> [%{REQUIRENAME}\n]\n'
^^space
The space will be printed, and will impact the check result, so remove it.
--- .../build-compare/build-compare_git.bb | 1 + .../files/functions.sh-remove-space-at-head.patch | 41 ++++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch
meta/recipes-devtools/build-compare/build-compare_git.bb +1 -0
@@ -9,6 +9,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \ file://Rename-rpm-check.sh-to-pkg-diff.sh.patch;striplevel=1 \ file://Ignore-DWARF-sections.patch;striplevel=1 \ file://0001-Add-support-for-deb-and-ipk-packaging.patch \ + file://functions.sh-remove-space-at-head.patch \ " SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
meta/recipes-devtools/build-compare/files/functions.sh-remove-space-at-head.patch new file +41 -0
41 lines not on this page · raw message
Signature · 1 line
2.9.0
PATCH v2 3/9 build-compare: make pkg-diff.sh 75% faster
In reply to Robert YangThe rpm tool is a heavy process, pkg-diff.sh ran 16 (or 17 for kernel) "rpm -qp" times when the pkgs are identical, now we only run "rpm -qp --qf <all we need>" twice (one is for old pkg, and one is for new), save the results to spec_old and spec_new, then use sed command to get what we need later, this can make it 75% faster when the pkgs are identical. Here is the rough data on my host Ubuntu 14.04.4, 32 cores CPU and 128G mem:
* When the pkgs are identical:
- Before the patch: 1s
- After the patch: 0.26s
I compare the whole spec firstly, and return 0 if they are the same, or go on checking one by one if not, without this, it would be 0.46s, the gain is great when there are lot of packages, usually, we have more than 10,000 rpms to compare.
* When the pkgs are different:
That depends on where is the different, if the different is at the comparing rpmtags stage:
- Before the patch: 0.26s
- After the patch: 0.29s
Increased 0.03s, but if the different is happend later than comparing rpmtags, it will save time.
--- .../build-compare/build-compare_git.bb | 1 + ...nctions.sh-run-rpm-once-to-make-it-faster.patch | 361 +++++++++++++++++++++ 2 files changed, 362 insertions(+) create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch
meta/recipes-devtools/build-compare/build-compare_git.bb +1 -0
@@ -10,6 +10,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \ file://Ignore-DWARF-sections.patch;striplevel=1 \ file://0001-Add-support-for-deb-and-ipk-packaging.patch \ file://functions.sh-remove-space-at-head.patch \ + file://functions.sh-run-rpm-once-to-make-it-faster.patch \ " SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
meta/recipes-devtools/build-compare/files/functions.sh-run-rpm-once-to-make-it-faster.patch new file +361 -0
361 lines not on this page · raw message
Signature · 1 line
2.9.0
PATCH v2 4/9 build-compare: fix checking for named pipe and others
In reply to Robert Yang* Fixed checking for named pipe
* Return at once when archives are the same
* Fix for type "directory"
--- .../build-compare/build-compare_git.bb | 3 ++ .../pkg-diff.sh-check-for-fifo-named-pipe.patch | 35 ++++++++++++++++++++ ...-check_single_file-return-at-once-when-sa.patch | 37 ++++++++++++++++++++++ ...diff.sh-remove-space-in-the-end-for-ftype.patch | 32 +++++++++++++++++++ 4 files changed, 107 insertions(+) create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch create mode 100644 meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch
meta/recipes-devtools/build-compare/build-compare_git.bb +3 -0
@@ -11,6 +11,9 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \ file://0001-Add-support-for-deb-and-ipk-packaging.patch \ file://functions.sh-remove-space-at-head.patch \ file://functions.sh-run-rpm-once-to-make-it-faster.patch \ + file://pkg-diff.sh-check-for-fifo-named-pipe.patch \ + file://pkg-diff.sh-check_single_file-return-at-once-when-sa.patch \ + file://pkg-diff.sh-remove-space-in-the-end-for-ftype.patch \ " SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
meta/recipes-devtools/build-compare/files/pkg-diff.sh-check-for-fifo-named-pipe.patch new file +35 -0
@@ -0,0 +1,35 @@ +From a78fe4f792a9ac9f4d364e836c8855f48561d6f2 Mon Sep 17 00:00:00 2001 +From: Robert Yang <[email protected]> +Date: Thu, 14 Jul 2016 19:52:18 -0700 +Subject: [PATCH 3/4] pkg-diff.sh: check for fifo(named pipe) + +Otherwise "cmp -s fifo1 fifo2" will wait for inputing forever. + +Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10] + +Signed-off-by: Robert Yang <[email protected]> +--- + pkg-diff.sh | 7 +++++++ + 1 file changed, 7 insertions(+) + +diff --git a/pkg-diff.sh b/pkg-diff.sh +index 5dd3a38..1f353aa 100644 +--- a/pkg-diff.sh ++++ b/pkg-diff.sh +@@ -735,6 +735,13 @@ check_single_file() + return 1 + fi + ;; ++ fifo*pipe*) ++ ftype_new="`/usr/bin/file new/$file | sed -e 's@^[^:]\+:[[:blank:]]*@@' -e 's@[[:blank:]]*$@@'`" ++ if [ "$ftype_new" = "$ftype" ]; then ++ return 0 ++ fi ++ return 1 ++ ;; + *) + if ! diff_two_files; then + return 1 +-- +2.9.0 +
meta/recipes-devtools/build-compare/files/pkg-diff.sh-check_single_file-return-at-once-when-sa.patch new file +37 -0
@@ -0,0 +1,37 @@ +From 657983ef9ca8f8354172682e17408c4f6b5bc667 Mon Sep 17 00:00:00 2001 +From: Robert Yang <[email protected]> +Date: Thu, 14 Jul 2016 19:46:08 -0700 +Subject: [PATCH 1/4] pkg-diff.sh: check_single_file(): return at once when + same + +If the two files are the same, return at once, this can save a lot of +time when there are archives inside archives. + +Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10] + +Signed-off-by: Robert Yang <[email protected]> +--- + pkg-diff.sh | 7 +++++++ + 1 file changed, 7 insertions(+) + +diff --git a/pkg-diff.sh b/pkg-diff.sh +index 3cf10aa..402d4a4 100644 +--- a/pkg-diff.sh ++++ b/pkg-diff.sh +@@ -293,6 +293,13 @@ check_compressed_file() + check_single_file() + { + local file="$1" ++ ++ # If the two files are the same, return at once. ++ if [ -f old/$file -a -f new/$file ]; then ++ if cmp -s old/$file new/$file; then ++ return 0 ++ fi ++ fi + case $file in + *.spec) + sed -i -e "s,Release:.*$release1,Release: @RELEASE@," old/$file +-- +2.9.0 +
meta/recipes-devtools/build-compare/files/pkg-diff.sh-remove-space-in-the-end-for-ftype.patch new file +32 -0
@@ -0,0 +1,32 @@ +From 836a6783df9c582a834fca239f227063a5687715 Mon Sep 17 00:00:00 2001 +From: Robert Yang <[email protected]> +Date: Thu, 14 Jul 2016 19:49:12 -0700 +Subject: [PATCH 2/4] pkg-diff.sh: remove space in the end for ftype + +Versions of file like 5.14 returns a " " in the end, for example: +ftype="directory ", but we need ftype="directory", remove the space to +fix the problem. + +Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10] + +Signed-off-by: Robert Yang <[email protected]> +--- + pkg-diff.sh | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg-diff.sh b/pkg-diff.sh +index 402d4a4..5dd3a38 100644 +--- a/pkg-diff.sh ++++ b/pkg-diff.sh +@@ -633,7 +633,7 @@ check_single_file() + ;; + esac + +- ftype=`/usr/bin/file old/$file | sed 's@^[^:]\+:[[:blank:]]*@@'` ++ ftype=`/usr/bin/file old/$file | sed -e 's@^[^:]\+:[[:blank:]]*@@' -e 's@[[:blank:]]*$@@'` + case $ftype in + PE32\ executable*Mono\/\.Net\ assembly*) + echo "PE32 Mono/.Net assembly: $file" +-- +2.9.0 +
Signature · 1 line
2.9.0
PATCH v2 5/9 build-compare: improve deb and ipk checking
In reply to Robert Yang* The deb and ipk's depends version string is like:
Depends: libc6 (>= 2.24)
Update trim_release_old and trim_release_new to match the bracket in the end ")".
* The deb's data tarball now is .tar.xz, and ipk's is .tar.gz.
* Update adjust_controlfile() to make ituse trim_release_old and trim_release_new.
--- .../build-compare/build-compare_git.bb | 1 + ...001-Add-support-for-deb-and-ipk-packaging.patch | 4 +- ...functions.sh-improve-deb-and-ipk-checking.patch | 117 +++++++++++++++++++++ 3 files changed, 120 insertions(+), 2 deletions(-) create mode 100644 meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch
meta/recipes-devtools/build-compare/build-compare_git.bb +1 -0
@@ -14,6 +14,7 @@ SRC_URI = "git://github.com/openSUSE/build-compare.git \ file://pkg-diff.sh-check-for-fifo-named-pipe.patch \ file://pkg-diff.sh-check_single_file-return-at-once-when-sa.patch \ file://pkg-diff.sh-remove-space-in-the-end-for-ftype.patch \ + file://functions.sh-improve-deb-and-ipk-checking.patch \ " SRCREV = "c5352c054c6ef15735da31b76d6d88620f4aff0a"
meta/recipes-devtools/build-compare/files/0001-Add-support-for-deb-and-ipk-packaging.patch +2 -2
@@ -3,7 +3,7 @@ From: Paul Eggleton <[email protected]> Date: Tue, 1 Sep 2015 12:04:33 +0100 Subject: [PATCH] Add support for deb and ipk packaging -Upstream-Status: Pending +Upstream-Status: Submitted [https://github.com/openSUSE/build-compare/pull/10] Signed-off-by: Paul Eggleton <[email protected]> ---
@@ -34,7 +34,7 @@ index 06079df..85c9003 100644 return $RES } + -+function adjust_controlfile () { ++function adjust_controlfile() { + cat $1/control | sed '/^Version: /d' > $1/control.fixed + mv $1/control.fixed $1/control + cat $2/control | sed '/^Version: /d' > $2/control.fixed
meta/recipes-devtools/build-compare/files/functions.sh-improve-deb-and-ipk-checking.patch new file +117 -0
117 lines not on this page · raw message
Signature · 1 line
2.9.0
PATCH v2 6/9 package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES
In reply to Robert YangThe dict.fromkeys() creates a dict without order, there might be a problem when build the same recipe again, for example:
- First build of make:
Provides: es-translation, make-locale
- Second build of acl:
Provides: make-locale, es-translation
They are exactly the same Provides, but tools like "diff" doesn't think so. Sort RPROVIDES will fix the problem.
--- meta/classes/package_deb.bbclass | 2 ++ meta/classes/package_ipk.bbclass | 2 ++ 2 files changed, 4 insertions(+)
meta/classes/package_deb.bbclass +2 -0
@@ -52,6 +52,7 @@ python do_package_deb () { import re, copy import textwrap import subprocess + import collections oldcwd = os.getcwd()
@@ -247,6 +248,7 @@ python do_package_deb () { debian_cmp_remap(rsuggests) # Deliberately drop version information here, not wanted/supported by deb rprovides = dict.fromkeys(bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or ""), []) + rprovides = collections.OrderedDict(sorted(rprovides.items(), key=lambda x: x[0])) debian_cmp_remap(rprovides) rreplaces = bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "") debian_cmp_remap(rreplaces)
meta/classes/package_ipk.bbclass +2 -0
@@ -20,6 +20,7 @@ python do_package_ipk () { import re, copy import textwrap import subprocess + import collections oldcwd = os.getcwd()
@@ -192,6 +193,7 @@ python do_package_ipk () { debian_cmp_remap(rsuggests) # Deliberately drop version information here, not wanted/supported by ipk rprovides = dict.fromkeys(bb.utils.explode_dep_versions2(localdata.getVar("RPROVIDES", True) or ""), []) + rprovides = collections.OrderedDict(sorted(rprovides.items(), key=lambda x: x[0])) debian_cmp_remap(rprovides) rreplaces = bb.utils.explode_dep_versions2(localdata.getVar("RREPLACES", True) or "") debian_cmp_remap(rreplaces)
Signature · 1 line
2.9.0
PATCH v2 7/9 packagefeed-stability: add class to help reduce package feed churn
In reply to Robert YangFrom: Paul Eggleton <[email protected]>
When a dependency causes a recipe to effectively be rebuilt, its output may in fact not change; but new packages (with an increased PR value, if using the PR server) will be generated nonetheless. There's no practical way for us to predict whether or not this is going to be the case based solely on the inputs, but we can compare the package output and see if that is materially different and based upon that decide to replace the old package with the new one.
This class effectively intercepts packages as they are written out by do_package_write_*, causing them to be written into a different directory where we can compare them to whatever older packages might be in the "real" package feed directory, and avoid copying the new package to the feed if it has not materially changed. We use build-compare to do the package comparison.
--- meta/classes/packagefeed-stability.bbclass | 270 +++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 meta/classes/packagefeed-stability.bbclass
meta/classes/packagefeed-stability.bbclass new file +270 -0
@@ -0,0 +1,270 @@ +# Class to avoid copying packages into the feed if they haven't materially changed +# +# Copyright (C) 2015 Intel Corporation +# Released under the MIT license (see COPYING.MIT for details) +# +# This class effectively intercepts packages as they are written out by +# do_package_write_*, causing them to be written into a different +# directory where we can compare them to whatever older packages might +# be in the "real" package feed directory, and avoid copying the new +# package to the feed if it has not materially changed. The idea is to +# avoid unnecessary churn in the packages when dependencies trigger task +# reexecution (and thus repackaging). Enabling the class is simple: +# +# INHERIT += "packagefeed-stability" +# +# Caveats: +# 1) Latest PR values in the build system may not match those in packages +# seen on the target (naturally) +# 2) If you rebuild from sstate without the existing package feed present, +# you will lose the "state" of the package feed i.e. the preserved old +# package versions. Not the end of the world, but would negate the +# entire purpose of this class. +# +# Note that running -c cleanall on a recipe will purposely delete the old +# package files so they will definitely be copied the next time. + +python() { + # Package backend agnostic intercept + # This assumes that the package_write task is called package_write_<pkgtype> + # and that the directory in which packages should be written is + # pointed to by the variable DEPLOY_DIR_<PKGTYPE> + for pkgclass in (d.getVar('PACKAGE_CLASSES', True) or '').split(): + if pkgclass.startswith('package_'): + pkgtype = pkgclass.split('_', 1)[1] + pkgwritefunc = 'do_package_write_%s' % pkgtype + sstate_outputdirs = d.getVarFlag(pkgwritefunc, 'sstate-outputdirs', False) + deploydirvar = 'DEPLOY_DIR_%s' % pkgtype.upper() + deploydirvarref = '${' + deploydirvar + '}' + pkgcomparefunc = 'do_package_compare_%s' % pkgtype + + if bb.data.inherits_class('image', d): + d.appendVarFlag('do_rootfs', 'recrdeptask', ' ' + pkgcomparefunc) + + if bb.data.inherits_class('populate_sdk_base', d): + d.appendVarFlag('do_populate_sdk', 'recrdeptask', ' ' + pkgcomparefunc) + + if bb.data.inherits_class('populate_sdk_ext', d): + d.appendVarFlag('do_populate_sdk_ext', 'recrdeptask', ' ' + pkgcomparefunc) + + d.appendVarFlag('do_build', 'recrdeptask', ' ' + pkgcomparefunc) + + if d.getVarFlag(pkgwritefunc, 'noexec', True) or (not d.getVarFlag(pkgwritefunc, 'task', True)) or pkgwritefunc in (d.getVar('__BBDELTASKS', True) or []): + # Packaging is disabled for this recipe, we shouldn't do anything + continue + + if deploydirvarref in sstate_outputdirs: + # Set intermediate output directory + d.setVarFlag(pkgwritefunc, 'sstate-outputdirs', sstate_outputdirs.replace(deploydirvarref, deploydirvarref + '-prediff')) + + d.setVar(pkgcomparefunc, d.getVar('do_package_compare', False)) + d.setVarFlags(pkgcomparefunc, d.getVarFlags('do_package_compare', False)) + d.appendVarFlag(pkgcomparefunc, 'depends', ' build-compare-native:do_populate_sysroot') + bb.build.addtask(pkgcomparefunc, 'do_build', 'do_packagedata ' + pkgwritefunc, d) +} + +# This isn't the real task function - it's a template that we use in the +# anonymous python code above +fakeroot python do_package_compare () { + currenttask = d.getVar('BB_CURRENTTASK', True) + pkgtype = currenttask.rsplit('_', 1)[1] + package_compare_impl(pkgtype, d) +} + +def package_compare_impl(pkgtype, d): + import errno + import fnmatch + import glob + import subprocess + import oe.sstatesig + + pn = d.getVar('PN', True) + deploydir = d.getVar('DEPLOY_DIR_%s' % pkgtype.upper(), True) + prepath = deploydir + '-prediff/' + + # Find out PKGR values are + pkgdatadir = d.getVar('PKGDATA_DIR', True) + packages = [] + try: + with open(os.path.join(pkgdatadir, pn), 'r') as f: + for line in f: + if line.startswith('PACKAGES:'): + packages = line.split(':', 1)[1].split() + break + except IOError as e: + if e.errno == errno.ENOENT: + pass + + if not packages: + bb.debug(2, '%s: no packages, nothing to do' % pn) + return + + pkgrvalues = {} + rpkgnames = {} + rdepends = {} + pkgvvalues = {} + for pkg in packages: + with open(os.path.join(pkgdatadir, 'runtime', pkg), 'r') as f: + for line in f: + if line.startswith('PKGR:'): + pkgrvalues[pkg] = line.split(':', 1)[1].strip() + if line.startswith('PKGV:'): + pkgvvalues[pkg] = line.split(':', 1)[1].strip() + elif line.startswith('PKG_%s:' % pkg): + rpkgnames[pkg] = line.split(':', 1)[1].strip() + elif line.startswith('RDEPENDS_%s:' % pkg): + rdepends[pkg] = line.split(':', 1)[1].strip() + + # Prepare a list of the runtime package names for packages that were + # actually produced + rpkglist = [] + for pkg, rpkg in rpkgnames.iteritems(): + if os.path.exists(os.path.join(pkgdatadir, 'runtime', pkg + '.packaged')): + rpkglist.append((rpkg, pkg)) + rpkglist.sort(key=lambda x: len(x[0]), reverse=True) + + pvu = d.getVar('PV', False) + if '$' + '{SRCPV}' in pvu: + pvprefix = pvu.split('$' + '{SRCPV}', 1)[0] + else: + pvprefix = None + + pkgwritetask = 'package_write_%s' % pkgtype + files = [] + copypkgs = [] + manifest, _ = oe.sstatesig.sstate_get_manifest_filename(pkgwritetask, d) + with open(manifest, 'r') as f: + for line in f: + if line.startswith(prepath): + srcpath = line.rstrip() + if os.path.isfile(srcpath): + destpath = os.path.join(deploydir, os.path.relpath(srcpath, prepath)) + + # This is crude but should work assuming the output + # package file name starts with the package name + # and rpkglist is sorted by length (descending) + pkgbasename = os.path.basename(destpath) + pkgname = None + for rpkg, pkg in rpkglist: + if pkgbasename.startswith(rpkg): + pkgr = pkgrvalues[pkg] + destpathspec = destpath.replace(pkgr, '*') + if pvprefix: + pkgv = pkgvvalues[pkg] + if pkgv.startswith(pvprefix): + pkgvsuffix = pkgv[len(pvprefix):] + if '+' in pkgvsuffix: + newpkgv = pvprefix + '*+' + pkgvsuffix.split('+', 1)[1] + destpathspec = destpathspec.replace(pkgv, newpkgv) + pkgname = pkg + break + else: + bb.warn('Unable to map %s back to package' % pkgbasename) + destpathspec = destpath + + oldfiles = glob.glob(destpathspec) + oldfile = None + docopy = True + if oldfiles: + oldfile = oldfiles[-1] + result = subprocess.call(['pkg-diff.sh', oldfile, srcpath]) + if result == 0: + docopy = False + + files.append((pkgname, pkgbasename, srcpath, oldfile, destpath)) + bb.debug(2, '%s: package %s %s' % (pn, files[-1], docopy)) + if docopy: + copypkgs.append(pkgname) + + # Ensure that dependencies on specific versions (such as -dev on the + # main package) are copied in lock-step + changed = True + while changed: + rpkgdict = {x[0]: x[1] for x in rpkglist} + changed = False + for pkgname, pkgbasename, srcpath, oldfile, destpath in files: + rdeps = rdepends.get(pkgname, None) + if not rdeps: + continue + rdepvers = bb.utils.explode_dep_versions2(rdeps) + for rdep, versions in rdepvers.iteritems(): + dep = rpkgdict.get(rdep, None) + for version in versions: + if version and version.startswith('= '): + if dep in copypkgs and not pkgname in copypkgs: + bb.debug(2, '%s: copying %s because it has a fixed version dependency on %s and that package is going to be copied' % (pn, pkgname, dep)) + changed = True + copypkgs.append(pkgname) + elif pkgname in copypkgs and not dep in copypkgs: + bb.debug(2, '%s: copying %s because %s has a fixed version dependency on it and that package is going to be copied' % (pn, dep, pkgname)) + changed = True + copypkgs.append(dep) + + # Read in old manifest so we can delete any packages we aren't going to replace or preserve + pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}')) + try: + with open(pcmanifest, 'r') as f: + knownfiles = [x[3] for x in files if x[3]] + for line in f: + fn = line.rstrip() + if fn: + if fn in knownfiles: + knownfiles.remove(fn) + else: + try: + os.remove(fn) + bb.warn('Removed old package %s' % fn) + except OSError as e: + if e.errno == errno.ENOENT: + pass + except IOError as e: + if e.errno == errno.ENOENT: + pass + + # Create new manifest + with open(pcmanifest, 'w') as f: + for pkgname, pkgbasename, srcpath, oldfile, destpath in files: + if pkgname in copypkgs: + bb.warn('Copying %s' % pkgbasename) + destdir = os.path.dirname(destpath) + bb.utils.mkdirhier(destdir) + if oldfile: + try: + os.remove(oldfile) + except OSError as e: + if e.errno == errno.ENOENT: + pass + if (os.stat(srcpath).st_dev == os.stat(destdir).st_dev): + # Use a hard link to save space + os.link(srcpath, destpath) + else: + shutil.copyfile(srcpath, destpath) + f.write('%s\n' % destpath) + else: + bb.warn('Not copying %s' % pkgbasename) + f.write('%s\n' % oldfile) + + +do_cleanall_append() { + import errno + for pkgclass in (d.getVar('PACKAGE_CLASSES', True) or '').split(): + if pkgclass.startswith('package_'): + pkgtype = pkgclass.split('_', 1)[1] + deploydir = d.getVar('DEPLOY_DIR_%s' % pkgtype.upper(), True) + prepath = deploydir + '-prediff' + pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}')) + try: + with open(pcmanifest, 'r') as f: + for line in f: + fn = line.rstrip() + if fn: + try: + os.remove(fn) + except OSError as e: + if e.errno == errno.ENOENT: + pass + os.remove(pcmanifest) + except IOError as e: + if e.errno == errno.ENOENT: + pass +}
Signature · 1 line
2.9.0
PATCH v2 8/9 packagefeed-stability.bbclass: minor fixes
In reply to Robert Yang* fix for python3
iteritems() -> items()
* Return immediately for native and cross.
* Remove the usage of __BBDELTASKS, there is no such var in bitbake.
--- meta/classes/packagefeed-stability.bbclass | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/packagefeed-stability.bbclass +5 -3
@@ -25,6 +25,8 @@ # package files so they will definitely be copied the next time. python() { + if bb.data.inherits_class('native', d) or bb.data.inherits_class('cross', d): + return # Package backend agnostic intercept # This assumes that the package_write task is called package_write_<pkgtype> # and that the directory in which packages should be written is
@@ -49,7 +51,7 @@ python() { d.appendVarFlag('do_build', 'recrdeptask', ' ' + pkgcomparefunc) - if d.getVarFlag(pkgwritefunc, 'noexec', True) or (not d.getVarFlag(pkgwritefunc, 'task', True)) or pkgwritefunc in (d.getVar('__BBDELTASKS', True) or []): + if d.getVarFlag(pkgwritefunc, 'noexec', True) or not d.getVarFlag(pkgwritefunc, 'task', True): # Packaging is disabled for this recipe, we shouldn't do anything continue
@@ -118,7 +120,7 @@ def package_compare_impl(pkgtype, d): # Prepare a list of the runtime package names for packages that were # actually produced rpkglist = [] - for pkg, rpkg in rpkgnames.iteritems(): + for pkg, rpkg in rpkgnames.items(): if os.path.exists(os.path.join(pkgdatadir, 'runtime', pkg + '.packaged')): rpkglist.append((rpkg, pkg)) rpkglist.sort(key=lambda x: len(x[0]), reverse=True)
@@ -187,7 +189,7 @@ def package_compare_impl(pkgtype, d): if not rdeps: continue rdepvers = bb.utils.explode_dep_versions2(rdeps) - for rdep, versions in rdepvers.iteritems(): + for rdep, versions in rdepvers.items(): dep = rpkgdict.get(rdep, None) for version in versions: if version and version.startswith('= '):
Signature · 1 line
2.9.0
PATCH v2 9/9 packagefeed-stability.bbclass: copy all packages of a recipe
In reply to Robert YangA recipes can generate several rpms such as a.rpm, a-dev.rpm, a-dbg.rpm, when update one of them in the repo, we'd better update all of them, otherwise, there might be a-dev.r0.1.rpm and a-dbg.r0.3.rpm in the repo, which looks strange.
--- meta/classes/packagefeed-stability.bbclass | 103 ++++++++++------------------- 1 file changed, 35 insertions(+), 68 deletions(-)
meta/classes/packagefeed-stability.bbclass +35 -68
@@ -133,8 +133,10 @@ def package_compare_impl(pkgtype, d): pkgwritetask = 'package_write_%s' % pkgtype files = [] - copypkgs = [] + docopy = False manifest, _ = oe.sstatesig.sstate_get_manifest_filename(pkgwritetask, d) + # Copy recipe's all packages if one of the packages are different to make + # they have the same PR. with open(manifest, 'r') as f: for line in f: if line.startswith(prepath):
@@ -164,88 +166,53 @@ def package_compare_impl(pkgtype, d): bb.warn('Unable to map %s back to package' % pkgbasename) destpathspec = destpath - oldfiles = glob.glob(destpathspec) oldfile = None - docopy = True - if oldfiles: - oldfile = oldfiles[-1] - result = subprocess.call(['pkg-diff.sh', oldfile, srcpath]) - if result == 0: - docopy = False - - files.append((pkgname, pkgbasename, srcpath, oldfile, destpath)) - bb.debug(2, '%s: package %s %s' % (pn, files[-1], docopy)) - if docopy: - copypkgs.append(pkgname) - - # Ensure that dependencies on specific versions (such as -dev on the - # main package) are copied in lock-step - changed = True - while changed: - rpkgdict = {x[0]: x[1] for x in rpkglist} - changed = False - for pkgname, pkgbasename, srcpath, oldfile, destpath in files: - rdeps = rdepends.get(pkgname, None) - if not rdeps: - continue - rdepvers = bb.utils.explode_dep_versions2(rdeps) - for rdep, versions in rdepvers.items(): - dep = rpkgdict.get(rdep, None) - for version in versions: - if version and version.startswith('= '): - if dep in copypkgs and not pkgname in copypkgs: - bb.debug(2, '%s: copying %s because it has a fixed version dependency on %s and that package is going to be copied' % (pn, pkgname, dep)) - changed = True - copypkgs.append(pkgname) - elif pkgname in copypkgs and not dep in copypkgs: - bb.debug(2, '%s: copying %s because %s has a fixed version dependency on it and that package is going to be copied' % (pn, dep, pkgname)) - changed = True - copypkgs.append(dep) - - # Read in old manifest so we can delete any packages we aren't going to replace or preserve - pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}')) - try: - with open(pcmanifest, 'r') as f: - knownfiles = [x[3] for x in files if x[3]] - for line in f: - fn = line.rstrip() - if fn: - if fn in knownfiles: - knownfiles.remove(fn) - else: + if not docopy: + oldfiles = glob.glob(destpathspec) + if oldfiles: + oldfile = oldfiles[-1] + result = subprocess.call(['pkg-diff.sh', oldfile, srcpath]) + if result != 0: + docopy = True + bb.note("%s and %s are different, will copy packages" % (oldfile, srcpath)) + else: + docopy = True + bb.note("No old packages found for %s, will copy packages" % pkgname) + + files.append((pkgname, pkgbasename, srcpath, destpath)) + + # Remove all the old files and copy again if docopy + if docopy: + bb.plain('Copying packages for recipe %s' % pn) + pcmanifest = os.path.join(prepath, d.expand('pkg-compare-manifest-${MULTIMACH_TARGET_SYS}-${PN}')) + try: + with open(pcmanifest, 'r') as f: + for line in f: + fn = line.rstrip() + if fn: try: os.remove(fn) - bb.warn('Removed old package %s' % fn) + bb.note('Removed old package %s' % fn) except OSError as e: if e.errno == errno.ENOENT: pass - except IOError as e: - if e.errno == errno.ENOENT: - pass + except IOError as e: + if e.errno == errno.ENOENT: + pass - # Create new manifest - with open(pcmanifest, 'w') as f: - for pkgname, pkgbasename, srcpath, oldfile, destpath in files: - if pkgname in copypkgs: - bb.warn('Copying %s' % pkgbasename) + # Create new manifest + with open(pcmanifest, 'w') as f: + for pkgname, pkgbasename, srcpath, destpath in files: destdir = os.path.dirname(destpath) bb.utils.mkdirhier(destdir) - if oldfile: - try: - os.remove(oldfile) - except OSError as e: - if e.errno == errno.ENOENT: - pass if (os.stat(srcpath).st_dev == os.stat(destdir).st_dev): # Use a hard link to save space os.link(srcpath, destpath) else: shutil.copyfile(srcpath, destpath) f.write('%s\n' % destpath) - else: - bb.warn('Not copying %s' % pkgbasename) - f.write('%s\n' % oldfile) - + else: + bb.plain('Not copying packages for %s' % pn) do_cleanall_append() { import errno
Signature · 1 line
2.9.0