package_ipk.bbclass: use "--force-arch" when install package
cover package_ipk.bbclass: use "--force-arch" when install package
Note:
* This is a RFC patch.
* This is for opkg_svn.bb, the opkg_0.1.8.bb also needs it, but needs re-work.
* Please feel free to give your comments.
// Robert
The following changes since commit 99c85ba93075a1aff789cafabbacf42e6a93bd32:
build-appliance-image: Update to latest poky master (2012-09-04 18:04:40 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/ipk_arch
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/ipk_arch
Robert Yang (2):
opkg svn: Add the --force-arch option package_ipk.bbclass: use "--force-arch" when install package
meta/classes/package_ipk.bbclass | 6 +- meta/recipes-devtools/opkg/opkg/force-arch.patch | 103 ++++++++++++++++++++++ meta/recipes-devtools/opkg/opkg_svn.bb | 3 +- 3 files changed, 109 insertions(+), 3 deletions(-) create mode 100644 meta/recipes-devtools/opkg/opkg/force-arch.patch
RFC PATCH 1/2 opkg svn: Add the --force-arch option
If there are more than one candidate which has the same pkg name in the candidate list, for example, the same pkg with different versions, then it will use the last one which is the highest version in the list
Add the "--force-arch" (just like the --force-downgrade) option to let it use the higher arch priority package when enabled. the default is no.
[YOCTO #2575]
--- meta/recipes-devtools/opkg/opkg/force-arch.patch | 103 ++++++++++++++++++++++ meta/recipes-devtools/opkg/opkg_svn.bb | 3 +- 2 files changed, 105 insertions(+), 1 deletions(-) create mode 100644 meta/recipes-devtools/opkg/opkg/force-arch.patch
meta/recipes-devtools/opkg/opkg/force-arch.patch new file +103 -0
@@ -0,0 +1,103 @@ +Add the --force-arch option + +If there are more than one candidate which has the same pkg name in the +candidate list, for example, the same pkg with different versions, then +it will use the last one which is the highest version in the list + +Add the "--force-arch" (just like the --force-downgrade) option to let +it use the higher arch priority package when enabled. the default is no. + +Upstream-Status: Pending + +Signed-off-by: Robert Yang <[email protected]> +--- + libopkg/opkg_conf.h | 1 + + libopkg/pkg_hash.c | 18 +++++++++++++++--- + src/opkg-cl.c | 8 ++++++++ + 3 files changed, 24 insertions(+), 3 deletions(-) + +diff --git a/libopkg/opkg_conf.h b/libopkg/opkg_conf.h +index 3a60bc5..c425219 100644 +--- a/libopkg/opkg_conf.h ++++ b/libopkg/opkg_conf.h +@@ -77,6 +77,7 @@ struct opkg_conf + int force_removal_of_essential_packages; + int force_postinstall; + int force_remove; ++ int force_arch; + int check_signature; + int nodeps; /* do not follow dependencies */ + char *offline_root; +diff --git a/libopkg/pkg_hash.c b/libopkg/pkg_hash.c +index a99cf6b..63a67d6 100644 +--- a/libopkg/pkg_hash.c ++++ b/libopkg/pkg_hash.c +@@ -376,10 +376,22 @@ pkg_hash_fetch_best_installation_candidate(abstract_pkg_t *apkg, + if (constraint_fcn(matching, cdata)) { + opkg_msg(DEBUG, "Candidate: %s %s.\n", + matching->name, matching->version) ; +- good_pkg_by_name = matching; + /* It has been provided by hand, so it is what user want */ +- if (matching->provided_by_hand == 1) +- break; ++ if (matching->provided_by_hand == 1) { ++ good_pkg_by_name = matching; ++ break; ++ } ++ /* Respect to the arch priorities when given alternatives */ ++ if (good_pkg_by_name && conf->force_arch) { ++ if (matching->arch_priority >= good_pkg_by_name->arch_priority) { ++ good_pkg_by_name = matching; ++ opkg_msg(DEBUG, "%s %s wins by priority.\n", ++ matching->name, matching->version) ; ++ } else ++ opkg_msg(DEBUG, "%s %s wins by priority.\n", ++ good_pkg_by_name->name, good_pkg_by_name->version) ; ++ } else ++ good_pkg_by_name = matching; + } + } + +diff --git a/src/opkg-cl.c b/src/opkg-cl.c +index 1b927e5..4d3995f 100644 +--- a/src/opkg-cl.c ++++ b/src/opkg-cl.c +@@ -42,6 +42,7 @@ enum { + ARGS_OPT_FORCE_SPACE, + ARGS_OPT_FORCE_POSTINSTALL, + ARGS_OPT_FORCE_REMOVE, ++ ARGS_OPT_FORCE_ARCH, + ARGS_OPT_ADD_ARCH, + ARGS_OPT_ADD_DEST, + ARGS_OPT_NOACTION, +@@ -83,6 +84,8 @@ static struct option long_options[] = { + {"force_postinstall", 0, 0, ARGS_OPT_FORCE_POSTINSTALL}, + {"force-remove", 0, 0, ARGS_OPT_FORCE_REMOVE}, + {"force_remove", 0, 0, ARGS_OPT_FORCE_REMOVE}, ++ {"force-arch", 0, 0, ARGS_OPT_FORCE_ARCH}, ++ {"force_arch", 0, 0, ARGS_OPT_FORCE_ARCH}, + {"noaction", 0, 0, ARGS_OPT_NOACTION}, + {"download-only", 0, 0, ARGS_OPT_DOWNLOAD_ONLY}, + {"nodeps", 0, 0, ARGS_OPT_NODEPS}, +@@ -173,6 +176,9 @@ args_parse(int argc, char *argv[]) + case ARGS_OPT_FORCE_REMOVE: + conf->force_remove = 1; + break; ++ case ARGS_OPT_FORCE_ARCH: ++ conf->force_arch= 1; ++ break; + case ARGS_OPT_NODEPS: + conf->nodeps = 1; + break; +@@ -281,6 +287,8 @@ usage() + printf("\t--force-space Disable free space checks\n"); + printf("\t--force-postinstall Run postinstall scripts even in offline mode\n"); + printf("\t--force-remove Remove package even if prerm script fails\n"); ++ printf("\t--force-arch Use the higher arch priority package rather than the\n"); ++ printf("\t higher version one if more than one candidate is found.\n"); + printf("\t--noaction No action -- test only\n"); + printf("\t--download-only No action -- download only\n"); + printf("\t--nodeps Do not follow dependencies\n"); +-- +1.7.1 +
meta/recipes-devtools/opkg/opkg_svn.bb +2 -1
@@ -7,6 +7,7 @@ SRC_URI = "svn://opkg.googlecode.com/svn;module=trunk;protocol=http \ file://offline_postinstall.patch\ file://track_parents.patch \ file://conf_override.patch \ + file://force-arch.patch \ " S = "${WORKDIR}/trunk"
@@ -14,4 +15,4 @@ S = "${WORKDIR}/trunk" SRCREV = "633" PV = "0.1.8+svnr${SRCPV}" -PR = "${INC_PR}.1" +PR = "${INC_PR}.2"
Signature · 1 line
1.7.1
Re: [RFC PATCH 1/2] opkg svn: Add the --force-arch option
Quoted 6 lines · Robert Yang
If there are more than one candidate which has the same pkg name in the candidate list, for example, the same pkg with different versions, then it will use the last one which is the highest version in the list
Add the "--force-arch" (just like the --force-downgrade) option to let it use the higher arch priority package when enabled. the default is no.
This seems like a reasonable feature to add to opkg, though I don't think "--force-arch" is a very good name for the switch. (That name implies to me "allow the package to be installed even if the architecture seems to be incompatible", which is quite different to what your option does.)
However, regarding the underlying problem:
a) for people who are not using O_P_M, it's becoming apparent that the whole concept of using opkg (or rpm, or any other external package manager) for rootfs construction is more of a liability than an asset because bitbake has more knowledge about which particular binaries ought to be installed. For those use-cases, it might be better to think in terms of abolishing opkg altogether which would solve this problem and a variety of others.
b) for people who _are_ using O_P_M, specifying --force-arch during rootfs construction is all very well but they might still lose during a subsequent "opkg upgrade". So for these use cases, it seems as though something a bit more sticky might be required.
p.
Re: [RFC PATCH 1/2] opkg svn: Add the --force-arch option
Quoted 7 lines · On Saturday 08 September 2012 21:08:55 Phil Blundell wrote:
a) for people who are not using
O_P_M, it's becoming apparent that the whole concept of using opkg (or rpm, or any other external package manager) for rootfs construction is more of a liability than an asset because bitbake has more knowledge about which particular binaries ought to be installed. For those use-cases, it might be better to think in terms of abolishing opkg altogether which would solve this problem and a variety of others.
On the other hand, using the package manager for rootfs construction for those that *are* using online package management allows us to have at least some confidence that a rootfs produced directly from the metadata and one produced by on-system upgrades will be the same; you can also have package management on in one image and off in another (or change it on the fly) and expect to have the same contents, apart from the package manager being removed. The potential for subtle differences in behaviour is too great to go down the path of implementing it ourselves IMO, not to mention the extra code paths to maintain.
Quoted 4 lines
b) for people who _are_ using
O_P_M, specifying --force-arch during rootfs construction is all very well but they might still lose during a subsequent "opkg upgrade". So for these use cases, it seems as though something a bit more sticky might be required.
In terms of a proper solution I agree with this - opkg needs to handle the package architecture configuration internally rather than us overriding it at rootfs construction time. If you're advocating spending effort implementing something I think that's where it should be spent.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [RFC PATCH 1/2] opkg svn: Add the --force-arch option
Quoted 9 lines · Paul Eggleton
Quoted 7 lines · On Saturday 08 September 2012 21:08:55 Phil Blundell wrote:
a) for people who are not using
O_P_M, it's becoming apparent that the whole concept of using opkg (or rpm, or any other external package manager) for rootfs construction is more of a liability than an asset because bitbake has more knowledge about which particular binaries ought to be installed. For those use-cases, it might be better to think in terms of abolishing opkg altogether which would solve this problem and a variety of others.
Quoted 9 lines
On the other hand, using the package manager for rootfs construction for those that *are* using online package management allows us to have at least some confidence that a rootfs produced directly from the metadata and one produced by on-system upgrades will be the same; you can also have package management on in one image and off in another (or change it on the fly) and expect to have the same contents, apart from the package manager being removed. The potential for subtle differences in behaviour is too great to go down the path of implementing it ourselves IMO, not to mention the extra code paths to maintain.
I agree that for the folks who do want O_P_M, using the same package manager for rootfs construction that they will use online is the only sensible way to build images. But it'd be interesting to consider what proportion of the oe-core userbase genuinely requires O_P_M for their usecase -- I suspect it's probably a minority nowadays and, personally, I would consider it a fairly bad idea to ship a product which relied on opkg-based O_P_M in anything other than a trivial way.
I've long been tempted to implement a pure bitbake-based rootfs construction system for micro and get rid of opkg altogether. This would:
a) improve build time, since it would eliminate the (relatively slow) opkg-make-index step. In principle I think one could eliminate the entire package_write_xx step as well since the necessary data for installation can be recovered directly from sstate_xxx_package.tgz.
b) not make package choice subject to opkg's whims, and ensure that the runtime dependency resolution matches what bitbake had decided to do during the build
c) avoid a class of problems to do with stale data in deploy/ipk if the package versions in the metadata go backwards (e.g. checking out an older version or a different branch).
As you say, there would be a certain amount of work involved in making this happen, but on the other hand bitbake does already carry out most of the necessary dependency analysis in order to do the right thing with statically-specified RDEPENDS in recipes.
p.
Re: [RFC PATCH 1/2] opkg svn: Add the --force-arch option
In reply to Paul EggletonHi All,
Thank you very much for your suggestions, do we have a final solution or work around please?
If we refer to the rpm, it doesn't care about the package's version, it just selects the newest build package, I had applied a patch to make it respect to the arch before.
It seems that we have no good solution for the PREFERRED_VERSION pkg during the package installation, what does the package management system knows is the arch's priority, the PREFERRED_VERSION pkg always has a higher priority than others in the feed. So maybe respect to the arch is the only way that we have current.
I'd like to submit such a patch if you are OK with it:
Let the arch priority win the higher version is the default way for opkg, and add an option (--select-higher-version) for it to make the higher version win the arch priority, so that the user can have another choice.
// Robert
Quoted 32 lines · Paul Eggleton
Quoted 7 lines · On Saturday 08 September 2012 21:08:55 Phil Blundell wrote:
a) for people who are not using
O_P_M, it's becoming apparent that the whole concept of using opkg (or rpm, or any other external package manager) for rootfs construction is more of a liability than an asset because bitbake has more knowledge about which particular binaries ought to be installed. For those use-cases, it might be better to think in terms of abolishing opkg altogether which would solve this problem and a variety of others.On the other hand, using the package manager for rootfs construction for those that *are* using online package management allows us to have at least some confidence that a rootfs produced directly from the metadata and one produced by on-system upgrades will be the same; you can also have package management on in one image and off in another (or change it on the fly) and expect to have the same contents, apart from the package manager being removed. The potential for subtle differences in behaviour is too great to go down the path of implementing it ourselves IMO, not to mention the extra code paths to maintain.
Quoted 4 lines
b) for people who _are_ using
O_P_M, specifying --force-arch during rootfs construction is all very well but they might still lose during a subsequent "opkg upgrade". So for these use cases, it seems as though something a bit more sticky might be required.In terms of a proper solution I agree with this - opkg needs to handle the package architecture configuration internally rather than us overriding it at rootfs construction time. If you're advocating spending effort implementing something I think that's where it should be spent.
Cheers,
Paul
RFC PATCH 2/2 package_ipk.bbclass: use "--force-arch" when install package
In reply to Robert YangThis is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbay
The qemux86's PACKAGE_ARCH is i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:
xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipk
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"
What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
This is caused by opkg's selecting mechanism: when more than one candidate is found, it will use the higher version one and ignore the arch priority.
we have several conf files which set the PREFERRED_VERSION_pkg = "..." , but there is no such a mechanism which can let us tell the opkg to install the preferred version. When the preferred version is higher, this is OK, but if the preferred version is lower, there would be problems:
1) Most of the packages are core2 in the image, but several of them are i586, though we have built the core2 ones, this seems strange.
2) What's worse is that the image may not work since the preferred version pkg is not installed.
We have set the arch priority clearly in the opkg.conf, I think that respect to the arch priority is reasonable during the image generation, use the "--force-arch" would fix the problem.
--- meta/classes/package_ipk.bbclass | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
meta/classes/package_ipk.bbclass +4 -2
@@ -71,7 +71,8 @@ package_tryout_install_multilib_ipk() { multilib_tryout_dirs="" for item in ${MULTILIB_VARIANTS}; do local target_rootfs="${MULTILIB_TEMP_ROOTFS}/${item}" - local ipkg_args="-f ${INSTALL_CONF_IPK} -o ${target_rootfs} --force_overwrite" + local ipkg_args="-f ${INSTALL_CONF_IPK} -o ${target_rootfs} \ + --force_overwrite --force-arch" local selected_pkg="" local pkgname_prefix="${item}-" local pkgname_len=${#pkgname_prefix}
@@ -139,7 +140,8 @@ package_install_internal_ipk() { mkdir -p ${target_rootfs}${localstatedir}/lib/opkg/ - local ipkg_args="-f ${conffile} -o ${target_rootfs} --force-overwrite --force_postinstall" + local ipkg_args="-f ${conffile} -o ${target_rootfs} --force-overwrite \ + --force_postinstall --force-arch" opkg-cl ${ipkg_args} update
Signature · 1 line
1.7.1
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 11 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipk
Quoted 6 lines
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
It is the correct behaviour, though.
Quoted 3 lines
This is caused by opkg's selecting mechanism: when more than one candidate is found, it will use the higher version one and ignore the arch priority.
As expected and wanted.
Quoted 3 lines
we have several conf files which set the
PREFERRED_VERSION_pkg= "..." , but there is no such a mechanism which can let us tell the opkg to install the preferred version.
As pointed out before, your DISTRO needs to fix this and you should NOT break opkg in the way you are doing now.
Quoted 13 lines · Koen Kooi
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 11 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipk
Quoted 8 lines
Quoted 6 lines
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
It is the correct behaviour, though.
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
So the current behaviour is totally broken and it needs to do one of:
a) Put the things the user asked for in the rootfs
b) Tell the user its not going to
c) Error out and ask the user to fix the problem
Builds are meant to be deterministic and this clearly isn't as what you get out depends on the order of what you build.
Quoted 5 lines
Quoted 3 lines
This is caused by opkg's selecting mechanism: when more than one candidate is found, it will use the higher version one and ignore the arch priority.
As expected and wanted.
Under certain situations but not this one.
Quoted 6 lines
Quoted 3 lines
we have several conf files which set the
PREFERRED_VERSION_pkg= "..." , but there is no such a mechanism which can let us tell the opkg to install the preferred version.As pointed out before, your DISTRO needs to fix this and you should NOT break opkg in the way you are doing now.
Maybe.
What you're suggesting is that we have to force every package on a given architecture to the lowest common denominator which makes no sense.
I understand about the "un-removable" package issue you previously mentioned. There is a difference between what we build at rootfs time and any ultimate disto package feeds however and I think we need to at least consider this proposal.
It would appear at face value to at least make builds deterministic and do what the users expects them to do. The side effect is making on target package management more difficult.
Going back to the un-removable package problem, I think that opkg probably should always prefer arch specific over version. The question is how do you get back to say arch specific rather than machine specific at some later date. I'd argue that removing the arch specific package from the feed would be the trigger for that. Could that work as a mechanism to get us out of this problematic situation?
Please try and help us figure out a way out of this. I will warn that above all else builds need to be deterministic and so Robert's patch is tempting right now as it fixes that.
Cheers,
Richard
Op 5 sep. 2012, om 13:44 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 21 lines · Koen Kooi
Quoted 20 lines
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 17 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipkThe crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
Quoted 5 lines
Quoted 1 line
It is the correct behaviour, though.
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
You forget that, you, as the user, instructed bitbake to build the other version when switching machine. Should bitbake refuse to build the packages in this scenario? That would make more sense than trying to clean up the mess in the package_ipk bbclass. If you have online feeds the problem will reappear anyway.
Quoted 8 lines
So the current behaviour is totally broken and it needs to do one of:
a) Put the things the user asked for in the rootfs
b) Tell the user its not going to
c) Error out and ask the user to fix the problemBuilds are meant to be deterministic and this clearly isn't as what you get out depends on the order of what you build.
How do we know what the user expects? The user already did something that isn't right by building compatible arch packages with different versions. And this is deterministic, the user instructed bitbake to build a more recent, but compatible version, which will end up in the rootfs. If would be non-deterministic if opkg would decide at random what to install.
So, fix this problem at the root and make bitbake error out if your build breaks PREFERRED_VERSION.
2012/9/5 Koen Kooi <[email protected]>:
Quoted 38 lines
Op 5 sep. 2012, om 13:44 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 21 lines · Koen Kooi
Quoted 20 lines
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 17 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipkThe crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
Quoted 5 lines
Quoted 1 line
It is the correct behaviour, though.
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
You forget that, you, as the user, instructed bitbake to build the other version when switching machine. Should bitbake refuse to build the packages in this scenario? That would make more sense than trying to clean up the mess in the
package_ipkbbclass. If you have online feeds the problem will reappear anyway.Quoted 6 lines
So the current behaviour is totally broken and it needs to do one of:
a) Put the things the user asked for in the rootfs
b) Tell the user its not going to
c) Error out and ask the user to fix the problem
Quoted 6 lines
Quoted 2 lines
Builds are meant to be deterministic and this clearly isn't as what you get out depends on the order of what you build.
How do we know what the user expects? The user already did something that isn't right by building compatible arch packages with different versions. And this is deterministic, the user instructed bitbake to build a more recent, but compatible version, which will end up in the rootfs. If would be non-deterministic if opkg would decide at random what to install.
So, fix this problem at the root and make bitbake error out if your build breaks
PREFERRED_VERSION.
+1
Even thought that this won't hit me again with a dedicated build machine that deletes the tmp/sstate dirs before each build.
I can remember building for the Pandaboard and Overo in a row which resulted in errors that could not be resolved with sstate clean, leading to unexpected runtime behavior but "stated" a completely sane build.
It turned out that the PREFERRED_VERSION for an Overo recipe was ignored, deleting the tmp/sstate directories did sort out the problem.
Signature · 2 lines
Regards Samuel
Quoted 25 lines · Koen Kooi
Op 5 sep. 2012, om 13:44 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 23 lines · Koen Kooi
Quoted 21 lines
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 11 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipkQuoted 6 lines
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
It is the correct behaviour, though.
Quoted 7 lines
Quoted 3 lines
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
You forget that, you, as the user, instructed bitbake to build the other version when switching machine. Should bitbake refuse to build the packages in this scenario?
That would be better than silently doing something non-deterministic.
The bit I hate about this is the fact that sometimes a build would result in one thing, sometimes another. It should always error out with an invalid configuration rather than do that.
Quoted 3 lines
That would make more sense than trying to clean up the mess in the
package_ipkbbclass. If you have online feeds the problem will reappear anyway.
I care more about builds being deterministic than online feeds. Sorry ;-).
Quoted 9 lines
Quoted 8 lines
So the current behaviour is totally broken and it needs to do one of:
a) Put the things the user asked for in the rootfs
b) Tell the user its not going to
c) Error out and ask the user to fix the problemBuilds are meant to be deterministic and this clearly isn't as what you get out depends on the order of what you build.
Quoted 6 lines
How do we know what the user expects? The user already did something that isn't right by building compatible arch packages with different versions. And this is deterministic, the user instructed bitbake to build a more recent, but compatible version, which will end up in the rootfs. If would be non-deterministic if opkg would decide at random what to install.
Having a different image depending on whether I build MACHINE A or B first is not what the user expects and is not deterministic in my world or I suspect in most user's. We can go and ask some if you really think we need to?
Quoted 2 lines
So, fix this problem at the root and make bitbake error out if your build breaks
PREFERRED_VERSION.
How do we detect this? I want deterministic builds so I need the error to appear regardless of whether I build A or B first (just like I expect a consistent image).
I also do want to support these situations where we need special versions. I appreciate its ugly but we have several significant users with this problem and pretending it doesn't exist doesn't work, much as I wish we could.
What I really need here is help with coming up with some working solution. Putting our heads in the sand and arguing whether its even a problem isn't going to go anywhere :(.
Its really easy to shoot down ideas on the mailing list. Its much harder to be creative and find ways to take the project to better places whilst addressing everyone's concerns. I'm starting to find I'm simply physically and mentally running out of bandwidth for some of these discussions. I try very hard to hear different opinions, explain decisions, come up with creative solutions and so on and I think this process is one of the better features of the project. I am going to need help in order to be able to continue to do this and scale the project though.
Cheers,
Richard
Quoted 3 lines · Richard Purdie
What I really need here is help with coming up with some working solution. Putting our heads in the sand and arguing whether its even a problem isn't going to go anywhere :(.
This doesn't make any sense. Deciding whether something is or isn't an actual problem / is or isn't expected behavior is a good thing to be doing, and is the opposite of putting heads in the sand, as it's debating user expectations vs tool behavior. If you really think it's better to blindly make changes before figuring out how the tool should behave relative to user expectations, or think figuring out what should be done first is "putting our heads in the sand", then I think we have serious problems with you in charge of the project.
Signature · 1 line
Christopher Larson
Quoted 6 lines · Chris Larson
Quoted 3 lines · Richard Purdie
What I really need here is help with coming up with some working solution. Putting our heads in the sand and arguing whether its even a problem isn't going to go anywhere :(.
Quoted 7 lines
This doesn't make any sense. Deciding whether something is or isn't an actual problem / is or isn't expected behavior is a good thing to be doing, and is the opposite of putting heads in the sand, as it's debating user expectations vs tool behavior. If you really think it's better to blindly make changes before figuring out how the tool should behave relative to user expectations, or think figuring out what should be done first is "putting our heads in the sand", then
This isn't what I was suggesting and is misinterpretation. Given the following comment I'm not sure its worth taking the discuss further.
Quoted 2 lines
I think
we have serious problems with you in charge of the project.
Your opinion is duly noted, thanks ;-).
Cheers,
Richard
Op 5 sep. 2012, om 23:19 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 40 lines · Koen Kooi
Quoted 25 lines
Op 5 sep. 2012, om 13:44 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 23 lines · Koen Kooi
Quoted 21 lines
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 11 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipkQuoted 6 lines
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
It is the correct behaviour, though.
Quoted 7 lines
Quoted 3 lines
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
You forget that, you, as the user, instructed bitbake to build the other version when switching machine. Should bitbake refuse to build the packages in this scenario?
That would be better than silently doing something non-deterministic.
The bit I hate about this is the fact that sometimes a build would result in one thing, sometimes another. It should always error out with an invalid configuration rather than do that.
Quoted 5 lines
Quoted 3 lines
That would make more sense than trying to clean up the mess in the
package_ipkbbclass. If you have online feeds the problem will reappear anyway.I care more about builds being deterministic than online feeds.
Thanks for going on the record for that.
Quoted 30 lines
Sorry ;-).
Quoted 9 lines
Quoted 8 lines
So the current behaviour is totally broken and it needs to do one of:
a) Put the things the user asked for in the rootfs
b) Tell the user its not going to
c) Error out and ask the user to fix the problemBuilds are meant to be deterministic and this clearly isn't as what you get out depends on the order of what you build.
Quoted 6 lines
How do we know what the user expects? The user already did something that isn't right by building compatible arch packages with different versions. And this is deterministic, the user instructed bitbake to build a more recent, but compatible version, which will end up in the rootfs. If would be non-deterministic if opkg would decide at random what to install.
Having a different image depending on whether I build MACHINE A or B first is not what the user expects and is not deterministic in my world or I suspect in most user's. We can go and ask some if you really think we need to?
Quoted 2 lines
So, fix this problem at the root and make bitbake error out if your build breaks
PREFERRED_VERSION.How do we detect this? I want deterministic builds so I need the error to appear regardless of whether I build A or B first (just like I expect a consistent image).
Quoted 8 lines
I also do want to support these situations where we need special versions. I appreciate its ugly but we have several significant users with this problem and pretending it doesn't exist doesn't work, much as I wish we could.
What I really need here is help with coming up with some working solution. Putting our heads in the sand and arguing whether its even a problem isn't going to go anywhere :(.
Where have I argued that it's not a problem? I've said time and time again that PREFERRED_VERSION problems are for the DISTRO to solve. You are arguing that we should band-aid it somewhere in a class where it would break package feeds. So my counter proposal for broken DISTROs was to have bitbake completely prevent the user from building such a conflicting version.
The low-tech solution would be to document that you can't build this combination of machines in the same TMPDIR. Wiping TMDIR between machines changes should be quite cheap with sstate-cache nowadays, no?
Quoted 42 lines · Koen Kooi
Op 5 sep. 2012, om 23:19 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 40 lines · Koen Kooi
Quoted 25 lines
Op 5 sep. 2012, om 13:44 heeft Richard Purdie <[email protected]> het volgende geschreven:
Quoted 23 lines · Koen Kooi
Quoted 21 lines
Op 5 sep. 2012, om 11:31 heeft Robert Yang <[email protected]> het volgende geschreven:
Quoted 11 lines
This is for fixing the problem:
1) bitbake core-image-sato-sdk with MACHINE=qemux86
2) bitbake core-image-sato with with MACHINE=crownbayThe qemux86's
PACKAGE_ARCHis i586, the crownbay's is core2, but several i586 packages will be installed into crownbay's rootfs though there are core2 packages. For example, there are:xserver-xorg_1.11.2-r4_i586.ipk
xserver-xorg_1.9.3-r1_core2.ipkQuoted 6 lines
The crownbay.conf says:
PREFERRED_VERSION_xserver-xorg ?= "1.9.3"What the crownbay's image needs is xserver-xorg_1.9.3-r1_core2.ipk, but the xserver-xorg_1.11.2-r4_i586.ipk will be installed, this is incorrect.
It is the correct behaviour, though.
Quoted 7 lines
Quoted 3 lines
No, it isn't. You told bitbake to build some specific versions, it did that, then put something else in the rootfs. Without mentioning any of this to the user.
You forget that, you, as the user, instructed bitbake to build the other version when switching machine. Should bitbake refuse to build the packages in this scenario?
That would be better than silently doing something non-deterministic.
The bit I hate about this is the fact that sometimes a build would result in one thing, sometimes another. It should always error out with an invalid configuration rather than do that.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
That would make more sense than trying to clean up the mess in the
package_ipkbbclass. If you have online feeds the problem will reappear anyway.I care more about builds being deterministic than online feeds.
Thanks for going on the record for that.
We're a build system and package feed creation has always been argued to be a second step based on the output of the build system, often by you iirc.
Quoted 17 lines
Quoted 8 lines
Quoted 3 lines
So, fix this problem at the root and make bitbake error out if your build breaks
PREFERRED_VERSION.How do we detect this? I want deterministic builds so I need the error to appear regardless of whether I build A or B first (just like I expect a consistent image).
Quoted 8 lines
I also do want to support these situations where we need special versions. I appreciate its ugly but we have several significant users with this problem and pretending it doesn't exist doesn't work, much as I wish we could.
What I really need here is help with coming up with some working solution. Putting our heads in the sand and arguing whether its even a problem isn't going to go anywhere :(.
Quoted 6 lines
Where have I argued that it's not a problem? I've said time and time again that
PREFERRED_VERSIONproblems are for the DISTRO to solve. You are arguing that we should band-aid it somewhere in a class where it would break package feeds. So my counter proposal for broken DISTROs was to have bitbake completely prevent the user from building such a conflicting version.
I'd argue its up to the DISTRO to solve at package feed generation time and that is the point it should error. I cannot see any way bitbake can reasonably figure out this problem and accurately error as it simply doesn't have the info. At package feed generation time, you do have that info.
On the other hand, the build system *can* generate images consistently.
Quoted 3 lines
The low-tech solution would be to document that you can't build this combination of machines in the same TMPDIR. Wiping TMDIR between machines changes should be quite cheap with sstate-cache nowadays, no?
By your own reasoning, this is unacceptable since the generated images would differ from what online package management would result in with image upgrades.
Documenting the builds aren't deterministic is a fail, we can do better.
Cheers,
Richard