PRServer's problem
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.
Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.
2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:PR_CHECKSUM = "${BB_TASKHASH}" (current way)
OrPR_CHECKSUM = "<recipe checksum>"
Signature · 3 lines
Thanks Robert
Hi Robert,
Quoted 8 lines · Robert Yang
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply
a CVE patch to pseudo-native or rpm-native, or do some slight changes in theirdo_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.
Paul began working to address the package feed churn issue some time back using the tool build-compare from the openSUSE project.
You can find links to various trees and a summary of the progress so far in this bugzilla entry:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318
It still needs some more work; most notably I'd like to see some tests that validates the functionality.
Regards,
Joshua
Hi Joshua,
Thanks, I read packagefeed-stability.bbclass just now. If we can change PRserver's checksum to consider RDEPENDS only (or make it optional) I will run a testing on that script.
// Robert
Quoted 27 lines · Joshua G Lock
Hi Robert,
Quoted 8 lines · Robert Yang
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply
a CVE patch to pseudo-native or rpm-native, or do some slight changes in theirdo_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Paul began working to address the package feed churn issue some time back using the tool build-compare from the openSUSE project.
You can find links to various trees and a summary of the progress so far in this bugzilla entry:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318
It still needs some more work; most notably I'd like to see some tests that validates the functionality.
Regards,
Joshua
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 34 lines · Robert Yang
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
OrPR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
// Robert
Quoted 40 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but it's quite annoying if you need to bump a lot of recipes you don't control :).
Quoted 39 lines
Quoted 39 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 12 lines · Martin Jansa
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
// Robert
Quoted 41 lines
Quoted 40 lines
Quoted 40 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 27 lines · Robert Yang
Quoted 12 lines · Martin Jansa
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 5 lines
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Hi Martin,
I found this patch in the bug:
Too many PR bumps and rebuildings are caused by this patch. I'm not very sure about what this patch tries to fix, it seems that it is trying to fix problems when 32bit and 64bit uses the same sstate cache? Would you please provide a simple reproducer, please?
Things will become much better after revert this patch. Be less stricter or more is a hard problem, if we still need the patch, can we leave such a choice to the end user? We can add some vars like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user who uses stable release like jethro or krogoth to make their distributions, and PR Service really matters here. Even if they switch the build between 32 and 64 builds (This is unlikely to happen for production environment) and got problems, they still can fix the problem by rebuilding, this is still much better than current status: run a "smart/opkg/apt-get upgrade" on the target, *all* of the packages are downloaded and installed again after a CVE patch applies on native recipes like pseudo-native or rpm-native, but in fact, nothing is changed on the target this is really a bad experience.
// Robert
Quoted 80 lines · Martin Jansa
Quoted 27 lines · Robert Yang
Quoted 12 lines · Martin Jansa
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 5 lines
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 6 lines · Robert Yang
Hi Martin,
I found this patch in the bug:
And this patch causes another inconsistent:
PACKAGE_CLASSES = "package_ipk"
1) After make some changes in opkg-utils-native, foo.do_package_write_ipk will re-run, but PR doesn't bump, this is because PR is bumped in do_package, and do_package doesn't depend on opkg-utils-native, but do_package_write_ipk does, so do_package_write_ipk will rerun but no PR is bumped.
2) Another more funny problem is, after make some changes in rpm-native (PACKAGE_CLASSES = "package_ipk"), both foo.do_package and foo.do_package_write_ipk will run and PR is bumped. This is because do_package depends on rpm-native (for FILERDEPENDS), so do_package re-runs and PR bumps, thus caused do_package_write_ipk re-runs.
Revert the patch will fix both 1) and 2).
For 2), the only one uses FILERDEPENDS is do_package_qa, maybe we need move the generation of FILERDEPENDS there. This is another bug.
// Robert
Quoted 107 lines
Too many PR bumps and rebuildings are caused by this patch. I'm not very sure about what this patch tries to fix, it seems that it is trying to fix problems when 32bit and 64bit uses the same sstate cache? Would you please provide a simple reproducer, please?
Things will become much better after revert this patch. Be less stricter or more is a hard problem, if we still need the patch, can we leave such a choice to the end user? We can add some vars like
DROP_NATIVE_SIG? = "0/1". This would be good to the end user who uses stable release like jethro or krogoth to make their distributions, and PR Service really matters here. Even if they switch the build between 32 and 64 builds (This is unlikely to happen for production environment) and got problems, they still can fix the problem by rebuilding, this is still much better than current status: run a "smart/opkg/apt-get upgrade" on the target, *all* of the packages are downloaded and installed again after a CVE patch applies on native recipes like pseudo-native or rpm-native, but in fact, nothing is changed on the target this is really a bad experience.// Robert
Quoted 82 lines · Martin Jansa
Quoted 28 lines · Robert Yang
Quoted 13 lines · Martin Jansa
Quoted 12 lines · Robert Yang
Hi Martin,
Quoted 8 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to
automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 5 lines
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 39 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
The bottom line is that the system is setup to be sensitive to changes. Where we've had cases where we haven't reacted to changes, people have complained and we've ended up making sure we do react to them. The patch you reference was one such case where users complained we didn't react enough.
You can't have things both ways, where it reacts to all changes but also never reacts to changes which don't have any visible effect on the end result (how do you know?).
The binary diff tool is likely going to be the only way to ultimately control a binary package feed and is still the only way I can see of solving this problem. I'd love some help on making that work. Making deterministic binaries is a large part in making that tool effectively ultimate and we had some great work on that in the last release.
To be really clear, OE-Core will not have a different signature policy on release branches since that differing policy would break user expectations and also wouldn't get tested apart from on the branch so we'd have less confidence it was working.
Users are free to set their own policies, the system was designed to do that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.
Cheers,
Richard
Quoted 17 lines · Richard Purdie
The bottom line is that the system is setup to be sensitive to changes. Where we've had cases where we haven't reacted to changes, people have complained and we've ended up making sure we do react to them. The patch you reference was one such case where users complained we didn't react enough.
You can't have things both ways, where it reacts to all changes but also never reacts to changes which don't have any visible effect on the end result (how do you know?).
The binary diff tool is likely going to be the only way to ultimately control a binary package feed and is still the only way I can see of solving this problem. I'd love some help on making that work. Making deterministic binaries is a large part in making that tool effectively ultimate and we had some great work on that in the last release.
Quoted 4 lines
To be really clear, OE-Core will not have a different signature policy on release branches since that differing policy would break user expectations and also wouldn't get tested apart from on the branch so we'd have less confidence it was working.
Yes, I agree with this, I just used stable release as an example (big changes won't happen on a stable release).
Quoted 4 lines
Users are free to set their own policies, the system was designed to do that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.
Thanks, frankly speaking, not only WindRiver wants this. After cloud computing and virtualization gets hot, more and more users want to customize their own images (for saving disk space, memory and security reason), oe/yocto is very good at customizing images, so more and more people try to use it to build their own distros, where live upgrades becomes very important.
// Robert
Quoted 5 lines
Cheers,
Richard
Quoted 17 lines · Robert Yang
Quoted 6 lines · Richard Purdie
To be really clear, OE-Core will not have a different signature policy
on release branches since that differing policy would break user expectations and also wouldn't get tested apart from on the branch so
we'd have less confidence it was working.Yes, I agree with this, I just used stable release as an example (big changes won't happen on a stable release).
Quoted 5 lines
Users are free to set their own policies, the system was designed to do
that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.
Quoted 7 lines
Thanks, frankly speaking, not only WindRiver wants this. After cloud computing and virtualization gets hot, more and more users want to customize their own images (for saving disk space, memory and security
reason), oe/yocto is very good at customizing images, so more and more people try to use it to build their own distros, where live upgrades becomes very important.
I understand that. So are we going to get binary diff working? Its the only viable solution we have to the problem at the moment that I know of. Randomly hacking bits of the signature generation isn't going to help this unfortunately.
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 17 lines · Robert Yang
Quoted 6 lines · Richard Purdie
To be really clear, OE-Core will not have a different signature policy
on release branches since that differing policy would break user expectations and also wouldn't get tested apart from on the branch so
we'd have less confidence it was working.Yes, I agree with this, I just used stable release as an example (big changes won't happen on a stable release).
Quoted 5 lines
Users are free to set their own policies, the system was designed to do
that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.Quoted 7 lines
Thanks, frankly speaking, not only WindRiver wants this. After cloud computing and virtualization gets hot, more and more users want to customize their own images (for saving disk space, memory and security
reason), oe/yocto is very good at customizing images, so more and more people try to use it to build their own distros, where live upgrades becomes very important.
Quoted 1 line
I understand that. So are we going to get binary diff working? Its the
Thanks everyone, it seems that this is the best way we have, I will try packagefeed-stability class.
// Robert
Quoted 9 lines
only viable solution we have to the problem at the moment that I know of. Randomly hacking bits of the signature generation isn't going to help this unfortunately.
Cheers,
Richard
Quoted 6 lines · Robert Yang
Quoted 4 lines · Richard Purdie
Users are free to set their own policies, the system was designed to do
that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.
Quoted 8 lines
Thanks, frankly speaking, not only WindRiver wants this. After cloud computing and virtualization gets hot, more and more users want to customize their own images (for saving disk space, memory and security
reason), oe/yocto is very good at customizing images, so more and more people try to use it to build their own distros, where live upgrades becomes very important.
The desire for this is well understood, which is why work began on the packagefeed-stability class.
Paul's initial effort is solid and works for simple cases, I had a brief look at it and tried to identify some of the issues and wrote some minor patches (for things like versioned depends and copying all packages for a recipe as soon as a difference is found) but unfortunately I haven't had time to finish the work.
The main piece we're missing is to define tests (in the test suite) to validate the class works as expected with each of the package backends and through those tests identify areas where it doesn't work (and of course fix them).
Regards,
Joshua
Quoted 29 lines · On Thu, 19 May 2016 11:37:03 Joshua G Lock wrote:
Quoted 6 lines · Robert Yang
Quoted 4 lines · Richard Purdie
Users are free to set their own policies, the system was designed to do
that. If WindRiver wants to have a much more permissive policy, I'm more than happy for them to do so.Quoted 7 lines
Thanks, frankly speaking, not only WindRiver wants this. After cloud computing and virtualization gets hot, more and more users want to customize their own images (for saving disk space, memory and security
reason), oe/yocto is very good at customizing images, so more and more people try to use it to build their own distros, where live upgrades becomes very important.The desire for this is well understood, which is why work began on the packagefeed-stability class.
Paul's initial effort is solid and works for simple cases, I had a brief look at it and tried to identify some of the issues and wrote some minor patches (for things like versioned depends and copying all packages for a recipe as soon as a difference is found) but unfortunately I haven't had time to finish the work.
The main piece we're missing is to define tests (in the test suite) to validate the class works as expected with each of the package backends and through those tests identify areas where it doesn't work (and of course fix them).
Quoted 5 lines
Regards,
Joshua
Also as I believe I mentioned in my RFC (IIRC) whilst it does basically work I was not exactly impressed by the build-compare tool's error handling, there were errors produced during some of my runs and yet the script itself succeeded, which gives me less confidence about the results. I haven't looked to see if there's a new version upstream that is improved in this regard though, and even if there isn't we do have the option of patching it ourselves (and sending the fix upstream, naturally).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
As the commit says, small change in package.bbclass also causes all packages to be recreated with PR bump even when the content is most likely the same.
Fixing bug in gcc may at least provide different binaries so it might be useful to upgrade them on target (or at least distinguish if they were already rebuilt with fixed gcc-cross or not).
Reverting the commit doesn't fix the issue that sstate handler cannot compare if the change in signature produced "significantly different" binaries. If it's reverted in oe-core I'll just revert the revert in our builds, because we care about reproducible builds and we already gave up on working upgrade paths, which are broken too often anyway.
Reflash of "system" partitions and storing user data/configuration somewhere else is faster and safer since the sstate was invented (since we stopped using OE classic). You can find some rant e-mails from me about this e.g.
http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
and the Yocto bug I gave you.
Regards,
Quoted 140 lines · Robert Yang
Hi Martin,
I found this patch in the bug:
Too many PR bumps and rebuildings are caused by this patch. I'm not very sure about what this patch tries to fix, it seems that it is trying to fix problems when 32bit and 64bit uses the same sstate cache? Would you please provide a simple reproducer, please?
Things will become much better after revert this patch. Be less stricter or more is a hard problem, if we still need the patch, can we leave such a choice to the end user? We can add some vars like
DROP_NATIVE_SIG? = "0/1". This would be good to the end user who uses stable release like jethro or krogoth to make their distributions, and PR Service really matters here. Even if they switch the build between 32 and 64 builds (This is unlikely to happen for production environment) and got problems, they still can fix the problem by rebuilding, this is still much better than current status: run a "smart/opkg/apt-get upgrade" on the target, *all* of the packages are downloaded and installed again after a CVE patch applies on native recipes like pseudo-native or rpm-native, but in fact, nothing is changed on the target this is really a bad experience.// Robert
Quoted 107 lines · Martin Jansa
Quoted 39 lines · Robert Yang
Quoted 26 lines · Martin Jansa
Quoted 16 lines · Robert Yang
Hi Martin,
Quoted 11 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to
automatically upgrade the packages when one of the dependencies changes .so
version, which you won't detect from recipe checksum of the app which is just
using the library.Quoted 7 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's
PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.Quoted 2 lines
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep
update to date with upstream, the changes from upstream don't do the manual
PR, and I don't think that we have to if they can be done automatically.Quoted 8 lines
it's quite annoying if you need to bump a lot of recipes you don't
Quoted 2 lines
control :).
What's your opinion about only consider RDEPENDS for PR service's checksum,
please ?I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected]
Quoted 49 lines
Quoted 49 lines
Quoted 48 lines
Quoted 48 lines
<mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like
pseudo-native
and rpm-native is changed. It is a very bad user experience
when we
run "smart/opkg upgrade" on running target, for example, when
we apply
a CVE patch to pseudo-native or rpm-native, or do some slight
changes
in theirdo_compile, "smart/opkg upgrade" will download/install
*all*
the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please
feel
free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for
bumping
PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.I prefer the first way, and an alternative way maybe add a var
so thatthe user can configure it: PR_CHECKSUM = "${BB_TASKHASH}" (current way) Or PR_CHECKSUM = "<recipe checksum>" -- Thanks Robert -- _______________________________________________ Openembedded-core mailing list [email protected] <mailto:[email protected]>http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 14 lines · Martin Jansa
As the commit says, small change in package.bbclass also causes all packages to be recreated with PR bump even when the content is most likely the same.
Fixing bug in gcc may at least provide different binaries so it might be useful to upgrade them on target (or at least distinguish if they were already rebuilt with fixed gcc-cross or not).
Reverting the commit doesn't fix the issue that sstate handler cannot compare if the change in signature produced "significantly different" binaries. If it's reverted in oe-core I'll just revert the revert in our builds, because we care about reproducible builds and we already gave up on working upgrade paths, which are broken too often anyway.
Quoted 7 lines
Reflash of "system" partitions and storing user data/configuration somewhere else is faster and safer since the sstate was invented (since we stopped using OE classic). You can find some rant e-mails from me about this e.g.
http://lists.openembedded.org/pipermail/openembedded-core/2011-Novemb
er/051354.html
and the Yocto bug I gave you.
FWIW I've not given up on upgrade paths and I do try and ensure basic things do get spotted and addressed during review.
IMO, the only way we'll improve the upgrade situation is if someone writes some automated tests for it, such that when we test changes, we test the upgrade paths work. We have a ton of test automation now so we have the basic framework to build something like this on top of. I do appreciate there are some challenges since it does mean maintaining a baseline state which the upgrade is tested from (previous state and previous release being the two obvious ones).
Cheers,
Richard
Hi Martin,
Quoted 2 lines · Martin Jansa
As the commit says, small change in package.bbclass also causes all packages to be recreated with PR bump even when the content is most likely the same.
That is another case we need work on.
Quoted 4 lines
Fixing bug in gcc may at least provide different binaries so it might be useful to upgrade them on target (or at least distinguish if they were already rebuilt with fixed gcc-cross or not).
The binary might be different when the compiler fixed a bug, but that may don't make any sense to the end user. AFAIK, ccache doesn't check /path/to/gcc's hash value.
Quoted 6 lines
Reverting the commit doesn't fix the issue that sstate handler cannot compare if the change in signature produced "significantly different" binaries. If it's reverted in oe-core I'll just revert the revert in our builds, because we care about reproducible builds and we already gave up on working upgrade paths, which are broken too often anyway.
Be less or more stricter is a hard problem, so we'd better let's leave this choice to the end user, something like DROP_NATIVE_SIG ? = "0/1" would help.
I'm afraid that there is nearly nothing we can do to enhance PR server's user experience based on the current stricter mode. Think about the distros like Debian, CentOS, Opensuse and so on, their "apt-get/yum/yast upgrade" never re-install all the pkgs, but the distros built out by oe-core usually does (especially, when a patch is applied to a native recipe, then all pkgs are downloaded and re-installed, it's very hard to explain such a problem to the distro's user).
// Robert
Quoted 164 lines
Reflash of "system" partitions and storing user data/configuration somewhere else is faster and safer since the sstate was invented (since we stopped using OE classic). You can find some rant e-mails from me about this e.g.
http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
and the Yocto bug I gave you.Regards,
On Thu, May 19, 2016 at 4:33 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
Hi Martin,
I found this patch in the bug:
Too many PR bumps and rebuildings are caused by this patch. I'm not very sure about what this patch tries to fix, it seems that it is trying to fix problems when 32bit and 64bit uses the same sstate cache? Would you please provide a simple reproducer, please?
Things will become much better after revert this patch. Be less stricter or more is a hard problem, if we still need the patch, can we leave such a choice to the end user? We can add some vars like
DROP_NATIVE_SIG? = "0/1". This would be good to the end user who uses stable release like jethro or krogoth to make their distributions, and PR Service really matters here. Even if they switch the build between 32 and 64 builds (This is unlikely to happen for production environment) and got problems, they still can fix the problem by rebuilding, this is still much better than current status: run a "smart/opkg/apt-get upgrade" on the target, *all* of the packages are downloaded and installed again after a CVE patch applies on native recipes like pseudo-native or rpm-native, but in fact, nothing is changed on the target this is really a bad experience.// Robert
On 05/18/2016 06:15 PM, Martin Jansa wrote:
On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
On 05/18/2016 05:20 PM, Martin Jansa wrote:
On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
Hi Martin,
On 05/18/2016 03:39 PM, Martin Jansa wrote:
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to
automatically upgrade the packages when one of the dependencies changes .so
version, which you won't detect from recipe checksum of the app which is just
using the library.For the development branch like master, yes, that would happen. But for
the stable release like jethro and krogoth, it is unlikely that would
happen, and if if does, the user can manually bump the impacted recipe's
PR to fix the problem. The current problem is that when *all* recipes'
PR are bumped, there is no way to fix the problem.You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep
update to date with upstream, the changes from upstream don't do the manual
PR, and I don't think that we have to if they can be done automatically.it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum,
please ?I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote:
The PRServer bumps PR according to do_package's
task hash, that
causes it bumps *all* packages' PR when recipes
like pseudo-native
and rpm-native is changed. It is a very bad user
experience when we
run "smart/opkg upgrade" on running target, for
example, when we apply
a CVE patch to pseudo-native or rpm-native, or do
some slight changes
in theirdo_compile, "smart/opkg upgrade" will
download/install *all*
the packages since all of the packages' PR are
bumped.Here are some rough suggestions to fix this
problem, and please feel
free to give your suggestions.
1) Do not use do_package's task for bumping PR,
the easiest way
is simulate manually bump PR -- only bump PR
when the recipe
itself's checksum is changed.2) Add a new task for PRServer, redefine its task
hash for bumping
PR, for example, this task hash only
considers RDEPENDS (no
DEPENDS), and drop any native dependencies.I prefer the first way, and an alternative way
maybe add a var so that
the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way) Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list
[email protected] <mailto:[email protected]>
<mailto:[email protected]
<mailto:[email protected]>>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 7 lines · Robert Yang
Hi Martin,
Quoted 2 lines · Martin Jansa
As the commit says, small change in package.bbclass also causes all packages to be recreated with PR bump even when the content is most likely the same.
That is another case we need work on.
It's not another case, because the root cause is the same and you're asking to revert one symptom of it, instead of trying to resolve the root cause.
Quoted 7 lines
Quoted 3 lines
Fixing bug in gcc may at least provide different binaries so it might be useful to upgrade them on target (or at least distinguish if they were already rebuilt with fixed gcc-cross or not).
The binary might be different when the compiler fixed a bug, but that may don't make any sense to the end user. AFAIK, ccache doesn't check /path/to/gcc's hash value.
That's exactly the same case as small change in package.bbclass which often doesn't change the binary output at all, which is even worse for end user to re-download and re-install exactly the same bits just because of EXTENDPRAUTO bump.
Quoted 183 lines
Quoted 5 lines
Reverting the commit doesn't fix the issue that sstate handler cannot compare if the change in signature produced "significantly different" binaries. If it's reverted in oe-core I'll just revert the revert in our builds, because we care about reproducible builds and we already gave up on working upgrade paths, which are broken too often anyway.
Be less or more stricter is a hard problem, so we'd better let's leave this choice to the end user, something like
DROP_NATIVE_SIG? = "0/1" would help.I'm afraid that there is nearly nothing we can do to enhance PR server's user experience based on the current stricter mode. Think about the distros like Debian, CentOS, Opensuse and so on, their "apt-get/yum/yast upgrade" never re-install all the pkgs, but the distros built out by oe-core usually does (especially, when a patch is applied to a native recipe, then all pkgs are downloaded and re-installed, it's very hard to explain such a problem to the distro's user).
// Robert
Quoted 164 lines
Reflash of "system" partitions and storing user data/configuration somewhere else is faster and safer since the sstate was invented (since we stopped using OE classic). You can find some rant e-mails from me about this e.g.
http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
and the Yocto bug I gave you.Regards,
On Thu, May 19, 2016 at 4:33 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
Hi Martin,
I found this patch in the bug:
Too many PR bumps and rebuildings are caused by this patch. I'm not very sure about what this patch tries to fix, it seems that it is trying to fix problems when 32bit and 64bit uses the same sstate cache? Would you please provide a simple reproducer, please?
Things will become much better after revert this patch. Be less stricter or more is a hard problem, if we still need the patch, can we leave such a choice to the end user? We can add some vars like
DROP_NATIVE_SIG? = "0/1". This would be good to the end user who uses stable release like jethro or krogoth to make their distributions, and PR Service really matters here. Even if they switch the build between 32 and 64 builds (This is unlikely to happen for production environment) and got problems, they still can fix the problem by rebuilding, this is still much better than current status: run a "smart/opkg/apt-get upgrade" on the target, *all* of the packages are downloaded and installed again after a CVE patch applies on native recipes like pseudo-native or rpm-native, but in fact, nothing is changed on the target this is really a bad experience.// Robert
On 05/18/2016 06:15 PM, Martin Jansa wrote:
On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
On 05/18/2016 05:20 PM, Martin Jansa wrote:
On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
Hi Martin,
On 05/18/2016 03:39 PM, Martin Jansa wrote:
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to
automatically upgrade the packages when one of the dependencies changes .so
version, which you won't detect from recipe checksum of the app which is just
using the library.For the development branch like master, yes, that would happen. But for
the stable release like jethro and krogoth, it is unlikely that would
happen, and if if does, the user can manually bump the impacted recipe's
PR to fix the problem. The current problem is that when *all* recipes'
PR are bumped, there is no way to fix the problem.You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep
update to date with upstream, the changes from upstream don't do the manual
PR, and I don't think that we have to if they can be done automatically.it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum,
please ?I would like to have separate handler as described in that bug, option c).
Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote:
The PRServer bumps PR according to do_package's
task hash, that
causes it bumps *all* packages' PR when recipes
like pseudo-native
and rpm-native is changed. It is a very bad user
experience when we
run "smart/opkg upgrade" on running target, for
example, when we apply
a CVE patch to pseudo-native or rpm-native, or do
some slight changes
in theirdo_compile, "smart/opkg upgrade" will
download/install *all*
the packages since all of the packages' PR are
bumped.Here are some rough suggestions to fix this
problem, and please feel
free to give your suggestions.
1) Do not use do_package's task for bumping PR,
the easiest way
is simulate manually bump PR -- only bump PR
when the recipe
itself's checksum is changed.2) Add a new task for PRServer, redefine its task
hash for bumping
PR, for example, this task hash only
considers RDEPENDS (no
DEPENDS), and drop any native dependencies.I prefer the first way, and an alternative way
maybe add a var so that
the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way) Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list
[email protected] <mailto:[email protected]>
<mailto:[email protected]
<mailto:[email protected]>>
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Hi,
just wanted to say that we are facing the same issue with PRServer and live updates, looking forward to a possible solution.
Kind regards,
Sergey
Quoted 81 lines · Robert Yang
Quoted 12 lines · Martin Jansa
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
// Robert
Quoted 41 lines
Quoted 40 lines
Quoted 40 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksRobert
--
_______________________________________________
Openembedded-core mailing list [email protected] <mailto:[email protected]>
http://lists.openembedded.org/mailman/listinfo/openembedded-coreSignature · 4 lines
_______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Hi Sergey,
Quoted 4 lines · Sergey 'Jin' Bostandzhyan
Hi,
just wanted to say that we are facing the same issue with PRServer and live updates, looking forward to a possible solution.
You can try packagefeed-stability.bbclass on master branch now, and please feel free to let me know if you find any bugs.
Usage:
INHERIT += "packagefeed-stability"
// Robert
Quoted 87 lines
Kind regards,
SergeyQuoted 81 lines · Robert Yang
Quoted 12 lines · Martin Jansa
Quoted 11 lines · Robert Yang
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
// Robert
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksQuoted 10 lines
Quoted 9 lines
Quoted 9 lines
Robert -- _______________________________________________ Openembedded-core mailing list [email protected] <mailto:[email protected]> http://lists.openembedded.org/mailman/listinfo/openembedded-core--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Hi Robert,
Quoted 9 lines · Robert Yang
Quoted 2 lines
just wanted to say that we are facing the same issue with PRServer and live updates, looking forward to a possible solution.
You can try packagefeed-stability.bbclass on master branch now, and please feel free to let me know if you find any bugs.
Usage:
INHERIT += "packagefeed-stability"
thank you, this sounds great! Is it safe to pull in this class to 2.0.2? This is what we are currently using, so I could only try it by patching my 2.0.2 based tree.
Kind regards,
Sergey
Quoted 83 lines
Quoted 83 lines · Robert Yang
Quoted 81 lines · Martin Jansa
Quoted 12 lines · Robert Yang
Quoted 11 lines
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
// Robert
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksQuoted 10 lines
Quoted 9 lines
Quoted 9 lines
Robert -- _______________________________________________ Openembedded-core mailing list [email protected] <mailto:[email protected]> http://lists.openembedded.org/mailman/listinfo/openembedded-core--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 9 lines · Sergey 'Jin' Bostandzhyan
Hi Robert,
Quoted 6 lines · Robert Yang
Quoted 2 lines
just wanted to say that we are facing the same issue with PRServer and live updates, looking forward to a possible solution.
You can try packagefeed-stability.bbclass on master branch now, and please feel free to let me know if you find any bugs.
Quoted 7 lines
Quoted 3 lines
Usage:
INHERIT += "packagefeed-stability"
thank you, this sounds great! Is it safe to pull in this class to 2.0.2? This is what we are currently using, so I could only try it by patching my 2.0.2 based tree.
Maybe it's not very easy to backport to 2.0.2, these are patches on master branch:
packagefeed-stability.bbclass: copy all packages of a recipe packagefeed-stability.bbclass: minor fixes package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES
build-compare: improve deb and ipk checking
build-compare: fix checking for named pipe and others
build-compare: make pkg-diff.sh 75% faster
build-compare: remove space at head
rpm: make --nosignature work
And there is another one which I sent today for fixing rpm + multilib:
[OE-core] [PATCH 0/1] packagefeed-stability.bbclass: fix multilib + rpm
Also another patch in bitbake:
bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2
// Robert
Quoted 89 lines
Kind regards,
SergeyQuoted 83 lines
Quoted 83 lines · Robert Yang
Quoted 81 lines · Martin Jansa
Quoted 12 lines · Robert Yang
Quoted 11 lines
Hi Martin,
Quoted 7 lines · Martin Jansa
See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970Just using recipe checksum wont work, because the main reason for PR bumps is to automatically upgrade the packages when one of the dependencies changes .so version, which you won't detect from recipe checksum of the app which is just using the library.
Quoted 7 lines
Quoted 5 lines
For the development branch like master, yes, that would happen. But for the stable release like jethro and krogoth, it is unlikely that would happen, and if if does, the user can manually bump the impacted recipe's PR to fix the problem. The current problem is that when *all* recipes' PR are bumped, there is no way to fix the problem.
You can still stop using PR service and start doing manual PR bumps, but
We can't stop PR service and start doing manual PR bumps since we need keep update to date with upstream, the changes from upstream don't do the manual PR, and I don't think that we have to if they can be done automatically.
Quoted 2 lines
it's quite annoying if you need to bump a lot of recipes you don't control :).
What's your opinion about only consider RDEPENDS for PR service's checksum, please ?
// Robert
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
On Wed, May 18, 2016 at 8:09 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM= "<recipe checksum>"--
ThanksQuoted 10 lines
Quoted 9 lines
Quoted 9 lines
Robert -- _______________________________________________ Openembedded-core mailing list [email protected] <mailto:[email protected]> http://lists.openembedded.org/mailman/listinfo/openembedded-core--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
In addition, the introduction of PR server also increased the load on the OpenPLi "feed" servers with a few terabytes per month. Luckily, bandwidth is cheap nowadays :)
Quoted 18 lines · Robert Yang
The PRServer bumps PR according to do_package's task hash, that causes it bumps *all* packages' PR when recipes like pseudo-native and rpm-native is changed. It is a very bad user experience when we run "smart/opkg upgrade" on running target, for example, when we apply a CVE patch to pseudo-native or rpm-native, or do some slight changes in their
do_compile, "smart/opkg upgrade" will download/install *all* the packages since all of the packages' PR are bumped.Here are some rough suggestions to fix this problem, and please feel free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way is simulate manually bump PR -- only bump PR when the recipe itself's checksum is changed.2) Add a new task for PRServer, redefine its task hash for bumping PR, for example, this task hash only considers RDEPENDS (no DEPENDS), and drop any native dependencies.
Quoted 6 lines
I prefer the first way, and an alternative way maybe add a var so that the user can configure it:
PR_CHECKSUM= "${BB_TASKHASH}" (current way)
OrPR_CHECKSUM= "<recipe checksum>"
Kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: [email protected]
Website: www.topicproducts.com
Please consider the environment before printing this e-mail