QA check for defined packages
cover QA check for defined packages
Folks,
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtime
Thoughts?
Thanks,
Joshua
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.
The following changes since commit 486b17af3e4cced76e9852e7634f75ea87433db2:
shared-mime-info: Fix a parallel make race (2011-10-11 18:06:56 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rfcqacheck
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rfcqacheck
Joshua Lock (1):
insane.bbclass: add qa check to ensure declared packages exist
meta/classes/insane.bbclass | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-)
Signature · 1 line
1.7.6.4
RFC PATCH 1/1 insane.bbclass: add qa check to ensure declared packages exist
This patch adds an extra package sanity check to test whether any extra packages defined by the recipe (i.e. anything other than -dev, -dbg, -doc, -staticdev and -locale packages) are actually created.
--- meta/classes/insane.bbclass | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-)
meta/classes/insane.bbclass +8 -1
@@ -520,6 +520,8 @@ python do_package_qa () { g = globals() walk_sane = True rdepends_sane = True + exists = True + for package in packages.split(): skip = (bb.data.getVar('INSANE_SKIP_' + package, d, True) or "").split() if skip:
@@ -539,13 +541,18 @@ python do_package_qa () { bb.note("Checking Package: %s" % package) path = "%s/%s" % (pkgdest, package) + if not package.endswith("-dev") and not package.endswith("-dbg") and not package.endswith("-doc") and not package.endswith("-staticdev") and not package.endswith("-locale"): + contents_len = os.listdir(path) + if not contents_len: + bb.error("QA Issue: PACKAGE %s defined but not created" % package) + exists = False if not package_qa_walk(path, warnchecks, errorchecks, skip, package, d): walk_sane = False if not package_qa_check_rdepends(package, pkgdest, skip, d): rdepends_sane = False - if not walk_sane or not rdepends_sane: + if not walk_sane or not rdepends_sane or not exists: bb.fatal("QA run found fatal errors. Please consider fixing them.") bb.note("DONE with PACKAGE QA") }
Signature · 1 line
1.7.6.4
Quoted 16 lines · Joshua Lock
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtime
Quoted 1 line
Thoughts?
I think we probably do need to cleanup some of that data.
I have some thoughts about where we're at with hob and this is probably as good a time as any to share them.
Effectively the problem is that there is a large body of data we only compute during the build process. This includes what the exact runtime dependencies of packages are, which packages exactly are generated (PACKAGES_DYNAMIC), what the size of the packages are, how long they take to build and so on. Some things we can fix, some things we can hack around but at the end of the day there are some things we just plain can't change.
I'm beginning to think we need to have two phases of control of the system:
a) The build phase
This is the step that generates the package feeds.
b) The image construction phase
This is the step that takes the package feed data and turns it into an image.
Obviously, you can skip to b) if you already have a package feed.
So we'd be talking about two UI's that could effectively hand off to each other and share a "build in progress" feedback to the user system. The image construction dialog would have a "Missing Packages? Build them here" type switch. This would mean the build system can continue on at what it does best yet the UI can let the users do what they want to do, particularly on a prebuilt package feed.
Thoughts?
Cheers,
Richard
Quoted 10 lines · Joshua Lock
Folks,
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
Quoted 7 lines
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtime
You'd probably get much better results from this patch if you account for ALLOW_EMPTY_<pkgname> btw...
Cheers,
Richard
Op 12 okt. 2011, om 14:37 heeft Richard Purdie het volgende geschreven:
Quoted 46 lines · Joshua Lock
Quoted 16 lines
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtimeQuoted 1 line
Thoughts?
I think we probably do need to cleanup some of that data.
I have some thoughts about where we're at with hob and this is probably as good a time as any to share them.
Effectively the problem is that there is a large body of data we only compute during the build process. This includes what the exact runtime dependencies of packages are, which packages exactly are generated (
PACKAGES_DYNAMIC), what the size of the packages are, how long they take to build and so on. Some things we can fix, some things we can hack around but at the end of the day there are some things we just plain can't change.I'm beginning to think we need to have two phases of control of the system:
a) The build phase
This is the step that generates the package feeds.
b) The image construction phase
This is the step that takes the package feed data and turns it into an image.
Obviously, you can skip to b) if you already have a package feed.
Quoted 8 lines
So we'd be talking about two UI's that could effectively hand off to each other and share a "build in progress" feedback to the user system. The image construction dialog would have a "Missing Packages? Build them here" type switch. This would mean the build system can continue on at what it does best yet the UI can let the users do what they want to do, particularly on a prebuilt package feed.
Thoughts?
I really, really, REALLY like the attention packages feeds are getting lately on the oe-core and yocto mailinglists!
Quoted 35 lines · Richard Purdie
Quoted 16 lines · Joshua Lock
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtimeQuoted 1 line
Thoughts?
I think we probably do need to cleanup some of that data.
I have some thoughts about where we're at with hob and this is probably as good a time as any to share them.
Effectively the problem is that there is a large body of data we only compute during the build process. This includes what the exact runtime dependencies of packages are, which packages exactly are generated (
PACKAGES_DYNAMIC), what the size of the packages are, how long they take to build and so on. Some things we can fix, some things we can hack around but at the end of the day there are some things we just plain can't change.I'm beginning to think we need to have two phases of control of the system:
Quoted 8 lines
a) The build phase
This is the step that generates the package feeds.
b) The image construction phase
This is the step that takes the package feed data and turns it into an image.
I actually think this is a neat idea, in fact we have the beginnings of a Gtk+ GUI to create images from a package feed which Rob Bradford worked on some 3 years or so ago - puccho. It's no doubt bit-rotten but may be worth a look.
I think we can reuse pieces from puccho and hob to create a GUI per this high-level design and I think we'd be much better off for it.
Quoted 1 line
Obviously, you can skip to b) if you already have a package feed.
a), right? Indeed I expect that this will be more in line with certain proposed use cases.
Quoted 6 lines
So we'd be talking about two UI's that could effectively hand off to each other and share a "build in progress" feedback to the user system. The image construction dialog would have a "Missing Packages? Build them here" type switch. This would mean the build system can continue on at what it does best yet the UI can let the users do what they want to do, particularly on a prebuilt package feed.
I like it. I think we had to "write one to throw away" to realise quite how much data we're missing up front but I support the proposed design.
Regards,
Joshua
--
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology Centre
Quoted 19 lines · Richard Purdie
Quoted 10 lines · Joshua Lock
Folks,
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
Quoted 7 lines
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtime
Quoted 2 lines
You'd probably get much better results from this patch if you account for ALLOW_EMPTY_<pkgname> btw...
Indeed, that does look useful.
So, ignoring the Hob discussion I still think this QA check will be generally useful. Are others open to this if I work on it further?
Regards,
Joshua
--
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology Centre
Quoted 50 lines · Joshua Lock
Quoted 35 lines · Richard Purdie
Quoted 16 lines · Joshua Lock
I'm looking for some comments on this WIP patch.
The reason I've added this is because the hob GUI requires us to offer much more reliable metadata - we show the user available packages, as defined by each recipe, to add to their image. Should a recipe define a package and yet not actually create it and the user has included that in their image we cause errors at build time.
However whilst the idea seems sound enough, there are some caveats - running a world build with this patch I see failures fairly early on at least a few of which I'd expect we'll need to special-case:
* eglibc-local
* linux-yocto
* linux-libc-headers
* gcc-runtimeQuoted 1 line
Thoughts?
I think we probably do need to cleanup some of that data.
I have some thoughts about where we're at with hob and this is probably as good a time as any to share them.
Effectively the problem is that there is a large body of data we only compute during the build process. This includes what the exact runtime dependencies of packages are, which packages exactly are generated (
PACKAGES_DYNAMIC), what the size of the packages are, how long they take to build and so on. Some things we can fix, some things we can hack around but at the end of the day there are some things we just plain can't change.I'm beginning to think we need to have two phases of control of the system:
Quoted 8 lines
a) The build phase
This is the step that generates the package feeds.
b) The image construction phase
This is the step that takes the package feed data and turns it into an image.
I actually think this is a neat idea, in fact we have the beginnings of a Gtk+ GUI to create images from a package feed which Rob Bradford worked on some 3 years or so ago - puccho. It's no doubt bit-rotten but may be worth a look.
Quoted 7 lines
I think we can reuse pieces from puccho and hob to create a GUI per this high-level design and I think we'd be much better off for it.
Quoted 1 line
Obviously, you can skip to b) if you already have a package feed.
a), right? Indeed I expect that this will be more in line with certain proposed use cases.
No, you'd skip the package feed generation and just end up using it so skip a) and start from b).
Quoted 9 lines
Quoted 6 lines
So we'd be talking about two UI's that could effectively hand off to each other and share a "build in progress" feedback to the user system. The image construction dialog would have a "Missing Packages? Build them here" type switch. This would mean the build system can continue on at what it does best yet the UI can let the users do what they want to do, particularly on a prebuilt package feed.
I like it. I think we had to "write one to throw away" to realise quite how much data we're missing up front but I support the proposed design.
I'd not say thrown away. We've moved a lot of the code forward and significant pieces would get reused...
Cheers,
Richard
Quoted 7 lines · Richard Purdie
Quoted 4 lines
Quoted 1 line
Obviously, you can skip to b) if you already have a package feed.
a), right? Indeed I expect that this will be more in line with certain proposed use cases.
No, you'd skip the package feed generation and just end up using it so skip a) and start from b).
My bad, I read that as skip b), not skip to b). Apologies.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
So we'd be talking about two UI's that could effectively hand off to each other and share a "build in progress" feedback to the user system. The image construction dialog would have a "Missing Packages? Build them here" type switch. This would mean the build system can continue on at what it does best yet the UI can let the users do what they want to do, particularly on a prebuilt package feed.
Quoted 5 lines
Quoted 2 lines
I like it. I think we had to "write one to throw away" to realise quite how much data we're missing up front but I support the proposed design.
I'd not say thrown away. We've moved a lot of the code forward and significant pieces would get reused...
Quite right, I meant the spirit of the quote. I've tried where possible to create re-usable components for hob. The crumbs module of the bb ui is chock full these days and there were various changes to core BitBake that will hopefully be useful outside of hob.
Cheers,
Joshua
--
Joshua Lock
Yocto Project "Johannes factotum"
Intel Open Source Technology Centre