Add recipe whitelisting class
cover Add recipe whitelisting class
This adds a class to allow whitelisting recipes from one or more layers, working in a similar manner to blacklist.bbclass. Let me know if you think this looks reasonable, in particular the naming and usage of the variables (which by necessity has to be different than blacklist.bbclass).
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.
The following changes since commit 059db140885bad379534e6ec713f3ceb4e18faea:
adt-installer: use DEPLOY_DIR in ANT_DEPLOY expansion (2015-08-16 17:28:14 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/whitelist
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/whitelist
Paul Eggleton (1):
classes/whitelist: add class to allow whitelisting recipes from a
layer
meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclass
Signature · 1 line
2.1.0
RFC PATCH 1/1 classes/whitelist: add class to allow whitelisting recipes from a layer
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Implements [YOCTO #8150].
--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclass
meta/classes/whitelist.bbclass new file +28 -0
@@ -0,0 +1,28 @@ +# Class that allows you to restrict the recipes brought from a layer to +# a specified list. This is similar in operation to blacklist.bbclass +# but note the difference in how PNWHITELIST is set - we don't use varflags +# here, we use an override for the layer name. This layer name is actually +# the name that gets added to BBFILE_COLLECTIONS in the layer's layer.conf, +# which may differ from how the layer is otherwise known. +# +# INHERIT += "whitelist" +# PNWHITELIST_layername = "recipe1 recipe2" +# +# If you would prefer to set a reason message other than the default, you +# can do so: +# +# PNWHITELIST_REASON_layername = "not supported by ${DISTRO}" + +python() { + layer = bb.utils.get_file_layer(d.getVar('FILE', True), d) + if layer: + localdata = bb.data.createCopy(d) + localdata.setVar('OVERRIDES', layer) + whitelist = localdata.getVar('PNWHITELIST', True) + if whitelist: + if not d.getVar('BPN', True) in whitelist.split(): + reason = localdata.getVar('PNWHITELIST_REASON', True) + if not reason: + reason = 'not in PNWHITELIST set for layer %s' % layer + raise bb.parse.SkipRecipe(reason) +}
Signature · 1 line
2.1.0
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 6 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Worked for all of my tests. I added all layers in meta-openembedded and then white-listed just gmock in meta-oe (aka openembedded-layer):
INHERIT += "whitelist"
PNWHITELIST_efl-layer = "no-recipe-enabled" PNWHITELIST_filesystems-layer = "no-recipe-enabled" PNWHITELIST_gnome-layer = "no-recipe-enabled" PNWHITELIST_gpe-layer = "no-recipe-enabled" PNWHITELIST_meta-initramfs = "no-recipe-enabled" PNWHITELIST_meta-python = "no-recipe-enabled" PNWHITELIST_multimedia-layer = "no-recipe-enabled" PNWHITELIST_networking-layer = "no-recipe-enabled" PNWHITELIST_openembedded-layer = "gmock" PNWHITELIST_perl-layer = "no-recipe-enabled" PNWHITELIST_ruby-layer = "no-recipe-enabled" PNWHITELIST_systemd-layer = "no-recipe-enabled" PNWHITELIST_toolchain-layer = "no-recipe-enabled" PNWHITELIST_webserver = "no-recipe-enabled" PNWHITELIST_xfce-layer = "no-recipe-enabled"
I got warnings for several of the layers, but strangely not for all of them:
WARNING: No bb files matched BBFILE_PATTERN_efl-layer '^/work/meta-openembedded/meta-efl/'
WARNING: No bb files matched BBFILE_PATTERN_filesystems-layer '^/work/meta-openembedded/meta-filesystems/'
WARNING: No bb files matched BBFILE_PATTERN_gpe-layer '^/work/meta-openembedded/meta-gpe/'
WARNING: No bb files matched BBFILE_PATTERN_meta-initramfs '^/work/meta-openembedded/meta-initramfs/'
WARNING: No bb files matched BBFILE_PATTERN_multimedia-layer '^/work/meta-openembedded/meta-multimedia/'
WARNING: No bb files matched BBFILE_PATTERN_networking-layer '^/work/meta-openembedded/meta-networking/'
WARNING: No bb files matched BBFILE_PATTERN_perl-layer '^/work/meta-openembedded/meta-perl/'
WARNING: No bb files matched BBFILE_PATTERN_meta-python '^/work/meta-openembedded/meta-python/'
WARNING: No bb files matched BBFILE_PATTERN_ruby-layer '^/work/meta-openembedded/meta-ruby/'
WARNING: No bb files matched BBFILE_PATTERN_webserver '^/work/meta-openembedded/meta-webserver/'
WARNING: No bb files matched BBFILE_PATTERN_xfce-layer '^/work/meta-openembedded/meta-xfce/'
Note that gnome-layer is not warned about, although none of its recipes are enabled (checked with "bitbake-layers show-recipes -f | grep -v '(skipped)' | grep meta-gnome"). Any idea why?
One more comment: it would be slightly nicer if empty whitelist could be distinguished from no whitelist, with "empty" meaning "enable no recipes". In other words, replace "if whitelist" with "if whitelist is not None".
I want to list all PNWHITELIST_xxx values for meta-openembedded, even when the layer is not (yet) in bblayers.sample.conf, in order to be prepared for adding it later. Doing that with an empty string is more readable than with a fake recipe name to make the variable non-empty.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 19 lines
-----Original Message-----
From: [email protected] [mailto:openembedded-core- [email protected]] On Behalf Of Patrick Ohly
Sent: Thursday, August 20, 2015 9:47 PM
To: Paul Eggleton
Cc: [email protected]
Subject: Re: [OE-core] [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layerQuoted 6 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 2 lines
Worked for all of my tests. I added all layers in meta-openembedded and then white-listed just gmock in meta-oe (aka openembedded-layer):
This worked for my tests as well, there are 160 recipes in my whitelist and there are 3 different cases:
1) All recipes are needed: nothing need to be done here.
2) No recipe is needed:
PNWHITELIST_efl-layer = "Nothing" PNWHITELIST_filesystems-layer = "Nothing" PNWHITELIST_gpe-layer = "Nothing" PNWHITELIST_meta-initramfs = "Nothing" PNWHITELIST_multimedia-layer = "Nothing" PNWHITELIST_ruby-layer = "Nothing" PNWHITELIST_systemd-layer = "Nothing" PNWHITELIST_toolchain-layer = "Nothing"
3) Some of the recipes are whitelisted, take gnome-layer for example:
PNWHITELIST_gnome-layer = "\
gnome-disk-utility \
gnome-keyring \
gsettings-desktop-schemas \
gvfs \
gvfs-gdu-volume-monitor \
libgnome-keyring \
libgtop \
libwnck \
libwnck3 \
libxklavier \
metacity \"
I also test to put some recipes both in whitelist and blacklist, and it turned out that the blacklist's priority is higher than whitelist.
Quoted 47 lines
INHERIT += "whitelist"
PNWHITELIST_efl-layer = "no-recipe-enabled" PNWHITELIST_filesystems-layer = "no-recipe-enabled" PNWHITELIST_gnome-layer = "no-recipe-enabled" PNWHITELIST_gpe-layer = "no-recipe-enabled" PNWHITELIST_meta-initramfs = "no-recipe-enabled" PNWHITELIST_meta-python = "no-recipe-enabled" PNWHITELIST_multimedia-layer = "no-recipe-enabled" PNWHITELIST_networking-layer = "no-recipe-enabled" PNWHITELIST_openembedded-layer = "gmock" PNWHITELIST_perl-layer = "no-recipe-enabled" PNWHITELIST_ruby-layer = "no-recipe-enabled" PNWHITELIST_systemd-layer = "no-recipe-enabled" PNWHITELIST_toolchain-layer = "no-recipe-enabled"
PNWHITELIST_webserver= "no-recipe-enabled" PNWHITELIST_xfce-layer = "no-recipe-enabled"I got warnings for several of the layers, but strangely not for all of them:
WARNING: No bb files matched BBFILE_PATTERN_efl-layer '^/work/meta-openembedded/meta-efl/'
WARNING: No bb files matched BBFILE_PATTERN_filesystems-layer '^/work/meta- openembedded/meta-filesystems/'
WARNING: No bb files matched BBFILE_PATTERN_gpe-layer '^/work/meta-openembedded/meta- gpe/'
WARNING: No bb files matched BBFILE_PATTERN_meta-initramfs '^/work/meta- openembedded/meta-initramfs/'
WARNING: No bb files matched BBFILE_PATTERN_multimedia-layer '^/work/meta- openembedded/meta-multimedia/'
WARNING: No bb files matched BBFILE_PATTERN_networking-layer '^/work/meta- openembedded/meta-networking/'
WARNING: No bb files matched BBFILE_PATTERN_perl-layer '^/work/meta-openembedded/meta- perl/'
WARNING: No bb files matched BBFILE_PATTERN_meta-python '^/work/meta- openembedded/meta-python/'
WARNING: No bb files matched BBFILE_PATTERN_ruby-layer '^/work/meta-openembedded/meta- ruby/'
WARNING: No bb files matchedBBFILE_PATTERN_webserver'^/work/meta-openembedded/meta- webserver/'
WARNING: No bb files matched BBFILE_PATTERN_xfce-layer '^/work/meta-openembedded/meta- xfce/'Note that gnome-layer is not warned about, although none of its recipes are enabled (checked with "bitbake-layers show-recipes -f | grep -v '(skipped)' | grep meta-gnome"). Any idea why?
Quoted 7 lines
One more comment: it would be slightly nicer if empty whitelist could be distinguished from no whitelist, with "empty" meaning "enable no recipes". In other words, replace "if whitelist" with "if whitelist is not None".
I want to list all
PNWHITELIST_xxxvalues for meta-openembedded, even when the layer is not (yet) in bblayers.sample.conf, in order to be prepared for adding it later. Doing that with an empty string is more readable than with a fake recipe name to make the variable non-empty.
I vote for this suggestion, it's better than a fake recipe name.
Thanks,
Jackie
Quoted 15 lines
--
Best Regards, Patrick OhlyThe content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
In reply to Patrick OhlyQuoted 9 lines · Patrick Ohly
One more comment: it would be slightly nicer if empty whitelist could be distinguished from no whitelist, with "empty" meaning "enable no recipes". In other words, replace "if whitelist" with "if whitelist is not None".
I want to list all
PNWHITELIST_xxxvalues for meta-openembedded, even when the layer is not (yet) in bblayers.sample.conf, in order to be prepared for adding it later. Doing that with an empty string is more readable than with a fake recipe name to make the variable non-empty.
I'd like to retract that proposal. It's unusual to distinguish between "set" and "set to empty string", and once set, it is (as far as I know) impossible to unset a variable. I.e., when a distro conf enables white listing for a layer, a user cannot undo that in local.conf.
Perhaps there should be an explicit PNWHITELIST_LAYERS variable which lists all layers for which white listing is meant to be active, similar to PACKAGES and FILES_<package>?
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
In reply to Paul EggletonQuoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 7 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclass
I should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
Speaking of it, combo-layer was designed to be an alternative way of dealing with these issues (more pain to use but causing less of a quality impact). Sadly, whilst it has seen some improvements, it still doesn't handle every operation it would need to make it work for some users and isn't widely adopted. I simply don't have the time to go and push it forward.
So my objection is on record but that is likely about all I can do, other than hope I'm overly paranoid...
Cheers,
Richard
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 51 lines · Richard Purdie
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 7 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclassI should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
Speaking of it, combo-layer was designed to be an alternative way of dealing with these issues (more pain to use but causing less of a quality impact). Sadly, whilst it has seen some improvements, it still doesn't handle every operation it would need to make it work for some users and isn't widely adopted. I simply don't have the time to go and push it forward.
Quoted 2 lines
So my objection is on record but that is likely about all I can do, other than hope I'm overly paranoid...
I fully support your objection and I also nervous about this one. Often vendors want to narrow their responsibility and focus so this class opens the door for this to be done officially. :-(
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
In reply to Richard PurdieQuoted 30 lines · Richard Purdie
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 7 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclassI should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Quoted 8 lines
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I understand that concern. I don't know whether there are currently users who take the full meta-oe and would stop doing that once this class gets added. But I know that the inverse is also going to happen:
not using meta-oe at all, starting to use a subset of it with this whitelist.bbclass. Whether that's a net gain or loss overall I have no idea.
Quoted 3 lines
Speaking of it, combo-layer was designed to be an alternative way of dealing with these issues (more pain to use but causing less of a quality impact).
I disagree here. When using combo-layer with filter to pick "desired" or "supported" recipes, you have the exact same effect as with whitelisting (only those recipes get tested).
Without filter we have indeed the situation of someone taking the entire meta-oe and activating it, but that's independent of using combo-layer or not.
Quoted 4 lines
Sadly, whilst it has seen some improvements, it still doesn't handle every operation it would need to make it work for some users and isn't widely adopted. I simply don't have the time to go and push it forward.
I'm not sure what improvements you have in mind here. The "filter" mechanism certainly could be enhanced (filtering out a recipe and adding it later does not work), but that's conceptually the same as whitelisting.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
In reply to Richard PurdieQuoted 51 lines · Richard Purdie
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 7 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclassI should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
Speaking of it, combo-layer was designed to be an alternative way of dealing with these issues (more pain to use but causing less of a quality impact). Sadly, whilst it has seen some improvements, it still doesn't handle every operation it would need to make it work for some users and isn't widely adopted. I simply don't have the time to go and push it forward.
Quoted 2 lines
So my objection is on record but that is likely about all I can do, other than hope I'm overly paranoid...
All points here are valid. We already see this with distro's which use layers verbatim e.g. angstrom
I wish everyone derived their distros that way since that respects the layer boundaries, a good chunk of work
there is still we send patches to layers to keep them up to date with changes in other layers and there still are patches
pending since every layer maintainer doesnt respond in sam time frame, but this sort of facilities if added is just going to worsen that workload.
I think amalgmation of layers start with use of combo-tool itself. This patch just takes it a step further. If we want to preserve the layer model's health we have to work towards respecting layer boundaries and I would even go a step further and suggest to discourage use of combo-tool or any sort of layer squashing.
Quoted 11 lines
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 56 lines · Khem Raj
Quoted 51 lines · Richard Purdie
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 7 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>--- meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclassI should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
Speaking of it, combo-layer was designed to be an alternative way of dealing with these issues (more pain to use but causing less of a quality impact). Sadly, whilst it has seen some improvements, it still doesn't handle every operation it would need to make it work for some users and isn't widely adopted. I simply don't have the time to go and push it forward.
Quoted 2 lines
So my objection is on record but that is likely about all I can do, other than hope I'm overly paranoid...
Quoted 14 lines
All points here are valid. We already see this with distro's which use layers verbatim e.g. angstrom
I wish everyone derived their distros that way since that respects the layer boundaries, a good chunk of work
there is still we send patches to layers to keep them up to date with changes in other layers and there still are patches
pending since every layer maintainer doesnt respond in sam time frame, but this sort of facilities if added is just going to worsen that workload.
I think amalgmation of layers start with use of combo-tool itself. This patch just takes it a step further. If we want to preserve the layer model's health we have to work towards respecting layer boundaries and I would even go a step further and suggest to discourage use of combo-tool or any sort of layer squashing.
I fully agree; in fact Poky itself is a bad example that I often have to explain for vendors. People justify putting several layers together in same repository saying that Poky does that and this is a contradiction which we need to justify as Yocto Project promotes the use of layers as one of most preeminent features but Poky does the opposite ...
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 16 lines · Otavio Salvador
Quoted 6 lines · Khem Raj
All points here are valid. We already see this with distro's which use layers verbatim e.g. angstrom
I wish everyone derived their distros that way since that respects the layer boundaries, a good chunk of work
there is still we send patches to layers to keep them up to date with changes in other layers and there still are patchesQuoted 8 lines
pending since every layer maintainer doesnt respond in sam time frame, but this sort of facilities if added is just going to worsen that workload.
I think amalgmation of layers start with use of combo-tool itself. This patch just takes it a step further. If we want to preserve the layer model's health we have to work towards respecting layer boundaries and I would even go a step further and suggest to discourage use of combo-tool or any sort of layer squashing.
Quoted 6 lines
I fully agree; in fact Poky itself is a bad example that I often have to explain for vendors. People justify putting several layers together in same repository saying that Poky does that and this is a contradiction which we need to justify as Yocto Project promotes the use of layers as one of most preeminent features but Poky does the opposite ...
Another way of looking at the issues people are having is that meta-openembedded is simply too large and it really needs to get split up into separate repos so people can get more granularity.
That is both technically hard in some ways and controversial and nobody wants to step up and try and form a consensus about doing it though. I would note the internal splits are a good start though and there is some separation of maintainership happened there already.
On the subject of poky, right from the start poky was a subset of OpenEmbedded, for good reason if we remember OE from those days. That reason was originally that OpenedHand didn't want to support all of OE for a customer, only some subset. The OSVs using the Yocto Project still have this issue today. The big difference between combo-layer and the whitelist is that in one case you don't ship the recipe. This makes it really clear to the customer what is and what is not supported. This has pros and cons, obviously.
I will state for the record that poky only has complete layers in it though, it doesn't pick components of meta-oe, I've actively avoided it. Putting layers together in one accessible form is a different topic in some ways to filtering one layer into a sublayer.
Patrick commented that whitelist and combo-layer both have the same drawback to metadata quality and that is probably true, I wasn't trying to suggest otherwise, merely highlight the other options available and their relative merits (which I didn't do a good job of).
Cheers,
Richard
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
In reply to Richard PurdieQuoted 36 lines · On Friday 21 August 2015 11:45:06 Richard Purdie wrote:
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 8 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>---
meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 meta/classes/whitelist.bbclassI should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
Quoted 8 lines
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
I appreciate the concern, but I have to be honest, I don't see this as being a problem with this class per se, for a couple of reasons:
1) People already feel the need to be selective in some way about the recipes they pull from larger layers such as meta-oe - whether they do it by using combo-layer, this class, git-filter-branch or simply by copying the recipes they want into their own layers. Using the whitelist class has one advantage over all the rest for the simple filtering case - you are always using the recipes from upstream. (OK, with combo-layer you probably are, but here it's direct.) Besides, even if people aren't doing any active filtering on the layer, they almost certainly aren't building the entire thing in any case - so can we really say they are testing it now?
2) Responsibility - it's great when people do report issues they find, but we shouldn't need to rely on people randomly coming across an issue in order to find it, we should be testing it ourselves. The job of maintaining the quality of layers rests with us - i.e. the people who are involved in maintaining the layers and developing the tools needed to do so. Martin does regular builds with a variety of layers as do others. If it's simply parse checking that we feel we are losing, it would be trivial to add extra regular checks; if nothing else, the layer index is parsing every recipe in almost every public layer every three hours; with a little extra work we could log and expose the results of that through the interface. (Am I volunteering to do this? Sure, as it'll probably be on my own time I can't promise to do it soon, but I will get to it if nobody else does first.)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 59 lines · Paul Eggleton
Quoted 36 lines · On Friday 21 August 2015 11:45:06 Richard Purdie wrote:
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 8 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>---
meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)create mode 100644 meta/classes/whitelist.bbclass
I should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
Quoted 8 lines
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
I appreciate the concern, but I have to be honest, I don't see this as being a problem with this class per se, for a couple of reasons:
1) People already feel the need to be selective in some way about the recipes they pull from larger layers such as meta-oe - whether they do it by using combo-layer, this class, git-filter-branch or simply by copying the recipes they want into their own layers. Using the whitelist class has one advantage over all the rest for the simple filtering case - you are always using the recipes from upstream. (OK, with combo-layer you probably are, but here it's direct.) Besides, even if people aren't doing any active filtering on the layer, they almost certainly aren't building the entire thing in any case - so can we really say they are testing it now?
Quoted 12 lines
2) Responsibility - it's great when people do report issues they find, but we shouldn't need to rely on people randomly coming across an issue in order to find it, we should be testing it ourselves. The job of maintaining the quality of layers rests with us - i.e. the people who are involved in maintaining the layers and developing the tools needed to do so. Martin does regular builds with a variety of layers as do others. If it's simply parse checking that we feel we are losing, it would be trivial to add extra regular checks; if nothing else, the layer index is parsing every recipe in almost every public layer every three hours; with a little extra work we could log and expose the results of that through the interface. (Am I volunteering to do this? Sure, as it'll probably be on my own time I can't promise to do it soon, but I will get to it if nobody else does first.)
Makes sense to me. I don't see the whitelist.bbclass in master-next yet. Are you just waiting for the linux-4.1 and gcc-5.2 to go in?
../Randy
Quoted 4 lines
Cheers,
Paul
Signature · 2 lines
# Randy MacLeod. SMTS, Linux, Wind River Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, Canada, K2K 2W5
Re: [RFC PATCH 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer
Quoted 75 lines · On Tuesday 25 August 2015 12:20:40 Randy MacLeod wrote:
Quoted 60 lines · Paul Eggleton
Quoted 36 lines · On Friday 21 August 2015 11:45:06 Richard Purdie wrote:
Quoted 7 lines · Paul Eggleton
Allow restricting recipes brought from a layer to a specified list. This is similar in operation to blacklist.bbclass, but instead specifies a per-layer whitelist of recipes (matched by BPN) that are able to be built from the layer - anything else is skipped. This is potentially useful where you want to bring in a select set of recipes from a larger layer e.g. meta-oe.
Quoted 8 lines
Implements [YOCTO #8150].
Signed-off-by Paul Eggleton <[email protected]>---
meta/classes/whitelist.bbclass | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)create mode 100644 meta/classes/whitelist.bbclass
I should go on record as having some pretty strong reservations about this from a philosophical standpoint.
Why? I think its going to encourage behaviours which will result in a decrease in layer quality, particularly for meta-oe.
Taking a simple example, today if someone breaks parsing in meta-oe, its quickly known about and multiple people can see and resolve the issue. Once people start using this class, many users will simply never see/care about parsing breakage in less maintained parts of the layer.
I appreciate Martin will likely notice, however this shouldn't Martin's sole responsibility.
Since people's focus will be on to narrow parts of the layer, we'll start to see islands which are well maintained/used and islands which are not. Worse, just looking at the layer, we won't be able to tell which is which.
Quoted 8 lines
I'm nervous about anything which pushes responsibility onto already overloaded maintainers and encourages behaviour which is a net loss on "quality".
I've talked to several significant users and they all "love" the idea of this class and plan to adopt it ASAP over existing tools like combo-layer. I therefore can't see much advantage of not merging the class as people are going to use it regardless.
I appreciate the concern, but I have to be honest, I don't see this as being a problem with this class per se, for a couple of reasons:
1) People already feel the need to be selective in some way about the recipes they pull from larger layers such as meta-oe - whether they do it by using combo-layer, this class, git-filter-branch or simply by copying the recipes they want into their own layers. Using the whitelist class has one advantage over all the rest for the simple filtering case - you are always using the recipes from upstream. (OK, with combo-layer you probably are, but here it's direct.) Besides, even if people aren't doing any active filtering on the layer, they almost certainly aren't building the entire thing in any case - so can we really say they are testing it now?
Quoted 13 lines
2) Responsibility - it's great when people do report issues they find, but we shouldn't need to rely on people randomly coming across an issue in order to find it, we should be testing it ourselves. The job of maintaining the quality of layers rests with us - i.e. the people who are involved in maintaining the layers and developing the tools needed to do so. Martin does regular builds with a variety of layers as do others. If it's simply parse checking that we feel we are losing, it would be trivial to add extra regular checks; if nothing else, the layer index is parsing every recipe in almost every public layer every three hours; with a little extra work we could log and expose the results of that through the interface. (Am I volunteering to do this? Sure, as it'll probably be on my own time I can't promise to do it soon, but I will get to it if nobody else does first.)
Quoted 2 lines
Makes sense to me. I don't see the whitelist.bbclass in master-next yet. Are you just waiting for the linux-4.1 and gcc-5.2 to go in?
Patrick was asking me to address his remaining issue, so at minimum we're waiting for me to do that I believe.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre