Skip to contentSkip to conversation

RFC: FOO_subtract, the logical antidote to FOO_append.

DISCUSSION Peter Seebach 7 participants 9 messages May 15, 2012 – May 18, 2012

lore
Share & tools
lore
#0 lore raw

There's a few cases where something is a huge list of space-separated things, and it is desireable to remove one. The example currently afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with the distro features including all but one of the words in it.

It seems to me that a counterpart to _append would make sense. Here is my basic idea:

FOO_subtract = "..."

means that, when you expand FOO:

1. Fully expand it.
2. Fully expand FOO_subtract.
3. Remove any words in FOO_subtract from FOO.
4. Yield the result.

The rationale is that the semantics of things where we're using _append seem to be consistently of the form "this is a space-separated set", and being able to remove things from a set would be Super Handy.

So I'm proposing the semantics for consideration, and if people like them, I will go try to implement it in my Copious Free Time.

-s

Signature · 1 line
Listen, get this.  Nobody with a good compiler needs to be justified.
#1 lore raw

Just a few minutes ago I was wondering if such a feature did exist...

I ran into a situation where I wanted to remove something from a .bbappend that is added to a variable using VARIABLE_append = "this and that"

Manuel

Quoted 32 lines · Peter Seebach

There's a few cases where something is a huge list of space-separated things, and it is desireable to remove one.  The example currently afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with the distro features including all but one of the words in it.

It seems to me that a counterpart to _append would make sense.  Here is my basic idea:

FOO_subtract = "..."

means that, when you expand FOO:

1. Fully expand it.
2. Fully expand FOO_subtract.
3. Remove any words in FOO_subtract from FOO.
4. Yield the result.

The rationale is that the semantics of things where we're using _append seem to be consistently of the form "this is a space-separated set", and being able to remove things from a set would be Super Handy.

So I'm proposing the semantics for consideration, and if people like them, I will go try to implement it in my Copious Free Time.

-s
--
Listen, get this.  Nobody with a good compiler needs to be justified.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#2 lore raw
Quoted 5 lines · Manuel Bessler

Just a few minutes ago I was wondering if such a feature did exist...

I ran into a situation where I wanted to remove something from a .bbappend that is added to a variable using VARIABLE_append = "this and that"

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

For what you are trying to do in a .bbappend, can be done by using oe_filter_out() from utils.bbclass, it has to be done in anonymous code.

VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"

This might be what you want.

Sau!

Quoted 43 lines

Manuel

Quoted 24 lines · Peter Seebach

There's a few cases where something is a huge list of space-separated things, and it is desireable to remove one. The example currently afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with the distro features including all but one of the words in it.

It seems to me that a counterpart to _append would make sense. Here is my basic idea:

FOO_subtract = "..."

means that, when you expand FOO:

1. Fully expand it.
2. Fully expand FOO_subtract.
3. Remove any words in FOO_subtract from FOO.
4. Yield the result.

The rationale is that the semantics of things where we're using _append seem to be consistently of the form "this is a space-separated set", and being able to remove things from a set would be Super Handy.

So I'm proposing the semantics for consideration, and if people like them, I will go try to implement it in my Copious Free Time.

Quoted 8 lines

-s
--
Listen, get this. Nobody with a good compiler needs to be justified.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#3 lore raw
Quoted 9 lines · Saul Wold
Quoted 5 lines · Manuel Bessler

Just a few minutes ago I was wondering if such a feature did exist...

I ran into a situation where I wanted to remove something from a .bbappend that is added to a variable using VARIABLE_append = "this and that"

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

Quoted 6 lines

For what you are trying to do in a .bbappend, can be done by using oe_filter_out() from utils.bbclass, it has to be done in anonymous code.

VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"

This might be what you want.

There are two issues I believe Peter is having.. the first is trying to figure out how to filter stuff out of _append. I'm sure sure the above will be able to do that.

But the real problem we keep struggling with is .conf load order.. you try to set some nice value in local.conf, and some later configuration file adds to it, or simply overwrites it. He'd like a way to strip out things like that.

I suspect the real answer though is fix the various configuration settings to better allow them to be overridden, but I'm not sure exactly how.

--Mark

Quoted 50 lines

Sau!

Quoted 37 lines

Manuel

Quoted 24 lines · Peter Seebach

There's a few cases where something is a huge list of space-separated things, and it is desireable to remove one. The example currently afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with the distro features including all but one of the words in it.

It seems to me that a counterpart to _append would make sense. Here is my basic idea:

FOO_subtract = "..."

means that, when you expand FOO:

1. Fully expand it.
2. Fully expand FOO_subtract.
3. Remove any words in FOO_subtract from FOO.
4. Yield the result.

The rationale is that the semantics of things where we're using _append seem to be consistently of the form "this is a space-separated set", and being able to remove things from a set would be Super Handy.

So I'm proposing the semantics for consideration, and if people like them, I will go try to implement it in my Copious Free Time.

Quoted 8 lines

-s
--
Listen, get this. Nobody with a good compiler needs to be justified.

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Quoted 6 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#4 lore raw
Quoted 25 lines · Mark Hatle
Quoted 12 lines · Saul Wold
Quoted 7 lines · Manuel Bessler

Just a few minutes ago I was wondering if such a feature did exist...

I ran into a situation where I wanted to remove something from a .bbappend
that is added to a variable using VARIABLE_append = "this and that"

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

Quoted 6 lines

For what you are trying to do in a .bbappend, can be done by using oe_filter_out() from utils.bbclass, it has to be done in anonymous code.

VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"

This might be what you want.

There are two issues I believe Peter is having.. the first is trying to figure out how to filter stuff out of _append.  I'm sure sure the above will be able to do that.

Quoted 7 lines

But the real problem we keep struggling with is .conf load order.. you try to set some nice value in local.conf, and some later configuration file adds to it, or simply overwrites it.  He'd like a way to strip out things like that.

I suspect the real answer though is fix the various configuration settings to better allow them to be overridden, but I'm not sure exactly how.

One related thing that won't solve this, but is worth noting -- I think ??= is a much more sane choice than ?= in almost all cases. We should think about a janitorial task to switch to ??= after carefully considering potential implications for each.

Signature · 1 line
Christopher Larson
#5 lore raw
In reply to Mark Hatle
Quoted 3 lines · Mark Hatle

But the real problem we keep struggling with is .conf load order.. you try to set some nice value in local.conf, and some later configuration file adds to it, or simply overwrites it. He'd like a way to strip out things like that.

OVERRIDES .= ":local"
VAR_local = "I rule"

Or you could use the existing "_forcevariable" override although I have a feeling that was intended for something else.

p.

#6 lore raw
In reply to Saul Wold
Quoted 2 lines · On Wed, 16 May 2012 07:35:45 +0300 Saul Wold <[email protected]> wrote:

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

Yes.

But consider, if you will, the specific case of DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc except that it lacks RPC.

Anything I do that isn't processed at the tail end of everything, around the point where _appends are processed, will be unable to cleanly obtain "the value that would have been set by default if nothing else happened", and then remove a word from it. I can't set a value in advance, because if I do the ?= won't fire and none of those words will get set. I can't necessarily set a value later.

Overrides won't work either, because overrides also destroy the existing values.

It seems to me that for a subtraction to work, it *must* be the very last thing done.

Basically, the purpose of suggesting this as a formal behavior defined to be The Very Last Thing is to minimize the complexity of the ordering implications. You get exactly what you would have gotten otherwise, with these words removed.

-s

Signature · 1 line
Listen, get this.  Nobody with a good compiler needs to be justified.
#7 lore raw
Quoted 22 lines · Peter Seebach
Quoted 2 lines · On Wed, 16 May 2012 07:35:45 +0300 Saul Wold <[email protected]> wrote:

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

Yes.

But consider, if you will, the specific case of DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc except that it lacks RPC.

Anything I do that isn't processed at the tail end of everything, around the point where _appends are processed, will be unable to cleanly obtain "the value that would have been set by default if nothing else happened", and then remove a word from it.  I can't set a value in advance, because if I do the ?= won't fire and none of those words will get set.  I can't necessarily set a value later.

Overrides won't work either, because overrides also destroy the existing values.

Quoted 7 lines

It seems to me that for a subtraction to work, it *must* be the very last thing done.

Basically, the purpose of suggesting this as a formal behavior defined to be The Very Last Thing is to minimize the complexity of the ordering implications.  You get exactly what you would have gotten otherwise, with these words removed.

I agree. I think this is the only sane option for implementing this given bitbake's current behavior. Clearly a -= would be extremely problematic, and would have clear complications with variable expansion, as we discussed on IRC. I'd be interested in seeing a prototype implementation of this, to experiment with it and see how viable it is.

Signature · 1 line
Christopher Larson
#8 lore raw
Quoted 32 lines · Chris Larson
Quoted 22 lines · Peter Seebach
Quoted 2 lines · On Wed, 16 May 2012 07:35:45 +0300 Saul Wold <[email protected]> wrote:

My understanding is that a _subtract is fraught with danger, there all sorts of ordering implications.

Yes.

But consider, if you will, the specific case of DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc except that it lacks RPC.

Anything I do that isn't processed at the tail end of everything, around the point where _appends are processed, will be unable to cleanly obtain "the value that would have been set by default if nothing else happened", and then remove a word from it. I can't set a value in advance, because if I do the ?= won't fire and none of those words will get set. I can't necessarily set a value later.

Overrides won't work either, because overrides also destroy the existing values.

Quoted 7 lines

It seems to me that for a subtraction to work, it *must* be the very last thing done.

Basically, the purpose of suggesting this as a formal behavior defined to be The Very Last Thing is to minimize the complexity of the ordering implications. You get exactly what you would have gotten otherwise, with these words removed.

Quoted 6 lines

I agree. I think this is the only sane option for implementing this given bitbake's current behavior. Clearly a -= would be extremely problematic, and would have clear complications with variable expansion, as we discussed on IRC. I'd be interested in seeing a prototype implementation of this, to experiment with it and see how viable it is.

The _subtract or _remove format of this might be a viable way of adding this functionality. "-=" is the one I'd prefer to avoid as it will end up confusing people which is why we've all effectively been against it but the _xxxx syntax does match the behaviour/ordering of _append and _prepend and hence could be added with less confusion.

So yes, I think this could be the best way to address some of these cases. It has always bugged me that its near impossible to counteract an _append other than with an override.

I also agree we should audit the system and change to ??= where it makes sense which will reduce the need for this.

Cheers,

Richard