curl-native build fix
cover curl-native build fix
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/fix
Thanks,
Qing He
---
Qing He (1):
curl: fix native dependency
meta/recipes-support/curl/curl_7.21.2.bb | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
PATCH 1/1 curl: fix native dependency
disable gnutls for -native
also fix --with-gnutls parameter for target
--- meta/recipes-support/curl/curl_7.21.2.bb | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
meta/recipes-support/curl/curl_7.21.2.bb +5 -2
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://COPYING;beginline=7;md5=3a34942f4ae3fbf1a303160714e66 DEPENDS = "zlib gnutls" DEPENDS_virtclass-native = "zlib-native" DEPENDS_virtclass-nativesdk = "zlib-nativesdk" -PR = "r0" +PR = "r1" SRC_URI = "http://curl.haxx.se/download/curl-${PV}.tar.bz2 \ file://noldlibpath.patch \
@@ -17,7 +17,6 @@ SRC_URI = "http://curl.haxx.se/download/curl-${PV}.tar.bz2 \ inherit autotools pkgconfig binconfig EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ - --with-gnutls=${STAGING_BINDIR_CROSS}/ \ --without-ssl \ --without-libssh2 \ --with-random=/dev/urandom \
@@ -25,6 +24,10 @@ EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ --enable-crypto-auth \ " +EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" +EXTRA_OECONF_virtclass-native_append = " --without-gnutls" +EXTRA_OECONF_virtclass-nativesdk_append = " --without-gnutls" + do_configure_prepend() { sed -i s:OPT_GNUTLS/bin:OPT_GNUTLS:g configure.ac }
Signature · 1 line
1.7.0
Re: [PATCH 1/1] curl: fix native dependency
meta/recipes-support/curl/curl_7.21.2.bb EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ · line 33
Quoted 11 lines · Qing He
meta/recipes-support/curl/curl_7.21.2.bb +4
-0@@ -25,6 +24,10 @@ EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ --enable-crypto-auth \ " +EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" +EXTRA_OECONF_virtclass-native_append = " --without-gnutls" +EXTRA_OECONF_virtclass-nativesdk_append = " --without-gnutls" + do_configure_prepend() { sed -i s:OPT_GNUTLS/bin:OPT_GNUTLS:g configure.ac }
I'm going to push a fix but I wanted to note that the above is not correct. Let me walk through this since I expect various people find this confusing.
EXTRA_OECONF_virtclass-native_append = "X"
means append "X" to the variable EXTRA_OECONF_virtclass-native
So as bitbake expands this,
EXTRA_OECONF_virtclass-native = "X" (since EXTRA_OECONF_virtclass-native was empty)
When bitbake expands overrides, the virtclass-native can be removed and this overwrites the original EXTRA_OECONF variable so EXTRA_OECONF becomes just X.
Now taking:
EXTRA_OECONF_append_virtclass-native = "X"
This is different. In this case, when overrides are expanded, this becomes EXTRA_OECONF_append and then X is added to EXTRA_OECONF.
If the virtclass-native is never expanded (not in OVERRIDES), the append is never "seen" by bitbake.
Unfortunately rewriting the above to:
EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" EXTRA_OECONF_append_virtclass-native = " --without-gnutls" EXTRA_OECONF_append_virtclass-nativesdk = " --without-gnutls"
doesn't help since we end up with two appends in the native/nativesdk cases. There is a way out of this though:
+CURLGNUTLS = " --with-gnutls=${STAGING_LIBDIR}/../" +CURLGNUTLS_virtclass-native = "--without-gnutls" +CURLGNUTLS_virtclass-nativesdk = "--without-gnutls"
and then reference CURLGNUTLS in EXTRA_OECONF which is what my update does :)
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
Quoted 24 lines
From: Richard Purdie
Sent: Sunday, November 28, 2010 10:23 PMmeta/recipes-support/curl/curl_7.21.2.bb EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ · line 33
Quoted 11 lines · Qing He
meta/recipes-support/curl/curl_7.21.2.bb +4
-0@@ -25,6 +24,10 @@ EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ --enable-crypto-auth \ " +EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" +EXTRA_OECONF_virtclass-native_append = " --without-gnutls" +EXTRA_OECONF_virtclass-nativesdk_append = " --without-gnutls" + do_configure_prepend() { sed -i s:OPT_GNUTLS/bin:OPT_GNUTLS:g configure.ac }I'm going to push a fix but I wanted to note that the above is not correct. Let me walk through this since I expect various people find this confusing.
EXTRA_OECONF_virtclass-native_append = "X"
means append "X" to the variable EXTRA_OECONF_virtclass-native
Quoted 8 lines
So as bitbake expands this,
EXTRA_OECONF_virtclass-native = "X" (since EXTRA_OECONF_virtclass-native was empty)
When bitbake expands overrides, the virtclass-native can be removed and this overwrites the original
EXTRA_OECONFvariable soEXTRA_OECONFbecomes just X.
I guess this will not happen, since finalize() first expands override and then handle append/prepend. In that way virtclass-native has been expanded before append is handled, and thus finally we'll get a variable EXTRA_OECONF_virtclass-native which ends up to mean nothing in the database. Is it true?
Quoted 11 lines
Now taking:
EXTRA_OECONF_append_virtclass-native = "X"
This is different. In this case, when overrides are expanded, this becomes
EXTRA_OECONF_appendand then X is added toEXTRA_OECONF.If the virtclass-native is never expanded (not in OVERRIDES), the append is never "seen" by bitbake.
Quoted 8 lines
Unfortunately rewriting the above to:
EXTRA_OECONF_append= " --with-gnutls=${STAGING_LIBDIR}/../" EXTRA_OECONF_append_virtclass-native = " --without-gnutls" EXTRA_OECONF_append_virtclass-nativesdk = " --without-gnutls"doesn't help since we end up with two appends in the native/nativesdk cases. There is a way out of this though:
This also confuses me a bit. When virtclass-native is expanded, EXTRA_OECONF_append is simply a variable. In that case the expanded value should override the 1st assignment of EXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append = " --without-gnutls "
and then that's what we expect.
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
Thanks
Kevin
Quoted 16 lines
(quoted diff) +3
-0+CURLGNUTLS = " --with-gnutls=${STAGING_LIBDIR}/../" +CURLGNUTLS_virtclass-native = "--without-gnutls" +CURLGNUTLS_virtclass-nativesdk = "--without-gnutls"and then reference CURLGNUTLS in
EXTRA_OECONFwhich is what my update does :)Cheers,
Richard
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Re: [PATCH 1/1] curl: fix native dependency
Quoted 10 lines · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No, EXTRA_OECONF_append is never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One _append cannot override/replace another on the same variable, its always cumulative.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 1/1] curl: fix native dependency
Quoted 12 lines
From: Chris Larson
Sent: Monday, November 29, 2010 1:54 PMQuoted 8 lines · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.
yes, you're correct. Thanks for pointing it out.
Thanks,
Kevin
Re: [PATCH 1/1] curl: fix native dependency
In reply to Chris LarsonQuoted 9 lines · Chris Larson
Quoted 8 lines · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.
Right.
Interestingly though, if I add this to curl*.bb:
FOO = "A"FOO_append = "B"
FOO_append_virtclass-native = "C"
and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
2010/11/29 Richard Purdie <[email protected]>:
Quoted 22 lines · Chris Larson
Quoted 9 lines · Kevin
Quoted 8 lines
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.Right.
Interestingly though, if I add this to curl*.bb:
Quoted 9 lines
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Hm.
Please allow me the remark that, if this does not do what you as expert expect it to to, that probably for mere mortals like me, this is a construct that is way too complicated.
What is the rationale that _append is not a variable? Wouldn't it be more logical/orthogonal/clearer if it was (and get appended at the end as most people seem to expect/assume)
Puzzled. Frans
Re: [PATCH 1/1] curl: fix native dependency
Quoted 10 lines · Frans Meulenbroeks
2010/11/29 Richard Purdie <[email protected]>:
Quoted 9 lines
Interestingly though, if I add this to curl*.bb:
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
Quoted 7 lines
Quoted 2 lines
I see FOO = "AB" which is not what I thought it would do...
Hm.
Please allow me the remark that, if this does not do what you as expert expect it to to, that probably for mere mortals like me, this is a construct that is way too complicated.
Its not that its too complicated, its just that its not following what I've understood to be the rules that variables follow. This is probably a bug in the parser but I'm very careful about jumping to that conclusion as there could be another explanation, not least the way virtclass as an override is implemented. I'd be interested in Chris' opinion which is why I was non-committal in calling it a bug ;-).
Quoted 3 lines
What is the rationale that
_appendis not a variable? Wouldn't it be more logical/orthogonal/clearer if it was (and get appended at the end as most people seem to expect/assume)
The point is that appends should stack and this is exactly the same as normal variable rules, e.g.:
FOO = "A"
FOO += "B"
FOO += "C"
and you'd expect the result to be "A B C", not "A C" which it would be if += was the append operator did not stack.
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
Quoted 36 lines
From: Richard Purdie
Sent: Monday, November 29, 2010 11:17 PMQuoted 10 lines · Frans Meulenbroeks
2010/11/29 Richard Purdie <[email protected]>:
Quoted 9 lines
Interestingly though, if I add this to curl*.bb:
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
Quoted 7 lines
Quoted 2 lines
I see FOO = "AB" which is not what I thought it would do...
Hm.
Please allow me the remark that, if this does not do what you as expert expect it to to, that probably for mere mortals like me, this is a construct that is way too complicated.Its not that its too complicated, its just that its not following what I've understood to be the rules that variables follow. This is probably a bug in the parser but I'm very careful about jumping to that conclusion as there could be another explanation, not least the way virtclass as an override is implemented. I'd be interested in Chris' opinion which is why I was non-committal in calling it a bug ;-).
Quoted 3 lines
What is the rationale that
_appendis not a variable? Wouldn't it be more logical/orthogonal/clearer if it was (and get appended at the end as most people seem to expect/assume)The point is that appends should stack and this is exactly the same as normal variable rules, e.g.:
Quoted 7 lines
FOO = "A"
FOO += "B"
FOO += "C"and you'd expect the result to be "A B C", not "A C" which it would be if += was the append operator did not stack.
then what about:
FOO = "A"FOO_append = "B"
FOO_append_virtclass-native = "C"
should it be "A C" or "A B C"?
I think I find the problem in current code (sent out in another mail), but the fix would depend on the answer here. From the current code, it looks designed for "A C".
Thanks
Kevin
Re: [PATCH 1/1] curl: fix native dependency
In reply to Richard PurdieQuoted 22 lines · Richard Purdie
Quoted 9 lines · Chris Larson
Quoted 8 lines · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.Right.
Interestingly though, if I add this to curl*.bb:
Quoted 8 lines
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Hmm, this isn't what I'd expect either. It's certainly not what was originally intended -- _append_<foo> and _prepend_<foo> were always supposed to act like normal _append/_prepend, just conditional ones.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 1/1] curl: fix native dependency
Quoted 32 lines
From: Chris Larson
Sent: Monday, November 29, 2010 11:25 PMQuoted 3 lines · Richard Purdie
Quoted 2 lines · Chris Larson
Quoted 1 line · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendQuoted 1 line
Quoted 1 line
Quoted 1 line
is simply a variable. In that case the expanded value should override the 1st
assignment
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
of
EXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.Right.
Interestingly though, if I add this to curl*.bb:
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"
Quoted 8 lines
Quoted 4 lines
and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Hmm, this isn't what I'd expect either. It's certainly not what was originally intended -- _append_<foo> and _prepend_<foo> were always supposed to act like normal _append/_prepend, just conditional ones.
I guess it's related to below lines (from finalize()):
# maybe the OVERRIDE was not yet added so keep the append
if (o and o in overrides) or not o:
self.delVarFlag(append, "_append")
if o and not o in overrides:
continueFOO_append = "B" is parsed into ("B", NONE) FOO_append_virtclass-native = "C" is parsed into ("C", virtclass-native)
If ("C", virtclass-native) is enumerated first, the result is desired: "C" is appended to FOO with ("B", NONE) simply removed.
If ("B", NONE) is enumerated first, then the rest appendixes are removed too which is not desired.
The net effect is that the 1st appendix always takes effect over the rest unless it has an override which hasn't been found yet.
Besides this problem, there's also no consideration about override priority. It can't handle multiple overrides on same FOO_append.
To solve it, we still need to iterate override list from highest priority to lowest end, and then choose the value from the one firstly matching an override. My remote machine is down now and thus unable to verify it though.
Thanks
Kevin
Re: [PATCH 1/1] curl: fix native dependency
Quoted 9 lines · Kevin
Quoted 9 lines
From: Chris Larson
Sent: Monday, November 29, 2010 11:25 PMQuoted 6 lines · Richard Purdie
Interestingly though, if I add this to curl*.bb:
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Hmm, this isn't what I'd expect either. It's certainly not what was originally intended -- _append_<foo> and _prepend_<foo> were always supposed to act like normal _append/_prepend, just conditional ones.
This is what I'd have expected too.
Quoted 7 lines
I guess it's related to below lines (from
finalize()):# maybe the OVERRIDE was not yet added so keep the append if (o and o in overrides) or not o: self.delVarFlag(append, "_append") if o and not o in overrides: continue
Correct, it is these lines that are the problem.
Quoted 8 lines
FOO_append= "B" is parsed into ("B", NONE) FOO_append_virtclass-native = "C" is parsed into ("C", virtclass-native)If ("C", virtclass-native) is enumerated first, the result is desired: "C" is appended to FOO with ("B", NONE) simply removed.
If ("B", NONE) is enumerated first, then the rest appendixes are removed too which is not desired.
This is exactly the problem.
Quoted 2 lines
The net effect is that the 1st appendix always takes effect over the rest unless it has an override which hasn't been found yet.
Right. I had a play with the code and can confirm a patch as follows fixes this problem. I've also made it warn when it finds code that would effect behaviour. For Poky its throwing up a warning about EXTRA_OEMAKE but its actually fine as whilst there is an unexpanded override, there is never any _append or _prepend to trigger this bug (the warning can show false positives). It would be nice to see if OE is suffering from this problem anywhere before we accept this fix into bitbake upstream. I'll also remove this error before this goes into Poky.
Quoted 6 lines
Besides this problem, there's also no consideration about override priority. It can't handle multiple overrides on same
FOO_append.To solve it, we still need to iterate override list from highest priority to lowest end, and then choose the value from the one firstly matching an override. My remote machine is down now and thus unable to verify it though.
This is a good point but its not a simple one :)
FOO = "A"FOO_append_OVERA = "B"FOO_append_OVERB = "C"
could result in ABC or ACB depending on whether OVERA was added to OVERRIDES late on. There isn't much we can probably do about that though.
FOO = "A"FOO_append_OVERA_OVERB = "B"FOO_append_OVERB_OVERA = "C"
plain won't work at present if I read the code correctly. I'm not sure there is an ordering constraint here though as "ABC" is the correct answer and either an OVERRIDE is appended or it isn't and priority isn't an issue?
Chris: Any thoughts on any of this?
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
Quoted 57 lines
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, December 06, 2010 9:04 AMQuoted 9 lines · Kevin
Quoted 9 lines
From: Chris Larson
Sent: Monday, November 29, 2010 11:25 PMQuoted 6 lines · Richard Purdie
Interestingly though, if I add this to curl*.bb:
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
Hmm, this isn't what I'd expect either. It's certainly not what was originally intended -- _append_<foo> and _prepend_<foo> were always supposed to act like normal _append/_prepend, just conditional ones.
This is what I'd have expected too.
Quoted 7 lines
I guess it's related to below lines (from
finalize()):# maybe the OVERRIDE was not yet added so keep the append if (o and o in overrides) or not o: self.delVarFlag(append, "_append") if o and not o in overrides: continueCorrect, it is these lines that are the problem.
Quoted 8 lines
FOO_append= "B" is parsed into ("B", NONE) FOO_append_virtclass-native = "C" is parsed into ("C", virtclass-native)If ("C", virtclass-native) is enumerated first, the result is desired: "C" is appended to FOO with ("B", NONE) simply removed.
If ("B", NONE) is enumerated first, then the rest appendixes are removed too which is not desired.
This is exactly the problem.
Quoted 2 lines
The net effect is that the 1st appendix always takes effect over the rest unless it has an override which hasn't been found yet.
Right. I had a play with the code and can confirm a patch as follows fixes this problem. I've also made it warn when it finds code that would effect behaviour. For Poky its throwing up a warning about
EXTRA_OEMAKEbut its actually fine as whilst there is an unexpanded override, there is never any_appendor_prependto trigger this bug (the warning can show false positives). It would be nice to see if OE is suffering from this problem anywhere before we accept this fix into bitbake upstream. I'll also remove this error before this goes into Poky.
Quoted 2 lines
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake&id=c77410305f
e736aa195ce720c8317b39da057052
yes, that looks a clear fix.
Quoted 18 lines
Quoted 7 lines
Besides this problem, there's also no consideration about override priority. It can't handle multiple overrides on same
FOO_append.To solve it, we still need to iterate override list from highest priority to lowest end, and then choose the value from the one firstly matching an override. My remote machine is down now and thus unable to verify it though.
This is a good point but its not a simple one :)
FOO = "A"
FOO_append_OVERA= "B"FOO_append_OVERB= "C"could result in ABC or ACB depending on whether OVERA was added to OVERRIDES late on. There isn't much we can probably do about that though.
Quoted 8 lines
FOO = "A"
FOO_append_OVERA_OVERB= "B"FOO_append_OVERB_OVERA= "C"plain won't work at present if I read the code correctly. I'm not sure there is an ordering constraint here though as "ABC" is the correct answer and either an OVERRIDE is appended or it isn't and priority isn't an issue?
Actually my original thought was a little bit different. Atm I thought that only one appendix should take effect, i.e:
FOO = "A"FOO_append_OVERA = "B"FOO_append_OVERB = "C"
If OVERB is higher priority than OVERA, the result would be "AC" then.
Now obviously my earlier thought is wrong. Every _append appends a new value if the override is valid. :-)
I also thought that there's no ordering constraint in this design. either "A B C" or "A C B" could work, unless the appendix itself has order constraint, say a patch sequence. But I'm not sure whether there's real usage and we need to handle it or not.
Quoted 7 lines
Chris: Any thoughts on any of this?
Cheers,
Richard
Thanks
Kevin
Re: [PATCH 1/1] curl: fix native dependency
Quoted 5 lines · Kevin
Quoted 3 lines
From: Richard Purdie [mailto:[email protected]]
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake&id=c77410305fe736aa195ce720c8317b39da057052yes, that looks a clear fix.
In the absence of Chris saying no, I've merged this into Poky so it doesn't get lost.
Quoted 11 lines
Actually my original thought was a little bit different. Atm I thought that only one appendix should take effect, i.e:
FOO = "A"
FOO_append_OVERA= "B"FOO_append_OVERB= "C"If OVERB is higher priority than OVERA, the result would be "AC" then.
Now obviously my earlier thought is wrong. Every
_appendappends a new value if the override is valid. :-)
Right, there is just the question of ordering.
Quoted 4 lines
I also thought that there's no ordering constraint in this design. either "A B C" or "A C B" could work, unless the appendix itself has order constraint, say a patch sequence. But I'm not sure whether there's real usage and we need to handle it or not.
I think its not something we should worry too much about, just something to be aware of.
I do still think we should support FOO_append_OVERA_OVERB_OVERC though perhaps we should leave a bug open on that?
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Kevin
Quoted 3 lines
From: Richard Purdie [mailto:[email protected]]
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake&id=c77410305fe736aa195ce720c8317b39da057052yes, that looks a clear fix.
In the absence of Chris saying no, I've merged this into Poky so it doesn't get lost.
Thank you for the fix, I've been looking the smart data section recently, and the rationale becomes much clearer to me.
However, the following case still has some confusion:
FOO = "A"FOO_append = "B"
FOO_virtclass-native = "C"
when in virtclass-native, the output is simply "C", instead of "CB" as I expected
Quoted 15 lines
Quoted 4 lines
Actually my original thought was a little bit different. Atm I thought that only one appendix should take effect, i.e:
Quoted 8 lines
FOO = "A"
FOO_append_OVERA= "B"FOO_append_OVERB= "C"If OVERB is higher priority than OVERA, the result would be "AC" then.
Now obviously my earlier thought is wrong. Every
_appendappends a new value if the override is valid. :-)Right, there is just the question of ordering.
Quoted 7 lines
Quoted 4 lines
I also thought that there's no ordering constraint in this design. either "A B C" or "A C B" could work, unless the appendix itself has order constraint, say a patch sequence. But I'm not sure whether there's real usage and we need to handle it or not.
I think its not something we should worry too much about, just something to be aware of.
There is some occasion where the ordering may matter, for example, CFLAGS (and some other command line options). But even this may avoid the need of predicatable order if we are careful.
Quoted 3 lines
I do still think we should support
FOO_append_OVERA_OVERB_OVERCthough perhaps we should leave a bug open on that?
This question also arose when I read the code, then I realize that FOO_append_OVERA_OVERB_OVERC doesn't work in current design. Previously I expected something like a staged handling, i.e. peeling off the append/prepend first, and then handling override, but actually they are not that loosely coupled.
That said, it seems supporting multiple overrides in append is still ok, I tried to sketch a patch as follow, seems work for FOO_append_virtclass-native_qemux86 = "C"
---
bitbake/lib/bb/data_smart.py +9 -3
@@ -160,9 +160,15 @@ class DataSmart: for append in appends: keep = [] for (a, o) in self.getVarFlag(append, op) or []: - if o and not o in overrides: - keep.append((a ,o)) - continue + if o: + applicable = 1 + for override in o.split('_'): + if not override in overrides: + applicable = 0 + break + if not applicable: + keep.append((a ,o)) + continue if op is "_append": sval = self.getVar(append, False) or ""
---
Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
Quoted 11 lines · Qing He
Thank you for the fix, I've been looking the smart data section recently, and the rationale becomes much clearer to me.
However, the following case still has some confusion:
FOO = "A"
FOO_append= "B"
FOO_virtclass-native = "C"when in virtclass-native, the output is simply "C", instead of "CB" as I expected
This is a tricky one. It would be interesting to see if this applies with other overrides such as MACHINE. The reason I say that is the way the virtclass-native override is added (see native.bbclass).
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
I also thought that there's no ordering constraint in this design. either "A B C" or "A C B" could work, unless the appendix itself has order constraint, say a patch sequence. But I'm not sure whether there's real usage and we need to handle it or not.
I think its not something we should worry too much about, just something to be aware of.
There is some occasion where the ordering may matter, for example, CFLAGS (and some other command line options). But even this may avoid the need of predicatable order if we are careful.
Right, there are cases it will matter like patch application but we just need to be sensitive to that when writing metadata.
Quoted 8 lines
Quoted 2 lines
I do still think we should support
FOO_append_OVERA_OVERB_OVERCthough perhaps we should leave a bug open on that?This question also arose when I read the code, then I realize that
FOO_append_OVERA_OVERB_OVERCdoesn't work in current design. Previously I expected something like a staged handling, i.e. peeling off the append/prepend first, and then handling override, but actually they are not that loosely coupled.
I'd also actually thought the code worked differently to how it does before I looked at it!
Quoted 3 lines
That said, it seems supporting multiple overrides in append is still ok, I tried to sketch a patch as follow, seems work for FOO_append_virtclass-native_qemux86 = "C"
This looks reasonable. In python you should used "True" and "False" instead of "1" and "0".
I've just mentally been asking myself what if this was a variable like SRC_URI where the name contains an "_" but since we don't allow overrides to contain the character, I think we're safe.
I also think you can simplify the logic slightly to be:
+ for override in o.split('_'):
+ if not override in overrides:
+ keep.append((a ,o))
+ continue
?
Cheers,
Richard
Quoted 30 lines
---
bitbake/lib/bb/data_smart.py +9
-4@@ -160,9 +160,15 @@ class DataSmart: for append in appends: keep = [] for (a, o) in self.getVarFlag(append, op) or []: - if o and not o in overrides: - keep.append((a ,o)) - continue + if o: + applicable = 1 + for override in o.split('_'): + if not override in overrides: + applicable = 0 + break + if not applicable: + keep.append((a ,o)) + continue if op is "_append": sval = self.getVar(append, False) or "" ---Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
Quoted 10 lines · Richard Purdie
Quoted 9 lines · Qing He
Thank you for the fix, I've been looking the smart data section recently, and the rationale becomes much clearer to me.
However, the following case still has some confusion:
FOO = "A"
FOO_append= "B"
FOO_virtclass-native = "C"
Quoted 6 lines
Quoted 2 lines
when in virtclass-native, the output is simply "C", instead of "CB" as I expected
This is a tricky one. It would be interesting to see if this applies with other overrides such as MACHINE. The reason I say that is the way the virtclass-native override is added (see native.bbclass).
So you mean this is intended? The FOO_append is meant to be hidden by the use of an override? I'd rather have expected something like this:
---
bitbake/lib/bb/data_smart.py +5 -0
@@ -149,6 +149,11 @@ class DataSmart: for var in vars: name = var[:-l] try: + for op in __setvar_keyword__: + if op in self[name]: + sval = self.getVarFlag(name, op, False) + self.setVarFlag(var, op, sval) + self[name] = self[var] except Exception: bb.msg.note(1, bb.msg.domain.Data, "Untracked delVar")
---
Quoted 10 lines
SRC_URIwhere the name contains an "_" but since we don't allow overrides to contain the character, I think we're safe.I also think you can simplify the logic slightly to be:
(quoted diff) +4
-0+ for override in o.split('_'): + if not override in overrides: + keep.append((a ,o)) + continue
Quoted 2 lines
?
This continue' falls into the scope of for override in o.split' instead of the former one, chances are the (a, o) can be appended more than once if multiple overrides not present.
Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
Quoted 30 lines · Qing He
Quoted 10 lines · Richard Purdie
Quoted 9 lines · Qing He
Thank you for the fix, I've been looking the smart data section recently, and the rationale becomes much clearer to me.
However, the following case still has some confusion:
FOO = "A"
FOO_append= "B"
FOO_virtclass-native = "C"Quoted 6 lines
Quoted 2 lines
when in virtclass-native, the output is simply "C", instead of "CB" as I expected
This is a tricky one. It would be interesting to see if this applies with other overrides such as MACHINE. The reason I say that is the way the virtclass-native override is added (see native.bbclass).
So you mean this is intended? The
FOO_appendis meant to be hidden by the use of an override? I'd rather have expected something like this:---
bitbake/lib/bb/data_smart.py +0
-0@@ -149,6 +149,11 @@ class DataSmart: for var in vars: name = var[:-l] try:
bitbake/lib/bb/data_smart.py class DataSmart:
Quoted 10 lines
bitbake/lib/bb/data_smart.py +5
-1+ for op in __setvar_keyword__: + if op in self[name]: + sval = self.getVarFlag(name, op, False) + self.setVarFlag(var, op, sval) + self[name] = self[var] except Exception: bb.msg.note(1, bb.msg.domain.Data, "Untracked delVar") ---
never mind, I thought "self[name]" as "self.dict[name]", it turns out to be __setitem__ which is at last a setVar() call.
Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
In reply to Richard PurdieQuoted 22 lines · Richard Purdie
Quoted 9 lines · Chris Larson
Quoted 8 lines · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendis simply a variable. In that case the expanded value should override the 1st assignment ofEXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.Right.
Interestingly though, if I add this to curl*.bb:
Quoted 8 lines
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
This is very interesting, I didn't see "AC" when preparing the patch so went the other way. After seeing your experiment, I also tried this:
FOO = "A"
FOO_virtclass-native = "B"
FOO_virtclass-native_append = "C"
Then for curl-native, FOO = "BC", and there is an explicit variable FOO_virtclass-native = "BC", this does not exist in RP's test. So it's more clear to me now what "*_append is never a variable" means.
However, the gut feeling is that this construct introduces some inconsistency. If "_append" is never a variable, but "_virtclass-native" is, how is bitbake supposed to handle FOO_append_virtclass-native and translate it into a FOO_virtclass-native variable?
To be more specific, if append is to be handled earlier than virtclass, FOO may not be able to be evaluated at the time when append/prepend is processed, as the following test case:
FOO = "A"
FOO_virtclass-native = "B"FOO_append = "C"
FOO_append_virtclass-nativesdk = "D"
How should it be handled?
Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
Quoted 49 lines
From: He, Qing
Sent: Tuesday, November 30, 2010 9:44 AMQuoted 3 lines · Richard Purdie
Quoted 2 lines · Chris Larson
Quoted 1 line · Kevin
This also confuses me a bit. When virtclass-native is expanded,
EXTRA_OECONF_appendQuoted 1 line
Quoted 1 line
Quoted 1 line
is simply a variable. In that case the expanded value should override the 1st
assignment
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
of
EXTRA_OECONF_append, and then we should get:
EXTRA_OECONF_append= " --without-gnutls "and then that's what we expect.
Quoted 7 lines
Quoted 2 lines
I guess I may still overlook something here, and really appreciate your explanation on the whole flow which is helpful. :-)
No,
EXTRA_OECONF_appendis never a variable. _append/_prepend are operations, not part of the name. The value gets set aside in a list of appends for that variable. One_appendcannot override/replace another on the same variable, its always cumulative.Right.
Interestingly though, if I add this to curl*.bb:
Quoted 8 lines
FOO = "A"
FOO_append= "B"
FOO_append_virtclass-native = "C"and then "bitbake curl-native -e | grep FOO" (he recipe has a BBCLASSEXTEND native) what should I see?
I see FOO = "AB" which is not what I thought it would do...
This is very interesting, I didn't see "AC" when preparing the patch so went the other way. After seeing your experiment, I also tried this:
FOO = "A"
FOO_virtclass-native = "B"
FOO_virtclass-native_append = "C"Then for curl-native, FOO = "BC", and there is an explicit variable FOO_virtclass-native = "BC", this does not exist in RP's test. So it's more clear to me now what "
*_appendis never a variable" means.
Quoted 4 lines
However, the gut feeling is that this construct introduces some inconsistency. If "
_append" is never a variable, but "_virtclass-native" is, how is bitbake supposed to handle FOO_append_virtclass-native and translate it into a FOO_virtclass-native variable?
See finalize() in data_smart.py. the override for _append/_prepend is handled specifically which is different from normal overrides tweak. There's some bug though which I described in another mail. In a nutshell, FOO_append_virtclass-native is parsed into a flag for FOO, with name as 'append', the value as ("C", virtclass-native). when finalize() handles _append/_prepend, "C" is appended to FOO directly and thus there's no FOO_virtclass-native being generated in the middle.
I'm not sure whether FOO_virtclass-native_append = "C" is a valid usage though. At least it's not in current design. As in your example, finally a new variable is created (FOO_virtclass-native) with value as 'BC'. Though override is handled before _append/_prepend, finalize() may be invoked multiple times. The 1st time will give you:
FOO = "A"
FOO_virtclass-native = "BC"
and then later invocations will get:
FOO = "BC"
FOO_virtclass-native = "BC"
This though gives a right result in the end, it's dangerous as its value is volatile in the middle.
Quoted 5 lines
To be more specific, if append is to be handled earlier than virtclass, FOO may not be able to be evaluated at the time when append/prepend is processed, as the following test case:
Quoted 7 lines
FOO = "A"
FOO_virtclass-native = "B"FOO_append= "C"
FOO_append_virtclass-nativesdk = "D"How should it be handled?
The expected result will be:
a) if you build pkgname, you'll get FOO = "AC" as the effect of _append
b) if you build pkgname-native, you'll get FOO = "BC" because overrides are handled before _append/_prepend
a) if you build pkgname-nativesdk, you'll get FOO = "AD" because override appendix is used
Thanks
Kevin
Re: [PATCH 1/1] curl: fix native dependency
Quoted 16 lines · Kevin
Quoted 6 lines
From: He, Qing
Sent: Tuesday, November 30, 2010 9:44 AM
To be more specific, if append is to be handled earlier than virtclass, FOO may not be able to be evaluated at the time when append/prepend is processed, as the following test case:Quoted 7 lines
FOO = "A"
FOO_virtclass-native = "B"FOO_append= "C"
FOO_append_virtclass-nativesdk = "D"How should it be handled?
The expected result will be:
Quoted 7 lines
a) if you build pkgname, you'll get FOO = "AC" as the effect of
_appendb) if you build pkgname-native, you'll get FOO = "BC" because overrides are handled before _append/_prepend
a) if you build pkgname-nativesdk, you'll get FOO = "AD" because override appendix is used
but this last case should be ACD as I think we've determined there is a bug in bitbake :)
Cheers,
Richard
Re: [PATCH 1/1] curl: fix native dependency
Quoted 22 lines
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, December 06, 2010 9:10 AMQuoted 18 lines · Kevin
Quoted 6 lines
From: He, Qing
Sent: Tuesday, November 30, 2010 9:44 AM
To be more specific, if append is to be handled earlier than virtclass, FOO may not be able to be evaluated at the time when append/prepend is processed, as the following test case:Quoted 7 lines
FOO = "A"
FOO_virtclass-native = "B"FOO_append= "C"
FOO_append_virtclass-nativesdk = "D"How should it be handled?
The expected result will be:
a) if you build pkgname, you'll get FOO = "AC" as the effect of
_append
Quoted 9 lines
Quoted 5 lines
b) if you build pkgname-native, you'll get FOO = "BC" because overrides are handled before _append/_prepend
a) if you build pkgname-nativesdk, you'll get FOO = "AD" because override appendix is used
but this last case should be ACD as I think we've determined there is a bug in bitbake :)
yes, you are correct. :-)
Thanks
Kevin
Re: [PATCH 1/1] curl: fix native dependency
In reply to Richard PurdieQuoted 5 lines · Richard Purdie
Quoted 4 lines · Qing He
meta/recipes-support/curl/curl_7.21.2.bb +0
-0@@ -25,6 +24,10 @@ EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ --enable-crypto-auth \ "
Quoted 11 lines
meta/recipes-support/curl/curl_7.21.2.bb EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ · line 33
Quoted 7 lines
meta/recipes-support/curl/curl_7.21.2.bb +4
-0+EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" +EXTRA_OECONF_virtclass-native_append = " --without-gnutls" +EXTRA_OECONF_virtclass-nativesdk_append = " --without-gnutls" + do_configure_prepend() { sed -i s:OPT_GNUTLS/bin:OPT_GNUTLS:g configure.ac }I'm going to push a fix but I wanted to note that the above is not correct. Let me walk through this since I expect various people find this confusing.
Thanks for the explanation.
In fact, at first, I also tried to use EXTRA_OECONF_append_virtclass-native, in the hope to override variable EXTRA_OECONF_append for different virtual classes. However, the log.do_configure still showed "--with-gnutls=...", that's why I then changed to the form in the patch, and it looked working.
And I also see something in the smart data implementation:
__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend)(_(?P<add>.*))?')Does that mean append/prepend is expected to appear before everything else?
Thanks,
Qing
Re: [PATCH 1/1] curl: fix native dependency
Quoted 27 lines · Qing He
Quoted 5 lines · Richard Purdie
Quoted 4 lines · Qing He
meta/recipes-support/curl/curl_7.21.2.bb +0
-0@@ -25,6 +24,10 @@ EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ --enable-crypto-auth \ "Quoted 11 lines
meta/recipes-support/curl/curl_7.21.2.bb EXTRA_OECONF = "--with-zlib=${STAGING_LIBDIR}/../ \ · line 33
Quoted 7 lines
meta/recipes-support/curl/curl_7.21.2.bb +4
-0+EXTRA_OECONF_append = " --with-gnutls=${STAGING_LIBDIR}/../" +EXTRA_OECONF_virtclass-native_append = " --without-gnutls" +EXTRA_OECONF_virtclass-nativesdk_append = " --without-gnutls" + do_configure_prepend() { sed -i s:OPT_GNUTLS/bin:OPT_GNUTLS:g configure.ac }I'm going to push a fix but I wanted to note that the above is not correct. Let me walk through this since I expect various people find this confusing.
Thanks for the explanation.
In fact, at first, I also tried to use EXTRA_OECONF_append_virtclass-native, in the hope to override variable
EXTRA_OECONF_appendfor different virtual classes. However, the log.do_configure still showed "--with-gnutls=...", that's why I then changed to the form in the patch, and it looked working.
Quoted 5 lines
And I also see something in the smart data implementation:
__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend)(_(?P<add>.*))?')Does that mean append/prepend is expected to appear before everything else?
It means append and prepend only work if they're the last part of the variable name.
e.g. XXX_append = "YYY"
so XXX_append_ZZZ = "YYY" won't do anything
of course if ZZZ is an override, it will be peeled off but the append only happens after its been overridden and it becomes XXX_append
Cheers,
Richard
Quoted 5 lines · Qing He
Pull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: qhe/fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/fix
Quoted 7 lines
Thanks,
Qing He
---Qing He (1):
curl: fix native dependency
Can you give more details about why this change was needed at this point?
Thanks
Sau!
Quoted 9 lines
meta/recipes-support/curl/curl_7.21.2.bb | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Quoted 6 lines · Saul Wold
Quoted 5 lines · Qing He
Pull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: qhe/fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/fix
Quoted 8 lines
Quoted 7 lines
Thanks,
Qing He
---Qing He (1):
curl: fix native dependencyCan you give more details about why this change was needed at this point?
Saul,
This was a problem I had reported to Qing off-list. It causes curl-native to fail on my Ubuntu 10.10 hosts.
Scott
Signature · 2 lines
Scott Garman Embedded Linux Distro Engineer - Yocto Project
Quoted 12 lines · Scott Garman
Quoted 11 lines · Saul Wold
Quoted 10 lines · Qing He
Pull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: qhe/fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/fixThanks,
Qing He
---
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines
Qing He (1):
curl: fix native dependencyCan you give more details about why this change was needed at this point?
Saul,
This was a problem I had reported to Qing off-list. It causes curl-native to fail on my Ubuntu 10.10 hosts.
Scott,
So we root-caused this to be because you had the gnutls-dev package installed on your system, so I am still not conviced that disabling gnutls on the native side is correct. We might need to look at adding gnutls as a native dependency of curl-native.
I will investigate this further.
Sau!
Quoted 2 lines
Scott