autotools.bbclass: Add functionality to force a distclean when reconfiguring
PATCH 1/1 autotools.bbclass: Add functionality to force a distclean when reconfiguring
Unfortunately whilst reruning configure and make against a project will mostly work there are situations where it does not correctly do the right thing.
In particular, eglibc and gcc will fail out with errors where settings do not match a previously built configuration. It could be argued they are broken but the situation is what it is. There is the possibility of more subtle errors too.
This patch adds a "make distclean" call to recipes where configure is rerunning and the sstate checksum for do_configure has changed. We could simply use a stamp but saving out the previous configuration checksum adds some data at no real overhead.
If we find there are things out there which don't have a "distclean" target, we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe, or users could disable it globally.
[YOCTO #2774]
[YOCTO #2848]
---
meta/classes/autotools.bbclass +12 -0
@@ -89,7 +89,16 @@ oe_runconf () { AUTOTOOLS_AUXDIR ?= "${S}" +CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate" + autotools_do_configure() { + if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then + if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then + echo "Previously configured build detected, running make distclean" + oe_runmake distclean + fi + fi + case ${PN} in autoconf*) ;;
@@ -180,6 +189,9 @@ autotools_do_configure() { else bbnote "nothing to configure" fi + if [ -n "${CONFIGURESTAMPFILE}" ]; then + echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE} + fi } autotools_do_install() {
I'm curious, is there any [easy] way we can force a rerun of configure as a test pass over the system?
I'd like a way to verify that both this patch works as expected, and future recipes work as expected. (It would also be nice to test the things that don't use the autotools.bbclass..)
--Mark
Quoted 61 lines · Richard Purdie
Unfortunately whilst reruning configure and make against a project will mostly work there are situations where it does not correctly do the right thing.
In particular, eglibc and gcc will fail out with errors where settings do not match a previously built configuration. It could be argued they are broken but the situation is what it is. There is the possibility of more subtle errors too.
This patch adds a "make distclean" call to recipes where configure is rerunning and the sstate checksum for
do_configurehas changed. We could simply use a stamp but saving out the previous configuration checksum adds some data at no real overhead.If we find there are things out there which don't have a "distclean" target, we can disable this behaviour with CONFIGURESTAMPFILE = "" in the recipe, or users could disable it globally.
[YOCTO #2774]
[YOCTO #2848]Signed-off-by Richard Purdie <[email protected]>---
meta/classes/autotools.bbclass +12
-0@@ -89,7 +89,16 @@ oe_runconf () { AUTOTOOLS_AUXDIR ?= "${S}" +CONFIGURESTAMPFILE = "${WORKDIR}/configure.sstate" + autotools_do_configure() { + if [ -n "${CONFIGURESTAMPFILE}" -a -e "${CONFIGURESTAMPFILE}" ]; then + if [ "`cat ${CONFIGURESTAMPFILE}`" != "${BB_TASKHASH}" ]; then + echo "Previously configured build detected, running make distclean" + oe_runmake distclean + fi + fi + case ${PN} in autoconf*) ;;@@ -180,6 +189,9 @@ autotools_do_configure() { else bbnote "nothing to configure" fi + if [ -n "${CONFIGURESTAMPFILE}" ]; then + echo ${BB_TASKHASH} > ${CONFIGURESTAMPFILE} + fi } autotools_do_install() {_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 2 lines · Mark Hatle
I'm curious, is there any [easy] way we can force a rerun of configure as a test pass over the system?
rm tmp/stamps/*/*do_configure*
along with a rename of the sstate cache so it doesn't get used.
Quoted 3 lines
I'd like a way to verify that both this patch works as expected, and future recipes work as expected. (It would also be nice to test the things that don't use the autotools.bbclass..)
Cheers,
Richard
Quoted 6 lines · Mark Hatle
I'm curious, is there any [easy] way we can force a rerun of configure as a test pass over the system?
I'd like a way to verify that both this patch works as expected, and future recipes work as expected. (It would also be nice to test the things that don't use the autotools.bbclass..)
Yeah, I expect we'll discover some cases when upstream just don't expect a distclean. By generally taking from git and re-running the entire autotools we *should* be okay, but...
Ross
Quoted 11 lines · Ross
Quoted 6 lines · Mark Hatle
I'm curious, is there any [easy] way we can force a rerun of configure as a test pass over the system?
I'd like a way to verify that both this patch works as expected, and future recipes work as expected. (It would also be nice to test the things that don't use the autotools.bbclass..)
Yeah, I expect we'll discover some cases when upstream just don't expect a distclean. By generally taking from git and re-running the entire autotools we *should* be okay, but...
Further testing suggests we either going to need a whitelist or a blacklist for this :/
The key places people get bitten are eglibc and gcc so those should be straight forward to test, the question is how widely to deploy this initially. I think the mechanism is good, its now just a question of the implementation detail.
FWIW, libgpg-error fails with checksum issues (checksumming a generated file?!) and libtool has issues about cleaning directories that have makefiles that were never generated...
Cheers,
Richard
Quoted 4 lines · Richard Purdie
The key places people get bitten are eglibc and gcc so those should be straight forward to test, the question is how widely to deploy this initially. I think the mechanism is good, its now just a question of the implementation detail.
Eglibc and gcc, at least, support building with ${B} != ${S} so it might be easier/quicker to just blow away the whole ${B} tree rather than trying to distclean it.
I imagine that most modern-ish autotools-based packages will also build fine in that configuration, though there are bound to be some that don't. It's hard to say whether there are likely to be more or fewer of those than there are ones where "make distclean" fails.
p.
Quoted 9 lines · Phil Blundell
Quoted 4 lines · Richard Purdie
The key places people get bitten are eglibc and gcc so those should be straight forward to test, the question is how widely to deploy this initially. I think the mechanism is good, its now just a question of the implementation detail.
Eglibc and gcc, at least, support building with ${B} != ${S} so it might be easier/quicker to just blow away the whole ${B} tree rather than trying to distclean it.
yeah and I think separating B from S in general has merits in long run
Quoted 6 lines
I imagine that most modern-ish autotools-based packages will also build fine in that configuration, though there are bound to be some that don't. It's hard to say whether there are likely to be more or fewer of those than there are ones where "make distclean" fails.
either way there seems to be same amount of uncertainity.
Quoted 8 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 1 line
-Khem
Quoted 7 lines · Khem Raj
Quoted 6 lines · Phil Blundell
Quoted 4 lines · Richard Purdie
The key places people get bitten are eglibc and gcc so those should be straight forward to test, the question is how widely to deploy this initially. I think the mechanism is good, its now just a question of the implementation detail.
Quoted 5 lines
Quoted 3 lines
Eglibc and gcc, at least, support building with ${B} != ${S} so it might be easier/quicker to just blow away the whole ${B} tree rather than trying to distclean it.
yeah and I think separating B from S in general has merits in long run
In principal I agree completely, however in practice at least 20-30% of the packages I try won't deal w/ the B/S separation properly.
A good optimization is likely when B != S, blow away B and start over... if they are the same, the distclean is likely the best approach..
Quoted 18 lines
Quoted 6 lines
I imagine that most modern-ish autotools-based packages will also build fine in that configuration, though there are bound to be some that don't. It's hard to say whether there are likely to be more or fewer of those than there are ones where "make distclean" fails.
either way there seems to be same amount of uncertainity.
Quoted 8 lines
p.
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 2 lines · Mark Hatle
In principal I agree completely, however in practice at least 20-30% of the packages I try won't deal w/ the B/S separation properly.
So what I did in my most recent build system[1] is:
1) Default to srcdir != builddir
2) Tag modules which don't support that
See: http://people.gnome.org/~walters/docs/build-api.txt
If a module doesn't support srcdir != builddir, I simply copy the source tree to the builddir. This is can be a confusing trap from a developer perspective; I've caught myself editing the copied source files, and then had them blown away during a build.
But in general we should pressure upstreams to support srcdir != builddir, because the benefits are great and the drawbacks small.
It'd be awesome if you guys considered having the bitbake autotools class implement the same semantics.
[1]
http://git.gnome.org/browse/gnome-ostree/tree/src/ostbuild/pyostbuild/builtin_compile_one.py#n132
Quoted 8 lines · Richard Purdie
Unfortunately whilst reruning configure and make against a project will mostly work there are situations where it does not correctly do the right thing.
In particular, eglibc and gcc will fail out with errors where settings do not match a previously built configuration. It could be argued they are broken but the situation is what it is. There is the possibility of more subtle errors too.
Quoted 4 lines
This patch adds a "make distclean" call to recipes where configure is rerunning and the sstate checksum for
do_configurehas changed. We could simply use a stamp but saving out the previous configuration checksum adds some data at no real overhead.
The major problem with distclean is that it completely falls over in the scenario where the source code has changed; the new distclean won't know about *old* object files it no longer builds, files renamed, etc.
In GNOME we just this cycle landed this patch to use "git clean -dfx" instead of "make distclean" if possible:
https://bugzilla.gnome.org/show_bug.cgi?id=656081
It might be interesting to have an option to run:
"git init; git add .; git commit -a -m auto-import" on each tarball build in OE. I've been considering doing this for jhbuild.
(My latest build system *only* builds from git repositories, so
it works there =) )
Quoted 22 lines · Colin Walters
Quoted 8 lines · Richard Purdie
Unfortunately whilst reruning configure and make against a project will mostly work there are situations where it does not correctly do the right thing.
In particular, eglibc and gcc will fail out with errors where settings do not match a previously built configuration. It could be argued they are broken but the situation is what it is. There is the possibility of more subtle errors too.
Quoted 4 lines
This patch adds a "make distclean" call to recipes where configure is rerunning and the sstate checksum for
do_configurehas changed. We could simply use a stamp but saving out the previous configuration checksum adds some data at no real overhead.The major problem with distclean is that it completely falls over in the scenario where the source code has changed; the new distclean won't know about *old* object files it no longer builds, files renamed, etc.
In GNOME we just this cycle landed this patch to use "git clean -dfx" instead of "make distclean" if possible:
https://bugzilla.gnome.org/show_bug.cgi?id=656081
Quoted 6 lines
It might be interesting to have an option to run:
"git init; git add .; git commit -a -m auto-import" on each tarball build in OE. I've been considering doing this for jhbuild.(My latest build system *only* builds from git repositories, so
it works there =) )
While we're not managing the patching w/ git.. we are with quilt. Does quilt have any facilities for restoring to the pristine last copy vs whatever is in the current tree? (I don't think it does BTW.)
--Mark
Quoted 7 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core