Design issue with the out-of-tree support
Hello,
I've restarted the work on supporting the per-package out-of-tree support, and I stumbled across a problem about which I'd like to seek the community opinion.
Currently, the "autoreconf" is executed as a pre-configure hook, i.e it is part of the configure step, and this is wrong when you move to out-of-tree build per packages.
When you do out-of-tree build per package, for each package, you have:
* A source tree, output/src/<name>-<version>/
* A target build tree, output/build/<name>-<version>/
* Optionally, a host build tree, output/build/host-<name>-<version>/
Running the configure script is something that should be done in the build tree, so it's really part of the steps that should be executed once for the target build, once for the host build.
However, running autoreconf is something that should be done in the source tree. It's part of the "patching" process of the package, more than its configuration step.
So, in my out-of-tree branch, I've moved the autoreconf step as a post-patch hook rather than a pre-configure hook.
The problem is that dependencies of a package are only prepared before its configure step is executed. This means that host-autoconf, host-automake and host-libtool are not yet available at the "patch" step, so we can't autoreconf yet.
You could tell me: let's move the package dependencies as a dependency of the patch step rather than the configure step. Yes, you could tell me that. But it would break the RTAI integration that we have in Buildroot, which relies on the assumption that the patch step of a given package can be executed without its dependencies being pulled in. See a473a616d27e46951a23d90249a31b08006098bf (package: change ordering of steps).
So, what I've done in my prototype of out-of-tree support is something like this:
$(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)
...
$(1)-configure: $$($(2)_DEPENDENCIES)
The idea is that if the package needs host-autoconf, or host-automake or host-libtool in its dependencies, then we pull these dependencies before the "patch" step, and all other dependencies are pulled before the "configure" step.
This works for most packages, but not all! Some packages, for their autoreconf step, need more than host-autoconf, host-automake or host-libtool. They might require host-pkgconf or host-gettext, to install some .m4 macro files that are needed to make the autoreconf work. So in fact, discriminating between dependencies that should be pulled in before the "patch" step and the dependencies that can wait until the "configure" step is not simple.
Do you see solutions to this?
I see several possibilities:
* In all packages, split in two variables the dependencies needed at autoreconf time from the dependencies needed at configure time. But this seems horrible and difficult to maintain and understand for users.
* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:
$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
...
$(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))
...
* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:
$(1)-autoreconf: $$($(2)_DEPENDENCIES)
which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.
However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.
* Find a different solution for RTAI, so that we can pull the dependencies before the "patch" step.
Here is a quick explanation for the RTAI issue, to refresh your minds if needed:
RTAI is delivered as a tarball, which has a configure script to build its userspace libraries, so it is a autotools package, see package/rtai/.
This tarball also contains the kernel patches that must be applied.
In order for the kernel to be built, we need these patches to be available, so the package/rtai package must be extracted and patched. This explains why you have LINUX_DEPENDENCIES += rtai-patch in linux/linux-ext-rtai.mk.
And on the other hand, the 'rtai' package itself can't be configured/built *before* the Linux kernel has been built, because it builds kernel modules, and to build kernel modules, you need an already built kernel tree (and remember, to build the kernel, we must first apply the RTAI patches).
Ideas?
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 22 lines · Thomas Petazzoni
So, what I've done in my prototype of out-of-tree support is something like this:
$(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)
...$(1)-configure: $$($(2)_DEPENDENCIES)
The idea is that if the package needs host-autoconf, or host-automake or host-libtool in its dependencies, then we pull these dependencies before the "patch" step, and all other dependencies are pulled before the "configure" step.
This works for most packages, but not all! Some packages, for their autoreconf step, need more than host-autoconf, host-automake or host-libtool. They might require host-pkgconf or host-gettext, to install some .m4 macro files that are needed to make the autoreconf work. So in fact, discriminating between dependencies that should be pulled in before the "patch" step and the dependencies that can wait until the "configure" step is not simple.
Quoted 1 line
Do you see solutions to this?
Thomas, I might be missing something but it seems like all the needed dependencies are host dependencies. Can we not make all host dependencies be on the patch step and all others on the configure step. I think that would work for rtai wouldn't it?
Regards
Will
Signature · 5 lines
------------------------------------------------------------------------ Will Wagner will_wagner at carallon.com Development Manager Office Tel: +44 (0)20 7471 9224 Carallon Ltd, Studio G20, Shepherds Building, Rockley Rd, London W14 0DA ------------------------------------------------------------------------
Dear Will Wagner,
Quoted 6 lines · Will Wagner
Quoted 1 line
Do you see solutions to this?
Thomas, I might be missing something but it seems like all the needed dependencies are host dependencies. Can we not make all host dependencies be on the patch step and all others on the configure step. I think that would work for rtai wouldn't it?
Wow, thanks for taking the time to look into the problem, and enter the discussion!
It seems like what you're proposing would work. It is not worst than the solution of filtering out linux on one side, and including linux on the other side that I was proposing, but I still don't find this very clean.
While it solves the problem, the solution seems completely disconnected from it. In a few years: "Why the heck do we have host dependencies before patch, and other dependencies before configure, instead of having all of them together before patch? Ah, yes, because there is this crazy specific need for RTAI".
So certainly, what you're proposing solves the problem, it just doesn't really feel like a nice and clean solution. I know it's a very subjective opinion I'm giving here, so if others do agree with that proposal, I'll be fine with it, but I'd be interested in hearing the ideas and opinions of others to see if we can come up with something cleaner.
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Hi Thomas,
Quoted 5 lines
While it solves the problem, the solution seems completely disconnected from it. In a few years: "Why the heck do we have host dependencies before patch, and other dependencies before configure, instead of having all of them together before patch? Ah, yes, because there is this crazy specific need for RTAI".
In my opinion, having two kind of dependencies will create headaches and it will make the current
packaging process more complicated.
Quoted 8 lines
So certainly, what you're proposing solves the problem, it just doesn't really feel like a nice and clean solution. I know it's a very subjective opinion I'm giving here, so if others do agree with that proposal, I'll be fine with it, but I'd be interested in hearing the ideas and opinions of others to see if we can come up with something cleaner.
Like you said in your first e-mail, I am in favor of this approach:
* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:
$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
...
$(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))--
Regards,
Markos Chandras
Thomas, All,
On 2013-05-23 13:12 +0200, Thomas Petazzoni spake thusly:
Quoted 3 lines
I've restarted the work on supporting the per-package out-of-tree support, and I stumbled across a problem about which I'd like to seek the community opinion.
Thank you for this thorough explanations!
Quoted 3 lines
However, running autoreconf is something that should be done in the source tree. It's part of the "patching" process of the package, more than its configuration step.
Agreed.
Quoted 6 lines
I see several possibilities:
* In all packages, split in two variables the dependencies needed at autoreconf time from the dependencies needed at configure time. But this seems horrible and difficult to maintain and understand for users.
Nope.
Quoted 9 lines
* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:
$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES)) ... $(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES)) ...
What when we have another package like that? And a third? That's not really nice IMHO.
Quoted 7 lines
* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:
$(1)-autoreconf: $$($(2)_DEPENDENCIES)
I believe that's the sane approach.
Quoted 7 lines
which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.
However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.
Maybe have the autotools infra override the dependency chain, by tweaking $(2)_TARGET_CONFIGURE or something like that?
Quoted 2 lines
* Find a different solution for RTAI, so that we can pull the dependencies before the "patch" step.
I'd say we should keep it that way, and wait for another one or two packages (if they even exist) with similar requirements, before we try to get a generic solution for that RTAI issue.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Quoted 8 lines · Thomas Petazzoni
* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:
$(1)-autoreconf: $$($(2)_DEPENDENCIES)
Quoted 7 lines
which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.
However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.
I haven't read the thread in detail or thought about it very carefully,
but some time ago I already thought that this would be better.
It is not necessarily limited to autoreconf. For example, the way that
xenomai patches the linux sources fits in this as well (xenomai could add to LINUX_POST_AUTORECONF_HOOKS).
So I would propose to add another step to the generic infrastructure. I
would call it the PREPARE step, and make it a full step with CMDS and HOOKS.
The question is whether the OVERRIDE_SRCDIR infrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.
BTW the libtool fixup should probably be done in this prepare step as
well. But again, I haven't thought this through in detail.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Dear Arnout Vandecappelle,
Quoted 9 lines · Arnout Vandecappelle
I haven't read the thread in detail or thought about it very carefully,
but some time ago I already thought that this would be better.It is not necessarily limited to autoreconf. For example, the way that
xenomai patches the linux sources fits in this as well (xenomai could add toLINUX_POST_AUTORECONF_HOOKS).So I would propose to add another step to the generic infrastructure. I
would call it the PREPARE step, and make it a full step with CMDS and HOOKS.
Indeed. For now, it's only has a CMDS and no HOOKS, but it could be added later on.
Quoted 5 lines
The question is whether the
OVERRIDE_SRCDIRinfrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users ofOVERRIDE_SRCDIRfor packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.
For now, I consider doing the prepare step even on OVERRIDE_SRCDIR package, even if that means that we are modifying the source directory.
Quoted 2 lines
BTW the libtool fixup should probably be done in this prepare step as
well. But again, I haven't thought this through in detail.
This is already done:
Thanks for the feedback!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 3 lines · Thomas Petazzoni
Dear Arnout Vandecappelle,
On Mon, 27 May 2013 18:50:54 +0200, Arnout Vandecappelle wrote:
[snip]
Quoted 8 lines
Quoted 5 lines
The question is whether the
OVERRIDE_SRCDIRinfrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users ofOVERRIDE_SRCDIRfor packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.For now, I consider doing the prepare step even on
OVERRIDE_SRCDIRpackage, even if that means that we are modifying the source directory.
Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to use OVERRIDE_SRCDIR to define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.
And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".
Regards,
Arnout
[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
Dear Arnout Vandecappelle,
Quoted 11 lines · Arnout Vandecappelle
Quoted 2 lines
For now, I consider doing the prepare step even on
OVERRIDE_SRCDIRpackage, even if that means that we are modifying the source directory.Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to useOVERRIDE_SRCDIRto define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".
Right.
What about packages that are using the "local" site method (which basically, are equivalent to doing OVERRIDE_SRCDIR) ?
I know some people (or, said otherwise, customers) that use the local SITE_METHOD for autotools-based components. Those are version controlled, and of course, only configure.ac + Makefile.am are version controlled. So, when the build starts, "something" should do the autoreconf.
If Buildroot doesn't do it, who will do it? The user manually, when there are 30+ software components? Doesn't seem really practical and realistic.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 27 lines · Thomas Petazzoni
Dear Arnout Vandecappelle,
Quoted 11 lines · Arnout Vandecappelle
Quoted 2 lines
For now, I consider doing the prepare step even on
OVERRIDE_SRCDIRpackage, even if that means that we are modifying the source directory.Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to useOVERRIDE_SRCDIRto define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".Right.
What about packages that are using the "local" site method (which basically, are equivalent to doing
OVERRIDE_SRCDIR) ?I know some people (or, said otherwise, customers) that use the local
SITE_METHODfor autotools-based components. Those are version controlled, and of course, only configure.ac + Makefile.am are version controlled. So, when the build starts, "something" should do the autoreconf.
Quoted 3 lines
If Buildroot doesn't do it, who will do it? The user manually, when there are 30+ software components? Doesn't seem really practical and realistic.
Well, actually the autotools-generated makefile will itself detect that
configure.ac has changed and regenerate it in the source directory, so a non-writeable source directory will be broken anyway.
Okay then. If anyone has an issue with buildroot messing with theirOVERRIDE_SRCDIR, they can still add a OVERRIDE_SKIP_PREPARE variable that disables the prepare step.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F