Skip to contentSkip to conversation

Design issue with the out-of-tree support

Buildroot Thomas Petazzoni 5 participants 10 messages May 23, 2013 – May 28, 2013

lore
Share & tools
lore
#0 lore raw

Hello,

I've restarted the work on supporting the per-package out-of-tree support, and I stumbled across a problem about which I'd like to seek the community opinion.

Currently, the "autoreconf" is executed as a pre-configure hook, i.e it is part of the configure step, and this is wrong when you move to out-of-tree build per packages.

When you do out-of-tree build per package, for each package, you have:

* A source tree, output/src/<name>-<version>/
* A target build tree, output/build/<name>-<version>/
* Optionally, a host build tree, output/build/host-<name>-<version>/

Running the configure script is something that should be done in the build tree, so it's really part of the steps that should be executed once for the target build, once for the host build.

However, running autoreconf is something that should be done in the source tree. It's part of the "patching" process of the package, more than its configuration step.

So, in my out-of-tree branch, I've moved the autoreconf step as a post-patch hook rather than a pre-configure hook.

The problem is that dependencies of a package are only prepared before its configure step is executed. This means that host-autoconf, host-automake and host-libtool are not yet available at the "patch" step, so we can't autoreconf yet.

You could tell me: let's move the package dependencies as a dependency of the patch step rather than the configure step. Yes, you could tell me that. But it would break the RTAI integration that we have in Buildroot, which relies on the assumption that the patch step of a given package can be executed without its dependencies being pulled in. See a473a616d27e46951a23d90249a31b08006098bf (package: change ordering of steps).

So, what I've done in my prototype of out-of-tree support is something like this:

$(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)

		...

$(1)-configure: $$($(2)_DEPENDENCIES)

The idea is that if the package needs host-autoconf, or host-automake or host-libtool in its dependencies, then we pull these dependencies before the "patch" step, and all other dependencies are pulled before the "configure" step.

This works for most packages, but not all! Some packages, for their autoreconf step, need more than host-autoconf, host-automake or host-libtool. They might require host-pkgconf or host-gettext, to install some .m4 macro files that are needed to make the autoreconf work. So in fact, discriminating between dependencies that should be pulled in before the "patch" step and the dependencies that can wait until the "configure" step is not simple.

Do you see solutions to this?

I see several possibilities:

* In all packages, split in two variables the dependencies needed at autoreconf time from the dependencies needed at configure time. But this seems horrible and difficult to maintain and understand for users.

* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:

$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))

		...

$(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))

		...

* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:

   $(1)-autoreconf: $$($(2)_DEPENDENCIES)

which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.

However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.

* Find a different solution for RTAI, so that we can pull the dependencies before the "patch" step.

Here is a quick explanation for the RTAI issue, to refresh your minds if needed:

RTAI is delivered as a tarball, which has a configure script to build its userspace libraries, so it is a autotools package, see package/rtai/.

This tarball also contains the kernel patches that must be applied.

In order for the kernel to be built, we need these patches to be available, so the package/rtai package must be extracted and patched. This explains why you have LINUX_DEPENDENCIES += rtai-patch in linux/linux-ext-rtai.mk.

And on the other hand, the 'rtai' package itself can't be configured/built *before* the Linux kernel has been built, because it builds kernel modules, and to build kernel modules, you need an already built kernel tree (and remember, to build the kernel, we must first apply the RTAI patches).

Ideas?

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#1 lore raw
Quoted 22 lines · Thomas Petazzoni

So, what I've done in my prototype of out-of-tree support is something like this:

$(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)

		...

$(1)-configure: $$($(2)_DEPENDENCIES)

The idea is that if the package needs host-autoconf, or host-automake or host-libtool in its dependencies, then we pull these dependencies before the "patch" step, and all other dependencies are pulled before the "configure" step.

This works for most packages, but not all! Some packages, for their autoreconf step, need more than host-autoconf, host-automake or host-libtool. They might require host-pkgconf or host-gettext, to install some .m4 macro files that are needed to make the autoreconf work. So in fact, discriminating between dependencies that should be pulled in before the "patch" step and the dependencies that can wait until the "configure" step is not simple.

Quoted 1 line

Do you see solutions to this?

Thomas, I might be missing something but it seems like all the needed dependencies are host dependencies. Can we not make all host dependencies be on the patch step and all others on the configure step. I think that would work for rtai wouldn't it?

Regards
Will

Signature · 5 lines
------------------------------------------------------------------------
Will Wagner                                     will_wagner at carallon.com
Development Manager                      Office Tel: +44 (0)20 7471 9224
Carallon Ltd, Studio G20, Shepherds Building, Rockley Rd, London W14 0DA
------------------------------------------------------------------------
#2 lore raw

Dear Will Wagner,

Quoted 6 lines · Will Wagner
Quoted 1 line

Do you see solutions to this?

Thomas, I might be missing something but it seems like all the needed dependencies are host dependencies. Can we not make all host dependencies be on the patch step and all others on the configure step. I think that would work for rtai wouldn't it?

Wow, thanks for taking the time to look into the problem, and enter the discussion!

It seems like what you're proposing would work. It is not worst than the solution of filtering out linux on one side, and including linux on the other side that I was proposing, but I still don't find this very clean.

While it solves the problem, the solution seems completely disconnected from it. In a few years: "Why the heck do we have host dependencies before patch, and other dependencies before configure, instead of having all of them together before patch? Ah, yes, because there is this crazy specific need for RTAI".

So certainly, what you're proposing solves the problem, it just doesn't really feel like a nice and clean solution. I know it's a very subjective opinion I'm giving here, so if others do agree with that proposal, I'll be fine with it, but I'd be interested in hearing the ideas and opinions of others to see if we can come up with something cleaner.

Thanks!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#3 lore raw

Hi Thomas,

Quoted 5 lines

While it solves the problem, the solution seems completely disconnected from it. In a few years: "Why the heck do we have host dependencies before patch, and other dependencies before configure, instead of having all of them together before patch? Ah, yes, because there is this crazy specific need for RTAI".

In my opinion, having two kind of dependencies will create headaches and it will make the current
packaging process more complicated.

Quoted 8 lines

So certainly, what you're proposing solves the problem, it just doesn't really feel like a nice and clean solution. I know it's a very subjective opinion I'm giving here, so if others do agree with that proposal, I'll be fine with it, but I'd be interested in hearing the ideas and opinions of others to see if we can come up with something cleaner.

Like you said in your first e-mail, I am in favor of this approach:

* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:

   $(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
                ...

   $(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))

--
Regards,
Markos Chandras

#4 lore raw
In reply to Thomas Petazzoni

Thomas, All,

On 2013-05-23 13:12 +0200, Thomas Petazzoni spake thusly:

Quoted 3 lines

I've restarted the work on supporting the per-package out-of-tree support, and I stumbled across a problem about which I'd like to seek the community opinion.

Thank you for this thorough explanations!

Quoted 3 lines

However, running autoreconf is something that should be done in the source tree. It's part of the "patching" process of the package, more than its configuration step.

Agreed.

Quoted 6 lines

I see several possibilities:

* In all packages, split in two variables the dependencies needed at autoreconf time from the dependencies needed at configure time. But this seems horrible and difficult to maintain and understand for users.

Nope.

Quoted 9 lines

* Make $(1)-patch depend on all dependencies. This will break RTAI, so we can exclude 'linux' for the list of $(1)-patch dependencies. So something like:

   $(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
		...

   $(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))
		...

What when we have another package like that? And a third? That's not really nice IMHO.

Quoted 7 lines

* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:

   $(1)-autoreconf: $$($(2)_DEPENDENCIES)

I believe that's the sane approach.

Quoted 7 lines

which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.

However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.

Maybe have the autotools infra override the dependency chain, by tweaking $(2)_TARGET_CONFIGURE or something like that?

Quoted 2 lines

* Find a different solution for RTAI, so that we can pull the dependencies before the "patch" step.

I'd say we should keep it that way, and wait for another one or two packages (if they even exist) with similar requirements, before we try to get a generic solution for that RTAI issue.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw
In reply to Thomas Petazzoni
Quoted 8 lines · Thomas Petazzoni

* Make autoreconf a step on its own, instead of being either a pre-patch hook or a post-patch hook. This may also allow to do something like a 'make <pkg>-reautoreconf' target, like we have 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this autoreconf step would be the one that has:

$(1)-autoreconf: $$($(2)_DEPENDENCIES)

Quoted 7 lines

which would work ok, since the RTAI/Linux integration depends on rtai-patch, which wouldn't pull the dependencies of the rtai package.

However, I am not yet sure how to insert this step into the package logic, since this step is specific to autotools package, and therefore would normally not belong to the pkg-generic.mk infrastructure.

I haven't read the thread in detail or thought about it very carefully,
but some time ago I already thought that this would be better.

It is not necessarily limited to autoreconf. For example, the way that
xenomai patches the linux sources fits in this as well (xenomai could add to LINUX_POST_AUTORECONF_HOOKS).

So I would propose to add another step to the generic infrastructure. I
would call it the PREPARE step, and make it a full step with CMDS and HOOKS.

The question is whether the OVERRIDE_SRCDIR infrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.

BTW the libtool fixup should probably be done in this prepare step as
well. But again, I haven't thought this through in detail.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#6 lore raw

Dear Arnout Vandecappelle,

Quoted 9 lines · Arnout Vandecappelle

I haven't read the thread in detail or thought about it very carefully,
but some time ago I already thought that this would be better.

It is not necessarily limited to autoreconf. For example, the way that
xenomai patches the linux sources fits in this as well (xenomai could add to LINUX_POST_AUTORECONF_HOOKS).

So I would propose to add another step to the generic infrastructure. I
would call it the PREPARE step, and make it a full step with CMDS and HOOKS.

Indeed. For now, it's only has a CMDS and no HOOKS, but it could be added later on.

Quoted 5 lines

The question is whether the OVERRIDE_SRCDIR infrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.

For now, I consider doing the prepare step even on OVERRIDE_SRCDIR package, even if that means that we are modifying the source directory.

Quoted 2 lines

BTW the libtool fixup should probably be done in this prepare step as
well. But again, I haven't thought this through in detail.

This is already done:

http://git.free-electrons.com/users/thomas-petazzoni/buildroot/commit/?h=out-of-tree-v3&id=5cf6166d713a43cdf83cdf794502eb77be36d43f

Thanks for the feedback!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#7 lore raw
Quoted 3 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,

On Mon, 27 May 2013 18:50:54 +0200, Arnout Vandecappelle wrote:

[snip]

Quoted 8 lines
Quoted 5 lines

The question is whether the OVERRIDE_SRCDIR infrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do the autoreconf manually. Which is probably for the best anyway, but which is a change compared to current behaviour.

For now, I consider doing the prepare step even on OVERRIDE_SRCDIR package, even if that means that we are modifying the source directory.

Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to use OVERRIDE_SRCDIR to define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.

And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".

Regards,
Arnout

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#8 lore raw

Dear Arnout Vandecappelle,

Quoted 11 lines · Arnout Vandecappelle
Quoted 2 lines

For now, I consider doing the prepare step even on OVERRIDE_SRCDIR package, even if that means that we are modifying the source directory.

Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to use OVERRIDE_SRCDIR to define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.

And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".

Right.

What about packages that are using the "local" site method (which basically, are equivalent to doing OVERRIDE_SRCDIR) ?

I know some people (or, said otherwise, customers) that use the local SITE_METHOD for autotools-based components. Those are version controlled, and of course, only configure.ac + Makefile.am are version controlled. So, when the build starts, "something" should do the autoreconf.

If Buildroot doesn't do it, who will do it? The user manually, when there are 30+ software components? Doesn't seem really practical and realistic.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#9 lore raw
Quoted 27 lines · Thomas Petazzoni

Dear Arnout Vandecappelle,

Quoted 11 lines · Arnout Vandecappelle
Quoted 2 lines

For now, I consider doing the prepare step even on OVERRIDE_SRCDIR package, even if that means that we are modifying the source directory.

Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I think it's a valid use case to use OVERRIDE_SRCDIR to define the custom source of a package, which is located in some shared directory. Having random builds modifying that source is not appropriate.

And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".

Right.

What about packages that are using the "local" site method (which basically, are equivalent to doing OVERRIDE_SRCDIR) ?

I know some people (or, said otherwise, customers) that use the local SITE_METHOD for autotools-based components. Those are version controlled, and of course, only configure.ac + Makefile.am are version controlled. So, when the build starts, "something" should do the autoreconf.

Quoted 3 lines

If Buildroot doesn't do it, who will do it? The user manually, when there are 30+ software components? Doesn't seem really practical and realistic.

Well, actually the autotools-generated makefile will itself detect that
configure.ac has changed and regenerate it in the source directory, so a non-writeable source directory will be broken anyway.

Okay then. If anyone has an issue with buildroot messing with their
OVERRIDE_SRCDIR, they can still add a OVERRIDE_SKIP_PREPARE variable that disables the prepare step.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F