package/waf: add a blind Config.in.host

Now that waf has a blind Config.in.host we can require that packages setting <PKG>_NEEDS_EXTERNAL_WAF to YES slso select host-waf.

Signed-off-by Carlos Santos <[email protected]>

---
 docs/manual/adding-packages-waf.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

docs/manual/adding-packages-waf.txt +0 -0

index 43b557c070..f38c3a6f55 100644

The plan for the future is:

* All host packages have a Config.in.host option.

* The host packages that are only build dependencies of other packages have a blind Config.in.host option

* The host packages that are useful by themselves continue to have a visible Config.in.host option.

host-waf gets a blind Config.in.host, because it exists only to build Waf-based packages that set <PKG>_NEEDS_EXTERNAL_WAF to YES.

A help text is included to document the package, only, since it is not shown in the configuration menu.

Signed-off-by Carlos Santos <[email protected]>

---
Changes v1->v2:
- Explain the motivation in the commit message

Quoted 13 lines

The plan for the future is:

* All host packages have a Config.in.host option.

* The host packages that are only build dependencies of other packages have a blind Config.in.host option

* The host packages that are useful by themselves continue to have a visible Config.in.host option.

host-waf gets a blind Config.in.host, because it exists only to build Waf-based packages that set <PKG>_NEEDS_EXTERNAL_WAF to YES.

Quoted 7 lines

A help text is included to document the package, only, since it is not shown in the configuration menu.

Signed-off-by Carlos Santos <[email protected]>

---
Changes v1->v2:
- Explain the motivation in the commit message

In fact the commit message doesn't explain the actual motivation. What prompted you to add a blind Config.in.host option for waf specifically ? Is this just a part of "let's start adding blind option for all host packages", or is there a specific issue with host-waf that this blind Config.in.host option will allow to solve ?

I.e: I am all in favor of this change, but I'm just curious to understand why you did it in the first place.

package/Config.in.host menu "Host utilities" · line 70

Quoted 12 lines

package/Config.in.host +1 -0

index 16b474fc9d..3644436fe3 100644
@@ -64,6 +64,7 @@ menu "Host utilities"
 	source "package/uboot-tools/Config.in.host"
 	source "package/util-linux/Config.in.host"
 	source "package/vboot-utils/Config.in.host"
+	source "package/waf/Config.in.host"
 	source "package/xorriso/Config.in.host"
 	source "package/zip/Config.in.host"
 	source "package/zstd/Config.in.host"

I'm not sure it's really nice to include all the host packages Config.in.host files inside the "Host utilities" menu. In practice, it's a blind option so it doesn't matter much, but it will clutter a lot this list, making it difficult to see which one is there to be really visible in menuconfig, and which one is here just so that the kconfig machinery knows about all those blind options.

I think I would prefer a new list, at the bottom of package/Config.in.host, outside of the Host utilities menu.

Let's see what Arnout/Peter/Yann have to say about this.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Quoted 2 lines

I.e: I am all in favor of this change, but I'm just curious to understand why you did it in the first place.

Well, I am not really happy with that, though: do we really plan on having packages really select all the host tools they need?

If so, do we really envision autotools-based packages selecting host-autoconf, host-automake, host-libtool? And then packages that use host-pkgconf you should also select it...

Also, what about host-cmake, which is conditionally built, but for which we do not have the info in kconfig? (well, we can argue we'd have to do like we do for host-gcc, but still). Oh, and host-tar, host-flex, host-bison, and so on... :-/

So, no, I'm not happy with that direction...

config BR2_PACKAGE_FOO
bool "foo"
select BR2_PACKAGE_HOST_AUTOCONF select BR2_PACKAGE_HOST_AUTOMAKE select BR2_PACKAGE_HOST_LIBTOOL
select BR2_PACKAGE_MAYBE_HOST_TAR select BR2_PACKAGE_MAYBE_HOST_FLEX_FOR_KCONFIG select BR2_PACKAGE_MAYBE_HOST_BISON_FOR_KCONFIG select BR2_PACKAGE_HOST_PKGCONF

Unles we're planning on hiding that away into meta-config, like:

    config BR2_PACKAGE_FOO
        bool "foo"
        select BR2_AUTOTOOLS_PACKAGE # mimick $(eval $(autotools-package))
        select BR2_PACKAGE_HOST_PKGCONF_BECAUSE_IT_S_NOT_MANDATORY

And still, the optionally-required host packages like tar, flex et al. are not covered...

Meh... :-(

Quoted 11 lines

Well, I am not really happy with that, though: do we really plan on having packages really select all the host tools they need?

If so, do we really envision autotools-based packages selecting host-autoconf, host-automake, host-libtool? And then packages that use host-pkgconf you should also select it...

Also, what about host-cmake, which is conditionally built, but for which we do not have the info in kconfig? (well, we can argue we'd have to do like we do for host-gcc, but still). Oh, and host-tar, host-flex, host-bison, and so on... :-/

Meh, I hadn't thought of conditional packages like host-tar, host-cmake and so on.

Quoted 19 lines

So, no, I'm not happy with that direction...

config BR2_PACKAGE_FOO
bool "foo"
select BR2_PACKAGE_HOST_AUTOCONF select BR2_PACKAGE_HOST_AUTOMAKE select BR2_PACKAGE_HOST_LIBTOOL
select BR2_PACKAGE_MAYBE_HOST_TAR select BR2_PACKAGE_MAYBE_HOST_FLEX_FOR_KCONFIG select BR2_PACKAGE_MAYBE_HOST_BISON_FOR_KCONFIG select BR2_PACKAGE_HOST_PKGCONF

Unles we're planning on hiding that away into meta-config, like:

    config BR2_PACKAGE_FOO
        bool "foo"
        select BR2_AUTOTOOLS_PACKAGE # mimick $(eval $(autotools-package))
        select BR2_PACKAGE_HOST_PKGCONF_BECAUSE_IT_S_NOT_MANDATORY

Quoted 4 lines

And still, the optionally-required host packages like tar, flex et al. are not covered...

Meh... :-(

Indeed, I understand the "Meh" here. I hadn't really realized what it would mean to have Config.in.host options for all packages, and properly selected by all its users.

But still, there are a number of cases where it would really help, so that a given host package can be aware that another host package has been built with a given feature (or not). Or precisely to force that a certain host package is built with a given option. For example, in host-python, we had situations where only a given package needed host-python to be built with FOO support, and since we don't have any BR2_PACKAGE_HOST_PYTHON_FOO option, our only choice was to unconditionally enable FOO support in host-python, adding build time to everyone, even if FOO in host-python might only be needed for one obscure package.

It seems like we don't have a very conclusive decision on this topic at this point.

To be fait,

From: "Yann Morin" <[email protected]>
To: "Thomas Petazzoni" <[email protected]>
Cc: "Peter Korsgaard" <[email protected]>, "Mahyar Koshkouei" <[email protected]>, "Thomas De Schampheleire" <[email protected]>, "buildroot" <[email protected]>
Sent: Quinta-feira, 3 de janeiro de 2019 20:06:52
Subject: Re: [Buildroot] [PATCH v3 1/3] package/waf: add a blind Config.in.host

Thomas, All,

On 2019-01-03 22:50 +0100, Thomas Petazzoni spake thusly:

Quoted 35 lines · Yann E. MORIN

Quoted 11 lines

Well, I am not really happy with that, though: do we really plan on having packages really select all the host tools they need?

If so, do we really envision autotools-based packages selecting host-autoconf, host-automake, host-libtool? And then packages that use host-pkgconf you should also select it...

Also, what about host-cmake, which is conditionally built, but for which we do not have the info in kconfig? (well, we can argue we'd have to do like we do for host-gcc, but still). Oh, and host-tar, host-flex, host-bison, and so on... :-/

Meh, I hadn't thought of conditional packages like host-tar, host-cmake and so on.

Quoted 19 lines

So, no, I'm not happy with that direction...

config BR2_PACKAGE_FOO
bool "foo"
select BR2_PACKAGE_HOST_AUTOCONF select BR2_PACKAGE_HOST_AUTOMAKE select BR2_PACKAGE_HOST_LIBTOOL
select BR2_PACKAGE_MAYBE_HOST_TAR select BR2_PACKAGE_MAYBE_HOST_FLEX_FOR_KCONFIG select BR2_PACKAGE_MAYBE_HOST_BISON_FOR_KCONFIG select BR2_PACKAGE_HOST_PKGCONF

Unles we're planning on hiding that away into meta-config, like:

    config BR2_PACKAGE_FOO
        bool "foo"
        select BR2_AUTOTOOLS_PACKAGE # mimick $(eval $(autotools-package))
        select BR2_PACKAGE_HOST_PKGCONF_BECAUSE_IT_S_NOT_MANDATORY

Quoted 8 lines

Quoted 4 lines

And still, the optionally-required host packages like tar, flex et al. are not covered...

Meh... :-(

Indeed, I understand the "Meh" here. I hadn't really realized what it would mean to have Config.in.host options for all packages, and properly selected by all its users.

To be fait, I was initially also in favour of adding Config.in.host options for all host packages. But this waf patch made me change my mind, becasue of the above...

Quoted 10 lines

But still, there are a number of cases where it would really help, so that a given host package can be aware that another host package has been built with a given feature (or not). Or precisely to force that a certain host package is built with a given option. For example, in host-python, we had situations where only a given package needed host-python to be built with FOO support, and since we don't have any BR2_PACKAGE_HOST_PYTHON_FOO option, our only choice was to unconditionally enable FOO support in host-python, adding build time to everyone, even if FOO in host-python might only be needed for one obscure package.

In this case, maybe we could simply depart from the rule, and just have python/Config.in.host contain just:

# Select this if you need host-python to support 'stuff' config BR2_NEEDS_HOST_PYTHON_WITH_STUFF
bool

And then, python/python.mk would have:

   ifeq ($(BR2_NEEDS_HOST_PYTHON_WITH_STUFF),y)
   HOST_PYTHON_CONF_OPTS += --with-stuff
   else
   HOST_PYTHON_CONF_OPTS += --without-stuff
   endif

Regards,
Yann E. MORIN.

Quoted 2 lines

It seems like we don't have a very conclusive decision on this topic at this point.

Well, I'm happy that my work helped you guys to analyze the pros and cons of having Config.in.host for all host packages. Looks like it's not a good idea, so I will mark the patches as refused in patchwork.

From: "Thomas Petazzoni" <[email protected]>
To: "Carlos Santos" <[email protected]>
Cc: "Yann Morin" <[email protected]>, "Peter Korsgaard" <[email protected]>, "Mahyar Koshkouei" <[email protected]>, "Thomas De Schampheleire" <[email protected]>, "buildroot" <[email protected]>, "Arnout Vandecappelle" <[email protected]>
Sent: Sexta-feira, 4 de janeiro de 2019 6:58:11
Subject: Re: [Buildroot] [PATCH v3 1/3] package/waf: add a blind Config.in.host

Hello Carlos,

Quoted 3 lines · Carlos Santos

Well, I'm happy that my work helped you guys to analyze the pros and cons of having Config.in.host for all host packages. Looks like it's not a good idea, so I will mark the patches as refused in patchwork.

I don't think we can conclude yet "it's not a good idea". At this point, my feeling is more: the consequences are worse than I initially expected, which requires some more thoughts on how we want to handle that.

Again, what was your motivation for adding those Config.in.host options in the first place, specifically for waf ? Was it just about getting this whole discussion started, and use waf just as a simple first example ?

It started when I realized that the trailing slashes in WAF_SITE and other host packages. Then Arnout said "we need Config.in.host" and I decided to give it a try. It may be a silly explanation but sometimes we do things just for fun. :-)

Thomas, All,

On 2019-01-03 22:50 +0100, Thomas Petazzoni spake thusly:

Quoted 15 lines · Yann E. MORIN

Quoted 11 lines

Well, I am not really happy with that, though: do we really plan on having packages really select all the host tools they need?

If so, do we really envision autotools-based packages selecting host-autoconf, host-automake, host-libtool? And then packages that use host-pkgconf you should also select it...

Also, what about host-cmake, which is conditionally built, but for which we do not have the info in kconfig? (well, we can argue we'd have to do like we do for host-gcc, but still). Oh, and host-tar, host-flex, host-bison, and so on... :-/

Meh, I hadn't thought of conditional packages like host-tar, host-cmake and so on.

Quoted 13 lines

Quoted 13 lines

So, no, I'm not happy with that direction...

config BR2_PACKAGE_FOO
bool "foo"
select BR2_PACKAGE_HOST_AUTOCONF select BR2_PACKAGE_HOST_AUTOMAKE select BR2_PACKAGE_HOST_LIBTOOL
select BR2_PACKAGE_MAYBE_HOST_TAR select BR2_PACKAGE_MAYBE_HOST_FLEX_FOR_KCONFIG select BR2_PACKAGE_MAYBE_HOST_BISON_FOR_KCONFIG select BR2_PACKAGE_HOST_PKGCONF

Quoted 6 lines

Quoted 6 lines

Unles we're planning on hiding that away into meta-config, like:

    config BR2_PACKAGE_FOO
        bool "foo"
        select BR2_AUTOTOOLS_PACKAGE # mimick $(eval $(autotools-package))
        select BR2_PACKAGE_HOST_PKGCONF_BECAUSE_IT_S_NOT_MANDATORY

Quoted 2 lines

Quoted 2 lines

And still, the optionally-required host packages like tar, flex et al. are not covered...

Quoted 6 lines

Quoted 2 lines

Meh... :-(

Indeed, I understand the "Meh" here. I hadn't really realized what it would mean to have Config.in.host options for all packages, and properly selected by all its users.

To be fait, I was initially also in favour of adding Config.in.host options for all host packages. But this waf patch made me change my mind, becasue of the above...

Quoted 11 lines

But still, there are a number of cases where it would really help, so that a given host package can be aware that another host package has been built with a given feature (or not). Or precisely to force that a certain host package is built with a given option. For example, in host-python, we had situations where only a given package needed host-python to be built with FOO support, and since we don't have any BR2_PACKAGE_HOST_PYTHON_FOO option, our only choice was to unconditionally enable FOO support in host-python, adding build time to everyone, even if FOO in host-python might only be needed for one obscure package.

In this case, maybe we could simply depart from the rule, and just have python/Config.in.host contain just:

# Select this if you need host-python to support 'stuff' config BR2_NEEDS_HOST_PYTHON_WITH_STUFF
bool

And then, python/python.mk would have:

    ifeq ($(BR2_NEEDS_HOST_PYTHON_WITH_STUFF),y)
    HOST_PYTHON_CONF_OPTS += --with-stuff
    else
    HOST_PYTHON_CONF_OPTS += --without-stuff
    endif

Regards,
Yann E. MORIN.

Quoted 2 lines

It seems like we don't have a very conclusive decision on this topic at this point.

Quoted 2 lines

On 2019-01-03 22:50 +0100, Thomas Petazzoni spake thusly:

Quoted 1 line · Yann E. MORIN

Quoted 12 lines

Quoted 12 lines

Quoted 12 lines

So, no, I'm not happy with that direction...

config BR2_PACKAGE_FOO
bool "foo"
select BR2_PACKAGE_HOST_AUTOCONF select BR2_PACKAGE_HOST_AUTOMAKE select BR2_PACKAGE_HOST_LIBTOOL
select BR2_PACKAGE_MAYBE_HOST_TAR select BR2_PACKAGE_MAYBE_HOST_FLEX_FOR_KCONFIG select BR2_PACKAGE_MAYBE_HOST_BISON_FOR_KCONFIG select BR2_PACKAGE_HOST_PKGCONF

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Unles we're planning on hiding that away into meta-config, like:

    config BR2_PACKAGE_FOO
        bool "foo"
        select BR2_AUTOTOOLS_PACKAGE # mimick $(eval $(autotools-package))
        select BR2_PACKAGE_HOST_PKGCONF_BECAUSE_IT_S_NOT_MANDATORY

This looks acceptable to me...

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

And still, the optionally-required host packages like tar, flex et al. are not covered...

tar at least is rather easy to overcome: we can add environment variables that
pass the results of dependencies.mk to Kconfig. And for things that are conditional like lzip we can use a combination of the two.

Note, however, that the only reason to require Config.in.host for *all*
packages is that it allows us to have a (partial) check that for each make

dependency, the Kconfig dependency exists as well (i.e., check that all dependencies are in PACKAGES).

That's not a very strong reason. So I would be
perfectly fine with making this a "soft" requirement (possibly verified with checkpackage).

The tricky thing is that when a package does have a Config.in.host option, we
really want it to be selected in *all* cases that it is built. So ideally we should have a way to verify that...