Skip to contentSkip to conversation

multilib_header.bbclass: Do not install the stubs for non-multilib case

PATCH In review Khem Raj 2 participants 7 messages Oct 21, 2020 – Oct 22, 2020

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH 1/1 multilib_header.bbclass: Do not install the stubs for non-multilib case

We mark certain headers which are not usable in multilib case through this function, this however is a unnessary indirection when building distros without multilib feature. Additionally it also addresses a case where we compile baremetal software e.g. bootloaders or BPF binaries e.g. which are a different target effectively but can be built using same toolchain with some flags. This does not work when we always install the stubs since the stubs require a fixed wordsize definition and that may not be true for the BPF cases e.g. it will work fine when multilib is enabled since that would have provided the headers it needed say for 32bit multilib. Currently they go missing for a non-multilib case and we end up with errors like

recipe-sysroot/usr/include/bits/long-double.h:23:10: fatal error: 'bits/long-double-32.h' file not found
| #include <bits/long-double-32.h>

Therefore lets install these stubs only when using multilib in distro

---
 meta/classes/multilib_header.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/multilib_header.bbclass +3 -1
index e03f5b13b2..9ccdf46be5 100644
@@ -6,7 +6,9 @@ inherit siteinfo
 # all of the ABI variants for that given architecture.
 #
 oe_multilib_header() {
-
+	if [ -z "${MULTILIB_VARIANTS}" ]; then
+		return
+	fi
 	case ${HOST_OS} in
 	*-musl*)
 		return
Signature · 1 line
2.29.0
#1 lore raw
Quoted 34 lines · Khem Raj

We mark certain headers which are not usable in multilib case through this function, this however is a unnessary indirection when building distros without multilib feature. Additionally it also addresses a case
where we compile baremetal software e.g. bootloaders or BPF binaries e.g. which are a different target effectively but can be built using same toolchain with some flags. This does not work when we always install the stubs since the stubs require a fixed wordsize definition and that may not be true for the BPF cases e.g. it will work fine when
multilib is enabled since that would have provided the headers it needed
say for 32bit multilib. Currently they go missing for a non-multilib case and we end up with errors like

recipe-sysroot/usr/include/bits/long-double.h:23:10: fatal error:
'bits/long-double-32.h' file
not found

Quoted 1 line

#include <bits/long-double-32.h>

Therefore lets install these stubs only when using multilib in distro

---
 meta/classes/multilib_header.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/multilib_header.bbclass +0 -0
index e03f5b13b2..9ccdf46be5 100644

meta/classes/multilib_header.bbclass inherit siteinfo · line 14

Quoted 10 lines
meta/classes/multilib_header.bbclass +3 -1
@@ -6,7 +6,9 @@ inherit siteinfo
 # all of the ABI variants for that given architecture.
 #
 oe_multilib_header() {
-
+	if [ -z "${MULTILIB_VARIANTS}" ]; then
+		return
+	fi
 	case ${HOST_OS} in
 	*-musl*)
 		return

I have a feeling this was written specifically not to touch that variable so that a multilib build can reuse components from a non- multilib build and share the sstate if you configure it that way. This change would force *all* multilibs to be stores as their own sstate and require their own namespace.

We may therefore have to find a different control for this :/.

Cheers,

Richard

author
#2 lore raw
Quoted 43 lines · Richard Purdie
Quoted 42 lines · Khem Raj

We mark certain headers which are not usable in multilib case through this function, this however is a unnessary indirection when building distros without multilib feature. Additionally it also addresses a case
where we compile baremetal software e.g. bootloaders or BPF binaries e.g. which are a different target effectively but can be built using same toolchain with some flags. This does not work when we always install the stubs since the stubs require a fixed wordsize definition and that may not be true for the BPF cases e.g. it will work fine when
multilib is enabled since that would have provided the headers it needed
say for 32bit multilib. Currently they go missing for a non-multilib case and we end up with errors like

recipe-sysroot/usr/include/bits/long-double.h:23:10: fatal error:
'bits/long-double-32.h' file
not found

Quoted 1 line

#include <bits/long-double-32.h>

Therefore lets install these stubs only when using multilib in distro

---
 meta/classes/multilib_header.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/multilib_header.bbclass +3 -1
index e03f5b13b2..9ccdf46be5 100644
@@ -6,7 +6,9 @@ inherit siteinfo
 # all of the ABI variants for that given architecture.
 #
 oe_multilib_header() {
-
+     if [ -z "${MULTILIB_VARIANTS}" ]; then
+             return
+     fi
      case ${HOST_OS} in
Quoted 8 lines

meta/classes/multilib_header.bbclass inherit siteinfo · line 14

Quoted 2 lines
meta/classes/multilib_header.bbclass +0 -0
      *-musl*)
              return

I have a feeling this was written specifically not to touch that variable so that a multilib build can reuse components from a non- multilib build and share the sstate if you configure it that way. This change would force *all* multilibs to be stores as their own sstate and require their own namespace.

In practice either you use multilib or you don’t such a usecase is not that common besides perhaps testing scenarios and maybe some OS vendor test farms which considers all sort of combinations

And in some cases e.g x86-64 BASELIB gets defined differently if you enable multilib or not so I guess we already are not reusing sstate between non multilib and multilib builds for same arch

Quoted 11 lines

We may therefore have to find a different control for this :/.

Cheers,

Richard

author
#3 lore raw
Quoted 58 lines · Khem Raj

On Wed, Oct 21, 2020 at 2:47 PM Richard Purdie <[email protected] <mailto:[email protected]>> wrote:

On Wed, 2020-10-21 at 14:09 -0700, Khem Raj wrote:
> We mark certain headers which are not usable in multilib case through > this function, this however is a unnessary indirection when building > distros without multilib feature. Additionally it also addresses a > case
> where we compile baremetal software e.g. bootloaders or BPF binaries > e.g. which are a different target effectively but can be built using > same toolchain with some flags. This does not work when we always > install the stubs since the stubs require a fixed wordsize definition > and that may not be true for the BPF cases e.g. it will work fine > when
> multilib is enabled since that would have provided the headers it > needed
> say for 32bit multilib. Currently they go missing for a non-multilib > case and we end up with errors like
>
> recipe-sysroot/usr/include/bits/long-double.h:23:10: fatal error:
> 'bits/long-double-32.h' file
> not found
> > #include <bits/long-double-32.h>
>
> Therefore lets install these stubs only when using
> multilib in distro
>
> Signed-off-by: Khem Raj <[email protected]
<mailto:[email protected]>>
> ---
>  meta/classes/multilib_header.bbclass | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/multilib_header.bbclass
> b/meta/classes/multilib_header.bbclass
> index e03f5b13b2..9ccdf46be5 100644
> --- a/meta/classes/multilib_header.bbclass
> +++ b/meta/classes/multilib_header.bbclass
> @@ -6,7 +6,9 @@ inherit siteinfo
>  # all of the ABI variants for that given architecture.
>  #
>  oe_multilib_header() {
> -
> +     if [ -z "${MULTILIB_VARIANTS}" ]; then
> +             return
> +     fi
>       case ${HOST_OS} in
>       *-musl*)
>               return

I have a feeling this was written specifically not to touch that variable so that a multilib build can reuse components from a non- multilib build and share the sstate if you configure it that way. This change would force *all* multilibs to be stores as their own sstate and require their own namespace.

Quoted 5 lines

In practice either you use multilib or you don’t such a usecase is not that common besides perhaps testing scenarios and maybe some OS vendor test farms which considers all sort of combinations

And in some cases e.g x86-64 BASELIB gets defined differently if you enable multilib or not so I guess we already are not reusing sstate between non multilib and multilib builds for same arch

I ran a few combination builds without this patch applied.

build core-image-minimal/qemux86-64

then add just following to local.conf

require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

and rebuild core-image-minimal

here is buildhistory-diff output ( before and after adding multilib to local.conf)

http://sprunge.us/oM7Fg3

It ends up rebuilding all target packages inclduing some cross packages e.g. libtool-cross, native sstate it still re-used and that remains so with above patch applied as well.

Therefore this patch does not make things any different than they are currently.

Let me know if I should be checking more things.

Quoted 9 lines

We may therefore have to find a different control for this :/.

Cheers,

Richard

#4 lore raw
Quoted 17 lines · Khem Raj

I ran a few combination builds without this patch applied.

build core-image-minimal/qemux86-64

then add just following to local.conf

require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

and rebuild core-image-minimal

here is buildhistory-diff output ( before and after adding multilib to local.conf)

http://sprunge.us/oM7Fg3

Quoted 8 lines

It ends up rebuilding all target packages inclduing some cross packages e.g. libtool-cross, native sstate it still re-used and that remains so with above patch applied as well.

Therefore this patch does not make things any different than they are currently.

Let me know if I should be checking more things.

Imagine you configure 64 bit x86 to use /lib64 in all situations (which many do now, it probably should be the default).

Then rerun that test. I doubt core2-64 will rebuild in that case but it will after your patch.

Cheers,

Richard

author
#5 lore raw
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Khem Raj

I ran a few combination builds without this patch applied.

build core-image-minimal/qemux86-64

then add just following to local.conf

require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

and rebuild core-image-minimal

here is buildhistory-diff output ( before and after adding multilib to local.conf)

http://sprunge.us/oM7Fg3

It ends up rebuilding all target packages inclduing some cross packages e.g. libtool-cross, native sstate it still re-used and that remains so with above patch applied as well.

Quoted 7 lines
Quoted 4 lines

Therefore this patch does not make things any different than they are currently.

Let me know if I should be checking more things.

Imagine you configure 64 bit x86 to use /lib64 in all situations (which many do now, it probably should be the default).

interesting but its not default, and multilib is not default as well. Thirdly we are optimizing for a case where we want to switch distros between multilib and non-multilib case. are there practical usecases of these three conditions? I think we are optimizing for a relatively small usecase here if there are any.

Quoted 8 lines

Then rerun that test. I doubt core2-64 will rebuild in that case but it will after your patch.

Cheers,

Richard

#6 lore raw
Quoted 31 lines · Khem Raj
Quoted 29 lines · Richard Purdie
Quoted 28 lines · Khem Raj

I ran a few combination builds without this patch applied.

build core-image-minimal/qemux86-64

then add just following to local.conf

require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

and rebuild core-image-minimal

here is buildhistory-diff output ( before and after adding multilib to
local.conf)

http://sprunge.us/oM7Fg3

It ends up rebuilding all target packages inclduing some cross packages
e.g. libtool-cross, native sstate it still re-used and that remains so
with above patch applied as well.

Therefore this patch does not make things any different than they are
currently.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Let me know if I should be checking more things.

Imagine you configure 64 bit x86 to use /lib64 in all situations (which
many do now, it probably should be the default).

interesting but its not default, and multilib is not default as well.

I thought it was the default on other distros? We only have it as the default for nodistro and poky for historical reasons and I've not wanted the pain of switching.

Quoted 2 lines

Thirdly we are optimizing for a case where we want to switch distros between multilib and non-multilib case.

Ok, in 3.3, lets propose switching the default on 64 bit x86 and that in turn should let us optimise the autobuilder load to cut out a lot of these currently basically duplicated builds? If its not wanted for nodistro, I can propose we change poky.

Quoted 2 lines

are there practical usecases of these three conditions? I think we are optimizing for a relatively small usecase here if there are any.

See above, we can at the very least optimise the autobuilder. I'd actually thought it should be reusing now but it clearly isn't. We should fix that.

I do think its a sensible optimization to have working. I know it means we need to be a bit more careful with that patch.

Cheers,

Richard