toolchain: workaround musl/kernel headers conflict

Rick Felker suggested[1] this hack as a workaround to musl libc conflict with kernel headers:

The problem is linux/libc-compat.h, which should fix this, only works on glibc, by design. See:

  #ifndef _LIBC_COMPAT_H
  #define _LIBC_COMPAT_H

/* We have included glibc headers... */
#if defined(__GLIBC__)

  /* Coordinate with glibc netinet/in.h header. */
  #if defined(_NETINET_IN_H)

If you patch it like this:

-#if defined(__GLIBC__)
+#if 1

then it should mostly work but it's still all a big hack. I think that's what distros are doing. The problem is that the same header is trying to do two different things:

1. Provide extra linux-kernel-API stuff that's not in the libc/userspace headers.

2. Provide definitions of the standard types and constants for uClibc and klibc, which don't have complete libc headers and rely on the kernel headers for definitions.

These two uses really should be separated out into separate headers so that the latter only get included explicitly by uClibc and klibc and otherwise remain completely unused. But that would require coordinated changes/upgrades which are unlikely to happen. :(

Upstream musl still evaluates[2][3] a permanent solution.

With this in place we can revert (at least) commits a167081c5d (bridge-utils:
fix build with musl) and e74d4fc4932 (norm: add patch to fix musl build).

[1] http://www.openwall.com/lists/musl/2015/10/08/2
[2] http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
[3] http://www.openwall.com/lists/musl/2016/11/09/2

Signed-off-by Baruch Siach <[email protected]>

---
v2:
* Add Rick's explanation in the commit log (Arnout)
* Link to more recent upstream changes

---
 toolchain/toolchain/toolchain.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

toolchain/toolchain/toolchain.mk +14 -0

index c22713bfe349..d317e917d032 100644
@@ -12,6 +12,20 @@ endif
 
 TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 
+# Apply a hack that Rick Felker suggested[1] to avoid conflicts between libc
+# headers and kernel headers. This is a temporary measure until musl finds a
+# better solution.
+#
+# [1] http://www.openwall.com/lists/musl/2015/10/08/2
+ifeq ($(BR2_TOOLCHAIN_USES_MUSL),y)
+define TOOLCHAIN_MUSL_KERNEL_HEADERS_COMPATIBILITY_HACK
+	$(SED) 's/^#if defined(__GLIBC__)$$/#if 1/' \
+		$(STAGING_DIR)/usr/include/linux/libc-compat.h
+endef
+TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += TOOLCHAIN_MUSL_KERNEL_HEADERS_COMPATIBILITY_HACK
+TOOLCHAIN_INSTALL_STAGING = YES
+endif
+
 $(eval $(virtual-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake

However, if bridge-utils and norm are really the only packages that suffer from
this problem, is it really worthwhile to apply this workaround?

Quoted 2 lines · Arnout Vandecappelle

However, if bridge-utils and norm are really the only packages that suffer from
this problem, is it really worthwhile to apply this workaround?

I'm sure there are more packages that we have patched for this reason, and possibly other build failures still unfixed that would get fixed (or at least partially).

*However*, the musl folks have already committed a patch that apparently resolves the problem:

   http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258

Baruch, could you investigate whether this musl commit fixes the header conflict problem for bridge-utils and norm?

Quoted 2 lines

Baruch, could you investigate whether this musl commit fixes the header conflict problem for bridge-utils and norm?

The commit log links to this musl commit and to a mailing list report[1] that this fix is not enough on its own. As a result of this discussion Felix Janda posted a kernel patch[2] to address the issue. See also Rich's response[3]. Even if this patch is accepted, we'll most likely have to keep this workaround, or an equivalent, for as long as we support kernel headers v4.9 and older.

[1] http://www.openwall.com/lists/musl/2016/11/09/2
[2] http://www.openwall.com/lists/musl/2016/11/11/1
[3] http://www.openwall.com/lists/musl/2016/11/11/2

Quoted 10 lines · Baruch Siach

Quoted 2 lines

Baruch, could you investigate whether this musl commit fixes the header conflict problem for bridge-utils and norm?

The commit log links to this musl commit and to a mailing list report[1] that this fix is not enough on its own. As a result of this discussion Felix Janda posted a kernel patch[2] to address the issue. See also Rich's response[3]. Even if this patch is accepted, we'll most likely have to keep this workaround, or an equivalent, for as long as we support kernel headers v4.9 and older.

Quoted 3 lines

[1] http://www.openwall.com/lists/musl/2016/11/09/2
[2] http://www.openwall.com/lists/musl/2016/11/11/1
[3] http://www.openwall.com/lists/musl/2016/11/11/2

Thanks a lot for the explanation.

So I believe we should apply your proposed workaround. However, I'm not sure I want to take the risk of applying this to the master branch (even though I agree it could fix some build failures). Are you OK with applying it to the next branch?

Quoted 4 lines

So I believe we should apply your proposed workaround. However, I'm not sure I want to take the risk of applying this to the master branch (even though I agree it could fix some build failures). Are you OK with applying it to the next branch?

Of course. This is definitely not for master.

Quoted 6 lines · Baruch Siach

Quoted 4 lines

So I believe we should apply your proposed workaround. However, I'm not sure I want to take the risk of applying this to the master branch (even though I agree it could fix some build failures). Are you OK with applying it to the next branch?

Of course. This is definitely not for master.

And any thoughts about my concerns to identify the packages that should be re-enabled and patches that should be removed?

Rick Felker suggested[1] this hack as a workaround to musl libc conflict with kernel headers:

The problem is linux/libc-compat.h, which should fix this, only works on glibc, by design. See:

  #ifndef _LIBC_COMPAT_H
  #define _LIBC_COMPAT_H

/* We have included glibc headers... */
#if defined(__GLIBC__)

  /* Coordinate with glibc netinet/in.h header. */
  #if defined(_NETINET_IN_H)

If you patch it like this:

-#if defined(__GLIBC__)
+#if 1

then it should mostly work but it's still all a big hack. I think that's what distros are doing. The problem is that the same header is trying to do two different things:

1. Provide extra linux-kernel-API stuff that's not in the libc/userspace headers.

2. Provide definitions of the standard types and constants for uClibc and klibc, which don't have complete libc headers and rely on the kernel headers for definitions.

These two uses really should be separated out into separate headers so that the latter only get included explicitly by uClibc and klibc and otherwise remain completely unused. But that would require coordinated changes/upgrades which are unlikely to happen. :(

Upstream musl still evaluates[2][3] a permanent solution.

With this in place we can revert (at least) commits a167081c5d (bridge-utils:
fix build with musl) and e74d4fc4932 (norm: add patch to fix musl build).

[1] http://www.openwall.com/lists/musl/2015/10/08/2
[2] http://git.musl-libc.org/cgit/musl/commit/?id=04983f2272382af92eb8f8838964ff944fbb8258
[3] http://www.openwall.com/lists/musl/2016/11/09/2

Signed-off-by Baruch Siach <[email protected]>

---
v2:
* Add Rick's explanation in the commit log (Arnout)
* Link to more recent upstream changes

---
 toolchain/toolchain/toolchain.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

I'm reviving this old thread, because my colleague Florent (in Cc) has found an issue with this patch.

Review

Quoted 6 lines · Baruch Siach

(quoted diff) +6 -0

+# [1] http://www.openwall.com/lists/musl/2015/10/08/2
+ifeq ($(BR2_TOOLCHAIN_USES_MUSL),y)
+define TOOLCHAIN_MUSL_KERNEL_HEADERS_COMPATIBILITY_HACK
+	$(SED) 's/^#if defined(__GLIBC__)$$/#if 1/' \
+		$(STAGING_DIR)/usr/include/linux/libc-compat.h
+endef

The problem with this is that libc-compat.h was introduced only in kernel 3.12 (it seems). Therefore, trying to build a musl toolchain with kernel headers 3.10 currently fails, with this $(SED) command failing because $(STAGING_DIR)/usr/include/linux/libc-compat.h does not exist.

So, we could add a kernel headers >= 3.12 to the condition to make the build work, but then we wouldn't have the fix that these modifications provide to avoid conflicts between libc and kernel headers.

Do you know if kernel headers < 3.12 are unaffected by the libc/kernel headers conflict issue? If they are, do you know what the solution is for kernel headers < 3.12 ?

I have not tested kernel headers earlier than v3.12. Having a look at the headers and the log of kernel commit cfd280c91253cc28 (net: sync some IP headers with glibc), I'm pretty sure that the same issue also affects older kernels.

Should we just disallow build of musl toolchain with older kernels? Something like:

Quoted 5 lines · Baruch Siach

I have not tested kernel headers earlier than v3.12. Having a look at the headers and the log of kernel commit cfd280c91253cc28 (net: sync some IP headers with glibc), I'm pretty sure that the same issue also affects older kernels.

Quoted 2 lines

Should we just disallow build of musl toolchain with older kernels? Something like:

That's one option, though a bit annoying, no? Just for this "small" reason, we prevent people from using musl toolchains on kernels older than 3.12 ?

Would that be better (untested)?

toolchain/toolchain/toolchain.mk +0 -0

index e29837357a27..e15ceeb426fa 100644
@@ -21,8 +21,10 @@ TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 # IFF_DORMANT and IFF_ECHO, add another macro to suppress them in the

Quoted 14 lines · Baruch Siach

Would that be better (untested)?

toolchain/toolchain/toolchain.mk +2 -1

index e29837357a27..e15ceeb426fa 100644
@@ -21,8 +21,10 @@ TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
 # IFF_DORMANT and IFF_ECHO, add another macro to suppress them in the
 # kernel header, and avoid macro/enum conflict.
 #
+# Kernel version 3.12 introduced the libc-compat.h header.
+#
 # [1] http://www.openwall.com/lists/musl/2015/10/08/2
-ifeq ($(BR2_TOOLCHAIN_USES_MUSL),y)

toolchain/toolchain/toolchain.mk TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO · line 30

Quoted 4 lines

toolchain/toolchain/toolchain.mk +1 -0

+ifeq ($(BR2_TOOLCHAIN_USES_MUSL)$(BR2_TOOLCHAIN_HEADERS_AT_LEAST_3_12),yy)
 define TOOLCHAIN_MUSL_KERNEL_HEADERS_COMPATIBILITY_HACK
 	$(SED) 's/^#if defined(__GLIBC__)$$/#if 1/' \
 		$(STAGING_DIR)/usr/include/linux/libc-compat.h

Yes. The question is whether we will see those build failures due to kernel/userspace headers conflicts. Do you remember a simple package/scenario that triggers the problem?