Static linking fixes

PATCH New Gustavo Zacarias 3 participants 20 messages May 25, 2014 – Jul 30, 2014

Share & tools

Gustavo Zacarias

author

2014-05-25 22:12 UTC #0 lore raw

cover Static linking fixes

As discussed on the mailing list here is my current patchset to get static linking fixed.
This is a WIP (Work In Progress) patchset, that being said i consider patches 1 and 3 pretty safe and ready for the tree.
Patch 2 and 4 are medium confidence level, how things should be IMHO. Patch 5 is also how things should be however it's not extensively tested.

Gustavo Zacarias (5):
uclibc: don't install tools to staging when static
gcc-final: disable shared build for static
libiconv: disable preloadable for static builds package infra: revert 0a4bd19f
package infra: add -static to C/CXXFLAGS

 package/Makefile.in                |  4 +++-
 package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
 package/libiconv/libiconv.mk       |  8 ++++++++
 package/uclibc/uclibc.mk           |  4 ++--
 4 files changed, 32 insertions(+), 5 deletions(-)

Signature · 1 line

1.8.5.5

Gustavo Zacarias

author

2014-05-25 22:12 UTC #1 lore raw

PATCH 1/5 uclibc: don't install tools to staging when static

Don't try to install ld* tools to staging when it's a static build, there's no point and it'll be broken once we build a purely-static uclibc.

Signed-off-by Gustavo Zacarias <[email protected]>

---
 package/uclibc/uclibc.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

package/uclibc/uclibc.mk +2 -2

index 717cf53..ee87009 100644
@@ -522,8 +522,8 @@ define UCLIBC_INSTALL_TARGET_CMDS
 	$(UCLIBC_INSTALL_TEST_SUITE)
 endef
 
-# For FLAT binfmts (static) there are no host utils
-ifeq ($(BR2_BINFMT_FLAT),)
+# STATIC has no ld* tools, only getconf
+ifeq ($(BR2_PREFER_STATIC_LIB),)
 define UCLIBC_INSTALL_UTILS_STAGING
 	$(INSTALL) -D -m 0755 $(@D)/utils/ldd.host $(HOST_DIR)/usr/bin/ldd
 	ln -sf ldd $(HOST_DIR)/usr/bin/$(GNU_TARGET_NAME)-ldd

Signature · 1 line

1.8.5.5

Thomas Petazzoni

2014-05-26 09:35 UTC #2 lore raw

PATCH 1/5 uclibc: don't install tools to staging when static

Dear Gustavo Zacarias,

Quoted 8 lines · Gustavo Zacarias

Don't try to install ld* tools to staging when it's a static build, there's no point and it'll be broken once we build a purely-static uclibc.
Signed-off-by Gustavo Zacarias <[email protected]>
---
 package/uclibc/uclibc.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Acked-by Thomas Petazzoni <[email protected]>

(for -next).

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Thomas Petazzoni

Applied

2014-07-15 18:54 UTC #3 lore raw

PATCH 1/5 uclibc: don't install tools to staging when static

In reply to Gustavo Zacarias

Dear Gustavo Zacarias,

Quoted 8 lines · Gustavo Zacarias

Don't try to install ld* tools to staging when it's a static build, there's no point and it'll be broken once we build a purely-static uclibc.
Signed-off-by Gustavo Zacarias <[email protected]>
---
 package/uclibc/uclibc.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Applied, thanks.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-25 22:12 UTC #4 lore raw

PATCH 2/5 gcc-final: disable shared build for static

In reply to Gustavo Zacarias

Disable shared build for host-gcc-final when building for static targets. We really want static or shared, there's no such thing as "preferring static" since we can't choose with any degree of granularity for which packages. And it confuses linking scripts having both available at the same time. Fixes:
http://autobuild.buildroot.net/results/c54/c54bdf88eff6d60c7001cb0e2cb6792cc75178db/

Signed-off-by Gustavo Zacarias <[email protected]>

---
 package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

package/gcc/gcc-final/gcc-final.mk +19 -2

index 622dcf2..366dc42 100644
@@ -26,7 +26,7 @@ HOST_GCC_FINAL_SUBDIR = build
 
 HOST_GCC_FINAL_PRE_CONFIGURE_HOOKS += HOST_GCC_CONFIGURE_SYMLINK
 
-define  HOST_GCC_FINAL_CONFIGURE_CMDS
+define HOST_GCC_FINAL_CONFIGURE_CMDS
         (cd $(HOST_GCC_FINAL_SRCDIR) && rm -rf config.cache; \
                 $(HOST_CONFIGURE_OPTS) \
                 CFLAGS="$(HOST_CFLAGS)" \

@@ -35,7 +35,7 @@ define  HOST_GCC_FINAL_CONFIGURE_CMDS
                 ./configure \
                 --prefix="$(HOST_DIR)/usr" \
                 --sysconfdir="$(HOST_DIR)/etc" \
-                --enable-shared --enable-static \
+                --enable-static \
                 $(QUIET) $(HOST_GCC_FINAL_CONF_OPT) \
         )
 endef

@@ -54,6 +54,13 @@ HOST_GCC_FINAL_CONF_OPT = \
 	$(DISABLE_LARGEFILE) \
 	--with-build-time-tools=$(HOST_DIR)/usr/$(GNU_TARGET_NAME)/bin
 
+# Disable shared libs like libstdc++ if we do static since it confuses linking
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+HOST_GCC_FINAL_CONF_OPT += --disable-shared
+else
+HOST_GCC_FINAL_CONF_OPT += --enable-shared
+endif
+
 ifeq ($(BR2_GCC_ENABLE_OPENMP),y)
 HOST_GCC_FINAL_CONF_OPT += --enable-libgomp
 else

@@ -141,6 +148,15 @@ endif
 endif
 
 ifneq ($(HOST_GCC_FINAL_USR_LIBS),)
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+define HOST_GCC_FINAL_INSTALL_USR_LIBS
+	mkdir -p $(TARGET_DIR)/usr/lib
+	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \
+		cp -dpf $(HOST_DIR)/usr/$(GNU_TARGET_NAME)/lib*/$${i}.a \
+			$(STAGING_DIR)/usr/lib/ ; \
+	done
+endef
+else
 define HOST_GCC_FINAL_INSTALL_USR_LIBS
 	mkdir -p $(TARGET_DIR)/usr/lib
 	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \

@@ -152,6 +168,7 @@ define HOST_GCC_FINAL_INSTALL_USR_LIBS
 			$(TARGET_DIR)/usr/lib/ ; \
 	done
 endef
+endif
 HOST_GCC_FINAL_POST_INSTALL_HOOKS += HOST_GCC_FINAL_INSTALL_USR_LIBS
 endif

Signature · 1 line

1.8.5.5

Thomas Petazzoni

2014-05-26 09:37 UTC #5 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Dear Gustavo Zacarias,

Quoted 5 lines · Gustavo Zacarias

Disable shared build for host-gcc-final when building for static targets. We really want static or shared, there's no such thing as "preferring static" since we can't choose with any degree of granularity for which packages. And it confuses linking scripts having both available at the same time. Fixes:
http://autobuild.buildroot.net/results/c54/c54bdf88eff6d60c7001cb0e2cb6792cc75178db/

While on principle I'm probably fine with the patch, I kinda disagree that it fixes the autobuilder issues. Because the same autobuilder issue will appear with an external toolchain that has both libstdc++.so and libstdc++.a.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-26 10:04 UTC #6 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Quoted 4 lines · Thomas Petazzoni

While on principle I'm probably fine with the patch, I kinda disagree that it fixes the autobuilder issues. Because the same autobuilder issue will appear with an external toolchain that has both libstdc++.so and libstdc++.a.

This is a way to fix it for internal toolchains, i can build up to mpd static with audiofile support with this, but it's not complete, that's for sure.
Regards.

Thomas Petazzoni

2014-05-26 10:07 UTC #7 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Dear Gustavo Zacarias,

Quoted 8 lines · Gustavo Zacarias

Quoted 4 lines
While on principle I'm probably fine with the patch, I kinda disagree that it fixes the autobuilder issues. Because the same autobuilder issue will appear with an external toolchain that has both libstdc++.so and libstdc++.a.
This is a way to fix it for internal toolchains, i can build up to mpd static with audiofile support with this, but it's not complete, that's for sure.

Yeah, I read on the rest of your patches, and the patch changing --static to -static indeed makes it clear that it fixes the problem for external toolchains. IMO, switching from --static to -static is the real fix: the build process should not be confused by the presence of libstdc++.so even if we're doing a pure static build.

Also, about reverting back from --static to -static, would it be useful to contact Andy Kennedy to have more details about why he made the change? His commit log lacks the details on why he changed from -static to --static.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-26 11:02 UTC #8 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Quoted 10 lines · Thomas Petazzoni

Yeah, I read on the rest of your patches, and the patch changing --static to -static indeed makes it clear that it fixes the problem for external toolchains. IMO, switching from --static to -static is the real fix: the build process should not be confused by the presence of libstdc++.so even if we're doing a pure static build.
Also, about reverting back from --static to -static, would it be useful to contact Andy Kennedy to have more details about why he made the change? His commit log lacks the details on why he changed from -static to --static.

In tests for the internal toolchain audiofile builds but mpd fails to link with that change alone (+host-gcc-final static works, actually with host-gcc-fnial static alone it works).
For an external toolchain it works with the -static change. Regards.

Thomas Petazzoni

2014-05-26 09:38 UTC #9 lore raw

PATCH 2/5 gcc-final: disable shared build for static

In reply to Gustavo Zacarias

Dear Gustavo Zacarias,

package/gcc/gcc-final/gcc-final.mk endif · line 158

Quoted 9 lines · Gustavo Zacarias

ifneq ($(HOST_GCC_FINAL_USR_LIBS),)

package/gcc/gcc-final/gcc-final.mk +8 -0

+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+define HOST_GCC_FINAL_INSTALL_USR_LIBS
+	mkdir -p $(TARGET_DIR)/usr/lib
+	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \
+		cp -dpf $(HOST_DIR)/usr/$(GNU_TARGET_NAME)/lib*/$${i}.a \
+			$(STAGING_DIR)/usr/lib/ ; \
+	done
+endef

No, we want to install .a files as well in my opinion. In fact, the patch series I have to rename BR2_PREFER_STATIC_LIB makes things even clearer, with three possibilities: pure static, static and shared, pure shared.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-26 10:06 UTC #10 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Quoted 4 lines · Thomas Petazzoni

No, we want to install .a files as well in my opinion. In fact, the patch series I have to rename BR2_PREFER_STATIC_LIB makes things even clearer, with three possibilities: pure static, static and shared, pure shared.

Sorry but i didn't see your patchset so relax a bit :)
That being said do we really want a pure dyn scenario? Since we purge .a files from the target anyway since they're not required for runtime it wouldn't hurt a bit to keep the dual mode and just static i think. Regards.

Thomas Petazzoni

2014-05-26 10:24 UTC #11 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Dear Gustavo Zacarias,

Quoted 6 lines · Gustavo Zacarias

Quoted 4 lines
No, we want to install .a files as well in my opinion. In fact, the patch series I have to rename BR2_PREFER_STATIC_LIB makes things even clearer, with three possibilities: pure static, static and shared, pure shared.
Sorry but i didn't see your patchset so relax a bit :)

Yeah, no problem :)

Quoted 4 lines

That being said do we really want a pure dyn scenario? Since we purge .a files from the target anyway since they're not required for runtime it wouldn't hurt a bit to keep the dual mode and just static i think. Regards.

There is an advantage with the pure dynamic scenario: build time.

When you have dyn+static, you build all object files (for packages that do it correctly): once for static without -fPIC, and once for dynamic with -fPIC. Here is what I have in my commit log about this:

For example, a static+shared build of libglib2 takes 1 minutes and 59 seconds, with a final build directory of 96 MB. A shared-only build of libglib2 takes only 1 minutes and 31 seconds (almost a 25% reduction of the build time), and the final build directory weights 89 MB (a reduction of almost 8%).

A 25% reduction in build time is certainly nice for users who don't care about static libraries, no?

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-26 11:03 UTC #12 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Quoted 6 lines · Thomas Petazzoni

There is an advantage with the pure dynamic scenario: build time.
When you have dyn+static, you build all object files (for packages that do it correctly): once for static without -fPIC, and once for dynamic with -fPIC. Here is what I have in my commit log about this:

Quoted 8 lines

For example, a static+shared build of libglib2 takes 1 minutes and 59 seconds, with a final build directory of 96 MB. A shared-only build of libglib2 takes only 1 minutes and 31 seconds (almost a 25% reduction of the build time), and the final build directory weights 89 MB (a reduction of almost 8%).
A 25% reduction in build time is certainly nice for users who don't care about static libraries, no?

Ok, so then we can go for it, i don't think anyone will care about the space saving though, but time is a whole different matter. Regards.

Thomas Petazzoni

2014-05-26 12:05 UTC #13 lore raw

PATCH 2/5 gcc-final: disable shared build for static

Dear Gustavo Zacarias,

Quoted 11 lines · Gustavo Zacarias

Quoted 8 lines
For example, a static+shared build of libglib2 takes 1 minutes
and 59 seconds, with a final build directory of 96 MB. A shared-only
build of libglib2 takes only 1 minutes and 31 seconds (almost a
25% reduction of the build time), and the final build directory weights 89 MB (a reduction of almost 8%).
A 25% reduction in build time is certainly nice for users who don't care about static libraries, no?
Ok, so then we can go for it, i don't think anyone will care about the space saving though, but time is a whole different matter.

Agreed, the disk space saving nobody really cares, especially if it's only ~8%. But a build time saving of 25% is quite nice, in my opinion. Normally, all packages using libtool should be affected, but I haven't checked.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Gustavo Zacarias

author

2014-05-25 22:12 UTC #14 lore raw

PATCH 3/5 libiconv: disable preloadable for static builds

In reply to Gustavo Zacarias

Disable the preloadable libiconv variant when building for static targets since that's not possible.

Signed-off-by Gustavo Zacarias <[email protected]>

---
 package/libiconv/libiconv.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

package/libiconv/libiconv.mk +8 -0

index ee313ce..d2209bc 100644
@@ -8,6 +8,14 @@ LIBICONV_VERSION = 1.14
 LIBICONV_SITE = $(BR2_GNU_MIRROR)/libiconv
 LIBICONV_INSTALL_STAGING = YES
 
+# Preloadable libiconv is for dynamic targets only
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+define LIBICONV_DISABLE_PRELOAD
+	$(SED) '/preload/d' $(@D)/Makefile.in
+endef
+endif
+LIBICONV_PRE_CONFIGURE_HOOKS += LIBICONV_DISABLE_PRELOAD
+
 # Remove not used preloadable libiconv.so
 define LIBICONV_TARGET_REMOVE_PRELOADABLE_LIBS
 	rm -f $(TARGET_DIR)/usr/lib/preloadable_libiconv.so

Signature · 1 line

1.8.5.5

Gustavo Zacarias

author

2014-05-25 22:12 UTC #15 lore raw

PATCH 4/5 package infra: revert 0a4bd19f

In reply to Gustavo Zacarias

Revert commit 0a4bd19f4a136930c611027942ce43124a81c5cb Using --static is not documented and actually breaks static linking according to my tests.
Fixes:
http://autobuild.buildroot.net/results/327/327c18db4e5d0ddc2c72a4684e103c19a1405e50/
...for external toolchains

Signed-off-by Gustavo Zacarias <[email protected]>

---
 package/Makefile.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/Makefile.in +1 -1

index 2fc3aa7..ec1f22b 100644
@@ -373,7 +373,7 @@ endif
 
 ifeq ($(BR2_PREFER_STATIC_LIB),y)
 SHARED_STATIC_LIBS_OPTS=--enable-static --disable-shared
-TARGET_LDFLAGS += --static
+TARGET_LDFLAGS += -static
 else
 SHARED_STATIC_LIBS_OPTS=--enable-static --enable-shared
 endif

Signature · 1 line

1.8.5.5

Gustavo Zacarias

author

2014-05-25 22:13 UTC #16 lore raw

PATCH 5/5 package infra: add -static to C/CXXFLAGS

In reply to Gustavo Zacarias

Generic infra packages might not use LDFLAGS at all so add -static for static builds to CFLAGS and CXXFLAGS too.

Signed-off-by Gustavo Zacarias <[email protected]>

---
 package/Makefile.in | 2 ++
 1 file changed, 2 insertions(+)

package/Makefile.in +2 -0

index ec1f22b..ec8fb1b 100644
@@ -373,6 +373,8 @@ endif
 
 ifeq ($(BR2_PREFER_STATIC_LIB),y)
 SHARED_STATIC_LIBS_OPTS=--enable-static --disable-shared
+TARGET_CFLAGS += -static
+TARGET_CXXFLAGS += -static
 TARGET_LDFLAGS += -static
 else
 SHARED_STATIC_LIBS_OPTS=--enable-static --enable-shared

Signature · 1 line

1.8.5.5

Thomas De Schampheleire

2014-05-26 08:13 UTC #17 lore raw

cover Static linking fixes

In reply to Gustavo Zacarias

Hi Gustavo,

Quoted 14 lines · Gustavo Zacarias

As discussed on the mailing list here is my current patchset to get static linking fixed.
This is a WIP (Work In Progress) patchset, that being said i consider patches 1 and 3 pretty safe and ready for the tree.
Patch 2 and 4 are medium confidence level, how things should be IMHO. Patch 5 is also how things should be however it's not extensively tested.
Gustavo Zacarias (5):
uclibc: don't install tools to staging when static
gcc-final: disable shared build for static
libiconv: disable preloadable for static builds package infra: revert 0a4bd19f
package infra: add -static to C/CXXFLAGS

Quoted 5 lines

 package/Makefile.in                |  4 +++-
 package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
 package/libiconv/libiconv.mk       |  8 ++++++++
 package/uclibc/uclibc.mk           |  4 ++--
 4 files changed, 32 insertions(+), 5 deletions(-)

Your patches do not yet go as far as touching the
BR2_PREFER_STATIC_LIB variable. We have talked about this in the past
but as far as I know there is no-one that has actually wrote patches
for it.
See also http://www.elinux.org/Buildroot:DeveloperDaysFOSDEM2014#BR2_PREFER_STATIC_LIB_refactoring

It would be awesome if this refactoring could be taken up by someone in the 2014.08 cycle, but I don't know if anyone has time for this...

Best regards,
Thomas

Gustavo Zacarias

author

2014-05-26 10:02 UTC #18 lore raw

cover Static linking fixes

On 05/26/2014 05:13 AM, Thomas De Schampheleire wrote:

Hi.

Quoted 8 lines

Your patches do not yet go as far as touching the
BR2_PREFER_STATIC_LIB variable. We have talked about this in the past
but as far as I know there is no-one that has actually wrote patches
for it.
See also http://www.elinux.org/Buildroot:DeveloperDaysFOSDEM2014#BR2_PREFER_STATIC_LIB_refactoring
It would be awesome if this refactoring could be taken up by someone in the 2014.08 cycle, but I don't know if anyone has time for this...

Well, that's why it's called a WIP :)
Also Thomas P. told me he has done something about it so i would just be reinventing the wheel.
And if it's just a rename it's a pretty simple "find . -type f -exec sed -i -e "s/BR2_PREFER_STATIC_LIB/BR2_SOMETHING/" {} \;" with some hand editing of toplevel Config.in to adjust the description as well. Regards.

Thomas Petazzoni

2014-07-30 21:42 UTC #19 lore raw

cover Static linking fixes

In reply to Gustavo Zacarias

Dear Gustavo Zacarias,

Quoted 9 lines · Gustavo Zacarias

As discussed on the mailing list here is my current patchset to get static linking fixed.
This is a WIP (Work In Progress) patchset, that being said i consider patches 1 and 3 pretty safe and ready for the tree.
Patch 2 and 4 are medium confidence level, how things should be IMHO. Patch 5 is also how things should be however it's not extensively tested.
Gustavo Zacarias (5):
uclibc: don't install tools to staging when static

This one was already applied.

Quoted 4 lines

gcc-final: disable shared build for static
libiconv: disable preloadable for static builds
package infra: revert 0a4bd19f
package infra: add -static to C/CXXFLAGS

I've applied those four patches. On the gcc-final patch, I did a small change to factorize the static library installation logic in one place. On the libiconv patch, I did a little bit more changes: completely disable the build of the preloadable library, since we were anyway removing it from staging and target. It allowed to simplify the patch a bit.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com