Skip to contentSkip to conversation

autossh: honour LDFLAGS

PATCH New Waldemar Brodkorb 3 participants 13 messages Nov 25, 2016 – Dec 5, 2016

lore
Share & tools
lore

Waldemar Brodkorb

author
#0 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Fixes:
http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973

Signed-off-by Waldemar Brodkorb <[email protected]>

---
It is indeed a change from uClibc-ng 1.0.18 and upward, but I think it is the fault of the package not respecting LDFLAGS and so not passing -static. I am unsure why it just worked in the past. When it fails linker is doing following:
-lnsl -lgcc -lc -lgcc
When -static is used, following is passed:
-lnsl --start-group -lgcc -lc --end-group

The gcc -dumpspecs from both gcc's are the same. I have no idea why gcc behaves in another way when toolchain is compiled with 1.0.17.

---
 package/autossh/0001-honour-LDFLAGS.patch | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 package/autossh/0001-honour-LDFLAGS.patch
package/autossh/0001-honour-LDFLAGS.patch new file +16 -0
new file mode 100644
index 0000000..2e6160b
@@ -0,0 +1,16 @@
+Honour LDFLAGS, especially required when static linking
+
+Signed-off-by: Waldemar Brodkorb <[email protected]>
+
+diff -Nur autossh-1.4e.orig/Makefile.in autossh-1.4e/Makefile.in
+--- autossh-1.4e.orig/Makefile.in	2015-02-10 05:41:58.000000000 +0100
++++ autossh-1.4e/Makefile.in	2016-11-25 12:49:06.052122855 +0100
+@@ -31,7 +31,7 @@
+ 
+ 
+ $(TARGET):	$(OFILES)
+-		$(CC) $(CPPFLAGS) -o $(TARGET) $(OFILES) $(LIBS)
++		$(CC) $(CPPFLAGS) $(LDFLAGS) -o $(TARGET) $(OFILES) $(LIBS)
+ 
+ clean:
+ 		- /bin/rm -f *.o *.a *.core *~
Signature · 1 line
2.1.4
Applied
#1 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hello,

Quoted 4 lines · Waldemar Brodkorb

Fixes:
http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973

Signed-off-by Waldemar Brodkorb <[email protected]>
Quoted 8 lines

---
It is indeed a change from uClibc-ng 1.0.18 and upward, but I think it is the fault of the package not respecting LDFLAGS and so not passing -static. I am unsure why it just worked in the past. When it fails linker is doing following:
-lnsl -lgcc -lc -lgcc
When -static is used, following is passed:
-lnsl --start-group -lgcc -lc --end-group

Applied to master.

The difference in the gcc behavior is due to gcc/config/gnu-user.h:

#define LINK_GCC_C_SEQUENCE_SPEC \
"%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"

So, when you're linking dynamic, you have:

-lgcc -lc -lgcc

And when you're linking statically, you have:

	--start-group -lgcc -lc --end-group

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

However, this interestingly doesn't happen for other architectures such as ARM. Maybe dl_iterate_phdr is not used?

In any case, could you try to send your patch upstream to the autossh developers?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#2 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Quoted 29 lines · Thomas Petazzoni
Quoted 4 lines · Waldemar Brodkorb

Fixes:
http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973

Signed-off-by Waldemar Brodkorb <[email protected]>
Quoted 8 lines

---
It is indeed a change from uClibc-ng 1.0.18 and upward, but I think it is the fault of the package not respecting LDFLAGS and so not passing -static. I am unsure why it just worked in the past. When it fails linker is doing following:
-lnsl -lgcc -lc -lgcc
When -static is used, following is passed:
-lnsl --start-group -lgcc -lc --end-group

Applied to master.

The difference in the gcc behavior is due to gcc/config/gnu-user.h:

#define LINK_GCC_C_SEQUENCE_SPEC \
"%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"

So, when you're linking dynamic, you have:

-lgcc -lc -lgcc

And when you're linking statically, you have:

        --start-group -lgcc -lc --end-group
Quoted 7 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

However, this interestingly doesn't happen for other architectures such as ARM. Maybe dl_iterate_phdr is not used?

ARM libgcc has its own custom unwinder: unwind-arm.c, which doesn't reference dl_iteratre_phdr, and it doesn't use unwind-dw2-fde-dip.c, which does. But it is reproducible with MIPS.

Signature · 2 lines
Thanks.
-- Max
author
#3 lore raw

PATCH 1/1 autossh: honour LDFLAGS

In reply to Thomas Petazzoni

Hi Thomas,
Thomas Petazzoni wrote,

Quoted 27 lines

Hello,

Quoted 4 lines · Waldemar Brodkorb

Fixes:
http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973

Signed-off-by Waldemar Brodkorb <[email protected]>
Quoted 8 lines

---
It is indeed a change from uClibc-ng 1.0.18 and upward, but I think it is the fault of the package not respecting LDFLAGS and so not passing -static. I am unsure why it just worked in the past. When it fails linker is doing following:
-lnsl -lgcc -lc -lgcc
When -static is used, following is passed:
-lnsl --start-group -lgcc -lc --end-group

Applied to master.

The difference in the gcc behavior is due to gcc/config/gnu-user.h:

#define LINK_GCC_C_SEQUENCE_SPEC \
"%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"

So, when you're linking dynamic, you have:

-lgcc -lc -lgcc

Quoted 8 lines

And when you're linking statically, you have:

	--start-group -lgcc -lc --end-group

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

Quoted 2 lines

In any case, could you try to send your patch upstream to the autossh developers?

Already done.

best regards
Waldemar

#4 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hello,

Quoted 9 lines · Waldemar Brodkorb
Quoted 4 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

I would suspect it's a fallout of the "merge everything in libc" change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr symbol, enables unwinding support, which cause this extra reference from libgcc to libc?

You would have to compare the config.log of libgcc for BR+gcc5+uClibc-ng-1.0.17 and BR+gcc5+uClibc-ng-1.0.19 to see what's the difference.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#5 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Quoted 11 lines · Thomas Petazzoni
Quoted 9 lines · Waldemar Brodkorb
Quoted 4 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

Quoted 8 lines

I would suspect it's a fallout of the "merge everything in libc" change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr symbol, enables unwinding support, which cause this extra reference from libgcc to libc?

No. The difference is in the __pthread_initialize_minimal function. Prior to uclibc-1.0.18 release it used to work because a call to __pthread_initialize_minimal from __uClibc_init was resolved to a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c which didn't reference pthread_unwind.
Now it is resolved to a function from libpthread/linuxthreads/pthread.c, and that file pulls in other unused functions, which reference __GI___pthread_unwind, and all the way down to the dl_iterate_phdr.

More details:
http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

Signature · 2 lines
Thanks.
-- Max
author
#6 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hi Max,
Max Filippov wrote,

Quoted 32 lines
Quoted 11 lines · Thomas Petazzoni
Quoted 9 lines · Waldemar Brodkorb
Quoted 4 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

Quoted 8 lines

I would suspect it's a fallout of the "merge everything in libc" change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr symbol, enables unwinding support, which cause this extra reference from libgcc to libc?

No. The difference is in the __pthread_initialize_minimal function. Prior to uclibc-1.0.18 release it used to work because a call to __pthread_initialize_minimal from __uClibc_init was resolved to a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c which didn't reference pthread_unwind.
Now it is resolved to a function from libpthread/linuxthreads/pthread.c, and that file pulls in other unused functions, which reference __GI___pthread_unwind, and all the way down to the dl_iterate_phdr.

Quoted 2 lines

More details:
http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

Sorry that I didn't looked into your report in more detail last time. May be I was too confused ;)

I really can't imagine that any code from libpthread/linuxthreads can be involved when we compile Xtensa with NPTL support.

I believe there is a bug or a change in behavior regarding static linking prior to 1.0.18.

See:
wbx at helium:~/buildroot/output/build/uclibc-1.0.19 $ grep -r __pthread_initialize_minimal * Binary file lib/libc.a matches
Binary file libc/misc/internals/__uClibc_main.os matches libc/misc/internals/__uClibc_main.c:extern void weak_function __pthread_initialize_minimal(void); libc/misc/internals/__uClibc_main.c:extern void __pthread_initialize_minimal(void);
libc/misc/internals/__uClibc_main.c: * __pthread_initialize_minimal so we can use pthread_locks
libc/misc/internals/__uClibc_main.c: if (likely(__pthread_initialize_minimal!=NULL))
libc/misc/internals/__uClibc_main.c: __pthread_initialize_minimal();
libpthread/nptl_db/td_ta_thr_iter.c: /* __pthread_initialize_minimal has not run. There is just the main
libpthread/nptl_db/td_ta_map_lwp2thr.c: before __pthread_initialize_minimal has gotten far enough. They
libpthread/nptl_db/td_thr_validate.c: /* __pthread_initialize_minimal has not run yet.
libpthread/linuxthreads/internals.h:void __pthread_initialize_minimal (void); libpthread/linuxthreads/pthread.c:void __pthread_initialize_minimal(void) libpthread/nptl/init.c:void __pthread_initialize_minimal_internal (void) attribute_hidden; libpthread/nptl/init.c:__pthread_initialize_minimal_internal (void) libpthread/nptl/init.c:strong_alias (__pthread_initialize_minimal_internal,

libpthread/nptl/init.c:       __pthread_initialize_minimal)
libpthread/nptl/allocatestack.c:   initialize this, since it's done in __pthread_initialize_minimal.  */

libpthread/nptl/sysdeps/generic/libc-tls.c:extern void __pthread_initialize_minimal(void) __attribute__((weak)); libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void) libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
libpthread/nptl/sysdeps/alpha/elf/pt-initfini.c: bsr $26, __pthread_initialize_minimal_internal !samegp \n\
Binary file libpthread/nptl/sysdeps/xtensa/libc-tls.os matches
Binary file libpthread/nptl/init.os matches
Binary file utils/getconf matches

libpthread/nptl/sysdeps/xtensa/libc-tls.c includes libpthread/nptl/sysdeps/generic/libc-tls.c.

Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
LDFLAGS-libpthread.so += $(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so

$(top_builddir)lib/libdl-$(VERSION).so \
       -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal

May be this might be a reason we see different behavior.

best regards
Waldemar

author
#7 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hi,
Waldemar Brodkorb wrote,

Quoted 77 lines

Hi Max,
Max Filippov wrote,

Quoted 32 lines
Quoted 11 lines · Thomas Petazzoni
Quoted 9 lines · Waldemar Brodkorb
Quoted 4 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

Quoted 8 lines

I would suspect it's a fallout of the "merge everything in libc" change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr symbol, enables unwinding support, which cause this extra reference from libgcc to libc?

No. The difference is in the __pthread_initialize_minimal function. Prior to uclibc-1.0.18 release it used to work because a call to __pthread_initialize_minimal from __uClibc_init was resolved to a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c which didn't reference pthread_unwind.
Now it is resolved to a function from libpthread/linuxthreads/pthread.c, and that file pulls in other unused functions, which reference __GI___pthread_unwind, and all the way down to the dl_iterate_phdr.

Quoted 2 lines

More details:
http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

Sorry that I didn't looked into your report in more detail last time. May be I was too confused ;)

I really can't imagine that any code from libpthread/linuxthreads can be involved when we compile Xtensa with NPTL support.

I believe there is a bug or a change in behavior regarding static linking prior to 1.0.18.

See:
wbx at helium:~/buildroot/output/build/uclibc-1.0.19 $ grep -r __pthread_initialize_minimal * Binary file lib/libc.a matches
Binary file libc/misc/internals/__uClibc_main.os matches libc/misc/internals/__uClibc_main.c:extern void weak_function __pthread_initialize_minimal(void); libc/misc/internals/__uClibc_main.c:extern void __pthread_initialize_minimal(void);
libc/misc/internals/__uClibc_main.c: * __pthread_initialize_minimal so we can use pthread_locks
libc/misc/internals/__uClibc_main.c: if (likely(__pthread_initialize_minimal!=NULL))
libc/misc/internals/__uClibc_main.c: __pthread_initialize_minimal();
libpthread/nptl_db/td_ta_thr_iter.c: /* __pthread_initialize_minimal has not run. There is just the main
libpthread/nptl_db/td_ta_map_lwp2thr.c: before __pthread_initialize_minimal has gotten far enough. They
libpthread/nptl_db/td_thr_validate.c: /* __pthread_initialize_minimal has not run yet.
libpthread/linuxthreads/internals.h:void __pthread_initialize_minimal (void); libpthread/linuxthreads/pthread.c:void __pthread_initialize_minimal(void) libpthread/nptl/init.c:void __pthread_initialize_minimal_internal (void) attribute_hidden; libpthread/nptl/init.c:__pthread_initialize_minimal_internal (void) libpthread/nptl/init.c:strong_alias (__pthread_initialize_minimal_internal,

libpthread/nptl/init.c:       __pthread_initialize_minimal)
libpthread/nptl/allocatestack.c:   initialize this, since it's done in __pthread_initialize_minimal.  */

libpthread/nptl/sysdeps/generic/libc-tls.c:extern void __pthread_initialize_minimal(void) __attribute__((weak)); libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void) libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
libpthread/nptl/sysdeps/alpha/elf/pt-initfini.c: bsr $26, __pthread_initialize_minimal_internal !samegp \n\
Binary file libpthread/nptl/sysdeps/xtensa/libc-tls.os matches
Binary file libpthread/nptl/init.os matches
Binary file utils/getconf matches

libpthread/nptl/sysdeps/xtensa/libc-tls.c includes libpthread/nptl/sysdeps/generic/libc-tls.c.

Quoted 7 lines

Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
LDFLAGS-libpthread.so += $(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so

$(top_builddir)lib/libdl-$(VERSION).so \
       -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal

May be this might be a reason we see different behavior.

I think it is not the reason. I think it is just how pthread_cancel and some other code in NPTL is implemented. There is some circular dependency we can't change after we have integrated all the code in libc.a. Musl does not use any functionality from libgcc, so it has not the problem with the static linking issues we are seeing.

So as long as we have the pthread_cancel code which use libgcc internally, we can't avoid any breakage from packages not correctly passing -static while linking.

If anyone thinks the analysis is wrong, please speak up.

best regards
Waldemar

#8 lore raw

PATCH 1/1 autossh: honour LDFLAGS

In reply to Waldemar Brodkorb

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

Quoted 32 lines · Waldemar Brodkorb
Quoted 32 lines
Quoted 11 lines · Thomas Petazzoni
Quoted 9 lines · Waldemar Brodkorb
Quoted 4 lines

The --start-group/--end-group ask the linker to loop between -lgcc and -lc until all unresolved symbols have been resolved. So dl_iterate_phdr() being defined in the C library, but used in libgcc, I guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different? BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required BR+gcc5+musl+static - no extra -static in LDFLAGS required BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

Quoted 8 lines

I would suspect it's a fallout of the "merge everything in libc" change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr symbol, enables unwinding support, which cause this extra reference from libgcc to libc?

No. The difference is in the __pthread_initialize_minimal function. Prior to uclibc-1.0.18 release it used to work because a call to __pthread_initialize_minimal from __uClibc_init was resolved to a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c which didn't reference pthread_unwind.
Now it is resolved to a function from libpthread/linuxthreads/pthread.c, and that file pulls in other unused functions, which reference __GI___pthread_unwind, and all the way down to the dl_iterate_phdr.

Quoted 8 lines
Quoted 2 lines

More details:
http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

Sorry that I didn't looked into your report in more detail last time. May be I was too confused ;)

I really can't imagine that any code from libpthread/linuxthreads can be involved when we compile Xtensa with NPTL support.

Oops, wrong path, wrong searching tool. But the idea:
in the failing case __pthread_initialize_minimal is taken from libpthread/nptl/init.c and it's way bigger than __pthread_initialize_minimal used for linking with the uClibc-ng-1.0.17.

Now, I believe that adding the missing -static is the right solution, because even with uClibc-ng-1.0.17 the linking fails for a simple application that uses pthread:

$ cat hello.c
#include <pthread.h>

int main()

{
        pthread_create(NULL, NULL, NULL, NULL);
        return 0;
}
$ xtensa-buildroot-linux-uclibc-gcc -pthread hello.c -o hello
.../libgcc.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_lock':
.../gthr-default.h:748: undefined reference to `dl_iterate_phdr'
collect2: error: ld returned 1 exit status

The only thing that worries me at this point is that now we have a lot of pthread internals linked into the program image even when the program doesn't use them.

E.g. autossh built with uClibc-ng-1.0.17:

$ size -A autossh
autossh:
section            size      addr
.text             63771   4194484
.rodata           28112   4258256
.eh_frame           792   4286368
.tbss                 8   4291256
.ctors                8   4291256
.dtors                8   4291264
.jcr                  4   4291272
.data.rel.ro        160   4291276
.data               692   4291440
.bss              15524   4292136
.comment             31         0
.xtensa.info         56         0
.debug_aranges      288         0
.debug_info        2195         0
.debug_abbrev       398         0
.debug_line        6327         0
.debug_frame         40         0
.debug_str          468         0
.debug_loc           40         0
.xt.lit            3752         0
.xt.prop          58596         0
Total            181270
$ nm autossh  | grep pthread | wc -l
10

vs. autossh built with uClibc-ng-1.0.19:

$ size -A autossh
autossh:
section            size      addr
.text             83955   4194484
.rodata           28564   4278440
.eh_frame          1628   4307004
.tdata                4   4312728
.tbss                 8   4312732
.ctors                8   4312732
.dtors                8   4312740
.jcr                  4   4312748
.data.rel.ro        160   4312752
.data               736   4312912
.bss              24516   4313648
.comment             35         0
.xtensa.info         56         0
.debug_aranges      384         0
.debug_info       20096         0
.debug_abbrev      2925         0
.debug_line       14492         0
.debug_frame       1432         0
.debug_str         4789         0
.debug_loc        11212         0
.debug_ranges       608         0
.xt.lit            4384         0
.xt.prop          77364         0
Total            277368
$ nm autossh | grep pthread | wc -l
77

That makes me wonder, how pthread initialization is done in uClibc-ng-1.0.17 for applications that use pthread and that don't. So what I see is the following:

- an application that uses pthread_create references libpthread/nptl/init.c (because pthread_create.c includes allocatestack.c, which references __xidcmd defined in init.c from its __nptl_setxid) and thus gets __pthread_initialize_minimal from it.

- an application that doesn't use pthread_create gets its weakly defined __pthread_initialize_minimal from the libpthread/nptl/sysdeps/generic/libc-tls.c, which provides __libc_setup_tls referenced from the __uClibc_init.

So we're interested in the following three files:
- init.os (built from libpthread/nptl/init.c, with strong __pthread_initialize_minimal)
- libc-tls.os (built from libpthread/nptl/sysdeps/xtensa/libc-tls.c, with weak __pthread_initialize_minimal)
- __uClibc_main.os (built from libc/misc/internal/__uClibc_main.c with undefined reference to __pthread_initialize_minimal).

Comparing uClibc-ng-1.0.17 with uClibc-ng-1.0.19 I see the following:
1.0.17 had libc.a with libc-tls.os and __uClibc_main.os and libpthread.a with init.os.
1.0.19 has libc.a with libc-tls.os, __uClibc_main.os and init.os in it in that order.

I haven't found any related documentation, but apparently with 1.0.19 when a definition of __pthread_initialize_minimal is searched for the undefined symbol in __uClibc_main.os, the init.os is considered first. Manually removing and re-adding libc-tls.os and init.os so that the final order is __uClibc_main.os, libc-tls.os and init.os restores the 1.0.17 behavior.

Quoted 7 lines

Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
LDFLAGS-libpthread.so += $(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so

$(top_builddir)lib/libdl-$(VERSION).so \
       -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal

May be this might be a reason we see different behavior.

AFAICS it only affects shared library, i.e. not related to static linking case.

Signature · 2 lines
Thanks.
-- Max
author
#9 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hi Max,
Max Filippov wrote,

Quoted 3 lines

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

Thanks for the very detailed analysis.
Do you have any idea how we could restore the old order and keep the binaries smaller?

best regards
Waldemar

#10 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Quoted 7 lines · Waldemar Brodkorb
Quoted 3 lines

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

Thanks for the very detailed analysis.
Do you have any idea how we could restore the old order and keep the binaries smaller?

One ugly solution could be putting __uClibc-main.os before both libc-tls.os and init.os, something like the following:

libc/misc/internals/Makefile.in +1 -1
index ae094ee..d2400d2 100644
@@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
 else
 libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
 endif
-libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
+libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
 libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
   $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
   $(MISC_INTERNALS_OUT)/shared_flat_add_library.o
Signature · 2 lines
Thanks.
-- Max
author
#11 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Hi Max,
Max Filippov wrote,

Quoted 18 lines · Waldemar Brodkorb
Quoted 7 lines
Quoted 3 lines

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

Thanks for the very detailed analysis.
Do you have any idea how we could restore the old order and keep the binaries smaller?

One ugly solution could be putting __uClibc-main.os before both libc-tls.os and init.os, something like the following:

libc/misc/internals/Makefile.in +0 -0
index ae094ee..d2400d2 100644

libc/misc/internals/Makefile.in libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS · line 31

Quoted 8 lines
libc/misc/internals/Makefile.in +1 -1
@@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
 else
 libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
 endif
-libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
+libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
 libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
   $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
   $(MISC_INTERNALS_OUT)/shared_flat_add_library.o

Thanks for the patch, I tried it.
But unforunately it breaks other applications as busybox init. Not startup possible in Qemu to test the smaller autossh.

best regards
Waldemar

#12 lore raw

PATCH 1/1 autossh: honour LDFLAGS

Quoted 12 lines · Waldemar Brodkorb
Quoted 12 lines · Waldemar Brodkorb
Quoted 7 lines
Quoted 3 lines

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

Thanks for the very detailed analysis.
Do you have any idea how we could restore the old order and keep the binaries smaller?

One ugly solution could be putting __uClibc-main.os before both libc-tls.os and init.os, something like the following:

Quoted 16 lines

libc/misc/internals/Makefile.in libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS · line 31

Quoted 13 lines
libc/misc/internals/Makefile.in +1 -1
index ae094ee..d2400d2 100644
@@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
 else
 libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
 endif
-libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
+libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
 libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
   $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
   $(MISC_INTERNALS_OUT)/shared_flat_add_library.o

Thanks for the patch, I tried it.
But unforunately it breaks other applications as busybox init.

Well, it shows that the current uClibc-ng is broken without pthread code. The root cause of the init breakage is a call to _pthread_cleanup_push_defer from mallopt, which currently points to NULL, because _pthread_cleanup_push_defer is weak and its definition was removed from __uClibc_main.c in the unification patch. Restoring it fixes init for me.
I tried to follow the story of these definitions, but it seems to be a chain of failures and attempted fixes. So I'm not sure this fixes everything and doesn't break something else ): It sure needs testing.

(quoted diff) +0 -0

b/libc/misc/internals/__uClibc_main.c
index 46e24d8..0446b08 100644

libc/misc/internals/__uClibc_main.c +36 -0
@@ -68,6 +68,42 @@ uintptr_t __stack_chk_guard attribute_relro;
 
 void internal_function _dl_aux_init (ElfW(auxv_t) *av);
 
+#ifdef __UCLIBC_HAS_THREADS__
+/*
+ * uClibc internal locking requires that we have weak aliases
+ * for dummy functions in case libpthread.a is not linked in.
+ * This needs to be in compilation unit that is pulled always
+ * in or linker will disregard these weaks.
+ */
+
+static int __pthread_return_0 (pthread_mutex_t *unused) { return 0; }
+weak_alias (__pthread_return_0, __pthread_mutex_lock)
+weak_alias (__pthread_return_0, __pthread_mutex_trylock)
+weak_alias (__pthread_return_0, __pthread_mutex_unlock)
+
+int weak_function
+__pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *attr)
+{
+        return 0;
+}
+
+void weak_function
+_pthread_cleanup_push_defer(struct _pthread_cleanup_buffer *__buffer,
+                            void (*__routine) (void *), void *__arg)
+{
+        __buffer->__routine = __routine;
+        __buffer->__arg = __arg;
+}
+
+void weak_function
+_pthread_cleanup_pop_restore(struct _pthread_cleanup_buffer *__buffer,
+                             int __execute)
+{
+        if (__execute)
+                __buffer->__routine(__buffer->__arg);
+}
+#endif /* __UCLIBC_HAS_THREADS__ */
+
 #endif /* !SHARED */
 
 /* Defeat compiler optimization which assumes function addresses are

never NULL */

Signature · 2 lines
Thanks.
-- Max