Skip to contentSkip to conversation

speed up building of posix apps

PATCH In review Jan Kiszka 3 participants 11 messages Aug 18, 2006 – Aug 19, 2006

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

PATCH 1/1 speed up building of posix apps

Hi,

with the growing number of wrapped posix applications also their fairly slow build process became visible. It somehow scaled badly.

I had the idea to pass all wrapping commands to the linker via a file for quite some time. Now I tried it and it gives a nice speedup of roughly 400% for me when building typical single-file apps (mileage may vary, I'm building on vmware box...). Moreover, it beautifies the compiler output.

Tested on various setups, no regressions known so far.

Jan

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.

configure.in +1 -2
Index: configure.in
===================================================================
@@ -525,8 +525,7 @@ XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS
 XENO_USER_CFLAGS="$XENO_USER_CFLAGS -D__IN_XENO__ -Wstrict-prototypes"
 XENO_USER_APP_LDFLAGS=$XENO_USER_LDFLAGS
 
-XENO_POSIX_WRAPPERS=`while read symbol; do echo -n "-Wl,--wrap,$symbol "; done \
-        < $srcdir/src/skins/posix/posix.wrappers`
+XENO_POSIX_WRAPPERS="-Wl,@`cd $srcdir && pwd`/src/skins/posix/posix.wrappers"
 
 AC_MSG_CHECKING(whether POSIX skin library automatically calls mlockall)
 AC_ARG_ENABLE(posix-auto-mlockall,
skins/posix/posix.wrappers +89 -89

178 lines not on this page · raw message

xeno-config.in +4 -8
Index: scripts/xeno-config.in
===================================================================
@@ -89,14 +89,10 @@ while test $# -gt 0; do
             ;;
         --posix-ldflags)
             if test \! -r $XENO_POSIX_WRAPPERS; then
-	       echo "POSIX support is not available" >&2
-	       exit 1
-	    fi
-	    wrappers=`cat $XENO_POSIX_WRAPPERS | \
-	    while read symbol; do \
-                echo -n "-Wl,--wrap,$symbol " ; \
-	     done`
-            echo "$wrappers $XENO_POSIX_LDFLAGS"
+               echo "POSIX support is not available" >&2
+               exit 1
+            fi
+            echo "-Wl,@$XENO_POSIX_WRAPPERS $XENO_POSIX_LDFLAGS"
             ;;
         --lib*-dir|--libdir|--user-libdir)
             echo $XENO_LIBRARY_DIR
ChangeLog +10 -3
Index: ChangeLog
===================================================================
@@ -1,7 +1,14 @@
+2006-08-18  Jan Kiszka  <[email protected]>
+
+	* src/skins/posix/posix.wrappers, configure.in,
+	scripts/xeno-config.in: Read wrapping options from file instead of
+	passing them via command line.
+
 2006-08-16  Sebastian Smolorz  <[email protected]>
 
 	* include/rtdm/rtcan.h, ksrc/drivers/can/rtcan_{module,socket,raw}.c:
-	Get rid of RTCAN_TIMEOUT_* defines and replace them with RTDM_TIMEOUT_*
+	Get rid of RTCAN_TIMEOUT_* defines and replace them with
+	RTDM_TIMEOUT_*.
 
 2006-08-16  Jan Kiszka  <[email protected]>
 

@@ -36,8 +43,8 @@
 
 2006-08-12  Ulrich Schwab  <[email protected]>
  
- 	* ksrc/arch/i386/smi.c added id of Intel ICH7 to list of SMI chipsets
- 	
+	* ksrc/arch/i386/smi.c added id of Intel ICH7 to list of SMI chipsets
+
 2006-08-12  Gilles Chanteperdrix  <[email protected]>
 
 	* ksrc/skins/posix/thread.c (thread_destroy): Do not assume that
#1 lore raw
Quoted 3 lines · Jan Kiszka

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.

Still pondering the libtool --module issue.

Signature · 1 line
Philippe.
author
#2 lore raw
Quoted 7 lines · Philippe Gerum
Quoted 2 lines · Jan Kiszka

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.

Still pondering the libtool --module issue.

Try a grep -r shouldnotlink find /usr/lib/ -name "*.la" to get an impression what kind of libraries were linked with --module and what most likely not. I cannot imagine they all do such "patching" like we. :)

Jan

#3 lore raw
In reply to Jan Kiszka
Quoted 23 lines · Jan Kiszka

Hi,

with the growing number of wrapped posix applications also their fairly slow build process became visible. It somehow scaled badly.

I had the idea to pass all wrapping commands to the linker via a file for quite some time. Now I tried it and it gives a nice speedup of roughly 400% for me when building typical single-file apps (mileage may vary, I'm building on vmware box...). Moreover, it beautifies the compiler output.

Tested on various setups, no regressions known so far.

Jan

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.
Index: configure.in
===================================================================

configure.in XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS · line 528

Quoted 6 lines
configure.in +1 -2
@@ -525,8 +525,7 @@ XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS
 XENO_USER_CFLAGS="$XENO_USER_CFLAGS -D__IN_XENO__ -Wstrict-prototypes"
 XENO_USER_APP_LDFLAGS=$XENO_USER_LDFLAGS
 
-XENO_POSIX_WRAPPERS=`while read symbol; do echo -n "-Wl,--wrap,$symbol "; done \
-        < $srcdir/src/skins/posix/posix.wrappers`
+XENO_POSIX_WRAPPERS="-Wl,@`cd $srcdir && pwd`/src/skins/posix/posix.wrappers"

Does not work here: the @ is passed to ld which does not know how to handle it.

Signature · 1 line
					    Gilles Chanteperdrix.
author
#4 lore raw
Quoted 27 lines · Gilles Chanteperdrix
Quoted 26 lines · Jan Kiszka

Hi,

with the growing number of wrapped posix applications also their fairly slow build process became visible. It somehow scaled badly.

I had the idea to pass all wrapping commands to the linker via a file for quite some time. Now I tried it and it gives a nice speedup of roughly 400% for me when building typical single-file apps (mileage may vary, I'm building on vmware box...). Moreover, it beautifies the compiler output.

Tested on various setups, no regressions known so far.

Jan

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.
Index: configure.in
===================================================================

configure.in +0 -0
@@ -525,8 +525,7 @@ XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS
 XENO_USER_CFLAGS="$XENO_USER_CFLAGS -D__IN_XENO__ -Wstrict-prototypes"
 XENO_USER_APP_LDFLAGS=$XENO_USER_LDFLAGS
Quoted 8 lines

configure.in XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS · line 528

Quoted 3 lines
configure.in +1 -2
-XENO_POSIX_WRAPPERS=`while read symbol; do echo -n "-Wl,--wrap,$symbol "; done \
-        < $srcdir/src/skins/posix/posix.wrappers`
+XENO_POSIX_WRAPPERS="-Wl,@`cd $srcdir && pwd`/src/skins/posix/posix.wrappers"

Does not work here: the @ is passed to ld which does not know how to handle it.

Mmm, which version of ld do you have? If it's different from 2.16.91, please check if "man ld" says something about the "@file" option.

Also, could you mail me the output of the build run?

author
#5 lore raw
In reply to Gilles Chanteperdrix
Quoted 27 lines · Gilles Chanteperdrix
Quoted 26 lines · Jan Kiszka

Hi,

with the growing number of wrapped posix applications also their fairly slow build process became visible. It somehow scaled badly.

I had the idea to pass all wrapping commands to the linker via a file for quite some time. Now I tried it and it gives a nice speedup of roughly 400% for me when building typical single-file apps (mileage may vary, I'm building on vmware box...). Moreover, it beautifies the compiler output.

Tested on various setups, no regressions known so far.

Jan

PS: What about the silence-libtool patch? I've heard neither ack nor nack so far.
Index: configure.in
===================================================================

configure.in +0 -0
@@ -525,8 +525,7 @@ XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS
 XENO_USER_CFLAGS="$XENO_USER_CFLAGS -D__IN_XENO__ -Wstrict-prototypes"
 XENO_USER_APP_LDFLAGS=$XENO_USER_LDFLAGS
Quoted 7 lines

configure.in XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS · line 528

Quoted 3 lines
configure.in +1 -2
-XENO_POSIX_WRAPPERS=`while read symbol; do echo -n "-Wl,--wrap,$symbol "; done \
-        < $srcdir/src/skins/posix/posix.wrappers`
+XENO_POSIX_WRAPPERS="-Wl,@`cd $srcdir && pwd`/src/skins/posix/posix.wrappers"

Does not work here: the @ is passed to ld which does not know how to handle it.

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Thanks,
Jan

configure.in +21 -2
Index: configure.in
===================================================================
@@ -525,8 +525,27 @@ XENO_USER_APP_CFLAGS=$XENO_USER_CFLAGS
 XENO_USER_CFLAGS="$XENO_USER_CFLAGS -D__IN_XENO__ -Wstrict-prototypes"
 XENO_USER_APP_LDFLAGS=$XENO_USER_LDFLAGS
 
-XENO_POSIX_WRAPPERS=`while read symbol; do echo -n "-Wl,--wrap,$symbol "; done \
-        < $srcdir/src/skins/posix/posix.wrappers`
+AC_MSG_CHECKING([whether ld supports @file])
+AC_CACHE_VAL(ac_cv_ld_file_option,
+  AC_LANG_SAVE
+  AC_LANG_C
+  save_LDFLAGS="$LDFLAGS"
+  [LDFLAGS="-Wl,@/dev/null"]
+  AC_LINK_IFELSE([main(){}],
+    [ac_cv_ld_file_option=yes],
+    [ac_cv_ld_file_option=no])
+  LDFLAGS="$save_LDFLAGS"
+  AC_LANG_RESTORE)
+if [[ $ac_cv_ld_file_option = yes ]]; then
+	XENO_POSIX_WRAPPERS="-Wl,@`cd $srcdir && pwd`/src/skins/posix/posix.wrappers"
+else
+	XENO_POSIX_WRAPPERS=`while read wrap_option symbol; do \
+			echo -n "-Wl,$wrap_option,$symbol "; \
+		done < $srcdir/src/skins/posix/posix.wrappers`
+fi
+AC_MSG_RESULT(${ac_cv_ld_file_option:-no})
+LD_FILE_OPTION=$ac_cv_ld_file_option
+AC_SUBST(LD_FILE_OPTION)
 
 AC_MSG_CHECKING(whether POSIX skin library automatically calls mlockall)
 AC_ARG_ENABLE(posix-auto-mlockall,
skins/posix/posix.wrappers +89 -89

178 lines not on this page · raw message

xeno-config.in +13 -8
Index: scripts/xeno-config.in
===================================================================
@@ -17,6 +17,7 @@ XENO_BASE_LDFLAGS="-L${staging}${libdir}
 XENO_POSIX_CFLAGS="-I. -I${staging}${includedir} -I${staging}${includedir}/posix @XENO_USER_APP_CFLAGS@"
 XENO_POSIX_LDFLAGS="-L${staging}${libdir} -lpthread_rt -lpthread -lrt @XENO_USER_APP_LDFLAGS@"
 XENO_POSIX_WRAPPERS="${staging}${libdir}/posix.wrappers"
+XENO_POSIX_FAST_WRAPPING="@LD_FILE_OPTION@"
 XENO_LIBRARY_DIR="${staging}${libdir}"
 
 unset prefix exec_prefix libdir datadir pkgdatadir includedir

@@ -89,14 +90,18 @@ while test $# -gt 0; do
             ;;
         --posix-ldflags)
             if test \! -r $XENO_POSIX_WRAPPERS; then
-	       echo "POSIX support is not available" >&2
-	       exit 1
-	    fi
-	    wrappers=`cat $XENO_POSIX_WRAPPERS | \
-	    while read symbol; do \
-                echo -n "-Wl,--wrap,$symbol " ; \
-	     done`
-            echo "$wrappers $XENO_POSIX_LDFLAGS"
+               echo "POSIX support is not available" >&2
+               exit 1
+            fi
+            if test "$XENO_POSIX_FAST_WRAPPING" = "y"; then
+                echo "-Wl,@$XENO_POSIX_WRAPPERS $XENO_POSIX_LDFLAGS"
+            else
+                wrappers=`cat $XENO_POSIX_WRAPPERS | \
+                    while read wrap_option symbol; do \
+                        echo -n "-Wl,$wrap_option,$symbol " ; \
+                    done`
+                echo "$wrappers $XENO_POSIX_LDFLAGS"
+            fi
             ;;
         --lib*-dir|--libdir|--user-libdir)
             echo $XENO_LIBRARY_DIR
ChangeLog +10 -3
Index: ChangeLog
===================================================================
@@ -1,3 +1,9 @@
+2006-08-18  Jan Kiszka  <[email protected]>
+
+	* src/skins/posix/posix.wrappers, configure.in,
+	scripts/xeno-config.in: Read wrapping options from file instead of
+	passing them via command line.
+
 2006-08-18  Gilles Chanteperdrix  <[email protected]>
 
 	* include/asm-generic, include/asm-arm, ksrc/nucleus/pod.c:

@@ -9,7 +15,8 @@
 2006-08-16  Sebastian Smolorz  <[email protected]>
 
 	* include/rtdm/rtcan.h, ksrc/drivers/can/rtcan_{module,socket,raw}.c:
-	Get rid of RTCAN_TIMEOUT_* defines and replace them with RTDM_TIMEOUT_*
+	Get rid of RTCAN_TIMEOUT_* defines and replace them with
+	RTDM_TIMEOUT_*.
 
 2006-08-16  Jan Kiszka  <[email protected]>
 

@@ -44,8 +51,8 @@
 
 2006-08-12  Ulrich Schwab  <[email protected]>
  
- 	* ksrc/arch/i386/smi.c added id of Intel ICH7 to list of SMI chipsets
- 	
+	* ksrc/arch/i386/smi.c added id of Intel ICH7 to list of SMI chipsets
+
 2006-08-12  Gilles Chanteperdrix  <[email protected]>
 
 	* ksrc/skins/posix/thread.c (thread_destroy): Do not assume that
#6 lore raw
Quoted 4 lines · Jan Kiszka

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Seems to work. But maybe we could find the bottleneck and fix it in a general way. What part of the process is slow ?

Signature · 1 line
					    Gilles Chanteperdrix.
author
#7 lore raw
Quoted 9 lines · Gilles Chanteperdrix
Quoted 4 lines · Jan Kiszka

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Seems to work. But maybe we could find the bottleneck and fix it in a general way. What part of the process is slow ?

Don't know. Maybe it's some automake magic (the arguments look different between libtool invocation and linker run), maybe just the long list of arguments.

Anyway, it's also annoying how the screen is filled up with this irrelevant information. So besides looking for other optimisations (but things run fine now I think), I definitely vote for the @file approach for the sake of output reduction. I haven't build a larger POSIX project against Xenomai recently, but I could imagine that the dump size can become even more than just annoying...

Jan

#8 lore raw
Quoted 15 lines · Jan Kiszka
Quoted 9 lines · Gilles Chanteperdrix
Quoted 4 lines · Jan Kiszka

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Seems to work. But maybe we could find the bottleneck and fix it in a general way. What part of the process is slow ?

Don't know. Maybe it's some automake magic (the arguments look different between libtool invocation and linker run), maybe just the long list of arguments.

Quoted 6 lines

Anyway, it's also annoying how the screen is filled up with this irrelevant information. So besides looking for other optimisations (but things run fine now I think), I definitely vote for the @file approach for the sake of output reduction. I haven't build a larger POSIX project against Xenomai recently, but I could imagine that the dump size can become even more than just annoying...

Patch applied, thanks. Since we are talking about optimisation, in shell scripts, I prefer to set a variable to true (or better, the : builtin) or false, this way, it may be used directly as a condition.

Signature · 1 line
					    Gilles Chanteperdrix.
author
#9 lore raw
Quoted 23 lines · Gilles Chanteperdrix
Quoted 15 lines · Jan Kiszka
Quoted 9 lines · Gilles Chanteperdrix
Quoted 4 lines · Jan Kiszka

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Seems to work. But maybe we could find the bottleneck and fix it in a general way. What part of the process is slow ?

Don't know. Maybe it's some automake magic (the arguments look different between libtool invocation and linker run), maybe just the long list of arguments.

Quoted 6 lines

Anyway, it's also annoying how the screen is filled up with this irrelevant information. So besides looking for other optimisations (but things run fine now I think), I definitely vote for the @file approach for the sake of output reduction. I haven't build a larger POSIX project against Xenomai recently, but I could imagine that the dump size can become even more than just annoying...

Quoted 4 lines

Patch applied, thanks. Since we are talking about optimisation, in shell scripts, I prefer to set a variable to true (or better, the : builtin) or false, this way, it may be used directly as a condition.

Sorry, don't get yet what part you are referring to. If you would like to express something in that patch differently, go ahead. I was already happy when it worked. :)

Jan

#10 lore raw
Quoted 24 lines · Jan Kiszka
Quoted 23 lines · Gilles Chanteperdrix
Quoted 21 lines · Jan Kiszka
Quoted 9 lines · Gilles Chanteperdrix
Quoted 4 lines · Jan Kiszka

Ok, here comes version 2, now with detection of the required ld feature. Falls back to normal behaviour if ld is too old. Could you test it please?

[Grmbl, the fun stops where autoconf begins...]

Seems to work. But maybe we could find the bottleneck and fix it in a general way. What part of the process is slow ?

Don't know. Maybe it's some automake magic (the arguments look different between libtool invocation and linker run), maybe just the long list of arguments.

Anyway, it's also annoying how the screen is filled up with this irrelevant information. So besides looking for other optimisations (but things run fine now I think), I definitely vote for the @file approach for the sake of output reduction. I haven't build a larger POSIX project against Xenomai recently, but I could imagine that the dump size can become even more than just annoying...

Quoted 8 lines
Quoted 4 lines

Patch applied, thanks. Since we are talking about optimisation, in shell scripts, I prefer to set a variable to true (or better, the : builtin) or false, this way, it may be used directly as a condition.

Sorry, don't get yet what part you are referring to. If you would like to express something in that patch differently, go ahead. I was already happy when it worked. :)

I meant, instead of doing:
var=yes

if test $var = yes

do:

var=true

if $var

But it is not really important.

Signature · 1 line
					    Gilles Chanteperdrix.