Build Iterates Indefinitely on "Generating include/autoconf.mk"

While I generally develop against local file systems only, today I did a build against an NFS mounted file system and encountered new behavior in the u-boot build that I had never before seen.

That is, an indefinite iteration on "Generating include/autoconf.mk":

    $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake
-C u-boot-1.3.2/ O=${PWD}/.build/ haleakala_config
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Configuring for haleakala board...
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake
-C u-boot-1.3.2/ O=${PWD}/.build/ all
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/home/gerickson/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/home/gerickson/u-boot/u-boot-1.3.2'
    ...

This continues for 5, 10, 15 minutes or longer if I let it. Has anyone else seen this?

Thinking that this was a classic build machine / NFS server time synchronization disconnect, I confirmed that both the local machine and NFS server were slaved to the same NTP server and ran the following quick check:

    $ rm -f test; echo > test; ls -l --full-time test; date +'%T.%N'
    -rw-r--r--  1 gerickson gerickson 1 2008-03-26 16:26:03.635637000 -0700
    test
    16:26:03.648288000

The skew seemed reasonable given inherent latencies in just executing the commands, so I ruled out time skew as an issue.

By complete coincidence, I then attempted to retest things in /tmp (tmpfs) rather than the usual /build (ext3) and got the same results as with NFS:

    $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake
-C u-boot-1.3.2/ O=${PWD}/.build/ haleakala_config
    gmake: Entering directory `/tmp/u-boot/u-boot-1.3.2'
    Configuring for haleakala board...
    gmake: Leaving directory `/tmp/u-boot/u-boot-1.3.2'
    $ CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" gmake
-C u-boot-1.3.2/ O=${PWD}/.build/ all
    gmake: Entering directory `/tmp/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/tmp/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/tmp/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/tmp/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/tmp/u-boot/u-boot-1.3.2'
    Generating include/autoconf.mk
    gmake: Leaving directory `/tmp/u-boot/u-boot-1.3.2'
    gmake: Entering directory `/tmp/u-boot/u-boot-1.3.2'
    ...

I then decided to see if it was a make-specific issue and tested with make-3.79 rather than make-3.81. Sure enough, it worked in all locations (tmpfs, ext3, nfs).

I then repeated with make-3.80 and came up with the following matrix:

make-3.79 make-3.80 make-3.81
-----------------------------------------
ext3 SUCCESS SUCCESS SUCCESS
nfs SUCCESS FAILURE FAILURE
tmpfs SUCCESS FAILURE FAILURE
-----------------------------------------

If someone else has encountered this, are there any work-arounds in the u-boot build that you've experimented with?

Regards,

Grant Erickson

For those interested, I've done some additional research on this issue and have come up with the results shown below. All experiments were carried out on a i686-pc-linux-gnu 2.6.9-34.ELsmp machine. The '/usr/local/bin' versions of make are vendor-supplied.

Summary
-------

    ------------------------------------------------------------------------
    Shell   Make                                     NFS     tmpfs    ext3
    ========================================================================
    bash    /usr/bin/make (3.80)                   FAILURE  FAILURE  SUCCESS
    bash    /usr/local/bin/make-3.79 (3.79)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/make-3.80 (3.80)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/make-3.81 (3.81)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/gmake (3.81)            FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.79)            FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.80 w/ librt)   FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.80 w/o librt)  FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/ librt)   FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/o librt)  FAILURE  FAILURE  SUCCESS
    ========================================================================

Analysis
--------

The failure of two seemingly identical versions of make (/usr/local/bin/make-3.81 vs. /usr/local/bin/gmake) offered a chance to examine what differences might exist between the two versions.

The most notable difference is the libraries against which each is linked:

    % ldd /usr/bin/make
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/local/bin/make-3.79
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/local/bin/make-3.80
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/software/bin/gmake
            librt.so.1 => /lib/tls/librt.so.1 (0x00561000)
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00958000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd /usr/software/rats/bin/make-3.81
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

So, at first glance, it would appear that make 3.81's use of clock_gettime() and the subsequent link against librt and, in turn, libpthread is somehow related.

However, it turns out to not be as simple as that. If we look at the two local, private builds of make-3.81 that I generated we find that the one that uses clock_gettime() and librt as well as the one that does not BOTH FAIL:

    % ldd ~/tmp/bin/make-librt
            librt.so.1 => /lib/tls/librt.so.1 (0x00561000)
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00958000)
            /lib/ld-linux.so.2 (0x006f1000)

    % ldd ~/tmp/bin/make-nolibrt
            libc.so.6 => /lib/tls/libc.so.6 (0x0070e000)
            /lib/ld-linux.so.2 (0x006f1000)

Set-up
------

1) The 'bash' shell builds were invoked by first logging into my default shell, 'tcsh' and then invoking 'bash'.

2) I installed the DENX ELDK 4.1 to my home directory at ${HOME}/eldk/4.1.

3) The builds were performed by doing the following:

   $ for makebin in <Make to Invoke>; do rm -rf .build/*;
CROSS_COMPILE=ppc_4xx- PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" ${makebin} -C
u-boot-1.3.2/ O=${PWD}/.build haleakala_config; CROSS_COMPILE=ppc_4xx-
PATH="${PATH}:${HOME}/eldk/4.1/usr/bin" ${makebin} -C u-boot-1.3.2/
O=${PWD}/.build all; done

5) The local, private version of make (3.81) was created by:

   % ~/src/gnu/
   % wget http://ftp.gnu.org/gnu/make/make-3.81.tar.gz
   % cd source/
   % tar zxf ../make-3.81.tar.gz
   % cd ../build/
   % mkdir make-3.81
   % cd make-3.81/
   % ../../source/make-3.81/configure --prefix=${HOME}/tmp
   % make
   % make install

and was built against gcc-4.1.1. The non-librt version was created by removing '#define HAVE_CLOCK_GETTIME 1' from config.h and removing '-lrt' from LIBS in Makefile.

Any insights on what might be at play? I thought perhaps GCC, but recompiling all of the private versions against gcc-3.4.6 yields the same results.

Regards,

Grant Erickson

Grant Erickson wrote on :

Maybe `.LOW_RESOLUTION_TIME' could help? Also, it might be worth trying a
run with the -d flag, in case it shows up why make thinks include/autoconf.mk is never up-to-date; it might be informative to know WRT what precisely make thinks it is outdated.

cheers,
DaveK

Can't think of a witty .sigline today....

I think running with "-d" is a good idea, even though it will produce a disk full of output. But in theory the output should be identical for all setups.

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
When it is incorrect, it is, at least *authoritatively* incorrect.
                                    - Hitchiker's Guide To The Galaxy

Turning on '-d' was somewhat instructive in that in contrast to the previous tests without '-d' all builds I have of make v3.81 on all file system types fail (ext3fs, tmpfs, and nfs) whereas one version previously worked on all file systems and all versions worked on ext3fs:

Without '-d':
-------------

    ------------------------------------------------------------------------
    Shell   Make                                     NFS     tmpfs    ext3
    ========================================================================
    bash    /usr/local/bin/make-3.81 (3.81)        SUCCESS  SUCCESS  SUCCESS
    bash    /usr/local/bin/gmake (3.81)            FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/ librt)   FAILURE  FAILURE  SUCCESS
    bash    ${HOME}/tmp/bin/make (3.81 w/o librt)  FAILURE  FAILURE  SUCCESS
    ========================================================================

With '-d':
----------

    ------------------------------------------------------------------------
    Shell   Make                                     NFS     tmpfs    ext3
    ========================================================================
    bash    /usr/local/bin/make-3.81 (3.81)        FAILURE  FAILURE  FAILURE
    bash    /usr/local/bin/gmake (3.81)            FAILURE  FAILURE  FAILURE
    bash    ${HOME}/tmp/bin/make (3.81 w/ librt)   FAILURE  FAILURE  FAILURE
    bash    ${HOME}/tmp/bin/make (3.81 w/o librt)  FAILURE  FAILURE  FAILURE
    ========================================================================

While the log files from all 12 runs (4 versions x 3 file systems) are identical (with the obvious exceptions of paths, PIDs, etc.), what sticks out in each is that make never thinks `${ROOT}/u-boot/build/include/version_autogenerated.h' (aka VERSION_FILE) exists even though it gets remade successfully each time it is requested. This is, of course, a completely expected result since $(VERSION_FILE) is tagged '.PHONY' in the main Makefile.

Removing $(VERSION_FILE) from the .PHONY list results in successes across the entire 12 cell test matrix.

Any insights on why this very real file is tagged as .PHONY and why it should not be eliminated from the .PHONY list?

Based on the test matrix above, is there any agreement that this is a potential make bug in how .PHONY prerequisites are handled? Given this apparent root cause, I would have expected failures consistently in the ext3fs case as well as in the not-linked-against-librt-and-libpthread version case (top row of the test matrix).

Regards,

Grant Erickson