tinycbor: try to fix unreproducible errors

There is a lot of unreproducible issues with tinycbor, latest is:
http://autobuild.buildroot.net/results/8e8fd1e19d6d07212f29e19a305712e715fe781b

All build failures ends with:
.config:8: *** missing separator. Stop.

I'm not 100% sure but perhaps these issues could be the result of using echo without double quotes to print Makefile variables inside .config. For example, tinycbor calls: echo system-cjson-pass := 1 >&9; instead of: echo "system-cjson-pass := 1" >&9;
As a result, echo will not print a single string but three independent parameters, this could lead in a format error when using make with parallel tasks through make -j.

Even if there is no issue, there is no good reason of not using double quotes with echo

---
 ...le-quotes-with-echo-in-Makefile.configure.patch | 46 ++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 package/tinycbor/0003-Use-double-quotes-with-echo-in-Makefile.configure.patch

new file mode 100644
index 0000000..d66c5d3
@@ -0,0 +1,46 @@
+From 738f0347535a807a3b898018cc5d0afe6649d34c Mon Sep 17 00:00:00 2001
+From: Fabrice Fontaine <[email protected]>
+Date: Tue, 13 Dec 2016 20:15:20 +0100
+Subject: [PATCH] Use double quotes with echo in Makefile.configure
+
+There is a lot of unreproducible issues with tinycbor, latest is:
+http://autobuild.buildroot.net/results/8e8fd1e19d6d07212f29e19a305712e715fe781b
+
+All build failures ends with:
+.config:8: *** missing separator.  Stop.
+
+I'm not 100% sure but perhaps these issues could be the result of using
+echo without double quotes to print Makefile variables inside .config.
+For example, tinycbor calls: echo system-cjson-pass := 1 >&9;
+instead of: echo "system-cjson-pass := 1" >&9;
+As a result, echo will not print a single string but three independent
+parameters, this could lead in a format error when using make with
+parallel tasks through make -j.
+
+Even if there is no issue, there is no good reason of not using double
+quotes with echo
+
+Signed-off-by: Fabrice Fontaine <[email protected]>
+---
+ Makefile.configure | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+
+diff --git a/Makefile.configure b/Makefile.configure
+index 424efb3..ce81560 100644
+--- a/Makefile.configure
++++ b/Makefile.configure
+@@ -22,9 +22,9 @@ sink:
+ configure: $(foreach it,$(ALLTESTS),check-$(it))
+ 
+ check-%:
+-	@echo $(subst check-,,$@)-tested := 1 $(OUT)
++	@echo "$(subst check-,,$@)-tested := 1" $(OUT)
+ 	$(if $(V),,@)if printf "$($(subst check-,PROGRAM-,$@))" | \
+ 	    $(CC) -xc $($(subst check-,CCFLAGS-,$@)) -o /dev/null - $(if $(V),,>/dev/null 2>&1); \
+ 	then \
+-	    echo $(subst check-,,$@)-pass := 1 $(OUT); \
++	    echo "$(subst check-,,$@)-pass := 1" $(OUT); \
+ 	fi
+-- 
+2.5.0
+

2.5.0

Hello,

I'm confused: this one is the *only* error with tinycbor-v0.4. See the full list of failures for this package at:
http://autobuild.buildroot.net/?reason=tinycbor-v0.4.

Why do you say there are a "lot of unreproducible issues" ?

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

2016-12-13 22:37 GMT+01:00 Thomas Petazzoni <thomas.petazzoni@free- electrons.com>:

was 5 on v0.3.2: http://autobuild.buildroot.net/?reason=tinycbor-v0.3.2. On these 5 errors, at least 2 of them are related to the same issue that we have on 0.4 as the same error arises: .config:7: *** missing separator. Stop.

I really think that there is something broken with the Makefile.configure but as I can not reproduce the problem, it is hard to investigate. An other possibility would be that the .config that is generated through Makefile is interpreted before the file is completely filled. In this case, perhaps we should define a TINYCBOR_CONFIGURE_CMDS to create this file before building. Any advise is welcome :-)

Why do you say there are a "lot of unreproducible issues" ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161213/632d663b/attachment.html>

Hello,

Ah, OK.

I've tried to reproduce
http://autobuild.buildroot.net/results/8e8/8e8fd1e19d6d07212f29e19a305712e715fe781b//
on gcc20 where the error occurred, just to rule out any build machine specific issue. Building just cjson+tinycbor worked fine.

I've done restarted the full configuration of build 8e8fd1e19d6d07212f29e19a305712e715fe781b, to see if it helps reproducing the build failure or not. If it doesn't, then we clearly have a parallel build issue.

I'll keep you informed of the result (if I don't forget). Ping me if you don't have news tomorrow evening :)

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hello,

2016-12-14 0:02 GMT+01:00 Thomas Petazzoni <
[email protected]>:

Best Regards,

Fabrice
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161215/01690c92/attachment.html>

Hello,

So, the problem occurred, but did not abort the build (which overall completed successfully). Relevant part of the build log below. It is worth mentioning that this build has tinycbor enabled, but cjson disabled.

ESC[3m>>> tinycbor v0.4 BuildingESC[23m PATH="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/bin:/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/sbin:/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin:/home/thomas/8e8fd 1e19d6d07212f29e19a305712e715fe781b/output/host/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games" /usr/bin/make -j25 PATH="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/bin:/home/thomas/8e8fd1e19d6d0721 2f29e19a305712e715fe781b/output/host/sbin:/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin:/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/u sr/games" AR="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-ar" AS="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-as" LD="/home/thomas/8e 8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-ld" NM="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-nm" CC="/home/thomas/8e8fd1e19d6d07212f29e19a305712e 715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc" GCC="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc" CPP="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr /bin/arm-none-linux-gnueabi-cpp" CXX="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-g++" FC="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueab i-gfortran" F77="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gfortran" RANLIB="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-ranlib" RE ADELF="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-readelf" STRIP="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-strip" OBJCOPY="/home/ thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-objcopy" OBJDUMP="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-objdump" AR_FOR_BUILD="/usr/bin/a r" AS_FOR_BUILD="/usr/bin/as" CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc" CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld" CPPFLAGS_FOR_BUILD="-I/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/incl ude" CFLAGS_FOR_BUILD="-O2 -I/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/include" CXXFLAGS_FOR_BUILD="-O2 -I/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/include" LDFLAGS_FOR_BUILD="-L/home /thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/lib -L/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/lib -Wl,-rpath,/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/lib" FCFLAGS_FOR_ BUILD="" DEFAULT_ASSEMBLER="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-as" DEFAULT_LINKER="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnuea bi-ld" CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os " CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os " LDFLAGS="" FCFLAGS=" -Os " FFLAGS=" -Os " PKG_CONFIG="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/pkg-config" STAGING_DIR="/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/arm-buildr oot-linux-gnueabi/sysroot" INTLTOOL_PERL=/usr/bin/perl V=1 -C /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/build/tinycbor-v0.4
Makefile:79: warning: funopen and fopencookie unavailable, open_memstream can not be implemented and conversion to JSON will not work properly!
/usr/bin/make -f ./Makefile.configure OUT='>&9' configure 9> .config

if printf "extern int open_memstream(); int main() { return open_memstream(); }" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc  -o /dev/null - ; \
        then \
            echo open_memstream-pass := 1 >&9; \
        fi
if printf "extern int funopen(); int main() { return funopen(); }" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc  -o /dev/null - ; \
        then \
            echo funopen-pass := 1 >&9; \
        fi
if printf "extern int fopencookie(); int main() { return fopencookie(); }" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc  -o /dev/null - ; \
        then \
            echo fopencookie-pass := 1 >&9; \
        fi
if printf "int main() {}" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc -Wl,--gc-sections -o /dev/null - ; \
        then \
            echo gc_sections-pass := 1 >&9; \
        fi
if printf "#include <stdlib.h>\n #include <cjson/cJSON.h>\n int main() { return cJSON_False; }" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc -lcjson -o /dev/null - ; \
        then \
            echo system-cjson-pass := 1 >&9; \
        fi
if printf "#include <stdlib.h>\n #include <cjson/cJSON.h>\n int main() { return cJSON_False; }" | \
            /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -xc -I./src/cjson -o /dev/null - ; \
        then \
            echo cjson-pass := 1 >&9; \
        fi

<stdin>:2:26: fatal error: cjson/cJSON.h: No such file or directory compilation terminated.
<stdin>:2:26: fatal error: cjson/cJSON.h: No such file or directory compilation terminated.
/tmp/ccO2eRGv.o: In function `main':
:(.text+0x8): undefined reference to `funopen'
collect2: error: ld returned 1 exit status
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborerrorstrings.o src/cborerrorstrings.c
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborencoder.o src/cborencoder.c
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborencoder_close_container_checked.o src/cborencoder_close_container_checked.c
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborparser.o src/cborparser.c
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborparser_dup_string.o src/cborparser_dup_string.c
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cborpretty.o src/cborpretty.c
mkdir -p lib
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o src/cbortojson.o src/cbortojson.c
mkdir -p bin
/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I./src -DTINYCBOR_VERSION=\"0.4\" -std=c99 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -c -o tools/cbordump/cbordump.o tools/cbordump/cbordump.c

sed > tinycbor.pc < tinycbor.pc.in \
                -e 's, at prefix@,/usr/local,' \
                -e 's, at exec_prefix@,/usr/local,' \
                -e 's, at libdir@,/usr/local/lib,' \
                -e 's, at includedir@,/usr/local/include,' \
                -e 's,@version@,0.4,'

/home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-ar cqs lib/libtinycbor.a src/cborerrorstrings.o src/cborencoder.o src/cborencoder_close_container_checked.o src/cborparser.o src/cborparser_dup_string.o src/cborpretty.o src/cbortojson.o /home/thomas/8e8fd1e19d6d07212f29e19a305712e715fe781b/output/host/usr/bin/arm-none-linux-gnueabi-gcc -o bin/cbordump tools/cbordump/cbordump.o lib/libtinycbor.a -lm
ESC[3m>>> tinycbor v0.4 Installing to staging directoryESC[23m

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hello Thomas, Frabrice,

This error messages are just stderr output of the compile tests (results should be stored in the file '.config' (any chance to take a lookg at the .config file after failure?)..., the difference to [1] is that both cJSON.h test (system-cjson-pass and cjson-pass), [1] shows only one failing cJSON.h test. Mind that the funopen test is printed first, the stderr output is shown last...

Maybe a parallel build problem (could not reproduce it local)?

Workaround by disabling parallel build? Or write a pre-generated .config before starting the compile?

	$ cat build/tinycbor-v0.4/.config 
funopen-tested := 1
fopencookie-tested := 1
open_memstream-tested := 1
system-cjson-tested := 1
gc_sections-tested := 1
cjson-tested := 1
fopencookie-pass := 1
system-cjson-pass := 1
cjson-pass := 1
open_memstream-pass := 1
gc_sections-pass := 1

Regards,
Peter

[1] http://autobuild.buildroot.net/results/8e8/8e8fd1e19d6d07212f29e19a305712e715fe781b/build-end.log

Hello,

We can try to disable the parallel build, and see if this makes the problem disappear in the autobuilders.

Thanks,

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com