Float / Double issues

Hey gang - I have what seems to be two seperate issues on Mips/Linux (cobalt boxes).

1) Programs using doubles with pthreads get corrupted data in the doubles.
2) Floats and va_arg crash with an FPE

below are two snippets that show these, at least on our boxes. If someone could confirm or deny their existence on any other platform?

Tim

#include <stdio.h>
#include <pthread.h>
#include <stdarg.h>

void doit(int, double, unsigned int);
void vait(unsigned int, ...);
void *ptfn(void *);
void doit2(int, int);
void vait2(int, ...);
void *ptfn2(void *);

int main()

{
	pthread_t pt1;
	double d1 = 1.234; 

	printf("Without threads: \n");
	ptfn((void *)&d1);
	printf("\n");
	fflush(stdout);

	printf("With threads: \n");
	pthread_create(&pt1, NULL, ptfn, &d1);
	printf("\n");
	fflush(stdout);

	pthread_exit(0);
	return 0;
}

void *ptfn(void *x)

{
	char ar[16];
	unsigned int decimals = 5;
	double d = *(double *)x;

	doit(1, d, decimals);
	vait(decimals, d);
}

void doit(int x, double d, unsigned int ui)

{
	char buff[331];

	sprintf(buff,"double: %.*f",ui,d);
	printf("%s\n", buff);
}

void vait(unsigned int ui, ...)

{
	double d=-10.0;
	va_list va;

	va_start(va, ui);

	printf("double before va_arg: %f\n", d);
	d = va_arg(va, double);
	printf("double after va_arg: %f\n", d);

	va_end(va);

	doit(1, d, ui);
}

#include <stdio.h>
#include <stdarg.h>

void doit(int, float, unsigned int);
void vait(unsigned int, ...);
void *ptfn(void *);

int main()

{
	float d1 = 1.234; 

	printf("Without threads: \n");
	ptfn((void *)&d1);
	printf("\n");
	fflush(stdout);

	return 0;
}

void *ptfn(void *x)

{
	char ar[16];
	unsigned int decimals = 5;
	float d = *(float *)x;

	doit(1, d, decimals);
	vait(decimals, d);
}

void doit(int x, float d, unsigned int ui)

{
	char buff[331];

	sprintf(buff,"float: %.*f",ui,d);
	printf("%s\n", buff);
}

void vait(unsigned int ui, ...)

{
	float d=-10.0;
	va_list va;

	va_start(va, ui);

	printf("float before va_arg: %f\n", d);
	d = va_arg(va, float);
	printf("float after va_arg: %f\n", d);

	va_end(va);

	doit(1, d, ui);
}

That one is funny. It was/is a longstanding libc bug originally reported by Dong Liu. I actually thought it'd be fixed but now thanks to your report I see it's still broken. So the multithreaded variant of your double.c doesn't work at all on our libc.

The bug is in glibc/sysdeps/unix/sysv/linux/mips/clone.S; I suppose Cobalt's libc which I last worked on late October is still using my old clone.S from that time or somebody else there came up with a funky new bug completly on it's own - which might explain why double.c cannot even successfully create the threads.

More later ...

Ralf

It is later :-)

Attached are fixed versions of clone.S. I also attach Dong's old test program. While now threads are working dong.c still fails for a large number of threads, that is > 1000 or so with a SIGSEGV. This of course doesn't solve your problem yet, Tim - but now we've got the same problem ...

Ralf

/* Copyright (C) 1996 Free Software Foundation, Inc.
This file is part of the GNU C Library. Contributed by Ralf Baechle <[email protected]>, 1996.

The GNU C Library is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

The GNU C Library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details.

You should have received a copy of the GNU Library General Public License along with the GNU C Library; see the file COPYING.LIB. If not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. */

/* clone() is even more special than fork() as it mucks with stacks
and invokes a function in the right context after its all over. */

#include <sys/asm.h>
#include <asm/unistd.h>
#include <sysdep.h>
#define _ERRNO_H 1
#include <errnos.h>

/* int clone(int (*fn)(void *arg), void *child_stack, int flags, void *arg) */

.text
NESTED(__clone,4*SZREG,sp)

	.set	noreorder
	.cpload	$25
	.set	reorder
	subu	sp, 32
	.cprestore 16

#ifdef PROF
	.set	noat
	move	$1, ra
	jal	_mcount
	.set	at

#endif

	/* Sanity check arguments.  */
	li	v0, EINVAL
	beqz	a0, error	/* no NULL function pointers */
	beqz	a1, error	/* no NULL stack pointers */

	subu	a1, 32			# reserve argument save space
	sw	a0, 0(a1)		# save function pointer
	sw	a3, 4(a1)		# save argument pointer

	move	a0, a2			# Do the system call
	li	v0, __NR_clone
	syscall

	bnez	a3, error
	beqz	v0, __thread_start

	addiu	sp, 32
	ret

	/* Something bad happened -- no child created */
error:
	addiu	sp, 32
#ifdef __PIC__
	la	t9, __syscall_error
	jr	t9
#else
	j	__syscall_error
#endif
	END(__clone)

/* Load up the arguments to the function. Put this block of code in
its own function so that we can terminate the stack trace with our debug info. */

ENTRY(__thread_start)

	/* cp is already loaded.  */
	.cprestore 16
	lw	t9, 0(sp)		# func ptr
	lw	a0, 4(sp)		# arg ptr

	/* Call the user's function */
	jalr		t9

	/* Call _exit rather than doing it inline for breakpoint purposes */
	move		a0, v0
	jal		_exit
	/* Unreached  */
	END(__thread_start)

weak_alias(__clone, clone)

#include <pthread.h>
#include <stdio.h>

void* new_thread(void* arg)

{
	int i;
	printf("Thread[%s] stack at %x\n",arg, &i);
	for (i = 0; i< 4; i++) {
	    printf("Thread[%s] %d\n", arg, i);
	    sched_yield();
	}
	return(NULL);
}

#define NUM_OF_THREAD 10000
main(int argc, char **argv)

{
        int num = NUM_OF_THREAD;
	pthread_t thread[NUM_OF_THREAD];
	
	char *args[NUM_OF_THREAD];
	int i;
	int last;
	void *status;

	if (argc > 1)
	    num = atoi(argv[1]);
	if (num>NUM_OF_THREAD)
	    num = NUM_OF_THREAD;
	printf("Original thread stack at %x\n", &i);

	for (i = 0 ; i < num; i++) {
	    args[i] = (char *)malloc(80);
	    sprintf(args[i], "%04d", i);
	    if (pthread_create(&thread[i],
			       NULL,
			       new_thread, (void *)args[i])) {
		printf("Error: creating new thread[%d]\n", i);
		break;
	    }
	}

	last = i;

	for (i = 0 ; i < last; i++) {
	    pthread_join(thread[i], &status);
	    printf("thread[%d] return status' address %p\n",i, status);
	}
	printf("%d threads created\n", last);
	exit(0);
}

We have the fix, I think. Not clone.S, but in glibc-2.0.7/linuxthreads/internals.h line 86 we need to tell gcc that this struct (_pthread_descr_struct - which defines the offset of a thread's stack from a well-aligned, malloc() returned addr) be padded to be aligned correctly on an 8byte boundary:

-};
+} __attribute__ ((aligned(__alignof__(double))));

It's working for us now. Please confirm if it works for you folks, too. A bug report has been filed for glibc, though I doubt if anything official goes on for 2.0.x, now. Hopefully 2.1.x will fold this in. A thread's stack really should always be well aligned :) Funny, the doubles were actually calculated right, but stdio functions use va_arg - which expected double's to be aligned, and weren't. The work got done, but it couldn't tell us!

As for the va_arg "bug" - I realized it was pilot error shortly after sending it out. I just needed to look at something that was NOT this double bug for a while. :) So the new question is who wants to do thread support in gdb !!

Tim

It's dangerous though. If the code would have attempted to use a MIPS II ldc1 / sdc1 instruction to access the floating point number, the code would have been killed by arch/mips/kernel/unaligned.c.

The MIPS ABI wants a 8 byte alignment for the stack and even 16 byte for ABI64. Now we know what can happen if not :-)

Thanks for your fix,

Ralf

The float / va_arg bug is a pilot error. See ANSI C draft chapter 6.5.2.2, paragraph 6 and7. In short the problem is that float parameters will be promoted to double. You therefore have to to change the call of va_arg in your float.c to va_arg(va, float).

Obvious, heh ;-)

Btw, SGI's compiler also doesn't attempt to get va_arg(..., float) right.

Ralf