Skip to contentSkip to conversation

x86_64 frame pointer via thread context

DISCUSSION Andi Kleen 4 participants 10 messages Aug 6, 2005 – Aug 8, 2005 1 message not in the archive

lore
Share & tools
lore

Andi Kleen

author
#1 lore raw
In reply to a message not in the archive
Quoted 4 lines · Dave Jiang

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

-Andi

#2 lore raw
Quoted 8 lines
Quoted 4 lines

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

Note that there is -fomit-frame-pointer which might give different results than without the option (or explicitly -fno-omit-frame-pointer).

Jan Engelhardt

#3 lore raw
Quoted 9 lines · Jan Engelhardt
Quoted 7 lines
Quoted 3 lines

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

Quoted 4 lines

Note that there is -fomit-frame-pointer which might give different results than without the option (or explicitly -fno-omit-frame-pointer).

Jan Engelhardt

I had somebody else test it on FC4 and he observed the same issue. It may be timing sensitive? Both platforms tested are em64t based so I'm not sure if on amd64 platforms it varies or not.....

And you definitely have to include the -fno-omit-frame-pointer. x86_64 gcc by default has -fomit-frame-pointer on and without explicitly stating that you want frame pointer you won't get it in rBP.

It is possible that userspace or toolchain may be suspect, However, why is the value bad in kernel space when rBP from pt_regs is dumped?

Signature · 9 lines
Dave

------------------------------------------------------
Dave Jiang
Software Engineer          Phone: (480) 517-0372
MontaVista Software, Inc.    Fax: (480) 517-0262
2141 E Broadway Rd, St 108   Web: www.mvista.com
Tempe, AZ  85282          mailto:[email protected]
------------------------------------------------------
#4 lore raw
Quoted 16 lines · Dave Jiang
Quoted 14 lines · Jan Engelhardt
Quoted 8 lines
Quoted 3 lines

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

Note that there is -fomit-frame-pointer which might give different results than without the option (or explicitly -fno-omit-frame-pointer).

Quoted 10 lines
Quoted 1 line

Jan Engelhardt

I had somebody else test it on FC4 and he observed the same issue. It may be timing sensitive? Both platforms tested are em64t based so I'm not sure if on amd64 platforms it varies or not.....

And you definitely have to include the -fno-omit-frame-pointer. x86_64 gcc by default has -fomit-frame-pointer on and without explicitly stating that you want frame pointer you won't get it in rBP.

It is possible that userspace or toolchain may be suspect, However, why is the value bad in kernel space when rBP from pt_regs is dumped?

One additional note. The app dumps the rBP for two threads. One looks okay, and the other gives 0xffffffff.

Signature · 9 lines
Dave

------------------------------------------------------
Dave Jiang
Software Engineer          Phone: (480) 517-0372
MontaVista Software, Inc.    Fax: (480) 517-0262
2141 E Broadway Rd, St 108   Web: www.mvista.com
Tempe, AZ  85282          mailto:[email protected]
------------------------------------------------------
#5 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 3 lines · Dave Jiang

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

Quoted 5 lines

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

-Andi

I revised the app a little so that it would allow the threads to start, thus should prevent rBP w/ all 0's showing up. Below are some of results that I've gotten from various different distros and platforms. As you can see, the f's shows up on most of them, including Suse 9.2. The only one showed up looking ok is the Mandrake/Mandriva distro. I'm not sure how different SLES9 is from Suse9.2....

2.6.13-rc5 w/ custom rootfs (NPTL) and toolchain (gcc 3.4.3 based), em64t:
Thread 1 (0x407ff960) created
Thread 2 (0x40fff960) created
Main pid: 0x2aaaaaef3860
pid1[0x407ff960] signaled

tb_sig_handler entered
thread 0x407ff960 context
rIP: 00002aaaaad5fb15
rSP: 00000000407ff610
rBP: 00000000ffffffff

pid2[0x40fff960] signaled

tb_sig_handler entered
thread 0x40fff960 context
rIP: 00002aaaaad5fb15
rSP: 0000000040fff610
rBP: 00000000ffffffff
--------------------------------------------------------------

FC4, em64t
Thread 1 (0x40a00960) created Thread 2 (0x41401960) created Main pid: 0x2aaaaaad59e0 pid1[0x40a00960] signaled

tb_sig_handler entered
thread 0x40a00960 context
rIP: 0000003cdc9925d1
rSP: 00000000409fffc0
rBP: 00000000ffffffff

pid2[0x41401960] signaled

tb_sig_handler entered
thread 0x41401960 context
rIP: 0000003cdc9925d1
rSP: 0000000041400fc0
rBP: 00000000ffffffff
------------------------------------------------------------

Mandrake 10.2, em64t
Thread 1 (0x40800960) created Thread 2 (0x41001960) created Main pid: 0x2aaaaaf07ea0 pid1[0x40800960] signaled

tb_sig_handler entered
thread 0x40800960 context
rIP: 00002aaaaad643b5
rSP: 00000000407fffc0
rBP: 00000000408001d0

pid2[0x41001960] signaled

tb_sig_handler entered
thread 0x41001960 context
rIP: 00002aaaaad643b5
rSP: 0000000041000fc0
rBP: 00000000410011d0
------------------------------------------------------------

Suse 9.2, AMD64
Thread 1 (0x401ff960) created Thread 2 (0x403ff960) created Main pid: 0x2a959a3860 pid1[0x401ff960] signaled

tb_sig_handler entered
thread 0x401ff960 context
rIP: 0000002a9580d675
rSP: 00000000401ff5e0
rBP: 00000000ffffffff

pid2[0x403ff960] signaled

tb_sig_handler entered
thread 0x403ff960 context
rIP: 0000002a9580d675
rSP: 00000000403ff5e0
rBP: 00000000ffffffff

Signature · 9 lines
Dave

------------------------------------------------------
Dave Jiang
Software Engineer          Phone: (480) 517-0372
MontaVista Software, Inc.    Fax: (480) 517-0262
2141 E Broadway Rd, St 108   Web: www.mvista.com
Tempe, AZ  85282          mailto:[email protected]
------------------------------------------------------

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <signal.h>
#include <unistd.h>
#include <sys/ucontext.h> #include <malloc.h>
#include <errno.h>

#define SCP_FROM_UCONTEXT(uc) \
    ((struct sigcontext *) &((struct ucontext *)(uc))->uc_mcontext)

static int _traceback_signal = -1; static pthread_t pid1 = 0; static pthread_t pid2 = 0; static pthread_t pidmain = 0;

extern int __libc_allocate_rtsig(int high);

static void tb_sig_handler(int sig, siginfo_t *info, void *ucontext)

{
	struct sigcontext *scp;
	int ret, i;

	printf("\n%s entered\n", __func__);
	printf("thread 0x%lx context\n", pthread_self());
	scp = SCP_FROM_UCONTEXT(ucontext);

	printf("rIP: %16.16lx\n", scp->rip);
	printf("rSP: %16.16lx\n", scp->rsp);
	printf("rBP: %16.16lx\n", scp->rbp);

	printf("\n");
}

void * test_thread1(void *arg)

{
	while(1) {
		sleep(2);
	};
	return NULL;
}

void * test_thread2(void *arg)

{
	while(1) {
		sleep(2);
	};
	return NULL;
}

int main()

{
	struct sigaction act;
	int ret = 0;
	
	_traceback_signal = __libc_allocate_rtsig(1);
	act.sa_sigaction = tb_sig_handler;
	sigemptyset(&act.sa_mask);
	act.sa_flags = SA_RESTART | SA_SIGINFO;
	sigaction(_traceback_signal, &act, NULL);

	ret = pthread_create(&pid1, NULL, test_thread1, NULL);
	if(ret < 0) {
		fprintf(stderr, "thread 1 creation failed\n");
		return -1;
	}
	printf("Thread 1 (0x%lx) created\n", pid1);

	sleep(1);

	ret = pthread_create(&pid2, NULL, test_thread2, NULL);
	if(ret < 0) {
		fprintf(stderr, "thread 2 creation failed\n");
		return -1;
	}
	printf("Thread 2 (0x%lx) created\n", pid2);

	sleep(1);

	pidmain = pthread_self();

	printf("Main pid: 0x%lx\n", pidmain);
	
	ret = pthread_kill(pid1, _traceback_signal);
	if(ret >= 0) {
		printf("pid1[0x%lx] signaled\n", pid1);
	}

	sleep(1);
	
	ret = pthread_kill(pid2, _traceback_signal);
	if(ret >= 0) {
		printf("pid2[0x%lx] signaled\n", pid2);
	}

	sleep(5);
#if 0
	ret = pthread_kill(pidmain, _traceback_signal);
	if(ret >= 0) {
		printf("pidmain[0x%lx] signaled\n", pidmain);
	}

#endif

	return 0;
}
#6 lore raw
Quoted 10 lines · Dave Jiang
Quoted 8 lines · Andi Kleen
Quoted 3 lines · Dave Jiang

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

Quoted 8 lines
Quoted 5 lines

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

-Andi

I revised the app a little so that it would allow the threads to start, thus should prevent rBP w/ all 0's showing up. Below are some of results that I've gotten from various different distros and platforms. As you can see, the f's shows up on most of them, including Suse 9.2. The only one showed up looking ok is the Mandrake/Mandriva distro. I'm not sure how different SLES9 is from Suse9.2....

Replace call to sleep() with busy loop. Glibc's sleep() uses %ebp for its own data, so when you interrupt sleep(), you get rbp=(unsigned int)-1, as rbp really contains 0x0000.0000.ffff.ffff when nanosleep() syscall is issued.
Petr

#7 lore raw
Quoted 21 lines · Petr Vandrovec
Quoted 19 lines · Dave Jiang
Quoted 14 lines · Andi Kleen
Quoted 3 lines · Dave Jiang

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

-Andi

Quoted 9 lines
Quoted 1 line

I revised the app a little so that it would allow the threads to start, thus should prevent rBP w/ all 0's showing up. Below are some of results that I've gotten from various different distros and platforms. As you can see, the f's shows up on most of them, including Suse 9.2. The only one showed up looking ok is the Mandrake/Mandriva distro. I'm not sure how different SLES9 is from Suse9.2....

Replace call to sleep() with busy loop. Glibc's sleep() uses %ebp for its own data, so when you interrupt sleep(), you get rbp=(unsigned int)-1, as rbp really contains 0x0000.0000.ffff.ffff when nanosleep() syscall is issued.
Petr

From what I understand, when you signal a thread, the signal handler executes in the thread context and not the main process context. So therefore the rbp would be the thread's copy and not the one that sleep() just modified. So whatever sleep does to the main process context, there shouldn't be any effect on the thread context.... Also, what can I call to allow the threads to run? sleep() allows me to run the other threads. Busy wait does not.....

Signature · 9 lines
Dave

------------------------------------------------------
Dave Jiang
Software Engineer          Phone: (480) 517-0372
MontaVista Software, Inc.    Fax: (480) 517-0262
2141 E Broadway Rd, St 108   Web: www.mvista.com
Tempe, AZ  85282          mailto:[email protected]
------------------------------------------------------
#8 lore raw
Quoted 36 lines · Dave Jiang
Quoted 27 lines · Petr Vandrovec
Quoted 16 lines · Dave Jiang
Quoted 14 lines · Andi Kleen
Quoted 3 lines · Dave Jiang

Am I doing something wrong, or is this intended to be this way on x86_64, or is something incorrect in the kernel? This method works fine on i386. Thanks for any help!

I just tested your program on SLES9 with updated kernel and RBP looks correct to me. Probably something is wrong with your user space includes or your compiler.

Quoted 6 lines
Quoted 1 line

-Andi

I revised the app a little so that it would allow the threads to start, thus should prevent rBP w/ all 0's showing up. Below are some of results that I've gotten from various different distros and platforms. As you can see, the f's shows up on most of them, including Suse 9.2. The only one showed up looking ok is the Mandrake/Mandriva distro. I'm not sure how different SLES9 is from Suse9.2....

Quoted 6 lines

Replace call to sleep() with busy loop. Glibc's sleep() uses %ebp for its own data, so when you interrupt sleep(), you get rbp=(unsigned int)-1, as rbp really contains 0x0000.0000.ffff.ffff when nanosleep() syscall is issued.
Petr

Quoted 2 lines

From what I understand, when you signal a thread, the signal handler
executes in the thread context and not the main process context. So therefore the rbp would be the thread's copy and not the one that sleep() just modified. So whatever sleep does to the main process context, there shouldn't be any effect on the thread context.... Also, what can I call to allow the threads to run? sleep() allows me to run the other threads. Busy wait does not.....

I do not understand. You call sleep() from both threads you spawn (as well from main), so both threads are always interrupted in the sleep(2). Load your process to the debugger...

#0  tb_sig_handler (sig=33, info=0x407ff2f0, ucontext=0x407ff1c0) at ttest1.c:26
#1  <signal handler called>
#2  0x00002aaaaad81335 in nanosleep () from /lib/libc.so.6
#3  0x00002aaaaad811a5 in sleep () from /lib/libc.so.6
#4  0x0000000000400871 in test_thread1 (arg=0x0) at ttest1.c:40
#5  0x00002aaaaabc6b55 in start_thread () from /lib/libpthread.so.0
#6  0x00002aaaaada87f0 in clone () from /lib/libc.so.6

Petr

#9 lore raw
Quoted 13 lines · Petr Vandrovec
Quoted 7 lines
Quoted 5 lines

Replace call to sleep() with busy loop. Glibc's sleep() uses %ebp for its own data, so when you interrupt sleep(), you get rbp=(unsigned int)-1, as rbp really contains 0x0000.0000.ffff.ffff when nanosleep() syscall is issued.
Petr

From what I understand, when you signal a thread, the signal handler
executes in the thread context and not the main process context. So therefore the rbp would be the thread's copy and not the one that sleep() just modified. So whatever sleep does to the main process context, there shouldn't be any effect on the thread context.... Also, what can I call to allow the threads to run? sleep() allows me to run the other threads. Busy wait does not.....

I do not understand. You call sleep() from both threads you spawn (as well from main), so both threads are always interrupted in the sleep(2). Load your process to the debugger...

Quoted 7 lines
#0  tb_sig_handler (sig=33, info=0x407ff2f0, ucontext=0x407ff1c0) at ttest1.c:26
#1  <signal handler called>
#2  0x00002aaaaad81335 in nanosleep () from /lib/libc.so.6
#3  0x00002aaaaad811a5 in sleep () from /lib/libc.so.6
#4  0x0000000000400871 in test_thread1 (arg=0x0) at ttest1.c:40
#5  0x00002aaaaabc6b55 in start_thread () from /lib/libpthread.so.0
#6  0x00002aaaaada87f0 in clone () from /lib/libc.so.6

Ooops, you are right. I forgot about those ones in the threads. Yes you are right. Once the sleep goes away rBP displays the correct values. Is this issue due to glibc or because of the toolchain? I do not have this issues on 32bit x86.... I would assume that the reason it works on Mandrake is due to the toolchain they use versus other distros? The toolchain determines which registers to use and the -fno-omit-frame-pointer did not prevent some of them from clobbering rbp?

Signature · 9 lines
Dave

------------------------------------------------------
Dave Jiang
Software Engineer          Phone: (480) 517-0372
MontaVista Software, Inc.    Fax: (480) 517-0262
2141 E Broadway Rd, St 108   Web: www.mvista.com
Tempe, AZ  85282          mailto:[email protected]
------------------------------------------------------
#10 lore raw
Quoted 10 lines · Dave Jiang
Quoted 7 lines · Petr Vandrovec
#0  tb_sig_handler (sig=33, info=0x407ff2f0, ucontext=0x407ff1c0) at ttest1.c:26
#1  <signal handler called>
#2  0x00002aaaaad81335 in nanosleep () from /lib/libc.so.6
#3  0x00002aaaaad811a5 in sleep () from /lib/libc.so.6
#4  0x0000000000400871 in test_thread1 (arg=0x0) at ttest1.c:40
#5  0x00002aaaaabc6b55 in start_thread () from /lib/libpthread.so.0
#6  0x00002aaaaada87f0 in clone () from /lib/libc.so.6
Quoted 2 lines

Ooops, you are right. I forgot about those ones in the threads. Yes you are right. Once the sleep goes away rBP displays the correct values. Is this issue due to glibc or because of the toolchain? I do not have this issues on 32bit x86.... I would assume that the reason it works on Mandrake is due to the toolchain they use versus other distros? The toolchain determines which registers to use and the -fno-omit-frame-pointer did not prevent some of them from clobbering rbp?

You are building only your application with -fno-omit-frame-pointer, libraries you are using are just used as is. On 32bit x86 it works as -O2 on x86 does not imply -fomit-frame-pointer, as frame pointer is required (well, was) for debugging. On x86-64 frame pointer is not needed for debugging as it was always using DWARF debug info, and so -O2 on x86-64 implies -fomit-frame-pointer. Due to this most of libraries you'll find on your 64bit system are built without frame pointer. Mandrake either explicitly asks for -fno-omit-frame-pointer, or maybe their glibc is just sufficiently different that their sleep() does not need %rbp for sleep().

Loading their glibc to the debugger and inspecting sleep & nanosleep will reveal whether %rbp is just unchanged by these (and so it works due to luck), or whether their sleep uses pushq %rbp; movq %rsp,%rbp
- in which case they built glibc with frame pointers for no apparent reason.

You must use debugging informations to get stacktrace on x86-64. Blindly following %rbp does not work on this architecture (and lot of others).
Petr