Vger broken w.r.t. gdb · kernelthread

Daniel Jacobowitz

author

1999-07-29 02:26 UTC #0 lore raw

After compiling the current vger 2.3.10 (gcc 2.95 -fno-strict-aliasing), I discovered an odd problem. Strace works just fine, but gdb doesn't. Simply 'gdb ls', run it, quit. No processes will die but every new process will recieve a SIGTRAP immediately (not quite sure if it's before or after the exec() yet). Ignoring SIGTRAP seems to make no difference. We're reaching the function ProgramCheckException in arch/ppc/kernel/traps.c:

Jul 28 21:07:32 drow kernel: Program check exception at PC: 3000c1a8, SR: 2d032, vector=700

This function includes the code:

        if (regs->msr & 0x100000) {
                /* IEEE FP exception */
                _exception(SIGFPE, regs);
        } else if (regs->msr & 0x20000) {
                /* trap exception */
#if defined(CONFIG_XMON) || defined(CONFIG_KGDB)
                if (debugger_bpt(regs))
                        return;
#endif
printk("Program check exception at PC: %lx, SR: %lx, vector=%lx\n",
  regs->nip, regs->msr, regs->trap);
                _exception(SIGTRAP, regs);
        } else {
                _exception(SIGILL, regs);
        }

I don't understand what this is supposed to do. When are program check exceptions generated? And most of all, what are those MSR bits? The first one is (1 << 20), or bit 11; this is labeled as Reserved in my reference. The second is (1 << 17), or bit 14; this is marked Implementation Dependent, and a note adds it is unused on the 601.

I would guess that the first one is supposed to check bit 20, or (1 << 11); that's FE0 and controls recoverable and precise floating point interrupts, so that would make more sense. And my hunch is that the second should be checking MSR_SE (1 << 10, bit 21).

Of course, since this file hasn't changed in an age, that leaves me completely baffled as to why this was never a problem before.

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Paul Mackerras

1999-07-29 04:00 UTC #1 lore raw

Quoted 9 lines · Daniel Jacobowitz

After compiling the current vger 2.3.10 (gcc 2.95 -fno-strict-aliasing), I discovered an odd problem. Strace works just fine, but gdb doesn't. Simply 'gdb ls', run it, quit. No processes will die but every new process will recieve a SIGTRAP immediately (not quite sure if it's before or after the exec() yet). Ignoring SIGTRAP seems to make no difference. We're reaching the function ProgramCheckException in arch/ppc/kernel/traps.c:
Jul 28 21:07:32 drow kernel: Program check exception at PC: 3000c1a8, SR: 2d032, vector=700

That address looks to be in some shared library. I haven't been using 2.3.x lately (it dies somewhere near `starting crond' on my SMP powermac), but I would suspect that the ppc ptrace code needs some work. It looks like gdb is inserting a trap instruction in a shared library which is going into the in-memory copy of the page rather than into a copy of the page.

Quoted 2 lines

I don't understand what this is supposed to do. When are program check exceptions generated? And most of all, what are those MSR bits? The

The program check exception is generated for various conditions such as floating-point exceptions, illegal instructions, privileged instructions in user mode, or trap instructions. On a program check exception, the CPU copies the MSR into the SRR1 register and then sets one or more of the high bits of SRR1 to indicate the cause of the exception. The regs->msr value actually comes from the SRR1 register.

Quoted 3 lines

first one is (1 << 20), or bit 11; this is labeled as Reserved in my reference. The second is (1 << 17), or bit 14; this is marked Implementation Dependent, and a note adds it is unused on the 601.

You need to look at the SRR1 bit settings for the program check exception.

Paul.

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Daniel Jacobowitz

author

1999-07-29 05:38 UTC #2 lore raw

Quoted 11 lines · Paul Mackerras

Quoted 8 lines · Daniel Jacobowitz
After compiling the current vger 2.3.10 (gcc 2.95 -fno-strict-aliasing), I discovered an odd problem. Strace works just fine, but gdb doesn't. Simply 'gdb ls', run it, quit. No processes will die but every new process will recieve a SIGTRAP immediately (not quite sure if it's before or after the exec() yet). Ignoring SIGTRAP seems to make no difference. We're reaching the function ProgramCheckException in arch/ppc/kernel/traps.c:

Quoted 8 lines

Quoted 1 line
Jul 28 21:07:32 drow kernel: Program check exception at PC: 3000c1a8, SR: 2d032, vector=700
That address looks to be in some shared library. I haven't been using 2.3.x lately (it dies somewhere near `starting crond' on my SMP powermac), but I would suspect that the ppc ptrace code needs some work. It looks like gdb is inserting a trap instruction in a shared library which is going into the in-memory copy of the page rather than into a copy of the page.

Well, that came from running /bin/ls:

drow:/usr/src/kernel/linux/arch/ppc/kernel$ ldd /bin/ls
        libc.so.6 => /lib/libc.so.6 (0x016ba000)
        /lib/ld.so.1 => /lib/ld.so.1 (0x08000000)

And ls itself loads at 0x01800000 from what I can tell/remember.

0x30000000 seems to be where things are being mmap()'d. Well, more or less. Relevant snippets:

execve("/bin/ls", ["ls"], [/* 21 vars */]) = 0
brk(0) = 0x184d7cc
open("/etc/ld.so.cache", O_RDONLY) = 3
mmap(0, 30308, PROT_READ, MAP_PRIVATE, 3, 0) = 0x30014000
close(3) = 0

Ahah, the first mmap was above the address we died at.

open("/lib/libc.so.6", O_RDONLY) = 3
fstat(3, {st_mode=0200000, st_size=933225093, ...}) = 0
read(3, "\177ELF\1\2\1\0\0\0\0\0\0\0\0\0\0\3\0\24\0\0\0\1\0\2 \30"..., 4096) = 4096 mmap(0x16ba000, 1268792, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x16ba000

mprotect(0x1799000, 355384, PROT_NONE)  = 0
close(3)                                = 0
munmap(0x30014000, 30308)               = 0

libc itself gets mapped in lower.

So my question is, what is at 0x3000c1a8? It would appear, if I am not misreading binfmt_elf.c, to be the program itself in its original mmap'd location. I'm not at all confident of that conclusion, though.

Quoted 10 lines

Quoted 3 lines
I don't understand what this is supposed to do. When are program check exceptions generated? And most of all, what are those MSR bits? The
The program check exception is generated for various conditions such as floating-point exceptions, illegal instructions, privileged instructions in user mode, or trap instructions. On a program check exception, the CPU copies the MSR into the SRR1 register and then sets one or more of the high bits of SRR1 to indicate the cause of the exception. The regs->msr value actually comes from the SRR1 register.

So regs->msr is actually from SRR1 of the running program? Does it generally get stuffed back into the MSR when that task is running?

Quoted 6 lines

Quoted 3 lines
first one is (1 << 20), or bit 11; this is labeled as Reserved in my reference. The second is (1 << 17), or bit 14; this is marked Implementation Dependent, and a note adds it is unused on the 601.
You need to look at the SRR1 bit settings for the program check exception.

... which I don't have a reference to at the moment. Is this sort of thing in the Motorola documentation for the processor? I would hope so. I'll look through that tomorrow.

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Paul Mackerras

1999-07-29 05:48 UTC #3 lore raw

Quoted 3 lines · Daniel Jacobowitz

So my question is, what is at 0x3000c1a8? It would appear, if I am not misreading binfmt_elf.c, to be the program itself in its original mmap'd location. I'm not at all confident of that conclusion, though.

Cat /proc/<pid>/maps while you have the ls process stopped at an appropriate point.

Quoted 2 lines

So regs->msr is actually from SRR1 of the running program? Does it generally get stuffed back into the MSR when that task is running?

Yep, but the high bits get ignored.

Paul.

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Daniel Jacobowitz

author

1999-07-30 05:18 UTC #4 lore raw

Quoted 6 lines · Paul Mackerras

Quoted 3 lines · Daniel Jacobowitz
So my question is, what is at 0x3000c1a8? It would appear, if I am not misreading binfmt_elf.c, to be the program itself in its original mmap'd location. I'm not at all confident of that conclusion, though.

Quoted 7 lines

Cat /proc/<pid>/maps while you have the ls process stopped at an appropriate point.
Quoted 2 lines
So regs->msr is actually from SRR1 of the running program? Does it generally get stuffed back into the MSR when that task is running?
Yep, but the high bits get ignored.

Now I'm really confused.

Quoted 1 line

From my readings it appears that we should only reach this point if a

trap instruction was encountered. But from what I can see no trap instruction exists at that address. At
<http://www.them.org/~drow/check-core> is a core dump I obtained while my system was in this confusing state; check-test in the same directory is the program responsible. The instruction appears to be in _dl_debug_state from what I can tell, but no trap instruction was present there.

Perhaps something having to do with instruction caching by the processor? This is a completely wild guess, but if a trap instruction was encountered, and then gdb cleared it, and the instruction cache was not flushed...

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Paul Mackerras

1999-07-30 05:24 UTC #5 lore raw

Quoted 5 lines · Daniel Jacobowitz

Now I'm really confused.
Quoted 1 line
From my readings it appears that we should only reach this point if a
trap instruction was encountered. But from what I can see no trap instruction exists at that address. At

A common thing that gdb does is to put trap instructions into the memory image of a process being debugged. It sounds to me as though, with all the page cache and memory management changes that have gone on in 2.3, we now have the undesirable situation that a trap instruction put in by gdb hasn't stayed confined to the process being debugged but has leaked into the copy of the page that is being used by other processes.

Quoted 4 lines

Perhaps something having to do with instruction caching by the processor? This is a completely wild guess, but if a trap instruction was encountered, and then gdb cleared it, and the instruction cache was not flushed...

Well, whatever the problem is, it's something new in 2.3, it's not a long-standing omission of some cache flushing stuff. It's far more likely to be something in the ptrace syscall code which hasn't been correctly updated to cope with the recent page cache and VM changes.

Paul.

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Daniel Jacobowitz

author

1999-07-30 06:03 UTC #6 lore raw

Quoted 8 lines · Paul Mackerras

Quoted 5 lines · Daniel Jacobowitz
Now I'm really confused.
Quoted 1 line
From my readings it appears that we should only reach this point if a
trap instruction was encountered. But from what I can see no trap instruction exists at that address. At

Quoted 7 lines

A common thing that gdb does is to put trap instructions into the memory image of a process being debugged. It sounds to me as though, with all the page cache and memory management changes that have gone on in 2.3, we now have the undesirable situation that a trap instruction put in by gdb hasn't stayed confined to the process being debugged but has leaked into the copy of the page that is being used by other processes.

That makes sense, but wouldn't it have shown up in the core dump if that were the case?

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Daniel Jacobowitz

author

1999-07-30 07:07 UTC #7 lore raw

nasty powerpc mmap problems (was: Re: Vger broken w.r.t. gdb)

In reply to Paul Mackerras

[CC me in replies, I'm not on linux-kernel]

Background from the thread on linuxppc-dev: with the current vger tree, gdb's breakpoints are apparently being set in the wrong copy of a shared library, and not properly removed. After running gdb over any program, even with no extra breakpoints set, every dynamically linked program run inside or outside of gdb dies with SIGTRAP. The SIGTRAP appears to be in _dl_debug_state or thereabouts in ld.so.

However, it now appears that this is symptomizing some much larger problems with mmap().

Quoted 8 lines · Paul Mackerras

Quoted 5 lines · Daniel Jacobowitz
Now I'm really confused.
Quoted 1 line
From my readings it appears that we should only reach this point if a
trap instruction was encountered. But from what I can see no trap instruction exists at that address. At

Quoted 7 lines

A common thing that gdb does is to put trap instructions into the memory image of a process being debugged. It sounds to me as though, with all the page cache and memory management changes that have gone on in 2.3, we now have the undesirable situation that a trap instruction put in by gdb hasn't stayed confined to the process being debugged but has leaked into the copy of the page that is being used by other processes.

Woah!

I've found a much simpler test case for this problem. It seems to me to be more than ptrace being broken. I wrote two test programs; mmap1 merely opens a file, mmaps it, closes it, prints some debugging info, sleeps, and prints a little more. Notice the permissions:
int fd = open("testfile", O_RDONLY);
ptr = mmap(0, 0x4000, PROT_READ|PROT_EXEC, MAP_SHARED, fd, 0); close(fd);

'attach' attaches to mmap1 and changes one value via PTRACE_POKEDATA.

Regard this:

drow:~/mmap$ dd if=/dev/zero of=testfile bs=4096 count=4
4+0 records in
4+0 records out
drow:~/mmap$ od testfile
0000000 000000 000000 000000 000000 000000 000000 000000 000000
*
0040000
drow:~/mmap$ ./mmap1
ptr[0x3000] is 0.
ptr + 0x3000 is 0x30017000.
And I am 1446

(in another window)

drow:~/mmap$ ./attach 1446 0x3001700
Value read was 0.
Value reread was 1.

(in the first window)
ptr[0x3000] is 1.

drow:~/mmap$ od testfile
0000000 000000 000000 000000 000000 000000 000000 000000 000000
*
0030000 000000 000001 000000 000000 000000 000000 000000 000000
0030020 000000 000000 000000 000000 000000 000000 000000 000000
*
0040000

Now hold on one second here! We just changed the testfile that we'd opened readonly and mmaped read|exec. Something is definitely messed up here. Worse yet, this happens even if I have no permission to write to the testfile.

Hopefully this is just on powerpc; could anyone running vger unstable on another architecture test this? The test programs are at
<http://www.them.org/~drow/mmap/>. Can anyone offer insight?

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

[[ This message was sent via the linuxppc-dev mailing list. Replies are ]] [[ not forced back to the list, so be sure to Cc linuxppc-dev if your ]] [[ reply is of general interest. Please check http://lists.linuxppc.org/ ]]
[[ and http://www.linuxppc.org/ for useful information before posting. ]]

Daniel Jacobowitz

author

1999-08-02 02:45 UTC #8 lore raw

Re: nasty powerpc mmap problems (was: Re: Vger broken w.r.t. gdb)

I think I've found the problem. access_process_vm() would trigger a page fault for write if the page was unmapped, but not if it was already mapped for read. The attached patch fixed all my problems.

Quoted 8 lines · Daniel Jacobowitz

[CC me in replies, I'm not on linux-kernel]
Background from the thread on linuxppc-dev: with the current vger tree, gdb's breakpoints are apparently being set in the wrong copy of a shared library, and not properly removed. After running gdb over any program, even with no extra breakpoints set, every dynamically linked program run inside or outside of gdb dies with SIGTRAP. The SIGTRAP appears to be in _dl_debug_state or thereabouts in ld.so.

...

Quoted 7 lines

I've found a much simpler test case for this problem. It seems to me to be more than ptrace being broken. I wrote two test programs; mmap1 merely opens a file, mmaps it, closes it, prints some debugging info, sleeps, and prints a little more. Notice the permissions:
int fd = open("testfile", O_RDONLY);
ptr = mmap(0, 0x4000, PROT_READ|PROT_EXEC, MAP_SHARED, fd, 0); close(fd);

Dan

/--------------------------------\ /--------------------------------\

|       Daniel Jacobowitz        |__|        SCS Class of 2002       |
|   Debian GNU/Linux Developer    __    Carnegie Mellon University   |
|         [email protected]         |  |       [email protected]      |
\--------------------------------/  \--------------------------------/

Index: ptrace.c
===================================================================
RCS file: /cvs/linux/linux/kernel/ptrace.c,v
retrieving revision 1.2
diff -c -p -r1.2 ptrace.c
*** ptrace.c 1999/07/23 02:04:32 1.2
--- ptrace.c 1999/08/02 02:23:45
*************** repeat:
*** 38,43 ****
--- 38,45 ----
pgtable = pte_offset(pgmiddle, addr);
if (!pte_present(*pgtable))
goto fault_in_page;
+ if (write && !pte_write(*pgtable))
+ goto fault_in_page;
page = pte_page(*pgtable);
if (MAP_NR(page) >= max_mapnr)
return 0;