Running the gdb 7.12.1 testsuite breaks kernel 4.13.8 on C8000

DISCUSSION Rolf Eike Beer 3 participants 15 messages Jan 25, 2018 – Jul 17, 2019

Subsystems (2)

PA-RISC architecture ptrace support

lore

Share & tools

lore

Rolf Eike Beer

author

2018-01-25 08:59 UTC #0 lore raw

The last message I had for the crash below is:

Running /var/tmp/portage/sys-devel/gdb-7.12.1/work/gdb-7.12.1/gdb/testsuite/gdb.base/bp-permanent.exp ...

[  909.754182] Backtrace:
[  909.756303] 
[  909.756303] 
[  909.756303] Bad Address (null pointer deref?): Code=6 (Instruction TLB miss fault) regs=000000007c818a50 (Addr=0000000000000000)
[  909.756303] CPU: 0 PID: 5191 Comm: bp-permanent Not tainted 4.13.8 #2
[  909.756303] task: 000000007c8183d0 task.stack: 000000007aad8000
[  909.756303] 
[  909.756303]      YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
[  909.756303] PSW: 00000000000001000000011100001111 Not tainted
[  909.756303] r00-03  000000000004070f 00000000f8f07840 00000000000106bb 00000000f8f07880
[  909.756303] r04-07  00000000f815bc70 00000000000e87a0 0000000000000000 00000000000e8818
[  909.756303] r08-11  0000000000000000 0000000000000000 000000000000006c 0000000000000000
[  909.756303] r12-15  0000000000000000 0000000000000000 0000000000080a74 00000000000e83f8
[  909.756303] r16-19  00000000000c85c8 00000000000cbad8 00000000000c99e8 00000000f815bc70
[  909.756303] r20-23  00000000000120bc 0000000000000004 ffffffffffffe520 0000000000012022
[  909.756303] r24-27  00000000000120bc 00000000000105d8 00000000000120b8 0000000000012000
[  909.756303] r28-31  00000000000120b8 0000000000000001 00000000f8f078c0 00000000f8bffa9f
[  909.756303] sr00-03  00000000010a1800 00000000010a1800 0000000000000000 00000000010a1800
[  909.756303] sr04-07  00000000010a1800 00000000010a1800 00000000010a1800 00000000010a1800
[  909.756303] 
[  909.756303] IASQ: 00000000010a1800 00000000010a1800 IAOQ: 0000000000000000 0000000000000004
[  909.756303]  IIR: 43ffff80    ISR: 0000000010240000  IOR: 000000a43fb07880
[  909.756303]  CPU:        0   CR30: 000000007aad8000 CR31: ffffffffffffffff
[  909.756303]  ORIG_R28: 0000000000000000
[  909.756303]  IAOQ[0]:           (null)
[  909.756303]  IAOQ[1]: 0x4
[  909.756303]  RP(r2): 0x106bb
[  909.756303] Backtrace:
[  909.756303] 
[  909.756303] Kernel panic - not syncing: Bad Address (null pointer deref?)
[  909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)

This is actually the second place where it breaks, before that I had the same with this test (twice):

Running /var/tmp/portage/sys-devel/gdb-7.12.1/work/gdb-7.12.1/gdb/testsuite/gdb.base/async.exp ...

It's a Gentoo system, the kernel has been built with gcc 6.4.0.

John David Anglin

2018-01-25 13:01 UTC #1 lore raw

Quoted 5 lines · Rolf Eike Beer

[  909.756303] Kernel panic - not syncing: Bad Address (null pointer deref?)
[  909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)
This is actually the second place where it breaks, before that I had the same with this test (twice):

Would you post the PIM dump of the most recent HPMC in the service menu? Also needed is the assembler dump of the routine where the HPMC occurred in your kernel. You need the 64-bit version of objdump for this.

There's no info in the posted register dump about the panic other than it was probably generated by the exception.

Dave

Signature · 1 line

John David Anglin  [email protected]

Rolf Eike Beer

author

2018-01-25 15:36 UTC #2 lore raw

Quoted 8 lines · John David Anglin

Quoted 5 lines · Rolf Eike Beer
[  909.756303] Kernel panic - not syncing: Bad Address (null pointer deref?)
[  909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)
This is actually the second place where it breaks, before that I had the same with this test (twice):
Would you post the PIM dump of the most recent HPMC in the service menu?

The service menu does not give any information, as in this older bug: https://bugs.gentoo.org/481768

Quoted 4 lines

Also needed
is the assembler dump of the routine where the HPMC occurred in your kernel. You need the
64-bit version of objdump for this.

I have put the kernel on https://opensource.sf-tec.de/c8000-kernel.tar.xz

Eike

Helge Deller

maintainer

2018-01-26 22:31 UTC #3 lore raw

Quoted 11 lines · Rolf Eike Beer

Quoted 8 lines · John David Anglin
Quoted 5 lines · Rolf Eike Beer
[ 909.756303] Kernel panic - not syncing: Bad Address (null pointer deref?) [ 909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)
This is actually the second place where it breaks, before that I had the same with this test (twice):
Would you post the PIM dump of the most recent HPMC in the service menu?
The service menu does not give any information, as in this older bug: https://bugs.gentoo.org/481768

FWIW, I've tested the testcase from
https://bugs.gentoo.org/481768
on a debian system:
Linux panama.osuosl.org 4.14.0-2-parisc64-smp #1 SMP Debian 4.14.7-1 (2017-12-28) parisc64 GNU/Linux gdb was version 7.12-6+b1

On debian I can not reproduce the crash.

With this sequence:
(gdb) break gdb-crash.c:14
(gdb) run
(gdb) set tp = {0,0}

tp isn't initialized yet before you reach line 25, and as such it points to random memory. I you try to set tp, it depends on what happens (but agreed, it shouldn't crash the kernel).

The Debian kernel hasn't any additional hppa-specific patches.

Quoted 4 lines

Quoted 4 lines
Also needed
is the assembler dump of the routine where the HPMC occurred in your kernel. You need the
64-bit version of objdump for this.

I wonder why there isn't any kernel backtrace... Does gentoo uses special compiler-optimization flags?

Quoted 1 line

I have put the kernel on https://opensource.sf-tec.de/c8000-kernel.tar.xz

I'd suggest you put some debugging code in your kernel, e.g. in compat_ptrace_request() in kernel/ptrace.c.
I think gdb uses PTRACE_POKEDATA to set some userspace memory of a process. It's generic code, so I wonder why it should crash on parisc.

You may look at compat_arch_ptrace() in arch/parisc/kernel/ptrace.c too, but it doesn't touch memory as far as I can see...

Helge

Rolf Eike Beer

author

2018-01-27 17:42 UTC #4 lore raw

Am Freitag, 26. Januar 2018, 23:31:46 schrieb Helge Deller:

Quoted 28 lines · Rolf Eike Beer

Quoted 6 lines
Quoted 5 lines · John David Anglin
Quoted 4 lines · Rolf Eike Beer
[  909.756303] Kernel panic - not syncing: Bad Address (null pointer
deref?) [ 909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
This is actually the second place where it breaks, before that I had the same>>
with this test (twice):
Would you post the PIM dump of the most recent HPMC in the service menu?
The service menu does not give any information, as in this older bug:
https://bugs.gentoo.org/481768
FWIW, I've tested the testcase from
https://bugs.gentoo.org/481768
on a debian system:
Linux panama.osuosl.org 4.14.0-2-parisc64-smp #1 SMP Debian 4.14.7-1 (2017-12-28) parisc64 GNU/Linux gdb was version 7.12-6+b1
On debian I can not reproduce the crash.
With this sequence:
(gdb) break gdb-crash.c:14
(gdb) run
(gdb) set tp = {0,0}

Quoted 5 lines

tp isn't initialized yet before you reach line 25, and as such it points to random memory. I you try to set tp, it depends on what happens (but agreed, it shouldn't crash the kernel).
The Debian kernel hasn't any additional hppa-specific patches.

I'm using vanilla, so nothing on my side either.

Quoted 6 lines

Quoted 4 lines
Quoted 4 lines
Also needed
is the assembler dump of the routine where the HPMC occurred in your kernel. You need the
64-bit version of objdump for this.
I wonder why there isn't any kernel backtrace...

This was with "dmesg -n 8". Without it has shown only the "Bad address" and "end Kernel panic" lines, which is IMHO bad by itself.

Quoted 1 line

Does gentoo uses special compiler-optimization flags?

I have none.

Quoted 9 lines

Quoted 1 line
I have put the kernel on https://opensource.sf-tec.de/c8000-kernel.tar.xz
I'd suggest you put some debugging code in your kernel, e.g. in compat_ptrace_request() in kernel/ptrace.c.
I think gdb uses PTRACE_POKEDATA to set some userspace memory of a process. It's generic code, so I wonder why it should crash on parisc.
You may look at compat_arch_ptrace() in arch/parisc/kernel/ptrace.c too, but it doesn't touch memory as far as I can see...

I will not be able to touch any of this until at least mid of next week. If you want to give it a try, check gdb 7.12.1 testsuite. It was one of the first tests that hit this.

Eike

Helge Deller

maintainer

2018-01-27 18:41 UTC #5 lore raw

Hi Rolf,

Quoted 58 lines · Rolf Eike Beer

Am Freitag, 26. Januar 2018, 23:31:46 schrieb Helge Deller:
Quoted 28 lines · Rolf Eike Beer
Quoted 6 lines
Quoted 5 lines · John David Anglin
Quoted 4 lines · Rolf Eike Beer
[  909.756303] Kernel panic - not syncing: Bad Address (null pointer
deref?) [ 909.756303] ---[ end Kernel panic - not syncing: Bad Address (null pointer deref?)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
This is actually the second place where it breaks, before that I had the same>>
with this test (twice):
Would you post the PIM dump of the most recent HPMC in the service menu?
The service menu does not give any information, as in this older bug:
https://bugs.gentoo.org/481768
FWIW, I've tested the testcase from
https://bugs.gentoo.org/481768
on a debian system:
Linux panama.osuosl.org 4.14.0-2-parisc64-smp #1 SMP Debian 4.14.7-1 (2017-12-28) parisc64 GNU/Linux gdb was version 7.12-6+b1
On debian I can not reproduce the crash.
With this sequence:
(gdb) break gdb-crash.c:14
(gdb) run
(gdb) set tp = {0,0}
Quoted 5 lines
tp isn't initialized yet before you reach line 25, and as such it points to random memory. I you try to set tp, it depends on what happens (but agreed, it shouldn't crash the kernel).
The Debian kernel hasn't any additional hppa-specific patches.
I'm using vanilla, so nothing on my side either.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Also needed
is the assembler dump of the routine where the HPMC occurred in your kernel. You need the
64-bit version of objdump for this.
I wonder why there isn't any kernel backtrace...
This was with "dmesg -n 8". Without it has shown only the "Bad address" and "end Kernel panic" lines, which is IMHO bad by itself.
Quoted 1 line
Does gentoo uses special compiler-optimization flags?
I have none.
Quoted 7 lines
Quoted 1 line
I have put the kernel on https://opensource.sf-tec.de/c8000-kernel.tar.xz
I'd suggest you put some debugging code in your kernel, e.g. in compat_ptrace_request() in kernel/ptrace.c.
I think gdb uses PTRACE_POKEDATA to set some userspace memory of a process. It's generic code, so I wonder why it should crash on parisc.

Quoted 6 lines

Quoted 2 lines
You may look at compat_arch_ptrace() in arch/parisc/kernel/ptrace.c too, but it doesn't touch memory as far as I can see...
I will not be able to touch any of this until at least mid of next week. If you want to give it a try, check gdb 7.12.1 testsuite. It was one of the first tests that hit this.

As I said before, I can't reproduce this on debian.
gdb does not (and never has) crash(ed) the system, as can be seen in the various build logs:
https://buildd.debian.org/status/logs.php?pkg=gdb&arch=hppa
e.g.:
https://buildd.debian.org/status/fetch.php?pkg=gdb&arch=hppa&ver=7.12-6%2Bb1&stamp=1507997688&raw=0

Helge

John David Anglin

2018-01-28 19:01 UTC #6 lore raw

Quoted 3 lines · Helge Deller

As I said before, I can't reproduce this on debian.
gdb does not (and never has) crash(ed) the system, as can be seen in the various build logs:
https://buildd.debian.org/status/logs.php?pkg=gdb&arch=hppa

I just built binutils-gdb trunk successfully.

Dave

Signature · 1 line

John David Anglin  [email protected]

Rolf Eike Beer

author

2019-06-30 10:05 UTC #7 lore raw

In reply to Rolf Eike Beer

Am Donnerstag, 25. Januar 2018, 09:59:03 CEST schrieb Rolf Eike Beer:

Quoted 5 lines

The last message I had for the crash below is:
Running /var/tmp/portage/sys-devel/gdb-7.12.1/work/gdb-7.12.1/gdb/testsuite/gdb.bas e/bp-permanent.exp ...

I just have been told it's still happening with the gdb 8.3 testsuite on the Gentoo test machine:

Linux hake 4.19.52-gentoo #2 Tue Jun 18 13:51:01 EDT 2019 parisc PA8600 (PCX-W +) 9000/785/C3600 GNU/Linux

Eike

Rolf Eike Beer

author

2019-06-30 13:43 UTC #8 lore raw

Am Sonntag, 30. Juni 2019, 12:05:46 CEST schrieb Rolf Eike Beer:

Quoted 8 lines

Am Donnerstag, 25. Januar 2018, 09:59:03 CEST schrieb Rolf Eike Beer:
Quoted 6 lines
The last message I had for the crash below is:
Running /var/tmp/portage/sys-devel/gdb-7.12.1/work/gdb-7.12.1/gdb/testsuite/gdb.ba s
e/bp-permanent.exp ...

Quoted 5 lines

I just have been told it's still happening with the gdb 8.3 testsuite on the Gentoo test machine:
Linux hake 4.19.52-gentoo #2 Tue Jun 18 13:51:01 EDT 2019 parisc PA8600 (PCX-W +) 9000/785/C3600 GNU/Linux

This seems to be a minimal reproducer:

https://481768.bugs.gentoo.org/attachment.cgi?id=361728

Eike

Helge Deller

maintainer

2019-06-30 19:59 UTC #9 lore raw

Quoted 9 lines · Rolf Eike Beer

Am Sonntag, 30. Juni 2019, 12:05:46 CEST schrieb Rolf Eike Beer:
Quoted 8 lines
Am Donnerstag, 25. Januar 2018, 09:59:03 CEST schrieb Rolf Eike Beer:
Quoted 6 lines
The last message I had for the crash below is:
Running /var/tmp/portage/sys-devel/gdb-7.12.1/work/gdb-7.12.1/gdb/testsuite/gdb.ba s
e/bp-permanent.exp ...

Quoted 8 lines

Quoted 5 lines
I just have been told it's still happening with the gdb 8.3 testsuite on the Gentoo test machine:
Linux hake 4.19.52-gentoo #2 Tue Jun 18 13:51:01 EDT 2019 parisc PA8600 (PCX-W +) 9000/785/C3600 GNU/Linux
This seems to be a minimal reproducer:
https://481768.bugs.gentoo.org/attachment.cgi?id=361728

It doesn't crash for me on debian (other kernel, other userspace, older gdb). But there is something fishy anyway.
It seems the "set tp = {0,0}" doesn't work as expected.
gdb prints the set zero-values correctly, but it doesn't seem to be stored correctly to memory. "set tp.a = 10" does work though.

Helge

Helge Deller

maintainer

2019-07-02 15:59 UTC #10 lore raw

Quoted 2 lines

Quoted 2 lines
This seems to be a minimal reproducer:
https://481768.bugs.gentoo.org/attachment.cgi?id=361728

That's a REALLY nasty bug!
I see at least two issues here: A Linux kernel bug and a gdb (userspace) bug.

The kernel bug is, that a userspace process (running gdb) is able to crash the machine.
The attached patch does fix the crashes by preventing userspace to change IAOQ values to point to the gateway page. This is what made the kernel crash, which probably happened, when the kernel tried to switch_to() the process. I'm still trying to figure out if there is a better patch, e.g. in the switch_to() function or similar. So, I don't think it's the final patch.
Anyway, this patch can help to figure out what's else going wrong.

Then, the second issue is:
In gdb this crashes the machine: "set tp = { 0,0 }".
Correct would be: "set *tp = { 0,0 }", which does NOT crash the kernel and works correctly.
So, for the first test it seems that gdb tries to create a temporary variable on the stack for "tp" in the user process. For that gdb analyzes the code and thus somehow starts executing the attached process.
I think some gdb-expert might be able to fix this, esp. since gdb now prints:
: The program being debugged stopped while in a function called from GDB.
: Evaluation of the expression containing the function
: (malloc) will be abandoned.
: When the function is done executing, GDB will silently stop.
Maybe gdb tries to call the "malloc" function to allocate memory?

gdb uses ptrace() to control the attached process. So, adding a
printk("PTRACE_REQUEST_PARISC req=0x%lx addr=0x%lx data=0x%lx\n", request, addr, data);
to arch_ptrace() in arch/parisc/kernel/ptrace.c helps to debug what gdb does.

Helge

arch/parisc/kernel/ptrace.c +16 -0

index a3d2fb4e6dd2..1b1af8a6508f 100644
@@ -167,6 +175,14 @@ long arch_ptrace(struct task_struct *child, long request,
 		if ((addr & (sizeof(unsigned long)-1)) ||
 		     addr >= sizeof(struct pt_regs))
 			break;
+		/* Do not allow userspace to set IAOQ to gateway page. */
+		if (addr == PT_IAOQ0 || addr == PT_IAOQ1) {
+			if (data < GATEWAY_PAGE_SIZE) {
+				ret = 0; // return sucess to not disturb gdb
+				// printk("UGH!!!!!!!!!!!!!!!!!!!!\n");
+				break;
+			}
+		}
 		if ((addr >= PT_GR1 && addr <= PT_GR31) ||
 				addr == PT_IAOQ0 || addr == PT_IAOQ1 ||
 				(addr >= PT_FR0 && addr <= PT_FR31 + 4) ||

@@ -281,6 +297,14 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
 			addr = translate_usr_offset(addr);
 			if (addr >= sizeof(struct pt_regs))
 				break;
+			/* Do not allow userspace to set IAOQ to gateway page. */
+			if (addr == PT_IAOQ0 || addr == PT_IAOQ1) {
+				if (data < GATEWAY_PAGE_SIZE) {
+					ret = 0; // return sucess to not disturb gdb
+					// printk("UGH!!!!!!!!!!!!!!!!!!!!\n");
+					break;
+				}
+			}
 			if (addr >= PT_FR0 && addr <= PT_FR31 + 4) {
 				/* Special case, fp regs are 64 bits anyway */
 				*(__u64 *) ((char *) task_regs(child) + addr) = data;

Helge Deller

maintainer

2019-07-16 20:02 UTC #11 lore raw

Hi Rolf,

Quoted 4 lines · Helge Deller

Quoted 2 lines
Quoted 2 lines
This seems to be a minimal reproducer:
https://481768.bugs.gentoo.org/attachment.cgi?id=361728
That's a REALLY nasty bug!

I think I finally fixed the issues for 32- and 64-bit kernels.

Can you please test the three patches in my ptrace-bugfix3 git tree?
https://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux.git/log/?h=ptrace-bugfix3

Helge

Rolf Eike Beer

author

2019-07-17 17:52 UTC #12 lore raw

Quoted 6 lines · Helge

Hi Rolf,
Quoted 3 lines · Helge Deller
Quoted 2 lines
Quoted 2 lines
This seems to be a minimal reproducer:
https://481768.bugs.gentoo.org/attachment.cgi?id=361728

Quoted 7 lines

Quoted 1 line
That's a REALLY nasty bug!
I think I finally fixed the issues for 32- and 64-bit kernels.
Can you please test the three patches in my ptrace-bugfix3 git tree?
https://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux.git/log/
?h=ptrace-bugfix3

I applied them on top of 5.2.0 and tested it on my C8000, it survived the whole gdb 8.1.2 testsuite. The results are horrible, but the machine is still up and running, just a couple of these:

[ 4481.730278] INEQUIVALENT ALIASES 0x41000000 and 0x42e81000 in file kill-
detach-inferiors-cmd
[ 8944.224759] INEQUIVALENT ALIASES 0x41ed2000 and 0x4171c000 in file multi-

attach

So feel free to add my Tested-by.

Eike

John David Anglin

2019-07-17 18:21 UTC #13 lore raw

Quoted 3 lines · Rolf Eike Beer

I applied them on top of 5.2.0 and tested it on my C8000, it survived the whole gdb 8.1.2 testsuite. The results are horrible, but the machine is still up and running, just a couple of these:

Sadly, it hasn't been maintained for years.

Quoted 5 lines

[ 4481.730278] INEQUIVALENT ALIASES 0x41000000 and 0x42e81000 in file kill-
detach-inferiors-cmd
[ 8944.224759] INEQUIVALENT ALIASES 0x41ed2000 and 0x4171c000 in file multi-

attach

Theoretically, these could crash machine. It would be useful to try to debug how they occur. I would guess it's a mmap issue. It shouldn't happen with linker.

Dave

Signature · 1 line

John David Anglin  [email protected]

Helge Deller

maintainer

2019-07-17 21:00 UTC #14 lore raw

In reply to Rolf Eike Beer

Quoted 9 lines · Rolf Eike Beer

Quoted 8 lines · Helge
Hi Rolf,
Quoted 4 lines · Helge Deller
Quoted 2 lines
Quoted 2 lines
This seems to be a minimal reproducer:
https://481768.bugs.gentoo.org/attachment.cgi?id=361728
That's a REALLY nasty bug!

Quoted 8 lines

Quoted 5 lines
I think I finally fixed the issues for 32- and 64-bit kernels.
Can you please test the three patches in my ptrace-bugfix3 git tree?
https://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux.git/log/
?h=ptrace-bugfix3
I applied them on top of 5.2.0 and tested it on my C8000, it survived the whole gdb 8.1.2 testsuite.

Great. Thanks for testing.

Quoted 7 lines

The results are horrible, but the machine is still up and running, just a couple of these:
[ 4481.730278] INEQUIVALENT ALIASES 0x41000000 and 0x42e81000 in file kill-
detach-inferiors-cmd
[ 8944.224759] INEQUIVALENT ALIASES 0x41ed2000 and 0x4171c000 in file multi-
attach

Usually those are uncritical.

Quoted 1 line

So feel free to add my Tested-by.

I'll do.

Helge