Skip to contentSkip to conversation

powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx

PATCH In review Jia Hongtao 4 participants 25 messages Mar 4, 2013 – Apr 2, 2013 PowerPC

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.

This patch depends on patch:
powerpc/85xx: Add platform_device declaration to fsl_pci.h

---
V4 changed:
* Rebased the patch on kernel v3.8.0-rc5

The link for V3:
http://patchwork.ozlabs.org/patch/171120/

 arch/powerpc/kernel/cpu_setup_fsl_booke.S |    2 +-
 arch/powerpc/kernel/traps.c               |    3 ++
 arch/powerpc/sysdev/fsl_pci.c             |   41 +++++++++++++++++++++++++++++
 arch/powerpc/sysdev/fsl_pci.h             |    6 ++++
 4 files changed, 51 insertions(+), 1 deletions(-)
arch/powerpc/kernel/cpu_setup_fsl_booke.S +1 -1
index dcd8819..f1bde90 100644
@@ -66,7 +66,7 @@ _GLOBAL(__setup_cpu_e500v2)
 	bl	__e500_icache_setup
 	bl	__e500_dcache_setup
 	bl	__setup_e500_ivors
-#ifdef CONFIG_FSL_RIO
+#if defined(CONFIG_FSL_RIO) || defined(CONFIG_FSL_PCI)
 	/* Ensure that RFXE is set */
 	mfspr	r3,SPRN_HID1
 	oris	r3,r3,HID1_RFXE@h
arch/powerpc/kernel/traps.c +3 -0
index a008cf5..dd275a4 100644
@@ -59,6 +59,7 @@
 #include <asm/fadump.h>
 #include <asm/switch_to.h>
 #include <asm/debug.h>
+#include <sysdev/fsl_pci.h>
 
 #if defined(CONFIG_DEBUGGER) || defined(CONFIG_KEXEC)
 int (*__debugger)(struct pt_regs *regs) __read_mostly;
@@ -556,6 +557,8 @@ int machine_check_e500(struct pt_regs *regs)
 	if (reason & MCSR_BUS_RBERR) {
 		if (fsl_rio_mcheck_exception(regs))
 			return 1;
+		if (fsl_pci_mcheck_exception(regs))
+			return 1;
 	}
 
 	printk("Machine check in kernel mode.\n");
arch/powerpc/sysdev/fsl_pci.c +41 -0

53 lines not on this page · raw message

arch/powerpc/sysdev/fsl_pci.h +6 -0
index df66721..ee90a95 100644
@@ -115,5 +115,11 @@ static inline int mpc85xx_pci_err_probe(struct platform_device *op)
 }
 #endif
 
+#ifdef CONFIG_FSL_PCI
+extern int fsl_pci_mcheck_exception(struct pt_regs *);
+#else
+static inline int fsl_pci_mcheck_exception(struct pt_regs *regs) {return 0; }
+#endif
+
 #endif /* __POWERPC_FSL_PCI_H */
 #endif /* __KERNEL__ */
Signature · 1 line
1.7.5.1
#1 lore raw
Quoted 6 lines · Jia Hongtao

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.

Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.

And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?

Stuart

#2 lore raw
Quoted 7 lines · Stuart Yoder
Quoted 5 lines · On Mon, Mar 4, 2013 at 2:40 AM, Jia Hongtao <[email protected]> =20 wrote:

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the =20

Quoted 7 lines

handler

Quoted 1 line

which skips the instruction caused the stall.

=20
Can you explain at a high level how just skipping an instruction =20 solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.

If the instruction was a load, we probably at least want to fill the =20 destination register with 0xffffffff or similar.

Quoted 2 lines

And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?

Maybe the machine check unhangs the core?

Is there an erratum number for this?

-Scott=

#3 lore raw
Quoted 21 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 10 lines · Stuart Yoder
Quoted 5 lines · Jia Hongtao

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the

handler

Quoted 1 line

which skips the instruction caused the stall.

Quoted 7 lines
Quoted 4 lines

Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.

=20
If the instruction was a load, we probably at least want to fill the destination register with 0xffffffff or similar.

You discuss this with Liu Shuo about a year ago.
here is the log:

"

Quoted 2 lines · On 02/01/2012 02:18 AM, [email protected] wrote:

v3 : Skip the instruction only. Don't access the user space memory in=20
mechine check.=20

It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information (such as a piece of a crypto key that happened to be sitting in the register to be loaded into).

-Scott
"

-Hongtao.

Quoted 9 lines

=20

Quoted 2 lines

And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?

=20
Maybe the machine check unhangs the core?
=20
Is there an erratum number for this?
=20
-Scott

#4 lore raw
Quoted 41 lines · On 03/05/2013 04:12:30 AM, Jia Hongtao-B38951 wrote:

=20
=20

Quoted 6 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 6 lines

PCIe erratum on mpc85xx

Quoted 3 lines · Stuart Yoder
Quoted 1 line · Jia Hongtao

A PCIe erratum of mpc85xx may causes a core hang when a link of =20

PCIe

Quoted 1 line
Quoted 1 line
Quoted 1 line

goes down. when the link goes down, Non-posted transactions =20

issued

Quoted 1 line
Quoted 1 line
Quoted 1 line

via the ATMU requiring completion result in an instruction =20

stall.

Quoted 1 line
Quoted 1 line
Quoted 1 line

At the same time a machine-check exception is generated to the =20

core

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

to allow further processing by the handler. We implements the

handler

Quoted 1 line

which skips the instruction caused the stall.

Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like =20

nothing is

Quoted 4 lines
Quoted 1 line

wrong, isn't your system possibly in a really bad state.

If the instruction was a load, we probably at least want to fill the destination register with 0xffffffff or similar.

=20
You discuss this with Liu Shuo about a year ago.
here is the log:
=20
"

Quoted 1 line · On 02/01/2012 02:18 AM, [email protected] wrote:

v3 : Skip the instruction only. Don't access the user space memory =20

Quoted 8 lines

in

Quoted 1 line

mechine check.

=20
It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information =20 (such
as a piece of a crypto key that happened to be sitting in the register to be loaded into).

Yes, that's (one reason) why you'd want to fill in a known value. Note =20 the "for now". :-)

-Scott=

#5 lore raw
Quoted 60 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Wednesday, March 06, 2013 2:48 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; Stuart Yoder; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 40 lines · On 03/05/2013 04:12:30 AM, Jia Hongtao-B38951 wrote:
Quoted 8 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to

fix

Quoted 6 lines

PCIe erratum on mpc85xx

Quoted 3 lines · Stuart Yoder
Quoted 1 line · Jia Hongtao

A PCIe erratum of mpc85xx may causes a core hang when a link of

PCIe

Quoted 1 line
Quoted 1 line
Quoted 1 line

goes down. when the link goes down, Non-posted transactions

issued

Quoted 1 line
Quoted 1 line
Quoted 1 line

via the ATMU requiring completion result in an instruction

stall.

Quoted 1 line
Quoted 1 line
Quoted 1 line

At the same time a machine-check exception is generated to the

core

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

to allow further processing by the handler. We implements the

handler

Quoted 1 line

which skips the instruction caused the stall.

Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like

nothing is

Quoted 4 lines
Quoted 1 line

wrong, isn't your system possibly in a really bad state.

If the instruction was a load, we probably at least want to fill the destination register with 0xffffffff or similar.

You discuss this with Liu Shuo about a year ago.
here is the log:

"
On 02/01/2012 02:18 AM, [email protected] wrote:

Quoted 8 lines
Quoted 1 line

v3 : Skip the instruction only. Don't access the user space memory

in

Quoted 1 line

mechine check.

It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information (such as a piece of a crypto key that happened to be sitting in the register to be loaded into).

Quoted 5 lines

=20
Yes, that's (one reason) why you'd want to fill in a known value. Note the "for now". :-)
=20
-Scott

I think there is no overwhelming reason to fill the destination register with 0xffffffff.=20

There's a small chance that 0xffffffff is treated as regular data rather than an error sign.

Also setting this register may influence the user space under certain circumstance.

So I think just ignore the skipped instruction is an acceptable option for this fix.

-Hongtao.

#6 lore raw
Quoted 1 line
Quoted 1 line

Yes, that's (one reason) why you'd want to fill in a known value. =

Note

Quoted 5 lines
Quoted 3 lines

the "for now". :-)

-Scott

=20
I think there is no overwhelming reason to fill the destination =

register

Quoted 3 lines

with 0xffffffff.
=20
There's a small chance that 0xffffffff is treated as regular data =

rather

Quoted 6 lines

than an error sign.
=20
Also setting this register may influence the user space under certain circumstance.
=20
So I think just ignore the skipped instruction is an acceptable option =

for

Quoted 1 line

this fix.

The 'random' value is just as likely to affect the reader, but only for some values - so you'll get almost impossible to repeat bugs. If a fixed value (0 or ~0) has an adverse effect, at least it will have the same every time.

Read errors are also likely to affect device drivers reading status bits, since these are very likely 'write to clear' any driver would have to be willing to process the 'dummy' value in a manner that won't loop forever (especially in an ISR).

You don't need every access to be via a function that explicitly (somehow) detects that the fault happened, but knowing that a specific value might be caused by a dead PCIe bus, and being able to find out whether that is true (to avoid looping forever) is probably useful.

This is probably similar to what a driver needs to recover from an external PCIe list being unplugged.

David

#7 lore raw
Quoted 40 lines

-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Wednesday, March 06, 2013 6:24 PM
To: Jia Hongtao-B38951; Wood Scott-B07421
Cc: [email protected]; Stuart Yoder
Subject: RE: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 8 lines
Quoted 4 lines

Yes, that's (one reason) why you'd want to fill in a known value. Note the "for now". :-)

-Scott

I think there is no overwhelming reason to fill the destination register with 0xffffffff.

Quoted 8 lines

There's a small chance that 0xffffffff is treated as regular data rather than an error sign.

Also setting this register may influence the user space under certain circumstance.

So I think just ignore the skipped instruction is an acceptable option for this fix.

=20
The 'random' value is just as likely to affect the reader, but only for some values - so you'll get almost impossible to repeat bugs. If a fixed value (0 or ~0) has an adverse effect, at least it will have the same every time.
=20
Read errors are also likely to affect device drivers reading status bits, since these are very likely 'write to clear' any driver would have to be willing to process the 'dummy' value in a manner that won't loop forever (especially in an ISR).
=20
You don't need every access to be via a function that explicitly (somehow) detects that the fault happened, but knowing that a specific value might be caused by a dead PCIe bus, and being able to find out

Quoted 8 lines

whether that is true (to avoid looping forever) is probably useful. =20
This is probably similar to what a driver needs to recover from an external PCIe list being unplugged.
=20
David
=20
=20

In my understanding filling the register could warn the executing process an error occurred in some cases. But no way to fix the wrong behavior cause= d
by the instruction lost. So let's say that filling the register may benefit a little.

On the other side, we should not access to the addresses of unknown process in Linux kernel. We must get the instruction before filling the register. If the instruction is not in the cache we have to access to the unknown addresses to get it. For system security I think this is strictly forbidden= .

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will allow this, but the hardware does)?

What if it happened in a KVM guest? You can't access guest addresses directly.
"

Although I think filling the register have some advantages but it's should be forbidden for security reason.

-Hongtao.

#8 lore raw
Quoted 1 line

In my understanding filling the register could warn the executing =

process

Quoted 1 line

an error occurred in some cases. But no way to fix the wrong behavior =

caused

Quoted 1 line

by the instruction lost. So let's say that filling the register may =

benefit

Quoted 1 line

a little.

IIRC the only ppc instructions that should be accessing PCIe space are simple memory reads/writes, locked exchanges probably don't work. writes will be async - so we are talking about memory reads.

Quoted 1 line

On the other side, we should not access to the addresses of unknown =

process

Quoted 1 line

in Linux kernel. We must get the instruction before filling the =

register.

Quoted 1 line

If the instruction is not in the cache we have to access to the =

unknown

Quoted 1 line

addresses to get it. For system security I think this is strictly =

forbidden.

Eh???
The kernel fault handler will know whether the fault is from kernel or userspace (in which case it must be the current process), and will almost certainly already have code that looks at the faulting = instruction
sequence.
If the faulting code address is 'user', then the normal functions for reading user addresses have to be used.
If the faulting address is 'kernel' it might be in an ISR - restricting what can be done - but the instruction is definitely readable.

Quoted 1 line

Although I think filling the register have some advantages but it's =

should

Quoted 1 line

be forbidden for security reason.

I'm sure security would say exactly the opposite.

David

#9 lore raw
In reply to Jia Hongtao-B38951
Quoted 18 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

=20
Be careful about taking a fault here. A simple TLB miss should be =20 safe
given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver =20 had
its code munmap()ed or swapped out)? What if permissions allow =20 execute
but not read (not sure if Linux will allow this, but the hardware =20 does)?
=20
What if it happened in a KVM guest? You can't access guest addresses directly.
"

That means you need to be careful about how you read the instruction, =20 not that you shouldn't do it at all.

-Scott=

#10 lore raw
Quoted 24 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 12 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will allow this, but the hardware does)?

Quoted 8 lines
Quoted 3 lines

What if it happened in a KVM guest? You can't access guest addresses directly.
"

=20
That means you need to be careful about how you read the instruction, not that you shouldn't do it at all.
=20
-Scott

I agree.

Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?

Thanks.
-Hongtao

#11 lore raw
Quoted 33 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:

=20
=20

Quoted 7 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 13 lines

PCIe erratum on mpc85xx

Quoted 10 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will =20

allow

Quoted 3 lines
Quoted 3 lines

this, but the hardware does)?

What if it happened in a KVM guest? You can't access guest =20

addresses

Quoted 4 lines
Quoted 2 lines

directly.
"

That means you need to be careful about how you read the =20

instruction, not

Quoted 8 lines
Quoted 3 lines

that you shouldn't do it at all.

-Scott

=20
I agree.
=20
Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?

probe_kernel_address() should take care of userspace issues. As for =20 KVM, if you see MSR_GS set, bail out and don't apply the workaround. =20 Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest and =20 let its machine check handler do the skipping). On PR-mode KVM (e.g. =20 on e500v2-based chips) there is no MSR_GS and it just looks like =20 userspace code -- for now just pretend it is user mode.

-Scott=

#12 lore raw
Quoted 53 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 09, 2013 8:49 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 33 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:
Quoted 9 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to

fix

Quoted 13 lines

PCIe erratum on mpc85xx

Quoted 10 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will

allow

Quoted 3 lines
Quoted 3 lines

this, but the hardware does)?

What if it happened in a KVM guest? You can't access guest

addresses

Quoted 4 lines
Quoted 2 lines

directly.
"

That means you need to be careful about how you read the

instruction, not

Quoted 8 lines
Quoted 3 lines

that you shouldn't do it at all.

-Scott

I agree.

Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?

Quoted 9 lines

=20
probe_kernel_address() should take care of userspace issues. As for KVM, if you see MSR_GS set, bail out and don't apply the workaround. Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest and let its machine check handler do the skipping). On PR-mode KVM (e.g. on e500v2-based chips) there is no MSR_GS and it just looks like userspace code -- for now just pretend it is user mode. =20
-Scott

Hi Scott,

Is that OK if I use the following code?

	u32 inst;
	int ret;

	if (is_in_pci_mem_space(addr)) {
		if (!user_mode(regs)) {
			ret =3D probe_kernel_address(regs->nip, inst);

			if (!ret) {
				rd =3D get_rt(inst);
				regs->gpr[rd] =3D 0xffffffff;
			}
		}

		regs->nip +=3D 4;
		return 1;
	}

Thanks.
-Hongtao.

#13 lore raw
Quoted 1 line

Is that OK if I use the following code?

...

Quoted 9 lines
	if (is_in_pci_mem_space(addr)) {
		if (!user_mode(regs)) {
			ret =3D probe_kernel_address(regs->nip, inst);
=20
			if (!ret) {
				rd =3D get_rt(inst);
				regs->gpr[rd] =3D 0xffffffff;
			}
		}

Don't you need to check that the instruction is actually a memory read?

I also know that there are people mapping PCIe addresses directly into userspace for 'simple' access to things like fpga devices.

I suspect that such devices are the ones likely to generate the faulting cycles. So you probably do want to handle faults from normal userspace addresses.

David

#14 lore raw
In reply to Jia Hongtao-B38951
Quoted 75 lines · On 03/12/2013 02:40:39 AM, Jia Hongtao-B38951 wrote:

=20
=20

Quoted 7 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 09, 2013 8:49 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 10 lines

PCIe erratum on mpc85xx

Quoted 7 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:
Quoted 7 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; =20

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler =20

to

Quoted 10 lines
Quoted 10 lines

fix

Quoted 9 lines

PCIe erratum on mpc85xx

Quoted 6 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:

Here is the ideas from Scott:
"

Review

Quoted 2 lines
(quoted diff) +2 -0
+	if (is_in_pci_mem_space(addr)) {
+		inst =3D *(unsigned int *)regs->nip;

Be careful about taking a fault here. A simple TLB miss =20

should be

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

safe given that we shouldn't be accessing PCIe in the middle =20

of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? =20

What if

Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line

permissions allow execute but not read (not sure if Linux will

allow

Quoted 3 lines
Quoted 3 lines

this, but the hardware does)?

What if it happened in a KVM guest? You can't access guest

addresses

Quoted 4 lines
Quoted 2 lines

directly.
"

That means you need to be careful about how you read the

instruction, not

Quoted 3 lines

that you shouldn't do it at all.

-Scott

Quoted 8 lines
Quoted 4 lines

I agree.

Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?

probe_kernel_address() should take care of userspace issues. As for KVM, if you see MSR_GS set, bail out and don't apply the workaround. Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest =20

and

Quoted 1 line

let its machine check handler do the skipping). On PR-mode KVM =20

(e.g.

Quoted 4 lines

on e500v2-based chips) there is no MSR_GS and it just looks like userspace code -- for now just pretend it is user mode.

-Scott

=20
Hi Scott,
=20
Is that OK if I use the following code?
=20

Quoted 6 lines
	u32 inst;
	int ret;
=20
	if (is_in_pci_mem_space(addr)) {
		if (!user_mode(regs)) {
			ret =3D probe_kernel_address(regs->nip, inst);

Hmm, seems there's no probe_user_address() -- for userspace we =20 basically want the same thing minus the KERNEL_DS. See =20 arch/powerpc/perf/callchain.c for an example.

You also need to skip this if (regs->msr & MSR_GS) as I mentioned above.

Quoted 4 lines
			if (!ret) {
				rd =3D get_rt(inst);
				regs->gpr[rd] =3D 0xffffffff;
			}

Check whether the instruction is a load, as David pointed out. Also =20 check the size of the load, whether it was load with update =20 instruction, etc.

-Scott=

#15 lore raw
Quoted 3 lines

Hmm, seems there's no probe_user_address() -- for userspace we basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Isn't that just copy_from_user() ?

David

#16 lore raw
Quoted 5 lines · David Laight
Quoted 3 lines

Hmm, seems there's no probe_user_address() -- for userspace we basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

=20
Isn't that just copy_from_user() ?

Plus pagefault_disable/enable().

-Scott=

#17 lore raw
Quoted 15 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 4 lines · David Laight
Quoted 3 lines

Hmm, seems there's no probe_user_address() -- for userspace we basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Quoted 5 lines
Quoted 1 line

Isn't that just copy_from_user() ?

=20
Plus pagefault_disable/enable().
=20
-Scott

pagefault_disable() is identical to preempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read back. probe_kernel_address() also have the same issue.

Does this mean we can't just use pagefault_disable() to prevent from gettin= g
the wrong instruction?

-Hongtao.

#18 lore raw
Quoted 19 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:

=20

Quoted 5 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421; =20

[email protected];

Quoted 2 lines

Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 9 lines

PCIe erratum on mpc85xx

Quoted 5 lines · David Laight
Quoted 3 lines

Hmm, seems there's no probe_user_address() -- for userspace we basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Isn't that just copy_from_user() ?

Quoted 8 lines
Quoted 3 lines

Plus pagefault_disable/enable().

-Scott

=20
pagefault_disable() is identical to preempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read =20 back.
probe_kernel_address() also have the same issue.

That's not the point -- the point is to let the page fault handler know =20 that it should go directly to bad_page_fault(). Do not pass =20 handle_mm_fault(). Do not collect a page from disk.

Granted, we're already in atomic context which will have that effect =20 due to being in the machine check handler, but it's better to be =20 explicit about it and not depend on how pagefault_diasble() is =20 implemented.

-Scott=

#19 lore raw
Quoted 39 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 19 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;

[email protected];

Quoted 2 lines

Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to

fix

Quoted 9 lines

PCIe erratum on mpc85xx

Quoted 5 lines · David Laight
Quoted 3 lines

Hmm, seems there's no probe_user_address() -- for userspace we basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Isn't that just copy_from_user() ?

Quoted 8 lines
Quoted 3 lines

Plus pagefault_disable/enable().

-Scott

pagefault_disable() is identical to preempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read back.
probe_kernel_address() also have the same issue.

Quoted 11 lines

=20
That's not the point -- the point is to let the page fault handler know that it should go directly to bad_page_fault(). Do not pass handle_mm_fault(). Do not collect a page from disk.
=20
Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on how pagefault_diasble() is implemented.
=20
-Scott

Based on the comments I updated the machine check handler.

Changes from last version:
* Check MSR_GS state
* Check if the instruction is LD
* Handle the user space issue

The updated machine check handler is as following:

int fsl_pci_mcheck_exception(struct pt_regs *regs)

{
        unsigned int op, rd;
        u32 inst;
        int ret;
        phys_addr_t addr =3D 0;

        /* Let KVM/QEMU deal with the exception */
        if (regs->msr & MSR_GS)
                return 0;
#ifdef CONFIG_PHYS_64BIT
        addr =3D mfspr(SPRN_MCARU);
        addr <<=3D 32;
#endif
        addr +=3D mfspr(SPRN_MCAR);

        if (is_in_pci_mem_space(addr)) {
                if (user_mode(regs)) {
                        pagefault_disable();
                        ret =3D copy_from_user(&(inst), (u32 __user *)regs-=
Quoted 1 line

nip, sizeof(inst));

                        pagefault_enable();
                } else {
                        ret =3D probe_kernel_address(regs->nip, inst);
                }

                op =3D get_op(inst);
                /* Check if the instruction is LD */
                if (!ret && (op =3D=3D 111010)) {
                        rd =3D get_rt(inst);
                        regs->gpr[rd] =3D 0xffffffff;
                }

                regs->nip +=3D 4;
                return 1;
        }

        return 0;
}

BTW, I'm still not sure how to deal with LD instruction with update.

Any comments and suggestions are welcomed.

Thanks.
-Hongtao.

#20 lore raw
Quoted 82 lines · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:

=20
=20

Quoted 7 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 12 lines

PCIe erratum on mpc85xx

Quoted 9 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;

[email protected];

Quoted 2 lines

Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler =20

to

Quoted 5 lines
Quoted 5 lines

fix

Quoted 4 lines

PCIe erratum on mpc85xx

Quoted 1 line · David Laight
Quoted 1 line

Hmm, seems there's no probe_user_address() -- for userspace =20

we

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Isn't that just copy_from_user() ?

Plus pagefault_disable/enable().

-Scott

pagefault_disable() is identical to preempt_disable(). So I think =20

this

Quoted 1 line
Quoted 1 line

could not avoid other cpu to swap out the instruction we want to =20

read

Quoted 4 lines
Quoted 2 lines

back.
probe_kernel_address() also have the same issue.

That's not the point -- the point is to let the page fault handler =20

know

Quoted 9 lines

that it should go directly to bad_page_fault(). Do not pass handle_mm_fault(). Do not collect a page from disk.

Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on how pagefault_diasble() is implemented.

-Scott

=20
=20
Based on the comments I updated the machine check handler. =20
Changes from last version:
* Check MSR_GS state
* Check if the instruction is LD
* Handle the user space issue
=20
The updated machine check handler is as following:
=20
int fsl_pci_mcheck_exception(struct pt_regs *regs)

{
        unsigned int op, rd;
        u32 inst;
        int ret;
        phys_addr_t addr =3D 0;
=20
        /* Let KVM/QEMU deal with the exception */
        if (regs->msr & MSR_GS)
                return 0;
=20
#ifdef CONFIG_PHYS_64BIT
        addr =3D mfspr(SPRN_MCARU);
        addr <<=3D 32;
#endif
Quoted 8 lines
        addr +=3D mfspr(SPRN_MCAR);
=20
        if (is_in_pci_mem_space(addr)) {
                if (user_mode(regs)) {
                        pagefault_disable();
                        ret =3D copy_from_user(&(inst), (u32 __user =20
*)regs->nip, sizeof(inst));
                        pagefault_enable();

You could use get_user() instead of copy_from_user().

Quoted 17 lines
                } else {
                        ret =3D probe_kernel_address(regs->nip, inst);
                }
=20
                op =3D get_op(inst);
                /* Check if the instruction is LD */
                if (!ret && (op =3D=3D 111010)) {
                        rd =3D get_rt(inst);
                        regs->gpr[rd] =3D 0xffffffff;
                }
=20
                regs->nip +=3D 4;
                return 1;
        }
=20
        return 0;
}
Quoted 2 lines

=20
BTW, I'm still not sure how to deal with LD instruction with update.

You would need to do the update yourself. Or just say that's a case =20 you don't handle, and return 0.

Again, please check for the size of the load operation.

-Scott=

#21 lore raw
Quoted 95 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 30, 2013 12:34 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 83 lines · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:
Quoted 9 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to

fix

Quoted 12 lines

PCIe erratum on mpc85xx

Quoted 9 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;

[email protected];

Quoted 2 lines

Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler

to

Quoted 5 lines
Quoted 5 lines

fix

Quoted 4 lines

PCIe erratum on mpc85xx

Quoted 1 line · David Laight
Quoted 1 line

Hmm, seems there's no probe_user_address() -- for userspace

we

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

basically want the same thing minus the KERNEL_DS. See arch/powerpc/perf/callchain.c for an example.

Isn't that just copy_from_user() ?

Plus pagefault_disable/enable().

-Scott

pagefault_disable() is identical to preempt_disable(). So I think

this

Quoted 1 line
Quoted 1 line

could not avoid other cpu to swap out the instruction we want to

read

Quoted 4 lines
Quoted 2 lines

back.
probe_kernel_address() also have the same issue.

That's not the point -- the point is to let the page fault handler

know

Quoted 9 lines

that it should go directly to bad_page_fault(). Do not pass handle_mm_fault(). Do not collect a page from disk.

Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on how pagefault_diasble() is implemented.

-Scott

Based on the comments I updated the machine check handler.

Changes from last version:
* Check MSR_GS state
* Check if the instruction is LD
* Handle the user space issue

The updated machine check handler is as following:

int fsl_pci_mcheck_exception(struct pt_regs *regs) {
        unsigned int op, rd;
        u32 inst;
        int ret;
        phys_addr_t addr =3D 0;

        /* Let KVM/QEMU deal with the exception */
        if (regs->msr & MSR_GS)
                return 0;
#ifdef CONFIG_PHYS_64BIT
        addr =3D mfspr(SPRN_MCARU);
        addr <<=3D 32;
#endif
        addr +=3D mfspr(SPRN_MCAR);
Quoted 8 lines
Quoted 6 lines
        if (is_in_pci_mem_space(addr)) {
                if (user_mode(regs)) {
                        pagefault_disable();
                        ret =3D copy_from_user(&(inst), (u32 __user
*)regs->nip, sizeof(inst));
                        pagefault_enable();

=20
You could use get_user() instead of copy_from_user().

Got it.

Quoted 19 lines

=20

Quoted 18 lines
                } else {
                        ret =3D probe_kernel_address(regs->nip, inst);
                }

                op =3D get_op(inst);
                /* Check if the instruction is LD */
                if (!ret && (op =3D=3D 111010)) {
                        rd =3D get_rt(inst);
                        regs->gpr[rd] =3D 0xffffffff;
                }

                regs->nip +=3D 4;
                return 1;
        }

        return 0;
}
Quoted 8 lines
Quoted 1 line

BTW, I'm still not sure how to deal with LD instruction with update.

=20
You would need to do the update yourself. Or just say that's a case you don't handle, and return 0.
=20
Again, please check for the size of the load operation.
=20
-Scott

For informing error to the process that hold the stall instruction we need to do:
1. Verify the instruction is load.
2. Fill the rd register with ~0UL.
3. Deal with the load instruction with update.

Here is the problems:
1. So many load instructions to handle. There are dozens of load instructio= ns
and most of them with different op code. Like:
=20
lbz: 1 0 0 0 1 0
lhz: 1 0 1 0 0 0
lwz: 1 0 0 0 0 0
ld : 1 1 1 0 1 0
...

Is there any available API for verifying the load instruction?

2. For different size of load operation could we just fill the rd register = with
~0UL?

3. A load instruction with update could not just verified by op code. I'd l= ike
to leave it along. I think we could not fix but just inform the error by filling the rd with ~0UL. Could you explain why should we care about the=
update?

The main problem is how to verifying the load instruction. I wonder if ther= e is
an easy way to do it. If not the code here will be so complicated.

Thanks.
-Hongtao.

#22 lore raw
Quoted 23 lines · On 04/02/2013 04:28:10 AM, Jia Hongtao-B38951 wrote:

=20
=20

Quoted 7 lines

-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 30, 2013 12:34 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20

fix

Quoted 4 lines

PCIe erratum on mpc85xx

Quoted 1 line · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:

BTW, I'm still not sure how to deal with LD instruction with =20

update.

Quoted 2 lines

You would need to do the update yourself. Or just say that's a =20

case you

Quoted 5 lines

don't handle, and return 0.

Again, please check for the size of the load operation.

-Scott

Quoted 11 lines

=20
For informing error to the process that hold the stall instruction we need to do:
1. Verify the instruction is load.
2. Fill the rd register with ~0UL.
3. Deal with the load instruction with update.
=20
Here is the problems:
1. So many load instructions to handle. There are dozens of load =20 instructions
and most of them with different op code. Like:

If you don't want to handle all of them, then don't, but in case you =20 run into an instruction you don't handle, don't skip it -- just let the =20 normal machine check handler run.

Quoted 8 lines

=20
lbz: 1 0 0 0 1 0
lhz: 1 0 1 0 0 0
lwz: 1 0 0 0 0 0
ld : 1 1 1 0 1 0
...
=20
Is there any available API for verifying the load instruction?

I don't know of anything in terms of an *API*... after all, you're not =20 just "verifying" it, you're extracting information to determine how to =20 emulate the instruction.

As for code you could borrow from, there's KVM emulation and probably =20 other places.

Quoted 3 lines

2. For different size of load operation could we just fill the rd =20 register with
~0UL?

Who knows in what ways the compiler is making assumptions about the =20 upper bits being zero after an lbz, etc...

Quoted 6 lines

3. A load instruction with update could not just verified by op code. =20 I'd like
to leave it along. I think we could not fix but just inform the =20
error by
filling the rd with ~0UL. Could you explain why should we care =20
about the update?

If you're emulating the instruction, you need to handle all of that =20 instruction's effects. If you're not going to emulate the instruction, =20 don't skip it.

-Scott=

#23 lore raw
In reply to Jia Hongtao
Quoted 6 lines

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.

Just skipping the instruction doesn't seem a good idea. But I suspect that re-initialising the PCI interface is also almost impossible.

Does the mpc83xx have the same errata?
We've seen machine-check faults using the CSB bridge on an 83xx doing a 'pio' access after a PEX_DMA transfer to certain target addresses stalls - software gives up waiting for the dma. The target is an fpga, nothing is mapped at those addesses - but we'd expect to get ~0u back as happens on other slave windows.

I also remember some problems with single word DMA.

David

#24 lore raw
Quoted 16 lines

-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Tuesday, March 05, 2013 1:16 AM
To: Jia Hongtao-B38951; [email protected]; [email protected]
Cc: Wood Scott-B07421
Subject: RE: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20

Quoted 6 lines

A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.

Quoted 4 lines

=20
Just skipping the instruction doesn't seem a good idea. But I suspect that re-initialising the PCI interface is also almost impossible.

This *skipping* is the best way I thought for this errata. It's not perfect but works.

-Hongtao.

Quoted 13 lines

=20
Does the mpc83xx have the same errata?
We've seen machine-check faults using the CSB bridge on an 83xx doing a 'pio' access after a PEX_DMA transfer to certain target addresses stalls
- software gives up waiting for the dma.
The target is an fpga, nothing is mapped at those addesses - but we'd expect to get ~0u back as happens on other slave windows. =20
I also remember some problems with single word DMA.
=20
David
=20
=20