powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
PATCH v4 1/1 powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.
This patch depends on patch:
powerpc/85xx: Add platform_device declaration to fsl_pci.h
---
V4 changed:
* Rebased the patch on kernel v3.8.0-rc5
The link for V3:
http://patchwork.ozlabs.org/patch/171120/
arch/powerpc/kernel/cpu_setup_fsl_booke.S | 2 +- arch/powerpc/kernel/traps.c | 3 ++ arch/powerpc/sysdev/fsl_pci.c | 41 +++++++++++++++++++++++++++++ arch/powerpc/sysdev/fsl_pci.h | 6 ++++ 4 files changed, 51 insertions(+), 1 deletions(-)
arch/powerpc/kernel/cpu_setup_fsl_booke.S +1 -1
@@ -66,7 +66,7 @@ _GLOBAL(__setup_cpu_e500v2) bl __e500_icache_setup bl __e500_dcache_setup bl __setup_e500_ivors -#ifdef CONFIG_FSL_RIO +#if defined(CONFIG_FSL_RIO) || defined(CONFIG_FSL_PCI) /* Ensure that RFXE is set */ mfspr r3,SPRN_HID1 oris r3,r3,HID1_RFXE@h
arch/powerpc/kernel/traps.c +3 -0
@@ -59,6 +59,7 @@#include <asm/fadump.h> #include <asm/switch_to.h> #include <asm/debug.h> +#include <sysdev/fsl_pci.h> #if defined(CONFIG_DEBUGGER ) || defined(CONFIG_KEXEC )int (*__debugger)(struct pt_regs *regs)__read_mostly ;
@@ -556,6 +557,8 @@ int machine_check_e500(struct pt_regs *regs)if (reason &MCSR_BUS_RBERR ) {if (fsl_rio_mcheck_exception (regs))return 1 ; +if (fsl_pci_mcheck_exception (regs)) +return 1 ; }printk ("Machine check in kernel mode.\n" );
arch/powerpc/sysdev/fsl_pci.c +41 -0
53 lines not on this page · raw message
arch/powerpc/sysdev/fsl_pci.h +6 -0
@@ -115,5 +115,11 @@ static inline int mpc85xx_pci_err_probe(struct platform_device *op) }#endif +#ifdef CONFIG_FSL_PCI +extern int fsl_pci_mcheck_exception (struct pt_regs *); +#else +static inline int fsl_pci_mcheck_exception (struct pt_regs *regs) {return 0 ; } +#endif +#endif /* __POWERPC_FSL_PCI_H */ #endif /* __KERNEL__ */
Signature · 1 line
1.7.5.1
Quoted 6 lines · Jia Hongtao
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.
Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.
And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?
Stuart
Quoted 7 lines · Stuart Yoder
Quoted 5 lines · On Mon, Mar 4, 2013 at 2:40 AM, Jia Hongtao <[email protected]> =20 wrote:
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the =20
Quoted 7 lines
handler
Quoted 1 line
which skips the instruction caused the stall.
=20
Can you explain at a high level how just skipping an instruction =20 solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.
If the instruction was a load, we probably at least want to fill the =20 destination register with 0xffffffff or similar.
Quoted 2 lines
And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?
Maybe the machine check unhangs the core?
Is there an erratum number for this?
-Scott=
Quoted 21 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 10 lines · Stuart Yoder
Quoted 5 lines · Jia Hongtao
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the
handler
Quoted 1 line
which skips the instruction caused the stall.
Quoted 7 lines
Quoted 4 lines
Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like nothing is
wrong, isn't your system possibly in a really bad state.=20
If the instruction was a load, we probably at least want to fill the destination register with0xffffffffor similar.
You discuss this with Liu Shuo about a year ago.
here is the log:
"
Quoted 2 lines · On 02/01/2012 02:18 AM, [email protected] wrote:
v3 : Skip the instruction only. Don't access the user space memory in=20
mechine check.=20
It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information (such as a piece of a crypto key that happened to be sitting in the register to be loaded into).
-Scott
"
-Hongtao.
Quoted 9 lines
=20
Quoted 2 lines
And if the core is already hung, due to the PCI link going down, isn't
it too late? How does skipping help?=20
Maybe the machine check unhangs the core?
=20
Is there an erratum number for this?
=20
-Scott
Quoted 41 lines · On 03/05/2013 04:12:30 AM, Jia Hongtao-B38951 wrote:
=20
=20Quoted 6 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 6 lines
PCIe erratum on mpc85xx
Quoted 3 lines · Stuart Yoder
Quoted 1 line · Jia Hongtao
A PCIe erratum of mpc85xx may causes a core hang when a link of =20
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
goes down. when the link goes down, Non-posted transactions =20
issued
Quoted 1 line
Quoted 1 line
Quoted 1 line
via the ATMU requiring completion result in an instruction =20
stall.
Quoted 1 line
Quoted 1 line
Quoted 1 line
At the same time a machine-check exception is generated to the =20
core
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
to allow further processing by the handler. We implements the
handler
Quoted 1 line
which skips the instruction caused the stall.
Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue like =20nothing is
Quoted 4 lines
Quoted 1 line
wrong, isn't your system possibly in a really bad state.
If the instruction was a load, we probably at least want to fill the destination register with
0xffffffffor similar.=20
You discuss this with Liu Shuo about a year ago.
here is the log:
=20
"Quoted 1 line · On 02/01/2012 02:18 AM, [email protected] wrote:
v3 : Skip the instruction only. Don't access the user space memory =20
Quoted 8 lines
in
Quoted 1 line
mechine check.
=20
It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information =20 (such
as a piece of a crypto key that happened to be sitting in the register to be loaded into).
Yes, that's (one reason) why you'd want to fill in a known value. Note =20 the "for now". :-)
-Scott=
Quoted 60 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Wednesday, March 06, 2013 2:48 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; Stuart Yoder; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 40 lines · On 03/05/2013 04:12:30 AM, Jia Hongtao-B38951 wrote:
Quoted 8 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Tuesday, March 05, 2013 7:46 AM
To: Stuart Yoder
Cc: Jia Hongtao-B38951; [email protected]; Kumar Gala
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler tofix
Quoted 6 lines
PCIe erratum on mpc85xx
Quoted 3 lines · Stuart Yoder
Quoted 1 line · Jia Hongtao
A PCIe erratum of mpc85xx may causes a core hang when a link of
PCIe
Quoted 1 line
Quoted 1 line
Quoted 1 line
goes down. when the link goes down, Non-posted transactions
issued
Quoted 1 line
Quoted 1 line
Quoted 1 line
via the ATMU requiring completion result in an instruction
stall.
Quoted 1 line
Quoted 1 line
Quoted 1 line
At the same time a machine-check exception is generated to the
core
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
to allow further processing by the handler. We implements the
handler
Quoted 1 line
which skips the instruction caused the stall.
Can you explain at a high level how just skipping an instruction solves
anything? If you just skip a load/store and continue likenothing is
Quoted 4 lines
Quoted 1 line
wrong, isn't your system possibly in a really bad state.
If the instruction was a load, we probably at least want to fill the destination register with
0xffffffffor similar.You discuss this with Liu Shuo about a year ago.
here is the log:"
On 02/01/2012 02:18 AM, [email protected] wrote:Quoted 8 lines
Quoted 1 line
v3 : Skip the instruction only. Don't access the user space memory
in
Quoted 1 line
mechine check.
It may be the least bad option for now, but be aware that there's a small chance that this will cause a leak of sensitive information (such as a piece of a crypto key that happened to be sitting in the register to be loaded into).
Quoted 5 lines
=20
Yes, that's (one reason) why you'd want to fill in a known value. Note the "for now". :-)
=20
-Scott
I think there is no overwhelming reason to fill the destination register with 0xffffffff.=20
There's a small chance that 0xffffffff is treated as regular data rather than an error sign.
Also setting this register may influence the user space under certain circumstance.
So I think just ignore the skipped instruction is an acceptable option for this fix.
-Hongtao.
Quoted 1 line
Quoted 1 line
Yes, that's (one reason) why you'd want to fill in a known value. =
Note
Quoted 5 lines
Quoted 3 lines
the "for now". :-)
-Scott
=20
I think there is no overwhelming reason to fill the destination =
register
Quoted 3 lines
with
0xffffffff.
=20
There's a small chance that0xffffffffis treated as regular data =
rather
Quoted 6 lines
than an error sign.
=20
Also setting this register may influence the user space under certain circumstance.
=20
So I think just ignore the skipped instruction is an acceptable option =
for
Quoted 1 line
this fix.
The 'random' value is just as likely to affect the reader, but only for some values - so you'll get almost impossible to repeat bugs. If a fixed value (0 or ~0) has an adverse effect, at least it will have the same every time.
Read errors are also likely to affect device drivers reading status bits, since these are very likely 'write to clear' any driver would have to be willing to process the 'dummy' value in a manner that won't loop forever (especially in an ISR).
You don't need every access to be via a function that explicitly (somehow) detects that the fault happened, but knowing that a specific value might be caused by a dead PCIe bus, and being able to find out whether that is true (to avoid looping forever) is probably useful.
This is probably similar to what a driver needs to recover from an external PCIe list being unplugged.
David
Quoted 40 lines
-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Wednesday, March 06, 2013 6:24 PM
To: Jia Hongtao-B38951; Wood Scott-B07421
Cc: [email protected]; Stuart Yoder
Subject: RE: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 8 lines
Quoted 4 lines
Yes, that's (one reason) why you'd want to fill in a known value. Note the "for now". :-)
-Scott
I think there is no overwhelming reason to fill the destination register with
0xffffffff.Quoted 8 lines
There's a small chance that
0xffffffffis treated as regular data rather than an error sign.Also setting this register may influence the user space under certain circumstance.
So I think just ignore the skipped instruction is an acceptable option for this fix.
=20
The 'random' value is just as likely to affect the reader, but only for some values - so you'll get almost impossible to repeat bugs. If a fixed value (0 or ~0) has an adverse effect, at least it will have the same every time.
=20
Read errors are also likely to affect device drivers reading status bits, since these are very likely 'write to clear' any driver would have to be willing to process the 'dummy' value in a manner that won't loop forever (especially in an ISR).
=20
You don't need every access to be via a function that explicitly (somehow) detects that the fault happened, but knowing that a specific value might be caused by a dead PCIe bus, and being able to find out
Quoted 8 lines
whether that is true (to avoid looping forever) is probably useful. =20
This is probably similar to what a driver needs to recover from an external PCIe list being unplugged.
=20
David
=20
=20
In my understanding filling the register could warn the executing process an error occurred in some cases. But no way to fix the wrong behavior cause= d
by the instruction lost. So let's say that filling the register may benefit a little.
On the other side, we should not access to the addresses of unknown process in Linux kernel. We must get the instruction before filling the register. If the instruction is not in the cache we have to access to the unknown addresses to get it. For system security I think this is strictly forbidden= .
Here is the ideas from Scott:
"
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;
Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will allow this, but the hardware does)?
What if it happened in a KVM guest? You can't access guest addresses directly.
"
Although I think filling the register have some advantages but it's should be forbidden for security reason.
-Hongtao.
Quoted 1 line
In my understanding filling the register could warn the executing =
process
Quoted 1 line
an error occurred in some cases. But no way to fix the wrong behavior =
caused
Quoted 1 line
by the instruction lost. So let's say that filling the register may =
benefit
Quoted 1 line
a little.
IIRC the only ppc instructions that should be accessing PCIe space are simple memory reads/writes, locked exchanges probably don't work. writes will be async - so we are talking about memory reads.
Quoted 1 line
On the other side, we should not access to the addresses of unknown =
process
Quoted 1 line
in Linux kernel. We must get the instruction before filling the =
register.
Quoted 1 line
If the instruction is not in the cache we have to access to the =
unknown
Quoted 1 line
addresses to get it. For system security I think this is strictly =
forbidden.
Eh???
The kernel fault handler will know whether the fault is from kernel or userspace (in which case it must be the current process), and will almost certainly already have code that looks at the faulting = instruction
sequence.
If the faulting code address is 'user', then the normal functions for reading user addresses have to be used.
If the faulting address is 'kernel' it might be in an ISR - restricting what can be done - but the instruction is definitely readable.
Quoted 1 line
Although I think filling the register have some advantages but it's =
should
Quoted 1 line
be forbidden for security reason.
I'm sure security would say exactly the opposite.
David
Quoted 18 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:
Here is the ideas from Scott:
"Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;=20
Be careful about taking a fault here. A simple TLB miss should be =20 safe
given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver =20 had
its code munmap()ed or swapped out)? What if permissions allow =20 execute
but not read (not sure if Linux will allow this, but the hardware =20 does)?
=20
What if it happened in a KVM guest? You can't access guest addresses directly.
"
That means you need to be careful about how you read the instruction, =20 not that you shouldn't do it at all.
-Scott=
Quoted 24 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 12 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:
Here is the ideas from Scott:
"Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will allow this, but the hardware does)?
Quoted 8 lines
Quoted 3 lines
What if it happened in a KVM guest? You can't access guest addresses directly.
"=20
That means you need to be careful about how you read the instruction, not that you shouldn't do it at all.
=20
-Scott
I agree.
Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?
Thanks.
-Hongtao
Quoted 33 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:
=20
=20Quoted 7 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 13 lines
PCIe erratum on mpc85xx
Quoted 10 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:
Here is the ideas from Scott:
"Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will =20
allow
Quoted 3 lines
Quoted 3 lines
this, but the hardware does)?
What if it happened in a KVM guest? You can't access guest =20
addresses
Quoted 4 lines
Quoted 2 lines
directly.
"That means you need to be careful about how you read the =20
instruction, not
Quoted 8 lines
Quoted 3 lines
that you shouldn't do it at all.
-Scott
=20
I agree.
=20
Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?
probe_kernel_address() should take care of userspace issues. As for =20 KVM, if you see MSR_GS set, bail out and don't apply the workaround. =20 Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest and =20 let its machine check handler do the skipping). On PR-mode KVM (e.g. =20 on e500v2-based chips) there is no MSR_GS and it just looks like =20 userspace code -- for now just pretend it is user mode.
-Scott=
Quoted 53 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 09, 2013 8:49 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 33 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:
Quoted 9 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler tofix
Quoted 13 lines
PCIe erratum on mpc85xx
Quoted 10 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:
Here is the ideas from Scott:
"Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;Be careful about taking a fault here. A simple TLB miss should be safe given that we shouldn't be accessing PCIe in the middle of exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? What if permissions allow execute but not read (not sure if Linux will
allow
Quoted 3 lines
Quoted 3 lines
this, but the hardware does)?
What if it happened in a KVM guest? You can't access guest
addresses
Quoted 4 lines
Quoted 2 lines
directly.
"That means you need to be careful about how you read the
instruction, not
Quoted 8 lines
Quoted 3 lines
that you shouldn't do it at all.
-Scott
I agree.
Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?
Quoted 9 lines
=20
probe_kernel_address()should take care of userspace issues. As for KVM, if you seeMSR_GSset, bail out and don't apply the workaround. Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest and let its machine check handler do the skipping). On PR-mode KVM (e.g. on e500v2-based chips) there is noMSR_GSand it just looks like userspace code -- for now just pretend it is user mode. =20
-Scott
Hi Scott,
Is that OK if I use the following code?
u32 inst;
int ret;
if (is_in_pci_mem_space(addr)) {
if (!user_mode(regs)) {
ret =3D probe_kernel_address(regs->nip, inst);
if (!ret) {
rd =3D get_rt(inst);
regs->gpr[rd] =3D 0xffffffff;
}
}
regs->nip +=3D 4;
return 1;
}Thanks.
-Hongtao.
Quoted 1 line
Is that OK if I use the following code?
...
Quoted 9 lines
if (is_in_pci_mem_space(addr)) {
if (!user_mode(regs)) {
ret =3D probe_kernel_address(regs->nip, inst);
=20
if (!ret) {
rd =3D get_rt(inst);
regs->gpr[rd] =3D 0xffffffff;
}
}Don't you need to check that the instruction is actually a memory read?
I also know that there are people mapping PCIe addresses directly into userspace for 'simple' access to things like fpga devices.
I suspect that such devices are the ones likely to generate the faulting cycles. So you probably do want to handle faults from normal userspace addresses.
David
Quoted 75 lines · On 03/12/2013 02:40:39 AM, Jia Hongtao-B38951 wrote:
=20
=20Quoted 7 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 09, 2013 8:49 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 10 lines
PCIe erratum on mpc85xx
Quoted 7 lines · On 03/08/2013 02:01:46 AM, Jia Hongtao-B38951 wrote:
Quoted 7 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Friday, March 08, 2013 12:38 AM
To: Jia Hongtao-B38951
Cc: David Laight; Wood Scott-B07421; =20Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler =20to
Quoted 10 lines
Quoted 10 lines
fix
Quoted 9 lines
PCIe erratum on mpc85xx
Quoted 6 lines · On 03/07/2013 02:06:05 AM, Jia Hongtao-B38951 wrote:
Here is the ideas from Scott:
"Review
Quoted 2 lines
(quoted diff) +2
-0+ if (is_in_pci_mem_space(addr)) { + inst =3D *(unsigned int *)regs->nip;Be careful about taking a fault here. A simple TLB miss =20
should be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
safe given that we shouldn't be accessing PCIe in the middle =20
of
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
exception code, but what if the mapping has gone away (e.g. a userspace driver had its code munmap()ed or swapped out)? =20
What if
Quoted 15 lines
Quoted 15 lines
Quoted 1 line
Quoted 1 line
permissions allow execute but not read (not sure if Linux will
allow
Quoted 3 lines
Quoted 3 lines
this, but the hardware does)?
What if it happened in a KVM guest? You can't access guest
addresses
Quoted 4 lines
Quoted 2 lines
directly.
"That means you need to be careful about how you read the
instruction, not
Quoted 3 lines
that you shouldn't do it at all.
-Scott
Quoted 8 lines
Quoted 4 lines
I agree.
Do you have a more secure way to get the instruction? Or what should be done to avoid permission break issue?
probe_kernel_address()should take care of userspace issues. As for KVM, if you seeMSR_GSset, bail out and don't apply the workaround. Let KVM/QEMU deal with it as it wishes (e.g. reflect to the guest =20and
Quoted 1 line
let its machine check handler do the skipping). On PR-mode KVM =20
(e.g.
Quoted 4 lines
on e500v2-based chips) there is no
MSR_GSand it just looks like userspace code -- for now just pretend it is user mode.-Scott
=20
Hi Scott,
=20
Is that OK if I use the following code?
=20
Quoted 6 lines
u32 inst;
int ret;
=20
if (is_in_pci_mem_space(addr)) {
if (!user_mode(regs)) {
ret =3D probe_kernel_address(regs->nip, inst);Hmm, seems there's no probe_user_address() -- for userspace we =20 basically want the same thing minus the KERNEL_DS. See =20 arch/powerpc/perf/callchain.c for an example.
You also need to skip this if (regs->msr & MSR_GS) as I mentioned above.
Quoted 4 lines
if (!ret) {
rd =3D get_rt(inst);
regs->gpr[rd] =3D 0xffffffff;
}Check whether the instruction is a load, as David pointed out. Also =20 check the size of the load, whether it was load with update =20 instruction, etc.
-Scott=
Quoted 3 lines
Hmm, seems there's no
probe_user_address()-- for userspace we basically want the same thing minus theKERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.
Isn't that just copy_from_user() ?
David
Quoted 5 lines · David Laight
Quoted 3 lines
Hmm, seems there's no
probe_user_address()-- for userspace we basically want the same thing minus theKERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.=20
Isn't that justcopy_from_user()?
Plus pagefault_disable/enable().
-Scott=
Quoted 15 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 4 lines · David Laight
Quoted 3 lines
Hmm, seems there's no
probe_user_address()-- for userspace we basically want the same thing minus theKERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.
Quoted 5 lines
Quoted 1 line
Isn't that just
copy_from_user()?=20
Plus pagefault_disable/enable().
=20
-Scott
pagefault_disable() is identical to preempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read back. probe_kernel_address() also have the same issue.
Does this mean we can't just use pagefault_disable() to prevent from gettin= g
the wrong instruction?
-Hongtao.
Quoted 19 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
=20
Quoted 5 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421; =20Quoted 2 lines
Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 9 lines
PCIe erratum on mpc85xx
Quoted 5 lines · David Laight
Quoted 3 lines
Hmm, seems there's no
probe_user_address()-- for userspace we basically want the same thing minus theKERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.Isn't that just
copy_from_user()?
Quoted 8 lines
Quoted 3 lines
Plus pagefault_disable/enable().
-Scott
=20
pagefault_disable()is identical topreempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read =20 back.probe_kernel_address()also have the same issue.
That's not the point -- the point is to let the page fault handler know =20 that it should go directly to bad_page_fault(). Do not pass =20 handle_mm_fault(). Do not collect a page from disk.
Granted, we're already in atomic context which will have that effect =20 due to being in the machine check handler, but it's better to be =20 explicit about it and not depend on how pagefault_diasble() is =20 implemented.
-Scott=
Quoted 39 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 19 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;Quoted 2 lines
Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler tofix
Quoted 9 lines
PCIe erratum on mpc85xx
Quoted 5 lines · David Laight
Quoted 3 lines
Hmm, seems there's no
probe_user_address()-- for userspace we basically want the same thing minus theKERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.Isn't that just
copy_from_user()?Quoted 8 lines
Quoted 3 lines
Plus pagefault_disable/enable().
-Scott
pagefault_disable()is identical topreempt_disable(). So I think this could not avoid other cpu to swap out the instruction we want to read back.probe_kernel_address()also have the same issue.
Quoted 11 lines
=20
That's not the point -- the point is to let the page fault handler know that it should go directly tobad_page_fault(). Do not passhandle_mm_fault(). Do not collect a page from disk.
=20
Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on howpagefault_diasble()is implemented.
=20
-Scott
Based on the comments I updated the machine check handler.
Changes from last version:
* Check MSR_GS state
* Check if the instruction is LD
* Handle the user space issue
The updated machine check handler is as following:
int fsl_pci_mcheck_exception(struct pt_regs *regs)
{
unsigned int op, rd;
u32 inst;
int ret;
phys_addr_t addr =3D 0;
/* Let KVM/QEMU deal with the exception */
if (regs->msr & MSR_GS)
return 0;#ifdef CONFIG_PHYS_64BIT
addr =3D mfspr(SPRN_MCARU);
addr <<=3D 32;
#endif
addr +=3D mfspr(SPRN_MCAR);
if (is_in_pci_mem_space(addr)) {
if (user_mode(regs)) {
pagefault_disable();
ret =3D copy_from_user(&(inst), (u32 __user *)regs-=Quoted 1 line
nip, sizeof(inst));
pagefault_enable();
} else {
ret =3D probe_kernel_address(regs->nip, inst);
}
op =3D get_op(inst);
/* Check if the instruction is LD */
if (!ret && (op =3D=3D 111010)) {
rd =3D get_rt(inst);
regs->gpr[rd] =3D 0xffffffff;
}
regs->nip +=3D 4;
return 1;
}
return 0;
}BTW, I'm still not sure how to deal with LD instruction with update.
Any comments and suggestions are welcomed.
Thanks.
-Hongtao.
Quoted 82 lines · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:
=20
=20Quoted 7 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 12 lines
PCIe erratum on mpc85xx
Quoted 9 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;Quoted 2 lines
Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler =20to
Quoted 5 lines
Quoted 5 lines
fix
Quoted 4 lines
PCIe erratum on mpc85xx
Quoted 1 line · David Laight
Quoted 1 line
Hmm, seems there's no
probe_user_address()-- for userspace =20we
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
basically want the same thing minus the
KERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.Isn't that just
copy_from_user()?Plus pagefault_disable/enable().
-Scott
pagefault_disable()is identical topreempt_disable(). So I think =20this
Quoted 1 line
Quoted 1 line
could not avoid other cpu to swap out the instruction we want to =20
read
Quoted 4 lines
Quoted 2 lines
back.
probe_kernel_address()also have the same issue.That's not the point -- the point is to let the page fault handler =20
know
Quoted 9 lines
that it should go directly to
bad_page_fault(). Do not passhandle_mm_fault(). Do not collect a page from disk.Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on how
pagefault_diasble()is implemented.-Scott
=20
=20
Based on the comments I updated the machine check handler. =20
Changes from last version:
* CheckMSR_GSstate
* Check if the instruction is LD
* Handle the user space issue
=20
The updated machine check handler is as following:
=20
intfsl_pci_mcheck_exception(struct pt_regs *regs){ unsigned int op, rd; u32 inst; int ret; phys_addr_t addr =3D 0; =20 /* Let KVM/QEMU deal with the exception */ if (regs->msr & MSR_GS) return 0; =20 #ifdef CONFIG_PHYS_64BIT addr =3D mfspr(SPRN_MCARU); addr <<=3D 32; #endif
Quoted 8 lines
addr +=3D mfspr(SPRN_MCAR);
=20
if (is_in_pci_mem_space(addr)) {
if (user_mode(regs)) {
pagefault_disable();
ret =3D copy_from_user(&(inst), (u32 __user =20
*)regs->nip, sizeof(inst));
pagefault_enable();You could use get_user() instead of copy_from_user().
Quoted 17 lines
} else {
ret =3D probe_kernel_address(regs->nip, inst);
}
=20
op =3D get_op(inst);
/* Check if the instruction is LD */
if (!ret && (op =3D=3D 111010)) {
rd =3D get_rt(inst);
regs->gpr[rd] =3D 0xffffffff;
}
=20
regs->nip +=3D 4;
return 1;
}
=20
return 0;
}Quoted 2 lines
=20
BTW, I'm still not sure how to deal with LD instruction with update.
You would need to do the update yourself. Or just say that's a case =20 you don't handle, and return 0.
Again, please check for the size of the load operation.
-Scott=
Quoted 95 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 30, 2013 12:34 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 83 lines · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:
Quoted 9 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 16, 2013 12:35 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler tofix
Quoted 12 lines
PCIe erratum on mpc85xx
Quoted 9 lines · On 03/14/2013 09:47:58 PM, Jia Hongtao-B38951 wrote:
Quoted 6 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Thursday, March 14, 2013 12:38 AM
To: David Laight
Cc: Jia Hongtao-B38951; Wood Scott-B07421;Quoted 2 lines
Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handlerto
Quoted 5 lines
Quoted 5 lines
fix
Quoted 4 lines
PCIe erratum on mpc85xx
Quoted 1 line · David Laight
Quoted 1 line
Hmm, seems there's no
probe_user_address()-- for userspacewe
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
basically want the same thing minus the
KERNEL_DS. Seearch/powerpc/perf/callchain.cfor an example.Isn't that just
copy_from_user()?Plus pagefault_disable/enable().
-Scott
pagefault_disable()is identical topreempt_disable(). So I thinkthis
Quoted 1 line
Quoted 1 line
could not avoid other cpu to swap out the instruction we want to
read
Quoted 4 lines
Quoted 2 lines
back.
probe_kernel_address()also have the same issue.That's not the point -- the point is to let the page fault handler
know
Quoted 9 lines
that it should go directly to
bad_page_fault(). Do not passhandle_mm_fault(). Do not collect a page from disk.Granted, we're already in atomic context which will have that effect due to being in the machine check handler, but it's better to be explicit about it and not depend on how
pagefault_diasble()is implemented.-Scott
Based on the comments I updated the machine check handler.
Changes from last version:
* CheckMSR_GSstate
* Check if the instruction is LD
* Handle the user space issueThe updated machine check handler is as following:
int fsl_pci_mcheck_exception(struct pt_regs *regs) { unsigned int op, rd; u32 inst; int ret; phys_addr_t addr =3D 0; /* Let KVM/QEMU deal with the exception */ if (regs->msr & MSR_GS) return 0;#ifdef CONFIG_PHYS_64BIT addr =3D mfspr(SPRN_MCARU); addr <<=3D 32; #endif addr +=3D mfspr(SPRN_MCAR);
Quoted 8 lines
Quoted 6 lines
if (is_in_pci_mem_space(addr)) { if (user_mode(regs)) { pagefault_disable(); ret =3D copy_from_user(&(inst), (u32 __user *)regs->nip, sizeof(inst)); pagefault_enable();=20
You could useget_user()instead ofcopy_from_user().
Got it.
Quoted 19 lines
=20
Quoted 18 lines
} else { ret =3D probe_kernel_address(regs->nip, inst); } op =3D get_op(inst); /* Check if the instruction is LD */ if (!ret && (op =3D=3D 111010)) { rd =3D get_rt(inst); regs->gpr[rd] =3D 0xffffffff; } regs->nip +=3D 4; return 1; } return 0; }
Quoted 8 lines
Quoted 1 line
BTW, I'm still not sure how to deal with LD instruction with update.
=20
You would need to do the update yourself. Or just say that's a case you don't handle, and return 0.
=20
Again, please check for the size of the load operation.
=20
-Scott
For informing error to the process that hold the stall instruction we need to do:
1. Verify the instruction is load.
2. Fill the rd register with ~0UL.
3. Deal with the load instruction with update.
Here is the problems:
1. So many load instructions to handle. There are dozens of load instructio= ns
and most of them with different op code. Like:
=20
lbz: 1 0 0 0 1 0
lhz: 1 0 1 0 0 0
lwz: 1 0 0 0 0 0
ld : 1 1 1 0 1 0
...
Is there any available API for verifying the load instruction?
2. For different size of load operation could we just fill the rd register = with
~0UL?
3. A load instruction with update could not just verified by op code. I'd l= ike
to leave it along. I think we could not fix but just inform the error by filling the rd with ~0UL. Could you explain why should we care about the=
update?
The main problem is how to verifying the load instruction. I wonder if ther= e is
an easy way to do it. If not the code here will be so complicated.
Thanks.
-Hongtao.
Quoted 23 lines · On 04/02/2013 04:28:10 AM, Jia Hongtao-B38951 wrote:
=20
=20Quoted 7 lines
-----Original Message-----
From: Wood Scott-B07421
Sent: Saturday, March 30, 2013 12:34 AM
To: Jia Hongtao-B38951
Cc: Wood Scott-B07421; David Laight; [email protected]; Stuart Yoder
Subject: Re: [PATCH V4] powerpc/85xx: Add machine check handler to =20fix
Quoted 4 lines
PCIe erratum on mpc85xx
Quoted 1 line · On 03/29/2013 03:03:51 AM, Jia Hongtao-B38951 wrote:
BTW, I'm still not sure how to deal with LD instruction with =20
update.
Quoted 2 lines
You would need to do the update yourself. Or just say that's a =20
case you
Quoted 5 lines
don't handle, and return 0.
Again, please check for the size of the load operation.
-Scott
Quoted 11 lines
=20
For informing error to the process that hold the stall instruction we need to do:
1. Verify the instruction is load.
2. Fill the rd register with ~0UL.
3. Deal with the load instruction with update.
=20
Here is the problems:
1. So many load instructions to handle. There are dozens of load =20 instructions
and most of them with different op code. Like:
If you don't want to handle all of them, then don't, but in case you =20 run into an instruction you don't handle, don't skip it -- just let the =20 normal machine check handler run.
Quoted 8 lines
=20
lbz: 1 0 0 0 1 0
lhz: 1 0 1 0 0 0
lwz: 1 0 0 0 0 0
ld : 1 1 1 0 1 0
...
=20
Is there any available API for verifying the load instruction?
I don't know of anything in terms of an *API*... after all, you're not =20 just "verifying" it, you're extracting information to determine how to =20 emulate the instruction.
As for code you could borrow from, there's KVM emulation and probably =20 other places.
Quoted 3 lines
2. For different size of load operation could we just fill the rd =20 register with
~0UL?
Who knows in what ways the compiler is making assumptions about the =20 upper bits being zero after an lbz, etc...
Quoted 6 lines
3. A load instruction with update could not just verified by op code. =20 I'd like
to leave it along. I think we could not fix but just inform the =20
error by
filling the rd with ~0UL. Could you explain why should we care =20
about the update?
If you're emulating the instruction, you need to handle all of that =20 instruction's effects. If you're not going to emulate the instruction, =20 don't skip it.
-Scott=
Quoted 6 lines
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.
Just skipping the instruction doesn't seem a good idea. But I suspect that re-initialising the PCI interface is also almost impossible.
Does the mpc83xx have the same errata?
We've seen machine-check faults using the CSB bridge on an 83xx doing a 'pio' access after a PEX_DMA transfer to certain target addresses stalls - software gives up waiting for the dma. The target is an fpga, nothing is mapped at those addesses - but we'd expect to get ~0u back as happens on other slave windows.
I also remember some problems with single word DMA.
David
Quoted 16 lines
-----Original Message-----
From: David Laight [mailto:[email protected]]
Sent: Tuesday, March 05, 2013 1:16 AM
To: Jia Hongtao-B38951; [email protected]; [email protected]
Cc: Wood Scott-B07421
Subject: RE: [PATCH V4] powerpc/85xx: Add machine check handler to fix PCIe erratum on mpc85xx
=20Quoted 6 lines
A PCIe erratum of mpc85xx may causes a core hang when a link of PCIe goes down. when the link goes down, Non-posted transactions issued via the ATMU requiring completion result in an instruction stall. At the same time a machine-check exception is generated to the core to allow further processing by the handler. We implements the handler which skips the instruction caused the stall.
Quoted 4 lines
=20
Just skipping the instruction doesn't seem a good idea. But I suspect that re-initialising the PCI interface is also almost impossible.
This *skipping* is the best way I thought for this errata. It's not perfect but works.
-Hongtao.
Quoted 13 lines
=20
Does the mpc83xx have the same errata?
We've seen machine-check faults using the CSB bridge on an 83xx doing a 'pio' access after aPEX_DMAtransfer to certain target addresses stalls
- software gives up waiting for the dma.
The target is an fpga, nothing is mapped at those addesses - but we'd expect to get ~0u back as happens on other slave windows. =20
I also remember some problems with single word DMA.
=20
David
=20
=20