sparc64: sun4v TLB error power off events
Subsystems (3)
PATCH 1/1 sparc64: sun4v TLB error power off events
From: bob picco <[email protected]>
We've witnessed a few TLB events causing the machine to power off because of prom_halt. In one case it was some nfs related area during rmmod. Another was an mmapper of /dev/mem. A more recent one is an ITLB issue with a bad pagesize which could be a hardware bug. Bugs happen but we should attempt to not power off the machine and/or hang it when possible.
This is a DTLB error from an mmapper of /dev/mem:
[root@sparcie ~]# SUN4V-DTLB: Error at TPC[fffff80100903e6c], tl 1
SUN4V-DTLB: TPC<0xfffff80100903e6c>
SUN4V-DTLB: O7[fffff801081979d0]
SUN4V-DTLB: O7<0xfffff801081979d0>
SUN4V-DTLB: vaddr[fffff80100000000] ctx[1250] pte[98000000000f0610] error[2] .
This is recent mainline for ITLB:
[ 3708.179864] SUN4V-ITLB: TPC<0xfffffc010071cefc> [ 3708.188866] SUN4V-ITLB: O7[fffffc010071cee8] [ 3708.197377] SUN4V-ITLB: O7<0xfffffc010071cee8> [ 3708.206539] SUN4V-ITLB: vaddr[e0003] ctx[1a3c] pte[2900000dcc800eeb] error[4]
.
We've treated DTLB/ITLB error events identically within the patch. Should TL be <= 1 then proceed to die_if_kernel. Fully expect though that for a privileged access the machine must be reset when panic_on_oops is armed. Should panic_on_oops not be armed, then you remain up but the quality and duration will be subject to what the error condition caused. An unprivileged task is killed off with a SIGSEGV.
Power off of large sparc64 machines is painful. Plus die_if_kernel provides more context. A reset sequence isn't a brief period on large sparc64 but better than power-off/power-on sequence.
For TL > 1 the machine does abruptly enter power off like it has.
--- arch/sparc/kernel/traps_64.c | 16 ++++++++++++++-- 1 files changed, 14 insertions(+), 2 deletions(-)
arch/sparc/kernel/traps_64.c +14 -2
@@ -2104,6 +2104,18 @@ void sun4v_nonresum_overflow(struct pt_regs *regs)atomic_inc (&sun4v_nonresum_oflow_cnt); } +static void sun4v_tlb_error (struct pt_regs *regs,int tl,char *message) +{ +/* Should we be above TL=1 then we just prom_halt. Should +* pstate.priv have been true at trap time and panic_on_oops +* disabled then we proceed but YMMV. +*/ +if (tl >1 ) +prom_halt (); +else +die_if_kernel (message, regs); +} +unsigned long sun4v_err_itlb_vaddr;unsigned long sun4v_err_itlb_ctx;unsigned long sun4v_err_itlb_pte;
@@ -2125,7 +2137,7 @@ void sun4v_itlb_error_report(struct pt_regs *regs, int tl) sun4v_err_itlb_vaddr, sun4v_err_itlb_ctx, sun4v_err_itlb_pte, sun4v_err_itlb_error); -prom_halt (); +sun4v_tlb_error (regs, tl,"ITLB HV ERROR" ); }unsigned long sun4v_err_dtlb_vaddr;
@@ -2149,7 +2161,7 @@ void sun4v_dtlb_error_report(struct pt_regs *regs, int tl) sun4v_err_dtlb_vaddr, sun4v_err_dtlb_ctx, sun4v_err_dtlb_pte, sun4v_err_dtlb_error); -prom_halt (); +sun4v_tlb_error (regs, tl,"DTLB HV ERROR" ); }void hypervisor_tlbop_error (unsigned long err,unsigned long op)
Signature · 1 line
1.7.1
David Miller
maintainerFrom: Bob Picco <[email protected]>
Date: Sun, 7 Sep 2014 11:47:38 -0400
Quoted 5 lines
We've witnessed a few TLB events causing the machine to power off because of
prom_halt. In one case it was some nfs related area during rmmod. Another was an mmapper of/dev/mem. A more recent one is an ITLB issue with a bad pagesize which could be a hardware bug. Bugs happen but we should attempt to not power off the machine and/or hang it when possible.
prom_halt() should not power off the machine, but rather drop us to the OF command line "ok" prompt.
Why doesn't it do that?
We properly do a >tl1 vs. tl1 etrap call, so we should be at trap level zero when we call into the prom to "exit".
David Miller wrote: [Tue Sep 09 2014, 03:22:37PM EDT]
Quoted 11 lines
From: Bob Picco <[email protected]>
Date: Sun, 7 Sep 2014 11:47:38 -0400Quoted 5 lines
We've witnessed a few TLB events causing the machine to power off because of
prom_halt. In one case it was some nfs related area during rmmod. Another was an mmapper of/dev/mem. A more recent one is an ITLB issue with a bad pagesize which could be a hardware bug. Bugs happen but we should attempt to not power off the machine and/or hang it when possible.
prom_halt()should not power off the machine, but rather drop us to the OF command line "ok" prompt.
I didn't know this. This would be ideal.
For my nearly P0 T4-2 it always powers off.
Quoted 2 lines
Why doesn't it do that?
Don't know.
Quoted 3 lines
We properly do a >tl1 vs. tl1 etrap call, so we should be at trap level zero when we call into the prom to "exit".
I agree.
I just ran a quick experiment on my T5-2 which is supported hardware. The kernel is 3.17-rc3 without any modification from me - well ixgbe. As root mmap of /dev/mem at address 0UL. It powered off:
4 GNU/Linux
[root@t5-2 ~]# [31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1
[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48> [31732.380652] SUN4V-DTLB: O7[100970] [31732.387418] SUN4V-DTLB: O7<0x100970> [31732.394548] SUN4V-DTLB: vaddr[fffffc0100028000] ctx[1634] pte[9a00000000000610] error[2]
Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1
Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48>
Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.380652] SUN4V-DTLB: O7[102014-09-09 20:35:34 SP> NOTICE: Host is off
. Some firmware widget we are unaware of?
Should you like the code it is below.
thanx,
bob
<<CLIP HERE>>
#define _GNU_SOURCE
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#define PGSIZE (8192)
void main(int argc, char **argv)
{
unsigned long addr;
char buf[PGSIZE];
void *mmap_addr;
ssize_t size;
off_t offset;
int rc, fd;
if (argc != 2)
fprintf(stderr, "%s: 0xaddress\n", argv[0]), exit(1);
rc = sscanf(argv[1], "%lx", &addr);
if (rc != 1)
fprintf(stderr, "%s: address-format-invalid\n", argv[0]),
exit(1);
fd = open("/dev/mem", O_RDONLY);
if (fd < 0)
fprintf(stderr, "%s: failed to open /dev/mem\n", argv[0]),
exit(1);
offset = addr;
size = PGSIZE;
mmap_addr = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, offset);
if (mmap_addr = MAP_FAILED)
fprintf(stderr, "%s: failed mmap offset=0x%lx\n", argv[0],
offset), exit(1);
memcpy(buf, mmap_addr, sizeof (buf));
(void) munmap(mmap_addr, size);
close(fd);
}David Miller
maintainerFrom: Bob Picco <[email protected]>
Date: Tue, 9 Sep 2014 17:12:27 -0400
Quoted 3 lines
I just ran a quick experiment on my T5-2 which is supported hardware. The kernel is 3.17-rc3 without any modification from me - well ixgbe. As root mmap of
/dev/memat address 0UL. It powered off:
Just out of curiosity what ixgbe patches do you need that aren't upstream already?
Quoted 10 lines
4 GNU/Linux
[root@t5-2 ~]# [31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48> [31732.380652] SUN4V-DTLB: O7[100970] [31732.387418] SUN4V-DTLB: O7<0x100970> [31732.394548] SUN4V-DTLB: vaddr[fffffc0100028000] ctx[1634] pte[9a00000000000610] error[2]Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1
Quoted 6 lines
Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48>Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.380652] SUN4V-DTLB: O7[102014-09-09 20:35:34 SP> NOTICE: Host is off
. Some firmware widget we are unaware of?
Hmmm...
Oh I see, if LDOMs are enabled we do ldom_power_off() instead of doing an OF "exit".
That explains everything.
I seem to remember that for some reason after early boot it got to the point with LDOMs that you had to stop talking to the OF, and that's why for all of these interfaces that could be invoked after early boot, we revector to a ldom_*() routine if ldom_domaining_enabled is true.
So I don't think there is anything we can do about this, so perhaps we should just unconditionally avoid using prom_halt() here, and just do a die_if_kernel() regardless of the trap level.
Also, for the >tl1 case, it would be beneficial to print out the stack of trap state registers that etraptl1 saves on the stack right after pt_regs. The format is traps_64.c's "struct tl1_traplog", and there is a dump_tl1_traplog() helper there already.
Thanks for looking into this Bob.
David Miller wrote: [Tue Sep 09 2014, 05:52:46PM EDT]
Quoted 9 lines
From: Bob Picco <[email protected]>
Date: Tue, 9 Sep 2014 17:12:27 -0400Quoted 3 lines
I just ran a quick experiment on my T5-2 which is supported hardware. The kernel is 3.17-rc3 without any modification from me - well ixgbe. As root mmap of
/dev/memat address 0UL. It powered off:Just out of curiosity what ixgbe patches do you need that aren't upstream already?
It is really Martin's (mkp) from last year. I ported it over to mainline. Basically the mac is acquired with:
addr = of_get_property(dp, "local-mac-address", &len);
. I'll append at the end of this email what my local T5-2 is using.
Quoted 23 lines
Quoted 11 lines
4 GNU/Linux
[root@t5-2 ~]# [31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48> [31732.380652] SUN4V-DTLB: O7[100970] [31732.387418] SUN4V-DTLB: O7<0x100970> [31732.394548] SUN4V-DTLB: vaddr[fffffc0100028000] ctx[1634] pte[9a00000000000610] error[2]Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.360547] SUN4V-DTLB: Error at TPC[fffffc01001cac48], tl 1Quoted 6 lines
Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.371659] SUN4V-DTLB: TPC<0xfffffc01001cac48>Message from syslogd@t5-2 at Sep 9 16:53:25 ...
kernel:[31732.380652] SUN4V-DTLB: O7[102014-09-09 20:35:34 SP> NOTICE: Host is off
. Some firmware widget we are unaware of?Hmmm...
Oh I see, if LDOMs are enabled we do
ldom_power_off()instead of doing an OF "exit".
Quoted 7 lines
That explains everything.
I seem to remember that for some reason after early boot it got to the point with LDOMs that you had to stop talking to the OF, and that's why for all of these interfaces that could be invoked after early boot, we revector to a ldom_*() routine if
ldom_domaining_enabledis true.
I seem to remember encountering similar for kexec and start/stop strand but that was long ago too :)
Quoted 4 lines
So I don't think there is anything we can do about this, so perhaps we should just unconditionally avoid using
prom_halt()here, and just do adie_if_kernel()regardless of the trap level.
okay.
Quoted 5 lines
Also, for the >tl1 case, it would be beneficial to print out the stack of trap state registers that etraptl1 saves on the stack right after
pt_regs. The format is traps_64.c's "structtl1_traplog", and there is adump_tl1_traplog()helper there already.
okay.
Quoted 2 lines
Thanks for looking into this Bob.
You're welcome and thanx,
bob
<<ixgbe>>
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 32 +++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +32 -0
@@ -49,6 +49,10 @@#include <linux/if_bridge.h> #include <linux/prefetch.h> #include <scsi/fc/fc_fcoe.h> +#ifdef CONFIG_SPARC +#include <asm/idprom.h> +#include <asm/prom.h> +#endif #include "ixgbe.h" #include "ixgbe_common.h"
@@ -8063,6 +8067,33 @@ int ixgbe_wol_supported(struct ixgbe_adapter *adapter, u16 device_id,return is_wol_supported; } +#ifdef CONFIG_SPARC +/** +* ixgbe_mac_addr_sparc - Look up MAC address on SPARC +* @adapter: Pointer to adapter struct +*/ +static void ixgbe_mac_addr_sparc (struct ixgbe_adapter *adapter) +{ +struct device_node *dp =pci_device_to_OF_node (adapter->pdev); +struct ixgbe_hw *hw = &adapter->hw; +const unsigned char *addr; +int len; + + addr =of_get_property (dp,"local-mac-address" , &len); +if (addr && len =6 ) { +e_dev_info ("Using OpenPROM MAC address\n" ); +memcpy (hw->mac.perm_addr, addr,6 ); + } + +if (!is_valid_ether_addr (hw->mac.perm_addr)) { +e_dev_info ("Using IDPROM MAC address\n" ); +memcpy (hw->mac.perm_addr, idprom->id_ethaddr,6 ); + } +} +#else +static void ixgbe_mac_addr_sparc (struct ixgbe_adapter *adapter) {} +#endif +/** * ixgbe_probe - Device Initialization Routine * @pdev: PCI device information struct
@@ -8330,6 +8361,7 @@ static int ixgbe_probe(struct pci_dev *pdev, const struct pci_device_id *ent) goto err_sw_init; } +ixgbe_mac_addr_sparc (adapter);memcpy (netdev->dev_addr, hw->mac.perm_addr, netdev->addr_len);if (!is_valid_ether_addr (netdev->dev_addr)) {
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
David Miller
maintainerFrom: Bob Picco <[email protected]>
Date: Wed, 10 Sep 2014 10:18:22 -0400
Quoted 8 lines
David Miller wrote: [Tue Sep 09 2014, 05:52:46PM EDT]
Quoted 7 lines
From: Bob Picco <[email protected]>
Date: Tue, 9 Sep 2014 17:12:27 -0400Quoted 3 lines
I just ran a quick experiment on my T5-2 which is supported hardware. The kernel is 3.17-rc3 without any modification from me - well ixgbe. As root mmap of
/dev/memat address 0UL. It powered off:
Quoted 6 lines
Quoted 2 lines
Just out of curiosity what ixgbe patches do you need that aren't upstream already?
It is really Martin's (mkp) from last year. I ported it over to mainline. Basically the mac is acquired with:
addr =of_get_property(dp, "local-mac-address", &len);
. I'll append at the end of this email what my local T5-2 is using.
This really needs to: 1) get fixed to depend upon CONFIG_OF rather than a specific architecture like SPARC and 2) get submitted and accepted by the Intel ethernet driver maintainers.