x86-64 bad pmds in 2.6.11.6
Another datapoint for this thread. The box spewing the bad pmds messages is a dual opteron 246 on a TYAN S2885 Thunder K8W motherboard. Kernel is 2.6.11.4-20a-smp.
Approximately one hour after the bad pmd's, the box was completely unresponsive. This machine is either idle or heavily loaded, many threads, lots of io and nfs network traffic. Never see this when idle. When heavily loaded, it will invariably become unresponsive within 24 hrs. Looks reproducible. I'm willing to provide more information and test patches.
Output:
Sep 15 06:42:46 lakeport -- MARK -- Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680bc8 (00002aaaaaaaba98). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680bd0 (0000000000000002). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680bd8 (00007ffffffffdcc). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680be0 (00007ffffffffdcd). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680bf0 (00007ffffffffdce). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680bf8 (00007ffffffffdcf). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c00 (00007ffffffffdd0). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c08 (00007ffffffffdd1). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c10 (00007ffffffffdd2). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c18 (00007ffffffffdd3). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c20 (00007ffffffffdd4). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c28 (00007ffffffffdd5). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c30 (00007ffffffffdd6). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c38 (00007ffffffffdd7). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c40 (00007ffffffffdd8). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c48 (00007ffffffffdd9). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c50 (00007ffffffffdda). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c58 (00007ffffffffddb). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c60 (00007ffffffffddc). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c68 (00007ffffffffddd). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c70 (00007ffffffffdde). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c78 (00007ffffffffddf). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c80 (00007ffffffffde0). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c88 (00007ffffffffde1). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c90 (00007ffffffffde2). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680c98 (00007ffffffffde3). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680ca0 (00007ffffffffde4). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680ca8 (00007ffffffffde5). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cb0 (00007ffffffffde6). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cc0 (0000000000000010). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cc8 (00000000078bfbff). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cd0 (0000000000000006). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cd8 (0000000000001000). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680ce0 (0000000000000011). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680ce8 (0000000000000064). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cf0 (0000000000000003). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680cf8 (0000000000400040). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d00 (0000000000000004). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d08 (0000000000000038). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d10 (0000000000000005). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d18 (0000000000000009). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d20 (0000000000000007). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d28 (00002aaaaaaab000). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d30 (0000000000000008). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d40 (0000000000000009). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d48 (00000000004010f0). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d50 (000000000000000b). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d60 (000000000000000c). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d70 (000000000000000d). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d80 (000000000000000e). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680d90 (0000000000000017). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680da0 (000000000000000f). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680da8 (00007ffffffffdc5). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680dc0 (3638780000000000). Sep 15 06:58:44 lakeport kernel: mm/memory.c:97: bad pmd ffff81013c680dc8 (000000000034365f). Sep 15 07:22:47 lakeport -- MARK --
Quoted 4 lines · Charles McCreary
Another datapoint for this thread. The box spewing the bad pmds messages is a dual opteron 246 on a TYAN S2885 Thunder K8W motherboard. Kernel is 2.6.11.4-20a-smp.
This is quite possibly the result of an Opteron errata (tlb flush filtering is broken on SMP) that we worked around as of 2.6.14-rc4.
So either just try 2.6.14-rc2, or try the appended patch (it has since been confirmed by many more people).
Linus
---
diff-tree bc5e8fdfc622b03acf5ac974a1b8b26da6511c99 (from 61ffcafafb3d985e1ab8463be0187b421614775c)
Author: Linus Torvalds <[email protected]> Date: Sat Sep 17 15:41:04 2005 -0700
x86-64/smp: fix random SIGSEGV issues
They seem to have been due to AMD errata 63/122; the fix is to disable TLB flush filtering in SMP configurations.
Confirmed to fix the problem by Andrew Walrond <[email protected]>
[ Let's see if we'll have a better fix eventually, this is the Q&D
"let's get this fixed and out there" version ]
arch/x86_64/kernel/setup.c +15 -0
@@ -831,11 +831,26 @@ static void __init amd_detect_cmp(struct#endif } +#define HWCR 0xc0010015 +static int __init init_amd (struct cpuinfo_x86 *c) {int r;int level; +#ifdef CONFIG_SMP +unsigned long value; + +// Disable TLB flush filter by setting HWCR.FFDIS: +// bit 6 of msr C001_0015 +// +// Errata 63 for SH-B3 steppings +// Errata 122 for all(?) steppings +rdmsrl (HWCR , value); + value |=1 <<6 ; +wrmsrl (HWCR , value); +#endif +/* Bit 31 in normal CPUID used for nonstandard 3DNow ID; 3DNow is IDd by bit 31 in extended CPUID (1*32+31) anyway */ clear_bit (0 *32 +31 , &c->x86_capability);
Quoted 2 lines · Linus Torvalds
This is quite possibly the result of an Opteron errata (tlb flush filtering is broken on SMP) that we worked around as of 2.6.14-rc4.
It would be really interesting to know if this does help. I was told em64t also have the 'bad pmd' problem but I can't make it happen here on opteron on em64t.
Quoted 8 lines · Chris Wedgwood
Quoted 2 lines · Linus Torvalds
This is quite possibly the result of an Opteron errata (tlb flush filtering is broken on SMP) that we worked around as of 2.6.14-rc4.
It would be really interesting to know if this does help. I was told em64t also have the 'bad pmd' problem but I can't make it happen here on opteron on em64t.
In the dozens of reports of bad pmd that Fedora users filed, there wasn't a single EM64T user. In fact, most of the hits were from very similar product lines, from a handful of vendors (Tyan's seemed especially susceptable). It may be that other vendors updated their BIOS's to include this workaround already, and Tyan and a few others lagged behind.
Dave