AMD64: fix mce_cpu_quirks typos
Florian Weimer <[email protected]> writes:
First please send x86-64 patches cc to the maintainer, things can get lost in the noise of the list.
Quoted 4 lines
The spurious MCE is TLB-related. I *think* the bit for the correct status code is stored at position 10 HEX, not 10 DEC. At least I still get those MCEs on a two-way Opteron box, even though they are supposed to be filtered out.
No, 10 is the correct bit index. But normally it's set by BIOS anyways.
The reason you still see it is that setting the bit here only prevent MCE exceptions, but it's still logged and the regular polling picks them up anyways. I have not found a nice way to handle this (other than adding a ugly CPU specific special case in the middle of the nice cpu independent machine check handler, which I couldn't bring myself to do so far...)
-Andi
* Andi Kleen:
Quoted 4 lines
Florian Weimer <[email protected]> writes:
First please send x86-64 patches cc to the maintainer, things can get lost in the noise of the list.
Oops, sorry about that. Perhaps I should repeat that MCE for the sake of discuss@, as decoded by mcelog:
MCE 0
CPU 0 4 northbridge TSC 91ec03f09330
ADDR 104500000
Northbridge GART error
bit61 = error uncorrected
TLB error 'generic transaction, level generic'
STATUS a40000000005001b MCGSTATUS 0
Quoted 9 lines
Quoted 4 lines
The spurious MCE is TLB-related. I *think* the bit for the correct status code is stored at position 10 HEX, not 10 DEC. At least I still get those MCEs on a two-way Opteron box, even though they are supposed to be filtered out.
No, 10 is the correct bit index. But normally it's set by BIOS anyways.
The reason you still see it is that setting the bit here only prevent MCE exceptions,
And thus a kernel panic?
Quoted 5 lines
but it's still logged and the regular polling picks them up anyways. I have not found a nice way to handle this (other than adding a ugly CPU specific special case in the middle of the nice cpu independent machine check handler, which I couldn't bring myself to do so far...)
Someone tried to track these messages down together with someone else from AMD, but they never got it finished.
For reference, here's the lspci -n output for the system. It's a two-way Opteron box (248, 2.2 GHz, stepping 10) with 8 GB of RAM. (BIOS and chipset details are not available to me at the moment.) The MCEs only appeared after a switch to a 64-bit kernel (2.6.15.2), adding the second CPU, along with 4 GB of RAM. Previously, the box ran 2.6.13 in 32-bit mode, and no MCEs appeared regularly.
In the history of the system, there was one more MCE, but we thought at that time it was related to thermal issues (it happened after someone had switched off air conditioning in the server room *cough*).
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) 0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:01:05.0 RAID bus controller: 3ware Inc 3ware ATA-RAID
0000:02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) 0000:03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 0000:03:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 0000:03:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 10)
Quoted 8 lines · Florian Weimer
Quoted 5 lines
but it's still logged and the regular polling picks them up anyways. I have not found a nice way to handle this (other than adding a ugly CPU specific special case in the middle of the nice cpu independent machine check handler, which I couldn't bring myself to do so far...)
Someone tried to track these messages down together with someone else from AMD, but they never got it finished.
They could have saved themselves a lot of work by just asking at the right mailing lists (which is not l-k BTW)
Quoted 6 lines
For reference, here's the lspci -n output for the system. It's a two-way Opteron box (248, 2.2 GHz, stepping 10) with 8 GB of RAM. (BIOS and chipset details are not available to me at the moment.) The MCEs only appeared after a switch to a 64-bit kernel (2.6.15.2), adding the second CPU, along with 4 GB of RAM. Previously, the box ran 2.6.13 in 32-bit mode, and no MCEs appeared regularly.
The 64bit kernel uses the AGP aperture as IOMMU, the 32bit kernel doesn't. It's a known documented hardware bug that this causes spurious GART errors. That is why the BIOS and Linux disable them. Unfortunately the Linux MCE handler is too thorough and picks them up anyways as corrected events.
-Andi
* Andi Kleen:
Quoted 8 lines · Florian Weimer
Quoted 7 lines
Quoted 6 lines
but it's still logged and the regular polling picks them up anyways. I have not found a nice way to handle this (other than adding a ugly CPU specific special case in the middle of the nice cpu independent machine check handler, which I couldn't bring myself to do so far...)
Quoted 5 lines
Quoted 2 lines
Someone tried to track these messages down together with someone else from AMD, but they never got it finished.
They could have saved themselves a lot of work by just asking at the right mailing lists (which is not l-k BTW)
Marc Michelsen brought this up last year on <[email protected]> (which I suppose is the right list), but he didn't receive many comments (not publicly, at least).
Quoted 3 lines
The 64bit kernel uses the AGP aperture as IOMMU, the 32bit kernel doesn't. It's a known documented hardware bug that this causes spurious GART errors.
Someone from AMD told Marc that fixes in pci-gart.c (probably related to iommu_fullflush, see the comment there) are supposed to suppress the error in the first place. That's why we are a bit confused whether the errors are really harmless (our machines do run stable, though).
It also seems that the bug is not as well-documented as it deserves to be. (The search engines will pick up this thread, though.) Our vendor told us to have the RAM tested, for instance. 8->
Quoted 3 lines
That is why the BIOS and Linux disable them. Unfortunately the Linux MCE handler is too thorough and picks them up anyways as corrected events.
If the errors are really harmless, it probably makes sense to add a warning to the mcelog output that this MCE is expected, preferably with an AMD errata reference.
Filtering in the kernel seems to be overkill because the rate of those spurious MCEs is fairly low, and they won't lead to loss of other, more important MCEs.
Quoted 10 lines · Florian Weimer
Quoted 4 lines
The 64bit kernel uses the AGP aperture as IOMMU, the 32bit kernel doesn't. It's a known documented hardware bug that this causes spurious GART errors.
Someone from AMD told Marc that fixes in
pci-gart.c(probably related toiommu_fullflush, see the comment there) are supposed to suppress the error in the first place. That's why we are a bit confused whether the errors are really harmless (our machines do run stable, though).
Long ago there was a real bug in this area which caused these GART errors legitimately, but even what that one was fixed they still occurred occasionally.
I was told back then that there was a bug in the Northbridge that causes them occasionally - that is why BIOS turn them off. The kernel did that eventually too.
Of course there is some probability that you have a driver that accesses the buffers after unmapping. The GART is currently not flushed on unmapping because that would be
Normally such drivers are caught though because some other IOMMU implementations on other architectures have stronger checking in this area.
-Andi