amd64_edac: EDAC module for AMD64
W1DUG
Quoted 16 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Subject: Re: [RFC PATCH 00/21 v2]amd64_edac: EDAC module for AMD64
To: "Andi Kleen" <[email protected]>
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Date: Thursday, April 30, 2009, 5:57 AM
Hi,Quoted 1 line · Andi Kleen
Borislav Petkov <[email protected]>
Quoted 4 lines · writes:
Quoted 4 lines
Hi,
thanks to all reviewers of the previous
submission, here is the second
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
version of this series.
The classic problem of the previous versions of these
patches was that
Quoted 1 line
they consume the same error registers (even if using
pci config versus
Quoted 1 line
msrs as access methods) as the kernel machine check
poll/threshold
Quoted 1 line
interrupt code.
Even the recommendation of AMD of having a polling thread for CORRECTABLE ERROR has a race issue to the same error registers due to the fact that a MCE is an exception and cannot be deferred or blocked off. In the middle of any poll cycle a MCE could fire and touch the same registers. small but present.
Quoted 12 lines
Quoted 1 line
And with two logging agents
racing on the same
Quoted 1 line
registers you will always get junk results. Typically
with threshold
Quoted 1 line
enabled the mce code wins the race. I suspect this
patchkit has
Quoted 1 line
exactly the same fundamental design problem. EDAC
really is not
Quoted 1 line
particularly fitting for integrated memory controllers
that report
Quoted 1 line
their errors using standard machine check events.
Quoted 11 lines
ok, how about we remove tha MSR/PCI cfg space reading bits and leave
that task solely to the mce core. Then, iff you have edac turned on inKconfig, mce code delivers needed error info to edac which, in turn,
goes and decodes the error/does the mapping to DIMM blocks/supplies DRAM
error injection facility for testing purposes and similar things. That
way you have both and they don't overlap in functionality.
Adding the synchronization between the two is very doable. It is not yet in the current patch set, but a work in progress.
That is the solution we are pursuing, to have a mechanism to provide communication between MCE and EDAC providing the mapping operation to a DIMM label. The MCA exception fires retrieves the info and calls EDAC module for address mapping.
MCE polling handler calls the EDAC module for address mapping.
EDAC's basic model is a polling operation on the error registers at a 1 second (tunable) rate.
AMD's manual describes the UNCORRECTABLE MEMORY error handling via the MCE handler. It further recommends a polling thread to harvest CORRECTABLE MEMORY errors. Last time I checked the MCE poller was running on a 5 minute poll cycle.
That is where we have 2 different threads polling the same error registers without synchronization is problematic and where a "Listener" pattern can be created to provide callbacks for both or form into a single poller operation.
Quoted 6 lines
By the way, I think there's a similar attempt/proposal of letting mce
and edac talk to each other from Red Hat so I think this could be a
viable thing to try.
Exactly
Quoted 19 lines
Quoted 2 lines
-Andi (who thinks all of this decoding should be in
user space anyways)
Think of a big data center with a thousands of 2,4,8 socket blades
and the admin collecting mce output and running around decoding the
errors on his workstation. Even worse, the blades have different DIMM
configurations due to hw upgrades/newer machines. I'd much rather have
the complete decoding done in kernel, where all the information needed
for proper decoding is present and with the error landing in syslog or
some other monitored buffer instead of reconstructing it in userspace.
Quoted 5 lines
Thanks.
Signature · 2 lines
Regards/Gruss, Boris.
This model of clusters with thousands of multi-core nodes (5,000 in one case I think of) is used many times. The system console is tie to a serial port via a BIOS switch. The serial port is then attached to "conman" and all the consoles are funneled to a cluster controller which parses for a "bad memory" event.
In sites with EDAC deployed now the parser finds the node number, the CPU number on the node and extracts the EDAC DIMM label provided and generates a Repair Ticket. The technician proceeds to find the proper rack, blade and DIMM and takes that node out of service (for MCEs that are intermittent the node is reboot earlier). Then the bad DIMM is replaced - the one identified from EDAC - and the node quickly brought back online.
Without the DIMM Label provided by EDAC - or with just mce 'bad address' information - ALL the DIMMS are swapped out for off-line testing or all are return for warranty replacement. Getting the node back on line is the priority and reducing the time the technician spends on the rack floor.
Bare MCE information is logged on the cluster controller and no time is spent trying to retrieve the log and running a user space program. Cheaper (man hours) and faster to swapout out all the DIMMs. But that is frowned on, with EDAC solving the problem for themnow.
The requested feature from the customers is to provide the DIMM label WITH the MCE error information as well.
doug t