Preserving CMC/CPE records across reboot
CMC/CPE records (unlike MCA/INIT) are copied into kernel space and cleared from NVRAM as soon as they occur. That decision was made by Bjorn Helgaas some years ago. The idea is that if you do not have salinfo_decode or some equivalent program running then the correctable errors still need to be deleted from NVRAM. But if the system hangs while reading the CMC/CPE then we get no data at all.
SGI just had an example of this. A cpu took a CMC, salinfo_decode started running and hung while processing the CMC record, the system had to be rebooted. Because the CMC record had been cleared from NVRAM before handing a copy to salinfo_decode, the contents were lost.
We should be able to keep the first few CMC/CPE records for each cpu in NVRAM and discard the later ones if we start getting a backlog. Then if the system hangs while processing a CMC/CPE, the data will still be available in NVRAM and will be processed on the next boot. If the reboot hangs again in salinfo processing then we have a solid error, either cpu or SAL, so switch the offending cpu out of the system.
Any objections from other platforms?
Quoted 9 lines · Keith Owens
We should be able to keep the first few CMC/CPE records for each cpu in NVRAM and discard the later ones if we start getting a backlog. Then if the system hangs while processing a CMC/CPE, the data will still be available in NVRAM and will be processed on the next boot. If the reboot hangs again in salinfo processing then we have a solid error, either cpu or SAL, so switch the offending cpu out of the system.
Any objections from other platforms?
Keith,
Sorry, it's been a while since I've looked at this code, but how do
we determine how many records can be stored in NVRAM? I would guess that for CPEs at least, it's platform dependent. If it can be done w/o losing records, it's probably ok, but I'm not sure I understand the details. Thanks,
Alex
Signature · 1 line
Alex Williamson HP Linux & Open Source Lab
Quoted 11 lines · Keith Owens
CMC/CPE records (unlike MCA/INIT) are copied into kernel space and cleared from NVRAM as soon as they occur. That decision was made by Bjorn Helgaas some years ago. The idea is that if you do not have
salinfo_decodeor some equivalent program running then the correctable errors still need to be deleted from NVRAM. But if the system hangs while reading the CMC/CPE then we get no data at all.SGI just had an example of this. A cpu took a CMC,
salinfo_decodestarted running and hung while processing the CMC record, the system had to be rebooted. Because the CMC record had been cleared from NVRAM before handing a copy tosalinfo_decode, the contents were lost.
On SN, CMC/CPE records are never written to NVRAM. They are saved only in memory. If the system hangs trying to log a CMC/CPE & the system is reset, all CMC/CPE records are lost.
It is possible that some of this could be changed but it currently works this way. Also, writing error records to NVRAM is slow - something to avoid on performance critical paths. I suppose we could threshhold the error rate & would limit the rate of writing to NVRAM.
Quoted 10 lines
We should be able to keep the first few CMC/CPE records for each cpu in NVRAM and discard the later ones if we start getting a backlog. Then if the system hangs while processing a CMC/CPE, the data will still be available in NVRAM and will be processed on the next boot. If the reboot hangs again in salinfo processing then we have a solid error, either cpu or SAL, so switch the offending cpu out of the system.
Any objections from other platforms?
Quoted 5 lines
Sorry, it's been a while since I've looked at this code, but how do
we determine how many records can be stored in NVRAM? I would guess that for CPEs at least, it's platform dependent. If it can be done w/o losing records, it's probably ok, but I'm not sure I understand the details. Thanks,
Agreed. The SAL spec doesn't require any particular capacity for NVRAM, nor does it provide a mechanism to discover it.
-Tony
Quoted 9 lines · Alex Williamson (on Fri, 13 Jan 2006 08:05:07 -0700) wrote:
Quoted 8 lines · Keith Owens
We should be able to keep the first few CMC/CPE records for each cpu in NVRAM and discard the later ones if we start getting a backlog. Then if the system hangs while processing a CMC/CPE, the data will still be available in NVRAM and will be processed on the next boot. If the reboot hangs again in salinfo processing then we have a solid error, either cpu or SAL, so switch the offending cpu out of the system.
Quoted 7 lines
Quoted 1 line
Any objections from other platforms?
Sorry, it's been a while since I've looked at this code, but how do
we determine how many records can be stored in NVRAM? I would guess that for CPEs at least, it's platform dependent. If it can be done w/o losing records, it's probably ok, but I'm not sure I understand the details. Thanks,
By counting the number of interrupts and subtracting the number of 'clear' events issued by user space. It would be messy but possible.
Jack Steiner has pointed out that the SGI prom never saves CMC/CPE records anyway, which means that my idea would not solve the problem of records being lost due to reboot. So I am dropping this idea.