anyone interested in chip register error diagnostics?
On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system fatal error analysis (see my work-in-progress proposals: https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.
A message that is not in the archive (sent off-list or not yet received): 20190304203754.5piu2z2hj2yipomd@thinkpad
A message that is not in the archive (sent off-list or not yet received): AM4PR08MB2788F676F7BE8F10D8EFB90A80710@AM4PR08MB2788.eurprd08.prod.outlook.com
oops…I did not mean to take the list off copy. Adding it back on…
Quoted 49 lines · Supreeth Venkatesh
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors. The format of the errors are defined in Appendix N in UEFI specification
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdfI have not read the proposal in its entirety, but this seems similar to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.
I will take a look at the reviews posted and provide more feedback.
If this is something similar to RAS feature, I have in fact proposed in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.
Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in an architecture agnostic way.
Thanks,
Supreeth-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?Quoted 4 lines · zshelle
On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system
Quoted 8 lines
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?
thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 10 lines · Brad Bishop
oops…I did not mean to take the list off copy. Adding it back on…
Quoted 8 lines · Supreeth Venkatesh
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
Supreeth - Is the code that generates these open source?
Quoted 7 lines · Brad Bishop
Quoted 6 lines · Brad Bishop
oops…I did not mean to take the list off copy. Adding it back on…
Quoted 4 lines
On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
Supreeth - Is the code that generates these open source?
Yes.
arm-tf: https://github.com/ARM-software/arm-trusted-firmware
edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg
Quoted 12 lines · Supreeth Venkatesh
Quoted 11 lines · Brad Bishop
Quoted 10 lines · Brad Bishop
oops…I did not mean to take the list off copy. Adding it back on…
Quoted 8 lines
On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
[email protected]> wrote:Thanks Brad.
Hi Zane/Brad,
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
Supreeth - Is the code that generates these open source?
Yes.
arm-tf: https://github.com/ARM-software/arm-trusted-firmware
edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg
Thanks for the info Supreeth!
-brad
Quoted 51 lines · Supreeth Venkatesh
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specification
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdfI have not read the proposal in its entirety, but this seems similar to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.
I will take a look at the reviews posted and provide more feedback.
If this is something similar to RAS feature, I have in fact proposed in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.
Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in an architecture agnostic way.
Thanks,
Supreeth-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?Quoted 4 lines · zshelle
On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system
Quoted 8 lines
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?
Quoted 6 lines
thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Looks like CPER is similar to IBM's Platform Error Log (PEL). At this time, I am not really focused on the log format at the moment, but it is on my list of things to investigate. I heard a rumor that there may be a standard "OpenBMC" logging mechanism in the works. With those, you could convert them into the CPER or PEL format if needed.
My proposals are more focused on reading the registers from hardware and determining what caused the error. On POWER, we have hundreds of what we call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So my component will sort through all of those registers, find the active bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any services actions defined by our RAS team (and commit a log). Is there anything like this on ARM?
Quoted 11 lines · zshelle
Quoted 10 lines · Supreeth Venkatesh
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report
these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specification
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
Quoted 61 lines
Quoted 51 lines
I have not read the proposal in its entirety, but this seems similar
to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.I will take a look at the reviews posted and provide more feedback.
If this is something similar to RAS feature, I have in fact proposed
in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in
an architecture agnostic way.Thanks,
Supreeth-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?Quoted 9 lines · zshelle
On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers,
I/O chips, etc.) and performs service actions based on those errors. I
have been working on porting some of this code to the BMC for system
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ andQuoted 8 lines
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
of the new design, we are building a generic, data-driven register
error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?
Quoted 9 lines
thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.
Quoted 9 lines
Looks like CPER is similar to IBM's Platform Error Log (PEL). At this
time, I am not really focused on the log format at the moment, but it is
on my list of things to investigate. I heard a rumor that there may be a
standard "OpenBMC" logging mechanism in the works. With those, you could
convert them into the CPER or PEL format if needed.
Ok. You are not currently focussed on format of the log itself but rather on the error syndrome/fault registers. Right? However, at present, error syndrome information gathered from the fault registers are converted to CPER/PEL format for logging purposes in the host firmware/microcontroller firmware.
Quoted 17 lines
My proposals are more focused on reading the registers from hardware and
determining what caused the error. On POWER, we have hundreds of what we
call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So
my component will sort through all of those registers, find the active
bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any
services actions defined by our RAS team (and commit a log). Is there
anything like this on ARM?
On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.
After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.
Is there a security implication here?
Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.
Quoted 111 lines · Supreeth Venkatesh
Quoted 11 lines · zshelle
Quoted 10 lines · Supreeth Venkatesh
Thanks Brad.
Hi Zane/Brad,
On Arm Platforms, We use Common Platform Error Record (CPER) to report
these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specificationhttp://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf
Quoted 61 lines
Quoted 51 lines
I have not read the proposal in its entirety, but this seems similar
to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.I will take a look at the reviews posted and provide more feedback.
If this is something similar to RAS feature, I have in fact proposed
in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in
an architecture agnostic way.Thanks,
Supreeth-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?Quoted 9 lines · zshelle
On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers,
I/O chips, etc.) and performs service actions based on those errors. I
have been working on porting some of this code to the BMC for system
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ andQuoted 8 lines
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
of the new design, we are building a generic, data-driven register
error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?
Quoted 9 lines
thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.Quoted 9 lines
Looks like CPER is similar to IBM's Platform Error Log (PEL). At this
time, I am not really focused on the log format at the moment, but it is
on my list of things to investigate. I heard a rumor that there may be a
standard "OpenBMC" logging mechanism in the works. With those, you could
convert them into the CPER or PEL format if needed.Ok. You are not currently focussed on format of the log itself but rather on the error syndrome/fault registers. Right? However, at present, error syndrome information gathered from the fault registers are converted to CPER/PEL format for logging purposes in the host
firmware/microcontrollerfirmware.Quoted 17 lines
My proposals are more focused on reading the registers from hardware and
determining what caused the error. On POWER, we have hundreds of what we
call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So
my component will sort through all of those registers, find the active
bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any
services actions defined by our RAS team (and commit a log). Is there
anything like this on ARM?On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.
Quoted 7 lines
After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.
Is there a security implication here?Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.
On POWER, reading the FIR data has no security concerns. Writing data to those registers does have security concerns, but that should not be allowed from the BMC.
Quoted 10 lines · Supreeth Venkatesh
On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.
It isn't really moving - we run it on the host as well as the BMC, for situations where the POWER processor is not able to execute instructions due to a malfunction somewhere.
Quoted 1 line
Is there a security implication here?
Are you able to expand on this?
Quoted 3 lines
Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.
Thanks Supreeth!
Quoted 14 lines · Brad Bishop
Quoted 8 lines · Supreeth Venkatesh
On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
On Arm platforms also, There are several error syndrome registers which
are read.
This information leads to contruction of CPER record which will be used
by OS to take service actions as per RAS policy.Quoted 4 lines
After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault
isolation registers.
Quoted 5 lines
It isn't really moving - we run it on the host as well as the BMC, for
situations where the POWER processor is not able to execute instructions
due to a malfunction somewhere.
Thanks for the clarification Brad.
Quoted 4 lines
Quoted 2 lines
Is there a security implication here?
Are you able to expand on this?
Yes. I meant if this involves direct host/target register access from BMC, then there may be a security issue.
However, from Zane's reply, it does not seem to be the case.
Quoted 7 lines
Quoted 5 lines
Thank you for the proposal, I will read 18591 thoroughly to understand,
whether we can reuse this on arm architecture.Thanks Supreeth!