Skip to contentSkip to conversation

anyone interested in chip register error diagnostics?

DISCUSSION zshelle 3 participants 10 messages Mar 4, 2019 – Mar 7, 2019 2 messages not in the archive

lore
Share & tools
lore

zshelle

author
#0 lore raw

On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system fatal error analysis (see my work-in-progress proposals: https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.

A message that is not in the archive (sent off-list or not yet received): 20190304203754.5piu2z2hj2yipomd@thinkpad

A message that is not in the archive (sent off-list or not yet received): AM4PR08MB2788F676F7BE8F10D8EFB90A80710@AM4PR08MB2788.eurprd08.prod.outlook.com

#3 lore raw

oops…I did not mean to take the list off copy. Adding it back on…

Quoted 49 lines · Supreeth Venkatesh

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors. The format of the errors are defined in Appendix N in UEFI specification
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf

I have not read the proposal in its entirety, but this seems similar to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.

I will take a look at the reviews posted and provide more feedback.

If this is something similar to RAS feature, I have in fact proposed in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.

Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in an architecture agnostic way.

Thanks,
Supreeth

-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?

Quoted 4 lines · zshelle

On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system

Quoted 8 lines

fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.

Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?

thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#4 lore raw
Quoted 10 lines · Brad Bishop

oops…I did not mean to take the list off copy. Adding it back on…

Quoted 8 lines · Supreeth Venkatesh

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.

Supreeth - Is the code that generates these open source?

#5 lore raw
Quoted 7 lines · Brad Bishop
Quoted 6 lines · Brad Bishop

oops…I did not mean to take the list off copy. Adding it back on…

Quoted 4 lines

On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
[email protected]> wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.

Supreeth - Is the code that generates these open source?

Yes.
arm-tf: https://github.com/ARM-software/arm-trusted-firmware
edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg

#6 lore raw
Quoted 12 lines · Supreeth Venkatesh
Quoted 11 lines · Brad Bishop
Quoted 10 lines · Brad Bishop

oops…I did not mean to take the list off copy. Adding it back on…

Quoted 8 lines

On Mar 4, 2019, at 3:56 PM, Supreeth Venkatesh <
[email protected]> wrote:

Thanks Brad.

Hi Zane/Brad,

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.

Supreeth - Is the code that generates these open source?

Yes.
arm-tf: https://github.com/ARM-software/arm-trusted-firmware
edk2: https://github.com/tianocore/edk2/tree/master/StandaloneMmPkg

Thanks for the info Supreeth!

-brad

author
#7 lore raw
In reply to a message not in the archive
Quoted 51 lines · Supreeth Venkatesh

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specification
http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf

I have not read the proposal in its entirety, but this seems similar to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.

I will take a look at the reviews posted and provide more feedback.

If this is something similar to RAS feature, I have in fact proposed in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.

Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in an architecture agnostic way.

Thanks,
Supreeth

-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?

Quoted 4 lines · zshelle

On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers, I/O chips, etc.) and performs service actions based on those errors. I have been working on porting some of this code to the BMC for system

Quoted 8 lines

fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part of the new design, we are building a generic, data-driven register error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.

Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?

Quoted 6 lines

thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Looks like CPER is similar to IBM's Platform Error Log (PEL). At this time, I am not really focused on the log format at the moment, but it is on my list of things to investigate. I heard a rumor that there may be a standard "OpenBMC" logging mechanism in the works. With those, you could convert them into the CPER or PEL format if needed.

My proposals are more focused on reading the registers from hardware and determining what caused the error. On POWER, we have hundreds of what we call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So my component will sort through all of those registers, find the active bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any services actions defined by our RAS team (and commit a log). Is there anything like this on ARM?

#8 lore raw
Quoted 11 lines · zshelle
Quoted 10 lines · Supreeth Venkatesh

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report
these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specification

http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf

Quoted 61 lines
Quoted 51 lines

I have not read the proposal in its entirety, but this seems similar
to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.

I will take a look at the reviews posted and provide more feedback.

If this is something similar to RAS feature, I have in fact proposed
in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.

Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in
an architecture agnostic way.

Thanks,
Supreeth

-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?

Quoted 9 lines · zshelle

On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers,
I/O chips, etc.) and performs service actions based on those errors. I
have been working on porting some of this code to the BMC for system
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and

Quoted 8 lines

https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
of the new design, we are building a generic, data-driven register
error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.

Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?

Quoted 9 lines

thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.

Quoted 9 lines

Looks like CPER is similar to IBM's Platform Error Log (PEL). At this
time, I am not really focused on the log format at the moment, but it is
on my list of things to investigate. I heard a rumor that there may be a
standard "OpenBMC" logging mechanism in the works. With those, you could
convert them into the CPER or PEL format if needed.

Ok. You are not currently focussed on format of the log itself but rather on the error syndrome/fault registers. Right? However, at present, error syndrome information gathered from the fault registers are converted to CPER/PEL format for logging purposes in the host firmware/microcontroller firmware.

Quoted 17 lines

My proposals are more focused on reading the registers from hardware and
determining what caused the error. On POWER, we have hundreds of what we
call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So
my component will sort through all of those registers, find the active
bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any
services actions defined by our RAS team (and commit a log). Is there
anything like this on ARM?

On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.

After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.
Is there a security implication here?

Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.

author
#9 lore raw
Quoted 111 lines · Supreeth Venkatesh
Quoted 11 lines · zshelle
Quoted 10 lines · Supreeth Venkatesh

Thanks Brad.

Hi Zane/Brad,

On Arm Platforms, We use Common Platform Error Record (CPER) to report
these kinds of hardware errors.
The format of the errors are defined in Appendix N in UEFI specification

http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf

Quoted 61 lines
Quoted 51 lines

I have not read the proposal in its entirety, but this seems similar
to Reliability, Availability, Serviceability (RAS) feature using System Management Mode/Management Mode, but on the BMC side.

I will take a look at the reviews posted and provide more feedback.

If this is something similar to RAS feature, I have in fact proposed
in DMTF PMCI WG to include CPER formats to be added to one of the PLDM specifications.

Arm would be interested in the design of this component, if it can accommodate the above error formats and component can be designed in
an architecture agnostic way.

Thanks,
Supreeth

-----Original Message-----
From: Brad Bishop <[email protected]>
Sent: Monday, March 4, 2019 2:38 PM
To: zshelle <[email protected]>; Supreeth Venkatesh <[email protected]>; [email protected]
Subject: Re: anyone interested in chip register error diagnostics?

Quoted 9 lines · zshelle

On POWER, I work on a component that listens for hardware errors reported by registers in the system chips (processors, memory buffers,
I/O chips, etc.) and performs service actions based on those errors. I
have been working on porting some of this code to the BMC for system
fatal error analysis (see my work-in-progress proposals:
https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18591/ and

Quoted 8 lines

https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/18831/). As part
of the new design, we are building a generic, data-driven register
error isolator, which will be used by several applications within POWER. However, it has the potential to be useful on other architectures as well. I am curious if anyone in the community is interested in this.

Thanks Zane - I'll tag Ed(x86) and Supreeth(arm) on this one. Ed, Supreeth - do you understand the function being proposed here? How does this work on x86 and arm servers?

Quoted 9 lines

thx - brad
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or
copy the information in any medium. Thank you.

Quoted 9 lines

Looks like CPER is similar to IBM's Platform Error Log (PEL). At this
time, I am not really focused on the log format at the moment, but it is
on my list of things to investigate. I heard a rumor that there may be a
standard "OpenBMC" logging mechanism in the works. With those, you could
convert them into the CPER or PEL format if needed.

Ok. You are not currently focussed on format of the log itself but rather on the error syndrome/fault registers. Right? However, at present, error syndrome information gathered from the fault registers are converted to CPER/PEL format for logging purposes in the host firmware/microcontroller firmware.

Quoted 17 lines

My proposals are more focused on reading the registers from hardware and
determining what caused the error. On POWER, we have hundreds of what we
call fault isolation registers (FIRs), where each bit within those registers can signify a different hardware error event. It is also possible that there may be several active bits on at the same time. So
my component will sort through all of those registers, find the active
bits, and determine what is the root cause of the failure versus side-effect errors. Once root cause is determined, we then perform any
services actions defined by our RAS team (and commit a log). Is there
anything like this on ARM?

On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.

Quoted 7 lines

After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.
Is there a security implication here?

Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.

On POWER, reading the FIR data has no security concerns. Writing data to those registers does have security concerns, but that should not be allowed from the BMC.

#10 lore raw
In reply to Supreeth Venkatesh
Quoted 10 lines · Supreeth Venkatesh

On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:

On Arm platforms also, There are several error syndrome registers which are read.
This information leads to contruction of CPER record which will be used by OS to take service actions as per RAS policy.

After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault isolation registers.

It isn't really moving - we run it on the host as well as the BMC, for situations where the POWER processor is not able to execute instructions due to a malfunction somewhere.

Quoted 1 line

Is there a security implication here?

Are you able to expand on this?

Quoted 3 lines

Thank you for the proposal, I will read 18591 thoroughly to understand, whether we can reuse this on arm architecture.

Thanks Supreeth!

#11 lore raw
Quoted 14 lines · Brad Bishop
Quoted 8 lines · Supreeth Venkatesh

On Mon, 2019-03-04 at 16:44 -0600, zshelle wrote:
On Arm platforms also, There are several error syndrome registers which
are read.
This information leads to contruction of CPER record which will be used
by OS to take service actions as per RAS policy.

Quoted 4 lines

After reading 18831, it looks like you want to move error data collection to BMC from host firmware and for that you collect all fault
isolation registers.

Quoted 5 lines

It isn't really moving - we run it on the host as well as the BMC, for
situations where the POWER processor is not able to execute instructions
due to a malfunction somewhere.

Thanks for the clarification Brad.

Quoted 4 lines
Quoted 2 lines

Is there a security implication here?

Are you able to expand on this?

Yes. I meant if this involves direct host/target register access from BMC, then there may be a security issue.
However, from Zane's reply, it does not seem to be the case.

Quoted 7 lines
Quoted 5 lines

Thank you for the proposal, I will read 18591 thoroughly to understand,
whether we can reuse this on arm architecture.

Thanks Supreeth!