Skip to contentSkip to conversation

Parse PELs sent by OpenPOWER host firmware

DISCUSSION Deepak Kodihalli 5 participants 5 messages Mar 13, 2018 – Apr 12, 2018

lore
Share & tools
lore
#0 lore raw

Hi,

The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot today are not interpreted, and are dumped as-is in a property of the error log D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in the eSEL, admins/users of the error log need to depend on external tools to parse the PEL to human readable text.

Here's a proposal to have the BMC itself parse such eSELs (the motivation is to make the OpenBMC error logs more useful in this context, without needing the extra steps to parse such errors) :

- Separate this from phosphor-logging. This code can't go to a phosphor app, because it's OpenPOWER specific. Have, for example, an openpower-logging app implement a D-Bus interface to parse out a PEL from an eSEL. This app will host it's own D-Bus service.

- Such an app can watch for error D-Bus objects being created in the phosphor namespace. If it finds an eSEL in an error log, it can attempt to parse a PEL (if there's one), and store the PELs parsed text representation in a D-Bus property.

- The app can create a D-Bus object with the same path as the original object and have a D-Bus object-manager at the error log root path, so that when someone uses the OpenBMC REST API to read an error log, properties across all D-Bus services will be returned (phosphor-rest-server does this today already). This means with the existing REST API to enumerate error logs, a parsed PEL would be returned, if present.

- To parse the PEL, use opal-elog-parse.

I have some quickly hacked-up code based on the proposal above, and I can submit that for review. In general, this is appearing to me to be one of the ways to add host-specific information to phosphor D-Bus objects, without breaking the phosphor D-Bus API or changing the REST URIs.

Thanks,
Deepak

#1 lore raw
Quoted 6 lines · Deepak Kodihalli

The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot today are not interpreted, and are dumped as-is in a property of the error log D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in the eSEL, admins/users of the error log need to depend on external tools to parse the PEL to human readable text.

My lack of love for PELs is probably fairly well known, but even if we replace them down the line, I think there's opportunity here for making things better for users (including myself).

Quoted 8 lines

Here's a proposal to have the BMC itself parse such eSELs (the motivation is to make the OpenBMC error logs more useful in this context, without needing the extra steps to parse such errors) :

- Separate this from phosphor-logging. This code can't go to a phosphor app, because it's OpenPOWER specific. Have, for example, an openpower-logging app implement a D-Bus interface to parse out a PEL from an eSEL. This app will host it's own D-Bus service.

It would be a bonus if we could take different actions based on log format. If in the future we had a hypothetical other format, it'd be great if it was *really* easy to plug into OpenBMC. I suspect other platforms may have similar things (e.g. Google has done protobuf based logs in the past at least, which are structurally somewhat similar to PELs but different serialisation format).

Quoted 5 lines

- Such an app can watch for error D-Bus objects being created in the phosphor namespace. If it finds an eSEL in an error log, it can attempt to parse a PEL (if there's one), and store the PELs parsed text representation in a D-Bus property.

Quoted 7 lines

- The app can create a D-Bus object with the same path as the original object and have a D-Bus object-manager at the error log root path, so that when someone uses the OpenBMC REST API to read an error log, properties across all D-Bus services will be returned (phosphor-rest-server does this today already). This means with the existing REST API to enumerate error logs, a parsed PEL would be returned, if present.

Is thre any reason for this approach rather than something like phosphor-logging having a list of properties/formats and external parsing tools? Kind of like mime type/action lists?

Quoted 1 line

- To parse the PEL, use opal-elog-parse.

+1

Signature · 2 lines
Stewart Smith
OPAL Architect, IBM.
#2 lore raw
Quoted 10 lines · Stewart Smith
Quoted 6 lines · Deepak Kodihalli

The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot today are not interpreted, and are dumped as-is in a property of the error log D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in the eSEL, admins/users of the error log need to depend on external tools to parse the PEL to human readable text.

My lack of love for PELs is probably fairly well known, but even if we replace them down the line, I think there's opportunity here for making things better for users (including myself).

Just my 2 cents. The IPMI-standardized SELs are good first of all because of IPMI-standardized PEFs and PETs.

Being able to decode the OEM SEL records is not just convenient for the users, but will also let them easily attach PEFs to such events if they want to. That's of course when PEFs are supported in OpenBMC which is currently not true, AFAIK.

As a maintainer of ipmitool I can promise to promptly include the OpenPOWER event record decoder into ipmitool upstream code. All I need is the information on how to decode those events. So far I'm planning to try and deduce this information from hostboot and opal sources.

I guess these logs don't need to be fully decoded at BMC or ipmitool levels as they contain too much of a too deeply technical information, but some high level hints beyond "Unknown #0x0c" and "System Event #0x01 | Undetermined system hardware failure" that we see now in ipmitool need to be given to the end user.

Alexander.

Quoted 35 lines
Quoted 9 lines

Here's a proposal to have the BMC itself parse such eSELs (the motivation is to make the OpenBMC error logs more useful in this context, without needing the extra steps to parse such errors) :

- Separate this from phosphor-logging. This code can't go to a phosphor app, because it's OpenPOWER specific. Have, for example, an openpower-logging app implement a D-Bus interface to parse out a PEL from an eSEL. This app will host it's own D-Bus service.

It would be a bonus if we could take different actions based on log format. If in the future we had a hypothetical other format, it'd be great if it was *really* easy to plug into OpenBMC. I suspect other platforms may have similar things (e.g. Google has done protobuf based logs in the past at least, which are structurally somewhat similar to PELs but different serialisation format).

Quoted 5 lines

- Such an app can watch for error D-Bus objects being created in the phosphor namespace. If it finds an eSEL in an error log, it can attempt to parse a PEL (if there's one), and store the PELs parsed text representation in a D-Bus property.

Quoted 7 lines

- The app can create a D-Bus object with the same path as the original object and have a D-Bus object-manager at the error log root path, so that when someone uses the OpenBMC REST API to read an error log, properties across all D-Bus services will be returned (phosphor-rest-server does this today already). This means with the existing REST API to enumerate error logs, a parsed PEL would be returned, if present.

Is thre any reason for this approach rather than something like phosphor-logging having a list of properties/formats and external parsing tools? Kind of like mime type/action lists?

Quoted 1 line

- To parse the PEL, use opal-elog-parse.

+1

#3 lore raw
Quoted 20 lines · Alexander Amelkin
Quoted 10 lines · Stewart Smith
Quoted 6 lines · Deepak Kodihalli

The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot today are not interpreted, and are dumped as-is in a property of the error log D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in the eSEL, admins/users of the error log need to depend on external tools to parse the PEL to human readable text.

My lack of love for PELs is probably fairly well known, but even if we replace them down the line, I think there's opportunity here for making things better for users (including myself).

Just my 2 cents. The IPMI-standardized SELs are good first of all because of IPMI-standardized PEFs and PETs.

Being able to decode the OEM SEL records is not just convenient for the users, but will also let them easily attach PEFs to such events if they want to. That's of course when PEFs are supported in OpenBMC which is currently not true, AFAIK.

Quoted 4 lines

As a maintainer of ipmitool I can promise to promptly include the OpenPOWER event record decoder into ipmitool upstream code. All I need is the information on how to decode those events. So far I'm planning to try and deduce this information from hostboot and opal sources.

(Adding in Tim from Raptor as we were talking about this yesterday)

There's opal-elog-parse, which is part of ppc64-diag, and there's
https://github.com/open-power/libopalevents which is a quite bitrotted attempt to turn parsing it into a shared library rather than just a binary.

there's also a (internal but in the process of being open sourced) implementation called 'errl' which is what has shipped on IBM FSP systems forever (but that for whatever reason we ended up doing opal-elog-parse rather than open sourcing it back in 2014).

From hostboot, ./src/build/debug/eSEL.pl is the script that extracts things from the BMC IIRC.

Now, there *is* a spec for the Platfrom Error Log format, which I have previously tried to get out into the open... so I encourage you to find all your IBM contacts and bug them about it.

That being said, I'm not convinced PEL is the right thing to continue to exist. It's Yet Another Custom Binary Format, and there's probably enough of those in the world as it is.

Signature · 2 lines
Stewart Smith
OPAL Architect, IBM.
#4 lore raw
In reply to Deepak Kodihalli
Quoted 3 lines · Deepak Kodihalli

Hi,

The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot today are not interpreted, and are dumped as-is in a property of the error log D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in the eSEL, admins/users of the error log need to depend on external tools to parse the PEL to human readable text.

Yes. We are having hard time with this. Any improvement is welcome.

Quoted 6 lines

Here's a proposal to have the BMC itself parse such eSELs (the motivation is to make the OpenBMC error logs more useful in this context, without needing the extra steps to parse such errors) :

- Separate this from phosphor-logging. This code can't go to a phosphor app,
because it's OpenPOWER specific. Have, for example, an openpower-logging app implement a D-Bus interface to parse out a PEL from an eSEL. This app will host it's own D-Bus service.

Quoted 7 lines

- Such an app can watch for error D-Bus objects being created in the phosphor
namespace. If it finds an eSEL in an error log, it can attempt to parse a PEL (if there's one), and store the PELs parsed text representation in a D-Bus property.

- The app can create a D-Bus object with the same path as the original object
and have a D-Bus object-manager at the error log root path, so that when someone uses the OpenBMC REST API to read an error log, properties across all D-Bus services will be returned (phosphor-rest-server does this today already). This means with the existing REST API to enumerate error logs, a parsed PEL would be returned, if present.

- To parse the PEL, use opal-elog-parse.

+1. I think it has capability to detect eSEL wrapper as well. It should just work fine.

-Vasant