Skip to contentSkip to conversation

Creating event logs for blackout power losses

DISCUSSION Matt Spinler 2 participants 4 messages Jun 28, 2019 – Jul 25, 2019

lore
Share & tools
lore
#0 lore raw

Hi,
We've been asked by field support to create an OpenBMC event log when we detect that we suffered a power blackout, by which I mean the system is up and running, and then the BMC suddenly reboots and when it comes back up main power is off. The issue for it is
https://github.com/ibm-openbmc/dev/issues/677.

I have a proposal in there for how to detect it in the chassis state manager code, which basically just consists of persisting the chassis power state and then checking it against the actual value on startup, and an error proposed at
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791.

Does anyone have any other ideas or thoughts?  Is it OK if we make this the default behavior, or would it be preferred we enable it with a compile flag?

Thanks,
Matt

#1 lore raw
Quoted 6 lines · Matt Spinler

Hi,
We've been asked by field support to create an OpenBMC event log when we detect that we suffered a power blackout, by which I mean the system is up and running, and then the BMC suddenly reboots and when it comes back up main power is off. The issue for it is
https://github.com/ibm-openbmc/dev/issues/677.

Quoted 7 lines

I have a proposal in there for how to detect it in the chassis state manager code, which basically just consists of persisting the chassis power state and then checking it against the actual value on startup, and an error proposed at
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791.

Does anyone have any other ideas or thoughts?  Is it OK if we make this the default behavior, or would it be preferred we enable it with a compile flag?

Sorry for the late replay as I have been out of the office.

In my testing on this type of failure, I found that there is a race condition between the Chassis and BMC losing power after the blackout. If the Chassis loses power while the BMC is still active, it can sometimes detect it, change the Chassis State to Off, and persist it before losing power. When this happens, the next boot will see only the Off state persisted and not detect the blackout.

To work around this, we have added a delay when persisting the Chassis State. In a normal Off, it will be persisted after the delay; in a blackout off, the delay will prevent the BMC from persisting the Off state allowing the blackout to be detected.

Quoted 3 lines

Thanks,
Matt

#2 lore raw
Quoted 12 lines · Jason M
Quoted 11 lines · Matt Spinler

Hi,
We've been asked by field support to create an OpenBMC event log when we detect that we suffered a power blackout, by which I mean the system is up and running, and then the BMC suddenly reboots and when it comes back up main power is off. The issue for it is
https://github.com/ibm-openbmc/dev/issues/677.

I have a proposal in there for how to detect it in the chassis state manager code, which basically just consists of persisting the chassis power state and then checking it against the actual value on startup, and an error proposed at
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791.

Quoted 8 lines
Quoted 2 lines

Does anyone have any other ideas or thoughts?  Is it OK if we make this the default behavior, or would it be preferred we enable it with a compile flag?

Sorry for the late replay as I have been out of the office.

In my testing on this type of failure, I found that there is a race condition between the Chassis and BMC losing power after the blackout. If the Chassis loses power while the BMC is still active, it can sometimes detect it, change the Chassis State to Off, and persist it before losing power.  When this happens, the next boot will see only the Off state persisted and not detect the blackout.

To work around this, we have added a delay when persisting the Chassis State.  In a normal Off, it will be persisted after the delay; in a blackout off, the delay will prevent the BMC from persisting the Off state allowing the blackout to be detected.

Hi, thanks for the response.
Did you do this in a downstream phosphor-state-manager repository, or somewhere else?  If in phosphor-state-manager, would you consider sharing it?

Quoted 5 lines
Quoted 4 lines

Thanks,
Matt

#3 lore raw
Quoted 20 lines · Matt Spinler
Quoted 17 lines · Jason M
Quoted 6 lines · Matt Spinler

Hi,
We've been asked by field support to create an OpenBMC event log when we detect that we suffered a power blackout, by which I mean the system is up and running, and then the BMC suddenly reboots and when it comes back up main power is off. The issue for it is
https://github.com/ibm-openbmc/dev/issues/677.

Quoted 7 lines

I have a proposal in there for how to detect it in the chassis state manager code, which basically just consists of persisting the chassis power state and then checking it against the actual value on startup, and an error proposed at
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791.

Does anyone have any other ideas or thoughts?  Is it OK if we make this the default behavior, or would it be preferred we enable it with a compile flag?

Sorry for the late replay as I have been out of the office.

Quoted 8 lines
Quoted 3 lines

In my testing on this type of failure, I found that there is a race condition between the Chassis and BMC losing power after the blackout. If the Chassis loses power while the BMC is still active, it can sometimes detect it, change the Chassis State to Off, and persist it before losing power.  When this happens, the next boot will see only the Off state persisted and not detect the blackout.

To work around this, we have added a delay when persisting the Chassis State.  In a normal Off, it will be persisted after the delay; in a blackout off, the delay will prevent the BMC from persisting the Off state allowing the blackout to be detected.

Hi, thanks for the response.
Did you do this in a downstream phosphor-state-manager repository, or somewhere else?  If in phosphor-state-manager, would you consider sharing it?

It's in a downstream x86-power-control repository that I haven't had a chance to push to the upstream x86-power-control, yet.

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

Thanks,
Matt