Skip to contentSkip to conversation

phosphor-ipmi-flash state recovery

DISCUSSION Patrick Voelker 4 participants 8 messages May 22, 2020 – Jun 3, 2020

lore
Share & tools
lore
#0 lore raw

When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.

When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn't change. I don't have the cleanup-delete option enabled but it doesn't look like that cleans up the state anyhow.

Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don't have a way to obtain the sessionID after the fact. Also the stale session doesn't seem to expire (as mentioned in the readme.md) and I can't find the support for that in the code.

Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?

#1 lore raw

I was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.

Quoted 27 lines · Patrick Voelker

When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.

When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.

Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?

#2 lore raw

Do you have a timeline for upstreaming your changes?

10 minutes seems rather excessive for a session timeout. I’d prefer to see it set to something more like 30 seconds except for the update process itself if it involves writing to flash.

Alternately, what would you think of this option?

· Add a ‘force’ command line override to burn_my_bmc that would cause it to attempt to obtain the current SessionID and do a close on it if the specified image is already active.

· On the BMC side, I’d need to a method to obtain the session ID for a blob. It’d have been perfect to return that in BmcBlobStat’s response but maybe a new command called BmcBlobSession would be appropriate?

From: William Kennington [mailto:[email protected]]
Sent: Friday, May 22, 2020 3:15 PM
To: Patrick Voelker
Cc: OpenBMC ([email protected]); Patrick Venture; Benjamin Fair
Subject: Re: phosphor-ipmi-flash state recovery

I was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.

On Fri, May 22, 2020 at 3:06 PM Patrick Voelker <[email protected]<mailto:[email protected]>> wrote:
When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.

When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.

Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?

#3 lore raw
Quoted 37 lines · Patrick Voelker

Do you have a timeline for upstreaming your changes?

10 minutes seems rather excessive for a session timeout. I’d prefer to see it set to something more like 30 seconds except for the update process itself if it involves writing to flash.

Alternately, what would you think of this option?

· Add a ‘force’ command line override to burn_my_bmc that would cause it to attempt to obtain the current SessionID and do a close on it if the specified image is already active.

· On the BMC side, I’d need to a method to obtain the session ID for a blob. It’d have been perfect to return that in BmcBlobStat’s response but maybe a new command called BmcBlobSession would be appropriate?

From: William Kennington [mailto:[email protected]]
Sent: Friday, May 22, 2020 3:15 PM
To: Patrick Voelker
Cc: OpenBMC ([email protected]); Patrick Venture; Benjamin Fair
Subject: Re: phosphor-ipmi-flash state recovery

I was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.

On Fri, May 22, 2020 at 3:06 PM Patrick Voelker <[email protected]> wrote:

When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.

Quoted 5 lines

When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.

Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

https://github.com/openbmc/phosphor-ipmi-flash/blob/master/bmc/firmware_handler.cpp#L95

You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.

Quoted 6 lines

Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?

#4 lore raw
Quoted 7 lines
Quoted 3 lines

Internally, it looks like I need to get to abortProcess() but to do that I need

to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

Quoted 5 lines

https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95

You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.

As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.

#5 lore raw
Quoted 8 lines · Patrick Voelker
Quoted 8 lines
Quoted 4 lines

Internally, it looks like I need to get to abortProcess() but to do that I need

to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

Quoted 7 lines
Quoted 5 lines

https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95

You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.

As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.

Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.

#6 lore raw

Created https://github.com/openbmc/phosphor-ipmi-flash/issues/7 to track this issue.

Thanks,
Brandon

Quoted 28 lines · Patrick Venture
Quoted 4 lines · Patrick Voelker
Quoted 4 lines
Quoted 4 lines

Internally, it looks like I need to get to abortProcess() but to do

that I need

Quoted 1 line
Quoted 1 line

to close the current session but I don’t have a way to obtain the

sessionID

Quoted 1 line
Quoted 1 line

after the fact. Also the stale session doesn’t seem to expire (as

mentioned

Quoted 9 lines
Quoted 7 lines

in the readme.md) and I can’t find the support for that in the code.

https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95

You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.

As coded, that only works to abort when in in the verificationPending

and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.

Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.

#7 lore raw
Quoted 5 lines · Brandon Kim

Created https://github.com/openbmc/phosphor-ipmi-flash/issues/7 to track this issue.

Thanks,
Brandon

Thanks. I had some time while working on cleaning up things for a different build environment, and dropped in an expiration method (sorry forgot to reply to this email when I did it).

Quoted 25 lines · Patrick Venture
Quoted 23 lines
Quoted 8 lines · Patrick Voelker
Quoted 8 lines
Quoted 4 lines

Internally, it looks like I need to get to abortProcess() but to do that I need

to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.

Quoted 7 lines
Quoted 5 lines

https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95

You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.

As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.

Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.