phosphor-ipmi-flash state recovery
When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.
When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn't change. I don't have the cleanup-delete option enabled but it doesn't look like that cleans up the state anyhow.
Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don't have a way to obtain the sessionID after the fact. Also the stale session doesn't seem to expire (as mentioned in the readme.md) and I can't find the support for that in the code.
Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?
I was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.
Quoted 27 lines · Patrick Voelker
When running
burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.When attempting to run the update again I can see that the /flash/active/image blob is present.
burn_my_bmcopens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.Internally, it looks like I need to get to
abortProcess()but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?
Do you have a timeline for upstreaming your changes?
10 minutes seems rather excessive for a session timeout. I’d prefer to see it set to something more like 30 seconds except for the update process itself if it involves writing to flash.
Alternately, what would you think of this option?
· Add a ‘force’ command line override to burn_my_bmc that would cause it to attempt to obtain the current SessionID and do a close on it if the specified image is already active.
· On the BMC side, I’d need to a method to obtain the session ID for a blob. It’d have been perfect to return that in BmcBlobStat’s response but maybe a new command called BmcBlobSession would be appropriate?
From: William Kennington [mailto:[email protected]]
Sent: Friday, May 22, 2020 3:15 PM
To: Patrick Voelker
Cc: OpenBMC ([email protected]); Patrick Venture; Benjamin Fair
Subject: Re: phosphor-ipmi-flash state recovery
I was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.
On Fri, May 22, 2020 at 3:06 PM Patrick Voelker <[email protected]<mailto:[email protected]>> wrote:
When running burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.
When attempting to run the update again I can see that the /flash/active/image blob is present. burn_my_bmc opens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.
Internally, it looks like I need to get to abortProcess() but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.
Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?
Quoted 37 lines · Patrick Voelker
Do you have a timeline for upstreaming your changes?
10 minutes seems rather excessive for a session timeout. I’d prefer to see it set to something more like 30 seconds except for the update process itself if it involves writing to flash.
Alternately, what would you think of this option?
· Add a ‘force’ command line override to
burn_my_bmcthat would cause it to attempt to obtain the current SessionID and do a close on it if the specified image is already active.· On the BMC side, I’d need to a method to obtain the session ID for a blob. It’d have been perfect to return that in BmcBlobStat’s response but maybe a new command called BmcBlobSession would be appropriate?
From: William Kennington [mailto:[email protected]]
Sent: Friday, May 22, 2020 3:15 PM
To: Patrick Voelker
Cc: OpenBMC ([email protected]); Patrick Venture; Benjamin Fair
Subject: Re: phosphor-ipmi-flash state recoveryI was working on a change to fix this a couple of weeks ago and implement the needed expiry / cancellation mechanisms to make everything happy. Right now the easiest thing you can is reset the BMC or just the ipmi daemon.
On Fri, May 22, 2020 at 3:06 PM Patrick Voelker <[email protected]> wrote:
When running
burn_my_bmc, if I exit the program during image upload with ctrl-c, it seems that the bmc gets left in a state that is difficult to recover from.
Quoted 5 lines
When attempting to run the update again I can see that the /flash/active/image blob is present.
burn_my_bmcopens the cleanup blob, commits it, and then closes it but the state doesn’t change. I don’t have the cleanup-delete option enabled but it doesn’t look like that cleans up the state anyhow.Internally, it looks like I need to get to
abortProcess()but to do that I need to close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.
https://github.com/openbmc/phosphor-ipmi-flash/blob/master/bmc/firmware_handler.cpp#L95
You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.
Quoted 6 lines
Can you give me a pointer on the best known way to recover from this scenario without rebooting the BMC?
Quoted 7 lines
Quoted 3 lines
Internally, it looks like I need to get to
abortProcess()but to do that I needto close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.
Quoted 5 lines
https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.
As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.
Quoted 8 lines · Patrick Voelker
Quoted 8 lines
Quoted 4 lines
Internally, it looks like I need to get to
abortProcess()but to do that I needto close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.
Quoted 7 lines
Quoted 5 lines
https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.
As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.
Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.
Created https://github.com/openbmc/phosphor-ipmi-flash/issues/7 to track this issue.
Thanks,
Brandon
Quoted 28 lines · Patrick Venture
Quoted 4 lines · Patrick Voelker
Quoted 4 lines
Quoted 4 lines
Internally, it looks like I need to get to
abortProcess()but to dothat I need
Quoted 1 line
Quoted 1 line
to close the current session but I don’t have a way to obtain the
sessionID
Quoted 1 line
Quoted 1 line
after the fact. Also the stale session doesn’t seem to expire (as
mentioned
Quoted 9 lines
Quoted 7 lines
in the readme.md) and I can’t find the support for that in the code.
https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.
As coded, that only works to abort when in in the verificationPending
and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.
Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.
Quoted 5 lines · Brandon Kim
Created https://github.com/openbmc/phosphor-ipmi-flash/issues/7 to track this issue.
Thanks,
Brandon
Thanks. I had some time while working on cleaning up things for a different build environment, and dropped in an expiration method (sorry forgot to reply to this email when I did it).
Quoted 25 lines · Patrick Venture
Quoted 23 lines
Quoted 8 lines · Patrick Voelker
Quoted 8 lines
Quoted 4 lines
Internally, it looks like I need to get to
abortProcess()but to do that I needto close the current session but I don’t have a way to obtain the sessionID after the fact. Also the stale session doesn’t seem to expire (as mentioned in the readme.md) and I can’t find the support for that in the code.
Quoted 7 lines
Quoted 5 lines
https://github.com/openbmc/phosphor-ipmi-
flash/blob/master/bmc/firmware_handler.cpp#L95You should be able to just delete any of the blob Id paths, such as the active blob path, and that should trigger what you want.
As coded, that only works to abort when in in the verificationPending and updatePending states. That won't work if it's stuck in the middle of an aborted transfer with a session that's still open.
Right - because handlers don't control sessions, those are owned by the blob manager that's routing the commands. - then we'll need a patch to handle expiration, per the email chain - Sorry, I misunderstood the circumstance.