Drivers: hv: vmbus: hibernation: do not hang forever in vmbus_bus_resume()
Subsystems (2)
Dexuan Cui
maintainerPATCH 1/1 Drivers: hv: vmbus: hibernation: do not hang forever in vmbus_bus_resume()
After we Stop and later Start a VM that uses Accelerated Networking (NIC SR-IOV), currently the VF vmbus device's Instance GUID can change, so after vmbus_bus_resume() -> vmbus_request_offers(), vmbus_onoffer() can not find the original vmbus channel of the VF, and hence we can't complete() vmbus_connection.ready_for_resume_event in check_ready_for_resume_event(), and the VM hangs in vmbus_bus_resume() forever.
Fix the issue by adding a timeout, so the resuming can still succeed, and the saved state is not lost, and according to my test, the user can disable Accelerated Networking and then will be able to SSH into the VM for further recovery. Also prevent the VM in question from suspending again.
The host will be fixed so in future the Instance GUID will stay the same across hibernation.
--- drivers/hv/vmbus_drv.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/hv/vmbus_drv.c +7 -2
@@ -2382,7 +2382,10 @@ static int vmbus_bus_suspend(struct device *dev)if (atomic_read (&vmbus_connection.nr_chan_close_on_suspend) >0 )wait_for_completion (&vmbus_connection.ready_for_suspend_event); -WARN_ON (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) !=0 ); +if (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) !=0 ) { +pr_err ("Can not suspend due to a previous failed resuming\n" ); +return -EBUSY ; + }mutex_lock (&vmbus_connection.channel_mutex);
@@ -2456,7 +2459,9 @@ static int vmbus_bus_resume(struct device *dev)vmbus_request_offers (); -wait_for_completion (&vmbus_connection.ready_for_resume_event); +if (wait_for_completion_timeout ( + &vmbus_connection.ready_for_resume_event,10 *HZ ) ==0 ) +pr_err ("Some vmbus device is missing after suspending?\n" );/* Reset the event for the next suspend. */ reinit_completion (&vmbus_connection.ready_for_suspend_event);
Signature · 1 line
2.19.1
From: Dexuan Cui <[email protected]> Sent: Friday, September 4, 2020 7:56 PM
Quoted 13 lines
After we Stop and later Start a VM that uses Accelerated Networking (NIC SR-IOV), currently the VF vmbus device's Instance GUID can change, so after
vmbus_bus_resume()->vmbus_request_offers(),vmbus_onoffer()can not find the original vmbus channel of the VF, and hence we can'tcomplete()vmbus_connection.ready_for_resume_event incheck_ready_for_resume_event(), and the VM hangs invmbus_bus_resume()forever.Fix the issue by adding a timeout, so the resuming can still succeed, and the saved state is not lost, and according to my test, the user can disable Accelerated Networking and then will be able to SSH into the VM for further recovery. Also prevent the VM in question from suspending again.
Quoted 8 lines
The host will be fixed so in future the Instance GUID will stay the same across hibernation.
Fixes d8bd2d442bb2 ("Drivers: hv: vmbus: Resume after fixing up old primary channels")Signed-off-by Dexuan Cui <[email protected]>--- drivers/hv/vmbus_drv.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
Quoted 30 lines
drivers/hv/vmbus_drv.c +7
-3@@ -2382,7 +2382,10 @@ static int vmbus_bus_suspend(struct device *dev)if (atomic_read (&vmbus_connection.nr_chan_close_on_suspend) >0 )wait_for_completion (&vmbus_connection.ready_for_suspend_event); -WARN_ON (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) !=0 ); +if (atomic_read (&vmbus_connection.nr_chan_fixup_on_resume) !=0 ) { +pr_err ("Can not suspend due to a previous failed resuming\n" ); +return -EBUSY ; + }mutex_lock (&vmbus_connection.channel_mutex);@@ -2456,7 +2459,9 @@ static int vmbus_bus_resume(struct device *dev)vmbus_request_offers (); -wait_for_completion (&vmbus_connection.ready_for_resume_event); +if (wait_for_completion_timeout ( + &vmbus_connection.ready_for_resume_event,10 *HZ ) ==0 ) +pr_err ("Some vmbus device is missing after suspending?\n" );/* Reset the event for the next suspend. */ reinit_completion (&vmbus_connection.ready_for_suspend_event); --2.19.1
Quoted 17 lines · Michael Kelley
From: Dexuan Cui <[email protected]> Sent: Friday, September 4, 2020 7:56 PM
Quoted 16 lines
After we Stop and later Start a VM that uses Accelerated Networking (NIC SR-IOV), currently the VF vmbus device's Instance GUID can change, so after
vmbus_bus_resume()->vmbus_request_offers(),vmbus_onoffer()can not find the original vmbus channel of the VF, and hence we can'tcomplete()vmbus_connection.ready_for_resume_event incheck_ready_for_resume_event(), and the VM hangs invmbus_bus_resume()forever.Fix the issue by adding a timeout, so the resuming can still succeed, and the saved state is not lost, and according to my test, the user can disable Accelerated Networking and then will be able to SSH into the VM for further recovery. Also prevent the VM in question from suspending again.
The host will be fixed so in future the Instance GUID will stay the same across hibernation.
Quoted 8 lines
Quoted 5 lines
Fixes d8bd2d442bb2 ("Drivers: hv: vmbus: Resume after fixing up old primary channels")Signed-off-by Dexuan Cui <[email protected]>--- drivers/hv/vmbus_drv.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)Reviewed-by Michael Kelley <[email protected]>
Applied to hyperv-fixes. Thanks.