Skip to contentSkip to conversation

nvme: move stopping keep-alive into nvme_uninit_ctrl()

PATCH Superseded Ming Lei 5 participants 7 messages Aug 9, 2024 – Aug 12, 2024 NVMe

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 nvme: move stopping keep-alive into nvme_uninit_ctrl()

Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") moves starting keep-alive from nvme_start_ctrl() into nvme_init_ctrl_finish(), but don't move stopping keep-alive into nvme_uninit_ctrl(), so keep-alive work can be started and keep pending after failing to start controller, finally use-after-free is triggered if nvme host driver is unloaded.

This patch fixes kernel panic when running nvme/004 in case that connection failure is triggered, by moving stopping keep-alive into nvme_uninit_ctrl().

This way is reasonable because keep-alive is now started in nvme_init_ctrl_finish().

Fixes 4733b65d82bd ("nvme: start keep-alive after admin queue setup")
Cc Hannes Reinecke <[email protected]>
Cc Mark O'Donovan <[email protected]>
---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
index 053d5b4909cd..562afa71ea85 100644
@@ -4612,7 +4612,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
 	nvme_mpath_stop(ctrl);
 	nvme_auth_stop(ctrl);
-	nvme_stop_keep_alive(ctrl);
 	nvme_stop_failfast_work(ctrl);
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fw_act_work);
@@ -4648,6 +4647,7 @@ EXPORT_SYMBOL_GPL(nvme_start_ctrl);
 
 void nvme_uninit_ctrl(struct nvme_ctrl *ctrl)
 {
+	nvme_stop_keep_alive(ctrl);
 	nvme_hwmon_exit(ctrl);
 	nvme_fault_inject_fini(&ctrl->fault_inject);
 	dev_pm_qos_hide_latency_tolerance(ctrl->device);
Signature · 1 line
2.45.2
#3 lore raw
In reply to Ming Lei
Quoted 27 lines · Ming Lei

Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") moves starting keep-alive from nvme_start_ctrl() into nvme_init_ctrl_finish(), but don't move stopping keep-alive into nvme_uninit_ctrl(), so keep-alive work can be started and keep pending after failing to start controller, finally use-after-free is triggered if nvme host driver is unloaded.

This patch fixes kernel panic when running nvme/004 in case that connection failure is triggered, by moving stopping keep-alive into nvme_uninit_ctrl().

This way is reasonable because keep-alive is now started in nvme_init_ctrl_finish().

Fixes 4733b65d82bd ("nvme: start keep-alive after admin queue setup")
Cc Hannes Reinecke <[email protected]>
Cc Mark O'Donovan <[email protected]>
---
  drivers/nvme/host/core.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index 053d5b4909cd..562afa71ea85 100644

drivers/nvme/host/core.c void nvme_stop_ctrl(struct nvme_ctrl *ctrl) · line 4617

Quoted 7 lines
drivers/nvme/host/core.c +0 -1
@@ -4612,7 +4612,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
  {
  	nvme_mpath_stop(ctrl);
  	nvme_auth_stop(ctrl);
-	nvme_stop_keep_alive(ctrl);
  	nvme_stop_failfast_work(ctrl);
  	flush_work(&ctrl->async_event_work);
  	cancel_work_sync(&ctrl->fw_act_work);

Huh? What happened here?
Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") has _exactly_ the same hunk.
Someone else must've changed it afterwards, so please update the 'fixes' tag to refer to the correct commit.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author
#4 lore raw
Quoted 33 lines · Hannes Reinecke
Quoted 32 lines · Ming Lei

Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") moves starting keep-alive from nvme_start_ctrl() into nvme_init_ctrl_finish(), but don't move stopping keep-alive into nvme_uninit_ctrl(), so keep-alive work can be started and keep pending after failing to start controller, finally use-after-free is triggered if nvme host driver is unloaded.

This patch fixes kernel panic when running nvme/004 in case that connection failure is triggered, by moving stopping keep-alive into nvme_uninit_ctrl().

This way is reasonable because keep-alive is now started in nvme_init_ctrl_finish().

Fixes 4733b65d82bd ("nvme: start keep-alive after admin queue setup")
Cc Hannes Reinecke <[email protected]>
Cc Mark O'Donovan <[email protected]>
---
  drivers/nvme/host/core.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -1
index 053d5b4909cd..562afa71ea85 100644
@@ -4612,7 +4612,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
  {
  	nvme_mpath_stop(ctrl);
  	nvme_auth_stop(ctrl);
-	nvme_stop_keep_alive(ctrl);
  	nvme_stop_failfast_work(ctrl);
Quoted 8 lines

drivers/nvme/host/core.c void nvme_stop_ctrl(struct nvme_ctrl *ctrl) · line 4617

Quoted 2 lines
drivers/nvme/host/core.c +0 -0
  	flush_work(&ctrl->async_event_work);
  	cancel_work_sync(&ctrl->fw_act_work);

Huh? What happened here?
Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") has _exactly_ the same hunk.
Someone else must've changed it afterwards, so please update the 'fixes' tag to refer to the correct commit.

It is exactly 4733b65d82bd ("nvme: start keep-alive after admin queue setup"), which moves nvme_start_keep_alive() into nvme_init_ctrl_finish(), but not move nvme_stop_keep_alive() to nvme_uninit_ctrl().

Thanks,
Ming

maintainer
#5 lore raw
In reply to Hannes Reinecke
Quoted 15 lines · Hannes Reinecke

drivers/nvme/host/core.c void nvme_stop_ctrl(struct nvme_ctrl *ctrl) · line 4617

Quoted 8 lines · Ming Lei
drivers/nvme/host/core.c +0 -1
@@ -4612,7 +4612,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
  {
  	nvme_mpath_stop(ctrl);
  	nvme_auth_stop(ctrl);
-	nvme_stop_keep_alive(ctrl);
  	nvme_stop_failfast_work(ctrl);
  	flush_work(&ctrl->async_event_work);
  	cancel_work_sync(&ctrl->fw_act_work);

Huh? What happened here?
Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") has _exactly_ the same hunk.
Someone else must've changed it afterwards, so please update the 'fixes' tag to refer to the correct commit.

Yes, someone did change it :)

commit 3af755a46881c32fecaecfdeaf3a8f0a869deca5
Author: Hannes Reinecke <[email protected]>
Date: Tue Nov 21 09:01:03 2023 +0100

nvme: move nvme_stop_keep_alive() back to original position

So 4733b65d82bd moved it to the wrong place and Ming's patch is the correct one?

#6 lore raw
Quoted 19 lines · Keith Busch
Quoted 15 lines · Hannes Reinecke

drivers/nvme/host/core.c void nvme_stop_ctrl(struct nvme_ctrl *ctrl) · line 4617

Quoted 8 lines · Ming Lei
drivers/nvme/host/core.c +0 -1
@@ -4612,7 +4612,6 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
   {
   	nvme_mpath_stop(ctrl);
   	nvme_auth_stop(ctrl);
-	nvme_stop_keep_alive(ctrl);
   	nvme_stop_failfast_work(ctrl);
   	flush_work(&ctrl->async_event_work);
   	cancel_work_sync(&ctrl->fw_act_work);

Huh? What happened here?
Commit 4733b65d82bd ("nvme: start keep-alive after admin queue setup") has _exactly_ the same hunk.
Someone else must've changed it afterwards, so please update the 'fixes' tag to refer to the correct commit.

Yes, someone did change it :)

Quoted 8 lines

commit 3af755a46881c32fecaecfdeaf3a8f0a869deca5
Author: Hannes Reinecke <[email protected]>
Date: Tue Nov 21 09:01:03 2023 +0100

nvme: move nvme_stop_keep_alive() back to original position

So 4733b65d82bd moved it to the wrong place and Ming's patch is the correct one?

That's perfectly fine, it's just that the 'Fixes' tag doesn't tell the full story.
All I was asking it to update the 'Fixes' tag to the latest patch
(ie 3af755a46881 ("nvme: move nvme_stop_keep_alive() back to original position")) such that the poor people having to do backports can rewind via the 'Fixes' tag and get all necessary patches.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich