Skip to contentSkip to conversation

scsi: core: put LLD module refcnt after SCSI device is released

PATCH Superseded Ming Lei 2 participants 8 messages Sep 30, 2021 SCSI

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH v2 1/1 scsi: core: put LLD module refcnt after SCSI device is released

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/scsi.c +7 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);
 }
 EXPORT_SYMBOL(scsi_device_put);
 
drivers/scsi/scsi_sysfs.c +10 -0
index 86793259e541..ba6defac91ae 100644
@@ -449,9 +449,16 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 	struct scsi_vpd *vpd_pg80 = NULL, *vpd_pg83 = NULL;
 	struct scsi_vpd *vpd_pg0 = NULL, *vpd_pg89 = NULL;
 	unsigned long flags;
+	struct module *mod;
+	bool put_mod = false;
 
 	sdev = container_of(work, struct scsi_device, ew.work);
 
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0) {
+		put_mod = true;
+		mod = sdev->host->hostt->module;
+	}
+
 	scsi_dh_release_device(sdev);
 
 	parent = sdev->sdev_gendev.parent;
@@ -502,6 +509,9 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 
 	if (parent)
 		put_device(parent);
+
+	if (put_mod)
+		module_put(mod);
 }
 
 static void scsi_device_dev_release(struct device *dev)
include/scsi/scsi_device.h +2 -0
index 430b73bd02ac..76268c473c22 100644
@@ -111,6 +111,8 @@ struct scsi_device {
 	struct sbitmap budget_map;
 	atomic_t device_blocked;	/* Device returned QUEUE_FULL. */
 
+	atomic_t put_dev_cnt;	/* increased by 1 when we are putting device */
+
 	atomic_t restarts;
 	spinlock_t list_lock;
 	struct list_head starved_entry;
Signature · 1 line
2.31.1
author
#1 lore raw
Quoted 28 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/scsi.c +0 -0
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 10 lines
drivers/scsi/scsi.c +7 -1
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

oops, sdev can be freed now, so this approach isn't good too, :-(

Will think further about the solution.

thanks,
Ming

#2 lore raw
In reply to Ming Lei
Quoted 26 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/scsi.c +0 -0
index b241f9e3885c..b6612161587f 100644

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 7 lines
drivers/scsi/scsi.c +3 -1
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 5 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

thanks,

greg k-h

author
#3 lore raw
Quoted 22 lines · Greg Kroah-Hartman
Quoted 21 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
Quoted 15 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 12 lines
drivers/scsi/scsi.c +3 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

The reason is to make sure that the LLD module is only put from either scsi_device_put() and scsi_device_dev_release_usercontext().

Quoted 8 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 6 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

module refcnt is grabbed in scsi_device_get(), so it is valid.

Quoted 5 lines

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

See it from the commit log:

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

and the upstream report on the issue:

https://lore.kernel.org/linux-block/CAHj4cs8XNtkzbbiLnFmVu82wYeQpLkVp6_wCtrnbhODay+OP9w@mail.gmail.com/

Thanks,
Ming

#4 lore raw
Quoted 51 lines · Ming Lei
Quoted 22 lines · Greg Kroah-Hartman
Quoted 21 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
Quoted 15 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 12 lines
drivers/scsi/scsi.c +3 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

The reason is to make sure that the LLD module is only put from either scsi_device_put() and scsi_device_dev_release_usercontext().

Quoted 8 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 6 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

Quoted 1 line

module refcnt is grabbed in scsi_device_get(), so it is valid.

Then you don't need the extra atomic variable.

Quoted 11 lines
Quoted 5 lines

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

See it from the commit log:

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

What is "hostt"?

Quoted 2 lines

So put LLD module refcnt after SCSI device is released.

Why not just drop it explicitly when you drop the reference count of the device object? Like you tried to do here, but no need for the extra atomic variable.

thanks,

greg k-h

author
#5 lore raw
Quoted 62 lines · Greg Kroah-Hartman
Quoted 51 lines · Ming Lei
Quoted 22 lines · Greg Kroah-Hartman
Quoted 21 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
Quoted 15 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 12 lines
drivers/scsi/scsi.c +3 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

The reason is to make sure that the LLD module is only put from either scsi_device_put() and scsi_device_dev_release_usercontext().

Quoted 8 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 6 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

Quoted 1 line

module refcnt is grabbed in scsi_device_get(), so it is valid.

Then you don't need the extra atomic variable.

Quoted 6 lines
Quoted 5 lines

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

Quoted 7 lines
Quoted 5 lines

See it from the commit log:

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

What is "hostt"?

hostt is 'struct scsi_host_template' which is defined in LLD module, and often allocated as static global variable, that is what try_get_module() tries to protect.

Quoted 7 lines
Quoted 3 lines

So put LLD module refcnt after SCSI device is released.

Why not just drop it explicitly when you drop the reference count of the device object? Like you tried to do here, but no need for the extra atomic variable.

scsi_device_dev_release_usercontext() may be scheduled via schedule_work from the device object's release handler for releasing the scsi_device, which may trigger scsi host's release handler in which hostt is required.

If we simply call module_put() after put_device() simply, the module refcnt may be dropped earlier than running scsi_device_dev_release_usercontext(), then the kernel panic still can't be addressed.

Thanks,
Ming

#6 lore raw
Quoted 79 lines · Ming Lei
Quoted 62 lines · Greg Kroah-Hartman
Quoted 51 lines · Ming Lei
Quoted 22 lines · Greg Kroah-Hartman
Quoted 21 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
Quoted 15 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 12 lines
drivers/scsi/scsi.c +3 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

The reason is to make sure that the LLD module is only put from either scsi_device_put() and scsi_device_dev_release_usercontext().

Quoted 8 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 6 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

Quoted 1 line

module refcnt is grabbed in scsi_device_get(), so it is valid.

Then you don't need the extra atomic variable.

Quoted 6 lines
Quoted 5 lines

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

Quoted 7 lines
Quoted 5 lines

See it from the commit log:

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

What is "hostt"?

hostt is 'struct scsi_host_template' which is defined in LLD module, and often allocated as static global variable, that is what try_get_module() tries to protect.

Quoted 4 lines
Quoted 3 lines

So put LLD module refcnt after SCSI device is released.

Quoted 7 lines
Quoted 3 lines

Why not just drop it explicitly when you drop the reference count of the device object? Like you tried to do here, but no need for the extra atomic variable.

scsi_device_dev_release_usercontext() may be scheduled via schedule_work from the device object's release handler for releasing the scsi_device, which may trigger scsi host's release handler in which hostt is required.

If a release handler can be called from the device release function, then that is when you need to drop the reference, after that function is finished being called, right?

thanks,

greg k-h

author
#7 lore raw
Quoted 84 lines · Greg Kroah-Hartman
Quoted 83 lines · Ming Lei
Quoted 62 lines · Greg Kroah-Hartman
Quoted 51 lines · Ming Lei
Quoted 22 lines · Greg Kroah-Hartman
Quoted 21 lines · Ming Lei

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

So put LLD module refcnt after SCSI device is released.

The real release handler can be run from wq context in case of in_interrupt(), so add one atomic counter for serializing putting module via current and wq context. This way is fine since we don't call scsi_device_put() in fast IO path.

---
 drivers/scsi/scsi.c        |  8 +++++++-
 drivers/scsi/scsi_sysfs.c  | 10 ++++++++++
 include/scsi/scsi_device.h |  2 ++
 3 files changed, 19 insertions(+), 1 deletion(-)
Quoted 15 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 558

Quoted 12 lines
drivers/scsi/scsi.c +3 -1
index b241f9e3885c..b6612161587f 100644
@@ -553,8 +553,14 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
+	atomic_inc(&sdev->put_dev_cnt);

Ick, no! Why are you making a new lock and reference count for no reason?

The reason is to make sure that the LLD module is only put from either scsi_device_put() and scsi_device_dev_release_usercontext().

Quoted 8 lines

drivers/scsi/scsi.c EXPORT_SYMBOL(scsi_device_get); · line 563

Quoted 6 lines
drivers/scsi/scsi.c +4 -0
+
 	put_device(&sdev->sdev_gendev);
+
+	if (atomic_dec_if_positive(&sdev->put_dev_cnt) >= 0)
+		module_put(mod);

How do you know if your module pointer is still valid here?

Quoted 1 line

module refcnt is grabbed in scsi_device_get(), so it is valid.

Then you don't need the extra atomic variable.

Quoted 6 lines
Quoted 5 lines

Why do you care?

What problem are you trying to solve and why is it unique to scsi devices?

Quoted 7 lines
Quoted 5 lines

See it from the commit log:

SCSI host release is triggered when SCSI device is freed, and we have to make sure that LLD module won't be unloaded before SCSI host instance is released because shost->hostt is required in host release handler.

What is "hostt"?

hostt is 'struct scsi_host_template' which is defined in LLD module, and often allocated as static global variable, that is what try_get_module() tries to protect.

Quoted 7 lines
Quoted 3 lines

So put LLD module refcnt after SCSI device is released.

Why not just drop it explicitly when you drop the reference count of the device object? Like you tried to do here, but no need for the extra atomic variable.

Quoted 7 lines
Quoted 3 lines

scsi_device_dev_release_usercontext() may be scheduled via schedule_work from the device object's release handler for releasing the scsi_device, which may trigger scsi host's release handler in which hostt is required.

If a release handler can be called from the device release function, then that is when you need to drop the reference, after that function is finished being called, right?

No.

Not like device object refcnt, module refcnt is special, it is grabbed when someone is using the device, then module won't be unloaded when there is active user. When no one uses the device, we should allow rmmod to unload the module.

SCSI stack models the device uses via scsi_device_get() and scsi_device_put(), such as, when one disk is opened, scsi_device_get() is called, and scsi_device_put() is called when the disk is closed.

Wrt. this issue, we can think that the LLD module holds one device refcnt of the scsi_host, and the refcnt is dropped when running module_exit(). But one disk attached to the host may still be opened by userspace, so the scsi_host won't be released until the disk is closed, and the LLD module won't be unloaded until the disk is closed too. When the disk is closed, scsi_device_put() is called and module_put() can unload the LLD module immediately if rmmod is started.

So what the patch is doing is that we need to make sure that module refcnt is put after the device is released since the host is released from its release handler directly if rmmod has been started.

Thanks,
Ming