Skip to contentSkip to conversation

Re-order scsi_remove_host and sas_remove_host in SAS HBA LLDDs

PATCH In review Johannes Thumshirn 4 participants 14 messages Apr 21, 2017 – Apr 22, 2017

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover Re-order scsi_remove_host and sas_remove_host in SAS HBA LLDDs

This series re-orders the calls to scsi_remove_host() and sas_remove_host() in all SAS HBA drivers (apart from mpt3sas which is doing it correctly). This is for two reasons:
1) After the change to recursive removal of sysfs entries, we're trying to remove already removed kobjects when doing a sas_remove_host() _after_ a scsi_remove_host()
2) the documentation mandates it even (becuase of 1)

Unfortunately this does not completely solve issues with recursive sysfs removals in SAS, as libsas has asynchronous behaviour where strong ordering would be needed. But I am working on it and I do know other do as well. So if anyone else (James, Christoph, Bart, I'm looking at you) has an idea, I do have test setups and I'm willing to take input in form of ideas and patches.

I also dropped the SDEV_CANCEL state change for now. We re-evaluate it once we have an idea how to tackle the ordering issues and place it into sas_unregister_ha() as per James' comment.

Thanks,
Johannes

Johannes Thumshirn (5):
scsi: isci: remove the SAS host after the SCSI host
aic94xx: remove the SAS host after the SCSI host
scsi: hisi_sas: remove the SAS host after the SCSI host
mvsas: remove the SAS host after the SCSI host
scsi: pm8001: remove the SAS host after the SCSI host

 drivers/scsi/aic94xx/aic94xx_init.c   | 7 ++++---
 drivers/scsi/hisi_sas/hisi_sas_main.c | 4 ++--
 drivers/scsi/isci/init.c              | 3 ++-
 drivers/scsi/mvsas/mv_init.c          | 6 ++++--
 drivers/scsi/pm8001/pm8001_init.c     | 8 ++++++--
 5 files changed, 18 insertions(+), 10 deletions(-)
Signature · 1 line
2.12.0
#1 lore raw

PATCH v2 1/5 scsi: isci: remove the SAS host after the SCSI host

After commit bcdde7e ("sysfs: make __sysfs_remove_dir() recursive") changed the removal path of kernfs to make it recursive we have to remove the SAS host before the SCSI host or we will see sysfs warnings on not found sysfs groups for kobjects.

---
 drivers/scsi/isci/init.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/isci/init.c +2 -1
index 0b5b5db0d0f8..e5824758aece 100644
@@ -272,10 +272,11 @@ static void isci_unregister(struct isci_host *isci_host)
 		return;
 
 	shost = to_shost(isci_host);
-	scsi_remove_host(shost);
+
 	sas_unregister_ha(&isci_host->sas_ha);
 
 	sas_remove_host(shost);
+	scsi_remove_host(shost);
 	scsi_host_put(shost);
 }
 
Signature · 1 line
2.12.0
#2 lore raw

PATCH v2 2/5 aic94xx: remove the SAS host after the SCSI host

In reply to Johannes Thumshirn

After commit bcdde7e ("sysfs: make __sysfs_remove_dir() recursive") changed the removal path of kernfs to make it recursive we have to remove the SAS host before the SCSI host or we will see sysfs warnings on not found sysfs groups for kobjects.

---
 drivers/scsi/aic94xx/aic94xx_init.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/scsi/aic94xx/aic94xx_init.c +4 -3
index 662b2321d1b0..8b9c6b9ea67d 100644
@@ -701,13 +701,14 @@ static int asd_register_sas_ha(struct asd_ha_struct *asd_ha)
 
 static int asd_unregister_sas_ha(struct asd_ha_struct *asd_ha)
 {
+	struct Scsi_Host *shost = asd_ha->sas_ha.core.shost;
 	int err;
 
-	scsi_remove_host(asd_ha->sas_ha.core.shost);
 	err = sas_unregister_ha(&asd_ha->sas_ha);
 
-	sas_remove_host(asd_ha->sas_ha.core.shost);
-	scsi_host_put(asd_ha->sas_ha.core.shost);
+	sas_remove_host(shost);
+	scsi_remove_host(shost);
+	scsi_host_put(shost);
 
 	kfree(asd_ha->sas_ha.sas_phy);
 	kfree(asd_ha->sas_ha.sas_port);
Signature · 1 line
2.12.0
#3 lore raw

PATCH v2 3/5 scsi: hisi_sas: remove the SAS host after the SCSI host

In reply to Johannes Thumshirn

After commit bcdde7e ("sysfs: make __sysfs_remove_dir() recursive") changed the removal path of kernfs to make it recursive we have to remove the SAS host before the SCSI host or we will see sysfs warnings on not found sysfs groups for kobjects.

---
 drivers/scsi/hisi_sas/hisi_sas_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/scsi/hisi_sas/hisi_sas_main.c +2 -2
index 53637a941b94..9ae39226d454 100644
@@ -1583,9 +1583,9 @@ int hisi_sas_remove(struct platform_device *pdev)
 	struct hisi_hba *hisi_hba = sha->lldd_ha;
 	struct Scsi_Host *shost = sha->core.shost;
 
-	scsi_remove_host(sha->core.shost);
 	sas_unregister_ha(sha);
-	sas_remove_host(sha->core.shost);
+	sas_remove_host(shost);
+	scsi_remove_host(shost);
 
 	hisi_sas_free(hisi_hba);
 	kfree(shost);
Signature · 1 line
2.12.0
#4 lore raw

PATCH v2 4/5 mvsas: remove the SAS host after the SCSI host

In reply to Johannes Thumshirn

After commit bcdde7e ("sysfs: make __sysfs_remove_dir() recursive") changed the removal path of kernfs to make it recursive we have to remove the SAS host before the SCSI host or we will see sysfs warnings on not found sysfs groups for kobjects.

---
 drivers/scsi/mvsas/mv_init.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/scsi/mvsas/mv_init.c +4 -2
index 8280046fd1f0..4592a915c2b9 100644
@@ -634,17 +634,19 @@ static void mvs_pci_remove(struct pci_dev *pdev)
 	unsigned short core_nr, i = 0;
 	struct sas_ha_struct *sha = pci_get_drvdata(pdev);
 	struct mvs_info *mvi = NULL;
+	struct Scsi_Host *shost;
 
 	core_nr = ((struct mvs_prv_info *)sha->lldd_ha)->n_host;
 	mvi = ((struct mvs_prv_info *)sha->lldd_ha)->mvi[0];
+	shost = mvi->shost;
 
 #ifdef CONFIG_SCSI_MVSAS_TASKLET
 	tasklet_kill(&((struct mvs_prv_info *)sha->lldd_ha)->mv_tasklet);
 #endif
 
-	scsi_remove_host(mvi->shost);
 	sas_unregister_ha(sha);
-	sas_remove_host(mvi->shost);
+	sas_remove_host(shost);
+	scsi_remove_host(shost);
 
 	MVS_CHIP_DISP->interrupt_disable(mvi);
 	free_irq(mvi->pdev->irq, sha);
Signature · 1 line
2.12.0
#5 lore raw

PATCH v2 5/5 scsi: pm8001: remove the SAS host after the SCSI host

In reply to Johannes Thumshirn

After commit bcdde7e ("sysfs: make __sysfs_remove_dir() recursive") changed the removal path of kernfs to make it recursive we have to remove the SAS host before the SCSI host or we will see sysfs warnings on not found sysfs groups for kobjects.

---
 drivers/scsi/pm8001/pm8001_init.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/scsi/pm8001/pm8001_init.c +6 -2
index 417368ccb686..e8f3d5ada201 100644
@@ -1086,11 +1086,15 @@ static void pm8001_pci_remove(struct pci_dev *pdev)
 {
 	struct sas_ha_struct *sha = pci_get_drvdata(pdev);
 	struct pm8001_hba_info *pm8001_ha;
+	struct Scsi_Host *shost;
 	int i, j;
 	pm8001_ha = sha->lldd_ha;
-	scsi_remove_host(pm8001_ha->shost);
+	shost = pm8001_ha->shost;
+
 	sas_unregister_ha(sha);
-	sas_remove_host(pm8001_ha->shost);
+	sas_remove_host(shost);
+	scsi_remove_host(shost);
+
 	list_del(&pm8001_ha->list);
 	PM8001_CHIP_DISP->interrupt_disable(pm8001_ha, 0xFF);
 	PM8001_CHIP_DISP->chip_soft_rst(pm8001_ha);
Signature · 1 line
2.12.0
#6 lore raw
In reply to Johannes Thumshirn

Thanks Johannes.

@wangyijing, can you test this patchset please (specifically 3/5)? I know that you have the modified version of libsas which you dabbled with upstreaming.

Quoted 8 lines · Johannes Thumshirn

This series re-orders the calls to scsi_remove_host() and sas_remove_host() in all SAS HBA drivers (apart from mpt3sas which is doing it correctly). This is for two reasons:
1) After the change to recursive removal of sysfs entries, we're trying to remove already removed kobjects when doing a sas_remove_host() _after_ a scsi_remove_host()
2) the documentation mandates it even (becuase of 1)

Quoted 6 lines

Unfortunately this does not completely solve issues with recursive sysfs removals in SAS, as libsas has asynchronous behaviour where strong ordering would be needed. But I am working on it and I do know other do as well. So if anyone else (James, Christoph, Bart, I'm looking at you) has an idea, I do have test setups and I'm willing to take input in form of ideas and patches.

wangyijing already sent an RFC for fixing this issue (mentioned above), which was a signifiagnt rewrite of some of libsas.
I am hoping that he would retry, and that community would support/shepherd this activity, or at least say it will be accepted so effort is not wasted.

Quoted 6 lines

I also dropped the SDEV_CANCEL state change for now. We re-evaluate it once we have an idea how to tackle the ordering issues and place it into sas_unregister_ha() as per James' comment.

Thanks,
Johannes

Thanks

Quoted 15 lines

Johannes Thumshirn (5):
scsi: isci: remove the SAS host after the SCSI host
aic94xx: remove the SAS host after the SCSI host
scsi: hisi_sas: remove the SAS host after the SCSI host
mvsas: remove the SAS host after the SCSI host
scsi: pm8001: remove the SAS host after the SCSI host

 drivers/scsi/aic94xx/aic94xx_init.c   | 7 ++++---
 drivers/scsi/hisi_sas/hisi_sas_main.c | 4 ++--
 drivers/scsi/isci/init.c              | 3 ++-
 drivers/scsi/mvsas/mv_init.c          | 6 ++++--
 drivers/scsi/pm8001/pm8001_init.c     | 8 ++++++--
 5 files changed, 18 insertions(+), 10 deletions(-)
#7 lore raw
Quoted 22 lines · John Garry

Thanks Johannes.

@wangyijing, can you test this patchset please (specifically 3/5)? I know that you have the modified version of libsas which you dabbled with upstreaming.

Quoted 8 lines · Johannes Thumshirn

This series re-orders the calls to scsi_remove_host() and sas_remove_host() in all SAS HBA drivers (apart from mpt3sas which is doing it correctly). This is for two reasons:
1) After the change to recursive removal of sysfs entries, we're trying to remove already removed kobjects when doing a sas_remove_host() _after_ a scsi_remove_host()
2) the documentation mandates it even (becuase of 1)

Quoted 6 lines

Unfortunately this does not completely solve issues with recursive sysfs removals in SAS, as libsas has asynchronous behaviour where strong ordering would be needed. But I am working on it and I do know other do as well. So if anyone else (James, Christoph, Bart, I'm looking at you) has an idea, I do have test setups and I'm willing to take input in form of ideas and patches.

Quoted 4 lines

wangyijing already sent an RFC for fixing this issue (mentioned above), which was a signifiagnt rewrite of some of libsas.
I am hoping that he would retry, and that community would support/shepherd this activity, or at least say it will be accepted so effort is not wasted.

Do you have a link to the series? No problem if not, I'll probably can find it in my archives. Anyways, if Wangyijing is re-sending his patches please Cc me on it.

Thanks,
Johannes

Signature · 6 lines
Johannes Thumshirn                                          Storage
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
#8 lore raw
Quoted 8 lines · Johannes Thumshirn
Quoted 4 lines

wangyijing already sent an RFC for fixing this issue (mentioned above),

Quoted 3 lines

which was a signifiagnt rewrite of some of libsas.
I am hoping that he would retry, and that community would support/shepherd this activity, or at least say it will be accepted so effort is not wasted.

Do you have a link to the series? No problem if not, I'll probably can find it in my archives. Anyways, if Wangyijing is re-sending his patches please Cc me on it.

https://marc.info/?l=linux-scsi&m=147494612310529&w=2

Quoted 2 lines

Thanks,
Johannes

John

#12 lore raw
In reply to Johannes Thumshirn
Quoted 14 lines · Johannes Thumshirn

This series re-orders the calls to scsi_remove_host() and sas_remove_host() in all SAS HBA drivers (apart from mpt3sas which is doing it correctly). This is for two reasons:
1) After the change to recursive removal of sysfs entries, we're trying to remove already removed kobjects when doing a sas_remove_host() _after_ a scsi_remove_host()
2) the documentation mandates it even (becuase of 1)

Unfortunately this does not completely solve issues with recursive sysfs removals in SAS, as libsas has asynchronous behaviour where strong ordering would be needed. But I am working on it and I do know other do as well. So if anyone else (James, Christoph, Bart, I'm looking at you) has an idea, I do have test setups and I'm willing to take input in form of ideas and patches.

Quoted 3 lines

I also dropped the SDEV_CANCEL state change for now. We re-evaluate it once we have an idea how to tackle the ordering issues and place it into sas_unregister_ha() as per James' comment.

Any reason to not just make sas_remove_host call scsi_remove_host to ensure we get the ordering right?

#9 lore raw
In reply to John Garry
Quoted 11 lines · John Garry
Quoted 8 lines · Johannes Thumshirn
Quoted 4 lines

wangyijing already sent an RFC for fixing this issue (mentioned above),

Quoted 3 lines

which was a signifiagnt rewrite of some of libsas.
I am hoping that he would retry, and that community would support/shepherd this activity, or at least say it will be accepted so effort is not wasted.

Do you have a link to the series? No problem if not, I'll probably can find it in my archives. Anyways, if Wangyijing is re-sending his patches please Cc me on it.

https://marc.info/?l=linux-scsi&m=147494612310529&w=2

Please repost the series against the current tree. Also a cover letter with a bit more explanation would be good.

#13 lore raw
In reply to Christoph Hellwig
Quoted 3 lines · Christoph Hellwig

Any reason to not just make sas_remove_host call scsi_remove_host to ensure we get the ordering right?

No other than "haven't thought of it". But let me double check the LLDDs fist.

Signature · 6 lines
Johannes Thumshirn                                          Storage
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
#10 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · John Garry
Quoted 8 lines · Johannes Thumshirn
Quoted 4 lines

wangyijing already sent an RFC for fixing this issue (mentioned above),

Quoted 3 lines

which was a signifiagnt rewrite of some of libsas.
I am hoping that he would retry, and that community would support/shepherd this activity, or at least say it will be accepted so effort is not wasted.

Do you have a link to the series? No problem if not, I'll probably can find it in my archives. Anyways, if Wangyijing is re-sending his patches please Cc me on it.

Quoted 5 lines
Quoted 1 line

https://marc.info/?l=linux-scsi&m=147494612310529&w=2

Please repost the series against the current tree. Also a cover letter with a bit more explanation would be good.

Great, I think that my colleagues can send an updated version next week.

Much appreciated,
John

Quoted 2 lines

.

#11 lore raw
Quoted 7 lines
Quoted 5 lines

Please repost the series against the current tree. Also a cover letter with a bit more explanation would be good.

Great, I think that my colleagues can send an updated version next week.

I will send a new version with a cover letter these days, before that, we need to do a detail test on real hardware based the current mainline.

Thannks!
Yijing.

Quoted 10 lines

Much appreciated,
John

Quoted 2 lines

.

.