Skip to contentSkip to conversation

scsi: enable reserved commands for LLDDs

RFC PATCH John Garry 2 participants 8 messages Aug 23, 2019 – Mar 10, 2020 1 message not in the archive SCSI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

John Garry

author
#1 lore raw
Quoted 16 lines · Hannes Reinecke

Hi all,

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

I was wondering if you have any plans to progress this series?

I don't mind helping out...

Cheers,
John

Quoted 55 lines

Hannes Reinecke (24):
block: disable elevator for reserved tags
scsi: add scsi_{get,put}_reserved_cmd()
scsi: add 'nr_reserved_cmds' field to the SCSI host template
csiostor: use reserved command for LUN reset
scsi: add scsi_cmd_from_priv()
virtio_scsi: use reserved commands for TMF
scsi: add host tagset busy iterator
fnic: use reserved commands
fnic: use scsi_host_tagset_busy_iter() to traverse commands
scsi: allocate separate queue for reserved commands
scsi: add scsi_host_get_reserved_cmd()
hpsa: move hpsa_hba_inquiry after scsi_add_host()
hpsa: use reserved commands
hpsa: use blk_mq_tagset_busy_iter() to traverse outstanding commands
hpsa: drop refcount field from CommandList
snic: use reserved commands
snic: use tagset iter for traversing commands
scsi: Implement scsi_is_reserved_cmd()
aacraid: move scsi_add_host()
aacraid: use private commands
aacraid: replace cmd_list with scsi_host_tagset_busy_iter()
aacraid: use scsi_host_tagset_busy_iter() to traverse outstanding
commands
dpt_i2o: drop cmd_list usage
scsi: drop scsi command list

 block/blk-mq.c                    |  22 +-
 drivers/scsi/aacraid/aachba.c     | 125 +++---
 drivers/scsi/aacraid/aacraid.h    |   6 +-
 drivers/scsi/aacraid/comminit.c   |  28 +-
 drivers/scsi/aacraid/commsup.c    | 134 +++---
 drivers/scsi/aacraid/linit.c      | 302 +++++++------
 drivers/scsi/csiostor/csio_init.c |   3 +-
 drivers/scsi/csiostor/csio_scsi.c |  48 +-
 drivers/scsi/dpt_i2o.c            |  23 +-
 drivers/scsi/fnic/fnic_scsi.c     | 916 ++++++++++++++++----------------------
 drivers/scsi/hosts.c              |  27 ++
 drivers/scsi/hpsa.c               | 310 ++++++-------
 drivers/scsi/hpsa.h               |   1 -
 drivers/scsi/hpsa_cmd.h           |   1 -
 drivers/scsi/scsi.c               |   1 -
 drivers/scsi/scsi_lib.c           |  69 +--
 drivers/scsi/scsi_scan.c          |   1 -
 drivers/scsi/snic/snic.h          |   2 +-
 drivers/scsi/snic/snic_main.c     |   3 +
 drivers/scsi/snic/snic_scsi.c     | 502 ++++++++++-----------
 drivers/scsi/virtio_scsi.c        | 100 ++---
 include/linux/blk-mq.h            |   2 +
 include/scsi/scsi_cmnd.h          |  12 +-
 include/scsi/scsi_device.h        |   1 -
 include/scsi/scsi_host.h          |  17 +-
 include/scsi/scsi_tcq.h           |  30 ++
 26 files changed, 1311 insertions(+), 1375 deletions(-)
#2 lore raw
Quoted 17 lines · John Garry
Quoted 16 lines · Hannes Reinecke

Hi all,

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 9 lines
Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

I was wondering if you have any plans to progress this series?

I don't mind helping out...

Thanks for the reminder.
I'll need to re-evaluate this where we stand now with shared tags; this patchset partially relies on them.
Will be sending an updated patchset.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      Teamlead Storage & Networking
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 247165 (AG München), GF: Felix Imendörffer
author
#3 lore raw
Quoted 21 lines · Hannes Reinecke
Quoted 20 lines · John Garry
Quoted 18 lines · Hannes Reinecke

Hi all,

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

As usual, comments and reviews are welcome.

Quoted 7 lines
Quoted 6 lines

Hi Hannes,

I was wondering if you have any plans to progress this series?

I don't mind helping out...

Thanks for the reminder.

Hi Hannes,

Quoted 1 line

I'll need to re-evaluate this where we stand now with shared tags;

As Ming Lei mentioned in https://www.spinics.net/lists/linux-block/msg43779.html, the future for hostwide shared tags doesn't look bright. I would tend to agree.

Quoted 1 line

this patchset partially relies on them.

In which way? I didn't think/hope it did.

Quoted 1 line

Will be sending an updated patchset.

For me, the way I see forward is to upstream this series, in addition to Ming's, linked above.

As for LLDDs and the unique tag problem, I suggest that they use sbitmap to generate the tag internally if they want to expose multiple queues. From our testing, using managed interrupts + sbitmap still far outperforms non-managed interrupts.

This approach would mean that we can still revisit hostwide shared tags or other some other approach in future.

Thanks,
John

Quoted 5 lines

Cheers,

Hannes

#4 lore raw
Quoted 47 lines · John Garry
Quoted 21 lines · Hannes Reinecke
Quoted 20 lines · John Garry
Quoted 18 lines · Hannes Reinecke

Hi all,

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

As usual, comments and reviews are welcome.

Quoted 7 lines
Quoted 6 lines

Hi Hannes,

I was wondering if you have any plans to progress this series?

I don't mind helping out...

Thanks for the reminder.

Hi Hannes,

Quoted 1 line

I'll need to re-evaluate this where we stand now with shared tags;

As Ming Lei mentioned in
https://www.spinics.net/lists/linux-block/msg43779.html, the future for hostwide shared tags doesn't look bright. I would tend to agree.

Quoted 1 line

this patchset partially relies on them.

In which way? I didn't think/hope it did.

Quoted 1 line

Will be sending an updated patchset.

For me, the way I see forward is to upstream this series, in addition to Ming's, linked above.

Quoted 8 lines

As for LLDDs and the unique tag problem, I suggest that they use sbitmap to generate the tag internally if they want to expose multiple queues. From our testing, using managed interrupts + sbitmap still far outperforms non-managed interrupts.

This approach would mean that we can still revisit hostwide shared tags or other some other approach in future.

Hmm. True. Will be checking.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      Teamlead Storage & Networking
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 247165 (AG München), GF: Felix Imendörffer
author
#5 lore raw
In reply to John Garry
Quoted 15 lines · John Garry
Quoted 15 lines

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 3 lines
Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

JFYI, I have rebased this series to 5.6-rc4 here:

https://github.com/hisilicon/kernel-dev/tree/private-topic-sas-5.6-resv-commands-v1

I am interested in enabling this for libsas and associated HBAs, so there are some patches on top for that.

No review comments have been addressed, apart from removing "block: disable elevator for reserved tags"

Please let me know your plan for this series.

Thanks,
John

Quoted 22 lines

I was wondering if you have any plans to progress this series?

I don't mind helping out...

Cheers,
John

Quoted 14 lines

Hannes Reinecke (24):
  block: disable elevator for reserved tags
  scsi: add scsi_{get,put}_reserved_cmd()
  scsi: add 'nr_reserved_cmds' field to the SCSI host template
  csiostor: use reserved command for LUN reset
  scsi: add scsi_cmd_from_priv()
  virtio_scsi: use reserved commands for TMF
  scsi: add host tagset busy iterator
  fnic: use reserved commands
  fnic: use scsi_host_tagset_busy_iter() to traverse commands
  scsi: allocate separate queue for reserved commands
  scsi: add scsi_host_get_reserved_cmd()
  hpsa: move hpsa_hba_inquiry after scsi_add_host()
  hpsa: use reserved commands

#6 lore raw
Quoted 26 lines · John Garry
Quoted 15 lines · John Garry
Quoted 15 lines

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 3 lines
Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

JFYI, I have rebased this series to 5.6-rc4 here:

https://github.com/hisilicon/kernel-dev/tree/private-topic-sas-5.6-resv-commands-v1

Quoted 8 lines

I am interested in enabling this for libsas and associated HBAs, so there are some patches on top for that.

No review comments have been addressed, apart from removing "block:
disable elevator for reserved tags"

Please let me know your plan for this series.

The plan is to wait for the shared tagset patchset to go in first :-)

Without it we'll be running on one tag bitmap per queue, and we risk duplicate tags when sending down reserved commands :-(

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
author
#7 lore raw
Quoted 30 lines · Hannes Reinecke
Quoted 29 lines · John Garry
Quoted 15 lines · John Garry
Quoted 15 lines

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 3 lines
Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

JFYI, I have rebased this series to 5.6-rc4 here:

https://github.com/hisilicon/kernel-dev/tree/private-topic-sas-5.6-resv-commands-v1

I am interested in enabling this for libsas and associated HBAs, so there are some patches on top for that.

Quoted 7 lines
Quoted 5 lines

No review comments have been addressed, apart from removing "block:
disable elevator for reserved tags"

Please let me know your plan for this series.

The plan is to wait for the shared tagset patchset to go in first :-)

I think that one should go last. Or, to be more specific, switching the drivers to expose multiple queues - which is part of that series - should go last.

Quoted 2 lines

Without it we'll be running on one tag bitmap per queue, and we risk duplicate tags when sending down reserved commands :-(

Until we expose multiple queues, we still have a single HW queue from blk-mq perspective, so will only have a single tagset per HBA, so no chance of duplicate tags.

This series should be able to go in without dependencies.

Thanks,
John

author
#8 lore raw
Quoted 40 lines · John Garry
Quoted 31 lines · Hannes Reinecke
Quoted 30 lines · John Garry
Quoted 15 lines · John Garry
Quoted 15 lines

quite some drivers use internal commands for various purposes, most commonly sending TMFs or querying the HBA status.
While these commands use the same submission mechanism than normal I/O commands, they will not be counted as outstanding commands, requiring those drivers to implement their own mechanism to figure out outstanding commands.
This patchset enables the use of reserved tags for the SCSI midlayer, enabling LLDDs to rely on the block layer for tracking outstanding commands.
More importantly, it allows LLDD to request a valid tag from the block layer without having to implement some tracking mechanism within the driver. This removes quite some hacks which were required for some drivers (eg. fnic or snic).

Quoted 3 lines
Quoted 2 lines

As usual, comments and reviews are welcome.

Hi Hannes,

JFYI, I have rebased this series to 5.6-rc4 here:

https://github.com/hisilicon/kernel-dev/tree/private-topic-sas-5.6-resv-commands-v1

I am interested in enabling this for libsas and associated HBAs, so there are some patches on top for that.

Quoted 7 lines
Quoted 5 lines

No review comments have been addressed, apart from removing "block:
disable elevator for reserved tags"

Please let me know your plan for this series.

The plan is to wait for the shared tagset patchset to go in first :-)

Quoted 8 lines

I think that one should go last. Or, to be more specific, switching the drivers to expose multiple queues - which is part of that series - should go last.

Quoted 2 lines

Without it we'll be running on one tag bitmap per queue, and we risk duplicate tags when sending down reserved commands :-(

Until we expose multiple queues, we still have a single HW queue from blk-mq perspective, so will only have a single tagset per HBA, so no chance of duplicate tags.

This series should be able to go in without dependencies.

BTW, for the other SCSI HBA drivers which use libsas, it would be better if they could stop managing their own tags and switch to use the request tag. That would mean that we need to reserve some tags, which is currently not done in those drivers (pm8xxx, anyway).

If I were to make this change, are there mvsas and aic9xxx cards around to test? I have a pm8xxx card.

Thanks,
John