Skip to content

Native SCSI Multipath support

John Garry dm-devel linux-nvme linux-scsi 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest29 patches 11 repliesIn review
  2. v1 RFC1 patch 3 repliesSuperseded

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/29scsi: core: Clear cmd->flags for terminal queuing errordrivers/scsi/scsi_lib.c+2 -23 replies
2/29libmultipath: Add initial frameworkinclude/linux/multipath.h lib/Kconfig lib/Makefile +1 more+104 -00 replies
3/29libmultipath: Add basic gendisk supportinclude/linux/multipath.h lib/multipath.c+141 -00 replies
4/29libmultipath: Add path selection supportinclude/linux/multipath.h lib/multipath.c+284 -00 replies
5/29libmultipath: Add bio handlinginclude/linux/multipath.h lib/multipath.c+128 -20 replies
6/29libmultipath: Add support for mpath_device managementinclude/linux/multipath.h lib/multipath.c+166 -00 replies
7/29libmultipath: Add delayed removal supportinclude/linux/multipath.h lib/multipath.c+108 -10 replies
8/29libmultipath: Add sysfs helpersinclude/linux/multipath.h lib/multipath.c+99 -00 replies
9/29libmultipath: Add support for block device IOCTLlib/multipath.c+42 -00 replies
10/29libmultipath: Add mpath_bdev_getgeo()lib/multipath.c+21 -00 replies
11/29libmultipath: Add mpath_bdev_get_unique_id()lib/multipath.c+22 -00 replies
12/29scsi-multipath: introduce basic SCSI device supportdrivers/scsi/Kconfig drivers/scsi/Makefile drivers/scsi/scsi.c +5 more+231 -10 replies
13/29scsi-multipath: introduce scsi_device head structuredrivers/scsi/scsi_multipath.c drivers/scsi/scsi_sysfs.c include/scsi/scsi_multipath.h+242 -20 replies
14/29scsi-multipath: provide sysfs link from to scsi_devicedrivers/scsi/scsi_multipath.c drivers/scsi/scsi_sysfs.c include/scsi/scsi_multipath.h+56 -00 replies
15/29scsi-multipath: support iopolicydrivers/scsi/scsi_multipath.c include/scsi/scsi_multipath.h+55 -00 replies
16/29scsi-multipath: clone each biodrivers/scsi/scsi_multipath.c include/scsi/scsi_multipath.h+34 -00 replies
17/29scsi-multipath: clear path when device is blockeddrivers/scsi/scsi_lib.c drivers/scsi/scsi_multipath.c include/scsi/scsi_multipath.h+18 -00 replies
18/29scsi-multipath: revalidate paths upon device unblockdrivers/scsi/scsi_lib.c drivers/scsi/scsi_multipath.c include/scsi/scsi_multipath.h+15 -00 replies
19/29scsi-multipath: failover handlingdrivers/scsi/scsi_multipath.c+14 -00 replies
20/29scsi-multipath: provide callbacks for path statedrivers/scsi/scsi_multipath.c+49 -00 replies
21/29scsi-multipath: add scsi_mpath_{start,end}_request()drivers/scsi/scsi_lib.c drivers/scsi/scsi_multipath.c include/scsi/scsi_cmnd.h +2 more+99 -20 replies
22/29scsi-multipath: update request for terminal request queuing errordrivers/scsi/scsi_lib.c drivers/scsi/scsi_multipath.c include/scsi/scsi_multipath.h+43 -00 replies
23/29scsi-multipath: add delayed disk removal supportdrivers/scsi/scsi_multipath.c include/scsi/scsi_driver.h+31 -00 replies
24/29scsi: sd: add multipath disk classdrivers/scsi/sd.c drivers/scsi/sd.h+45 -10 replies
25/29scsi: sd: add multipath disk attr groupsdrivers/scsi/sd.c+36 -00 replies
26/29scsi: sd: support multipath diskdrivers/scsi/sd.c+406 -200 replies
27/29scsi: sd: add mpath_dev filedrivers/scsi/sd.c+74 -10 replies
28/29scsi: sd: add mpath_numa_nodes dev attributedrivers/scsi/sd.c+14 -00 replies
29/29scsi: sd: add mpath_queue_depth dev attributedrivers/scsi/sd.c+19 -00 replies

Reviews trailers per patch

2 of 29 patches have a Reviewed-by or Acked-by.

PatchHannes Reinecke
1/29
2/29R-b
3/29
4/29
5/29
6/29
7/29
8/29
9/29
10/29
11/29
12/29
13/29
14/29R-b
15/29
16/29
17/29
18/29
19/29
20/29
21/29
22/29
23/29
24/29
25/29
26/29
27/29
28/29
29/29

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v9 thread

Cover letter [PATCH v9 00/29] Native SCSI Multipath support

NATIVE SCSI MULTIPATH:

This series introduces native SCSI multipath support. It is intended as an alternative to dm-mpath.

This support aims to provide a multipath-enabled SCSI block device/ gendisk.

For a SCSI device to support native multipath, either of the following conditions must be satisfied:
a. unique ID in VPD page 83 and ALUA support and scsi_multipath modparam
enabled
b. unique ID in VPD page 83 and scsi_multipath_always modparam enabled

This series relies on reading sdev->access_state to get path information. This path information would be provided by ALUA. ALUA support which does not rely on device handlers has already been discussed at
https://lore.kernel.org/linux-scsi/[email protected]/
and support will be added in the next phase.

New classes of devices are added:
- scsi_mpath_device
- scsi_mpath_disk

These are required since a multipath scsi_device has no common scsi host. An example of the sysfs files and directories for these new classes is as follows:

$ ls -l /sys/class/scsi_mpath_device/scsi_mpath_device0/
total 0
-rw-r--r--    1 root     root          4096 Feb 25 11:59 iopolicy
drwxr-xr-x    2 root     root             0 Feb 25 11:59 multipath
drwxr-xr-x    2 root     root             0 Feb 25 11:59 power
lrwxrwxrwx    1 root     root             0 Feb 25 11:59 subsystem ->
../../../../class/scsi_mpath_device
-rw-r--r--    1 root     root          4096 Feb 25 11:58 uevent
-r--r--r--    1 root     root          4096 Feb 25 11:59 vpd_id
$ ls -l /sys/class/scsi_mpath_device/scsi_mpath_device0/multipath/
total 0
lrwxrwxrwx    1 root     root             0 Feb 25 11:59 8:0:0:0 ->
../../../../platform/host8/session1/target8:0:0/8:0:0:0
lrwxrwxrwx    1 root     root             0 Feb 25 11:59 9:0:0:0 ->
../../../../platform/host9/session2/target9:0:0/9:0:0:0
$ cat /sys/class/scsi_mpath_device/scsi_mpath_device0/vpd_id
naa.600140505200a986f0043c9afa1fd077
$ cat /sys/class/scsi_mpath_device/scsi_mpath_device0/iopolicy
numa
$
$ ls -l /sys/class/scsi_mpath_disk/scsi_mpath_disk0/
total 0
drwxr-xr-x    2 root     root             0 Feb 25 12:00 power
drwxr-xr-x   11 root     root             0 Feb 25 11:58 sdc
lrwxrwxrwx    1 root     root             0 Feb 25 11:58 subsystem ->
../../../../class/scsi_mpath_disk
-rw-r--r--    1 root     root          4096 Feb 25 11:58 uevent
$ ls -l /sys/class/scsi_mpath_disk/scsi_mpath_disk0/sdc/multipath/
total 0
lrwxrwxrwx    1 root     root             0 Feb 25 12:00 sdc:0 ->
../../../../../platform/host8/session1/target8:0:0/8:0:0:0/block/sdc:0
lrwxrwxrwx    1 root     root             0 Feb 25 12:00 sdc:1 ->
../../../../../platform/host9/session2/target9:0:0/9:0:0:0/block/sdc:1
$ ls -l /dev/sdc
brw-rw----    1 root     disk        8,  32 Feb 25 11:58 /dev/sdc

The scsi_device and scsi_disk classes otherwise remain unmodified. However, the per-path block device is hidden in /dev/. Furthermore, multipathed block devices have a new naming scheme, sdX:Y, where X is the scsi multipath device index and Y is the path index.

No multipath sg support is added. We still have a per-path sg device. Since the SCSI block device is multipath enabled, we can access multipathed scsi_ioctl() through that block device.

For failover, we take the approach of cloning bio's and re-submitting them in full (for failover errors).

LIBMULTIPATH:

This series introduces also libmultipath. It is essentially a refactoring of NVME multipath support, so we can have a common library to also support native SCSI multipath.

Much of the code is taken directly from the NVMe multipath code. However, NVMe specifics are removed. A template structure is provided so the driver -may provide callbacks for driver specifics, like ANA support for NVMe.

Important new structures introduced include:

- mpath_head
These contain much of the multipath-specific functionality from nvme_ns_head, including a pointer to the gendisk structure and a path SRCU-based array.

- mpath_device
This is the per-path structure, and contains much the same multipath-specific functionality in nvme_ns

libmultipath provides functionality for path management, path selection, data path, and failover handling.

Full series also available at
https://github.com/johnpgarry/linux/tree/scsi-multipath-v7.4-v9

Differences to v8:
- Rebase
- Hardening changes advised by sashiko, including "Clear cmd->flags" change

Differences to v7:
- rebase
- simplify scsi_mpath_clone_end_io() (sashiko)
- rename bio_list_add_clone() -> bio_list_add_master()
- rework checks in mpath_alloc_disk_head() (sashiko)

John Garry (29):
scsi: core: Clear cmd->flags for terminal queuing error
libmultipath: Add initial framework
libmultipath: Add basic gendisk support
libmultipath: Add path selection support
libmultipath: Add bio handling
libmultipath: Add support for mpath_device management
libmultipath: Add delayed removal support
libmultipath: Add sysfs helpers
libmultipath: Add support for block device IOCTL
libmultipath: Add mpath_bdev_getgeo()
libmultipath: Add mpath_bdev_get_unique_id()
scsi-multipath: introduce basic SCSI device support
scsi-multipath: introduce scsi_device head structure
scsi-multipath: provide sysfs link from to scsi_device
scsi-multipath: support iopolicy
scsi-multipath: clone each bio
scsi-multipath: clear path when device is blocked
scsi-multipath: revalidate paths upon device unblock
scsi-multipath: failover handling
scsi-multipath: provide callbacks for path state
scsi-multipath: add scsi_mpath_{start,end}_request()
scsi-multipath: update request for terminal request queuing error
scsi-multipath: add delayed disk removal support
scsi: sd: add multipath disk class
scsi: sd: add multipath disk attr groups
scsi: sd: support multipath disk
scsi: sd: add mpath_dev file
scsi: sd: add mpath_numa_nodes dev attribute
scsi: sd: add mpath_queue_depth dev attribute

 drivers/scsi/Kconfig          |  10 +
 drivers/scsi/Makefile         |   1 +
 drivers/scsi/scsi.c           |   8 +-
 drivers/scsi/scsi_lib.c       |  20 +-
 drivers/scsi/scsi_multipath.c | 691 +++++++++++++++++++++++++
 drivers/scsi/scsi_scan.c      |   4 +
 drivers/scsi/scsi_sysfs.c     |  10 +
 drivers/scsi/sd.c             | 611 +++++++++++++++++++++-
 drivers/scsi/sd.h             |   3 +
 include/linux/multipath.h     | 156 ++++++
 include/scsi/scsi_cmnd.h      |   9 +-
 include/scsi/scsi_device.h    |   2 +
 include/scsi/scsi_driver.h    |   4 +
 include/scsi/scsi_host.h      |   4 +
 include/scsi/scsi_multipath.h | 129 +++++
 lib/Kconfig                   |   6 +
 lib/Makefile                  |   2 +
 lib/multipath.c               | 948 ++++++++++++++++++++++++++++++++++
 18 files changed, 2592 insertions(+), 26 deletions(-)
 create mode 100644 drivers/scsi/scsi_multipath.c
 create mode 100644 include/linux/multipath.h
 create mode 100644 include/scsi/scsi_multipath.h
 create mode 100644 lib/multipath.c
Signature · 1 line
2.43.0