Skip to content

RFC Target cluster implementation over DLM

Dmitry Bogdanov linux-scsi target-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest48 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/48scsi: target/core: Add a way to hide a port groupdrivers/target/target_core_alua.c drivers/target/target_core_configfs.c include/target/target_core_base.h+38 -00 replies
2/48scsi: target/core: Set MULTIP bit for se_device with multiple portsdrivers/target/target_core_spc.c+17 -00 replies
3/48scsi: target/core: Add cleanup sequence in core_tpg_register()drivers/target/target_core_tpg.c+6 -40 replies
4/48scsi: target/core: Add RTPI field to target portdrivers/target/target_core_tpg.c include/target/target_core_base.h+77 -50 replies
5/48scsi: target/core: Use RTPI from target portdrivers/target/target_core_alua.c drivers/target/target_core_device.c drivers/target/target_core_pr.c +2 more+11 -110 replies
6/48scsi: target/core: Drop device-based RTPIdrivers/target/target_core_device.c drivers/target/target_core_internal.h drivers/target/target_core_tpg.c +1 more+0 -520 replies
7/48scsi: target/core: Add common port attributesdrivers/target/target_core_configfs.c drivers/target/target_core_fabric_configfs.c drivers/target/target_core_internal.h +1 more+50 -30 replies
8/48scsi: target/core: Add RTPI attribute for target portdrivers/target/target_core_tpg.c+70 -00 replies
9/48target: core: check RTPI uniquity for enabled TPGdrivers/target/target_core_fabric_configfs.c drivers/target/target_core_internal.h drivers/target/target_core_tpg.c+56 -150 replies
10/48scsi: target/core: Unlock PR generation bumpdrivers/target/target_core_configfs.c drivers/target/target_core_pr.c include/target/target_core_base.h+9 -210 replies
11/48target: core: fix preempt and abort for allreg resdrivers/target/target_core_pr.c+1 -10 replies
12/48target: core: fix memory leak in preempt_and_abortdrivers/target/target_core_pr.c+5 -40 replies
13/48target: core: abort all preempted regs if requesteddrivers/target/target_core_pr.c+15 -10 replies
14/48target: core: new key must be used for moved PRdrivers/target/target_core_pr.c+6 -20 replies
15/48target: core: remove unused variable in se_dev_entrydrivers/target/target_core_device.c include/target/target_core_base.h+0 -60 replies
16/48target: core: undepend PR registrant of nacldrivers/target/target_core_configfs.c drivers/target/target_core_device.c drivers/target/target_core_fabric_lib.c +4 more+88 -1590 replies
17/48target: core: make some functions publicdrivers/target/target_core_pr.c drivers/target/target_core_pr.h+54 -210 replies
18/48target: core: proper clear reservation on LUN RESETdrivers/target/target_core_tmr.c+1 -20 replies
19/48target: core: remove superfluous checksdrivers/target/target_core_pr.c+1 -40 replies
20/48target: core: proper check of SCSI-2 reservationdrivers/target/target_core_configfs.c drivers/target/target_core_pr.c+14 -90 replies
21/48target: core: checks against peer node SCSI2 reservationdrivers/target/target_core_pr.c+10 -30 replies
22/48target: core: UA on all luns after resetdrivers/target/target_core_device.c drivers/target/target_core_internal.h drivers/target/target_core_transport.c+21 -20 replies
23/48target: core: refactor LUN_RESET codedrivers/target/target_core_internal.h drivers/target/target_core_pr.c drivers/target/target_core_tmr.c +1 more+26 -160 replies
24/48target: core: pr: use RTPI in APTPLdrivers/target/target_core_configfs.c drivers/target/target_core_pr.c drivers/target/target_core_pr.h +1 more+15 -370 replies
25/48target: core: pr: have Transport ID storeddrivers/target/target_core_fabric_lib.c drivers/target/target_core_internal.h drivers/target/target_core_pr.c +1 more+68 -920 replies
26/48target: core: pr: remove se_tpg from pr_regdrivers/target/target_core_configfs.c drivers/target/target_core_fabric_configfs.c drivers/target/target_core_internal.h +4 more+104 -920 replies
27/48target: core: fix parsing PR OUT TIDdrivers/target/target_core_fabric_lib.c drivers/target/target_core_internal.h drivers/target/target_core_pr.c+89 -480 replies
28/48target: core: add function to compare TransportIDdrivers/target/target_core_fabric_lib.c drivers/target/target_core_internal.h+25 -40 replies
29/48target: core: store proto_id in APTPLdrivers/target/target_core_configfs.c drivers/target/target_core_pr.c drivers/target/target_core_pr.h+22 -80 replies
30/48target: core: rethink APTPL registrationsdrivers/target/target_core_pr.c+43 -610 replies
31/48dlm_ckv: introduce DLM cluster key-value storagedrivers/target/Kconfig drivers/target/Makefile drivers/target/dlm_ckv.c +1 more+350 -00 replies
32/48dlm_ckv: add notification servicedrivers/target/dlm_ckv.c drivers/target/dlm_ckv.h+194 -00 replies
33/48dlm_ckv: add key-value storage servicedrivers/target/dlm_ckv.c drivers/target/dlm_ckv.h+137 -00 replies
34/48dlm_ckv: add KV get/set async APIdrivers/target/dlm_ckv.c drivers/target/dlm_ckv.h+115 -10 replies
35/48target: add virtual remote targetdrivers/target/Kconfig drivers/target/Makefile drivers/target/tcm_remote/Kconfig +3 more+446 -00 replies
36/48target: cluster: introduce cluster opsdrivers/target/target_core_configfs.c drivers/target/target_core_device.c drivers/target/target_core_internal.h +2 more+232 -80 replies
37/48target: cluster: introduce dlm clusterdrivers/target/Makefile drivers/target/target_cluster_dlm.c+176 -00 replies
38/48target: cluster: store PR data in DLM clusterdrivers/target/target_cluster_dlm.c drivers/target/target_core_device.c drivers/target/target_core_fabric_lib.c +3 more+268 -30 replies
39/48target: cluster: read PR data from clusterdrivers/target/target_cluster_dlm.c drivers/target/target_core_device.c drivers/target/target_core_fabric_lib.c +4 more+320 -70 replies
40/48target: cluster: sync PR for dynamic aclsdrivers/target/target_cluster_dlm.c drivers/target/target_core_pr.c+21 -60 replies
41/48target: cluster: sync-up PR data on cluster joindrivers/target/target_cluster_dlm.c+5 -00 replies
42/48target: cluster: sync SPC-2 reservationsdrivers/target/dlm_ckv.c drivers/target/dlm_ckv.h drivers/target/target_cluster_dlm.c +6 more+104 -150 replies
43/48target: cluster: allocate UAs on PR syncdrivers/target/target_cluster_dlm.c drivers/target/target_core_device.c drivers/target/target_core_pr.c +2 more+121 -60 replies
44/48target: cluster: support PR OUT preempt and abortdrivers/target/target_cluster_dlm.c drivers/target/target_core_internal.h drivers/target/target_core_pr.c +3 more+24 -50 replies
45/48target: cluster: add reset cluster functiondrivers/target/target_core_device.c drivers/target/target_core_tmr.c include/target/target_core_base.h+11 -80 replies
46/48target: cluster: implement LUN reset in DLM clusterdrivers/target/target_cluster_dlm.c drivers/target/target_core_device.c drivers/target/target_core_internal.h +1 more+32 -30 replies
47/48target: cluster: split cluster sync functiondrivers/target/target_cluster_dlm.c+74 -430 replies
48/48target: cluster: request data on initial syncdrivers/target/target_cluster_dlm.c+35 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/48] Target cluster implementation over DLM

Hi linux target comminity.

Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.

The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.

Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).

Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.

Patches 30-34:
DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.

Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.

Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.

Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.

It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster

How to test

1. Setup DLM over worked corosync & pacemaker cluster

 $ zypper install libdlm libdlm3
 $ crm configure primitive dlm ocf:pacemaker:controld args="-q0 -f0" allow_stonith_disabled=true  op monitor interval="60" timeout="60"
 $ crm configure clone clone-dlm dlm meta interleave=true target-role=Started
2. setup TCM cluster configuration
You can use my script [2] that I used to use while developing the patchset.
3. Test what you want.
For example, my test report [3].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://pastebin.com/HgCfjywh
[3] https://pastebin.com/AgLSgnWn

Dmitry Bogdanov (39):
target: core: check RTPI uniquity for enabled TPG
target: core: fix preempt and abort for allreg res
target: core: fix memory leak in preempt_and_abort
target: core: abort all preempted regs if requested
target: core: new key must be used for moved PR
target: core: remove unused variable in se_dev_entry
target: core: undepend PR registrant of nacl
target: core: make some functions public
target: core: proper clear reservation on LUN RESET
target: core: remove superfluous checks
target: core: proper check of SCSI-2 reservation
target: core: checks against peer node SCSI2 reservation
target: core: UA on all luns after reset
target: core: refactor LUN_RESET code
target: core: pr: use RTPI in APTPL
target: core: pr: have Transport ID stored
target: core: pr: remove se_tpg from pr_reg
target: core: fix parsing PR OUT TID
target: core: add function to compare TransportID
target: core: store proto_id in APTPL
target: core: rethink APTPL registrations
dlm_ckv: introduce DLM cluster key-value storage
dlm_ckv: add notification service
dlm_ckv: add key-value storage service
dlm_ckv: add KV get/set async API
target: add virtual remote target
target: cluster: introduce cluster ops
target: cluster: introduce dlm cluster
target: cluster: store PR data in DLM cluster
target: cluster: read PR data from cluster
target: cluster: sync PR for dynamic acls
target: cluster: sync-up PR data on cluster join
target: cluster: sync SPC-2 reservations
target: cluster: allocate UAs on PR sync
target: cluster: support PR OUT preempt and abort
target: cluster: add reset cluster function
target: cluster: implement LUN reset in DLM cluster
target: cluster: split cluster sync function
target: cluster: request data on initial sync

Konstantin Shelekhin (1):
scsi: target/core: Unlock PR generation bump

Roman Bolshakov (8):
scsi: target/core: Add a way to hide a port group
scsi: target/core: Set MULTIP bit for se_device with multiple ports
scsi: target/core: Add cleanup sequence in core_tpg_register()
scsi: target/core: Add RTPI field to target port
scsi: target/core: Use RTPI from target port
scsi: target/core: Drop device-based RTPI
scsi: target/core: Add common port attributes
scsi: target/core: Add RTPI attribute for target port

 drivers/target/Kconfig                       |    7 +
 drivers/target/Makefile                      |    4 +
 drivers/target/dlm_ckv.c                     |  757 +++++++++++++
 drivers/target/dlm_ckv.h                     |   44 +
 drivers/target/target_cluster_dlm.c          | 1012 ++++++++++++++++++
 drivers/target/target_core_alua.c            |   12 +-
 drivers/target/target_core_configfs.c        |  191 +++-
 drivers/target/target_core_device.c          |  244 ++++-
 drivers/target/target_core_fabric_configfs.c |   68 +-
 drivers/target/target_core_fabric_lib.c      |  262 +++--
 drivers/target/target_core_internal.h        |   29 +-
 drivers/target/target_core_pr.c              |  613 +++++------
 drivers/target/target_core_pr.h              |   33 +-
 drivers/target/target_core_sbc.c             |   12 +-
 drivers/target/target_core_spc.c             |   19 +-
 drivers/target/target_core_stat.c            |    6 +-
 drivers/target/target_core_tmr.c             |   38 +-
 drivers/target/target_core_tpg.c             |  193 +++-
 drivers/target/target_core_transport.c       |    9 +-
 drivers/target/target_core_ua.c              |    1 +
 drivers/target/tcm_remote/Kconfig            |    8 +
 drivers/target/tcm_remote/Makefile           |    2 +
 drivers/target/tcm_remote/tcm_remote.c       |  405 +++++++
 drivers/target/tcm_remote/tcm_remote.h       |   29 +
 include/target/target_core_base.h            |   56 +-
 25 files changed, 3436 insertions(+), 618 deletions(-)
 create mode 100644 drivers/target/dlm_ckv.c
 create mode 100644 drivers/target/dlm_ckv.h
 create mode 100644 drivers/target/target_cluster_dlm.c
 create mode 100644 drivers/target/tcm_remote/Kconfig
 create mode 100644 drivers/target/tcm_remote/Makefile
 create mode 100644 drivers/target/tcm_remote/tcm_remote.c
 create mode 100644 drivers/target/tcm_remote/tcm_remote.h
Signature · 1 line
2.25.1