RFC Target cluster implementation over DLM
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest48 patches 2 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/48 | scsi: target/core: Add a way to hide a port groupdrivers/ drivers/ include/ | +38 | 0 replies | |
| 2/48 | scsi: target/core: Set MULTIP bit for se_device with multiple portsdrivers/ | +17 | 0 replies | |
| 3/48 | scsi: target/core: Add cleanup sequence in core_tpg_register()drivers/ | +6 | 0 replies | |
| 4/48 | scsi: target/core: Add RTPI field to target portdrivers/ include/ | +77 | 0 replies | |
| 5/48 | scsi: target/core: Use RTPI from target portdrivers/ drivers/ drivers/ +2 more | +11 | 0 replies | |
| 6/48 | scsi: target/core: Drop device-based RTPIdrivers/ drivers/ drivers/ +1 more | +0 | 0 replies | |
| 7/48 | scsi: target/core: Add common port attributesdrivers/ drivers/ drivers/ +1 more | +50 | 0 replies | |
| 8/48 | scsi: target/core: Add RTPI attribute for target portdrivers/ | +70 | 0 replies | |
| 9/48 | target: core: check RTPI uniquity for enabled TPGdrivers/ drivers/ drivers/ | +56 | 0 replies | |
| 10/48 | scsi: target/core: Unlock PR generation bumpdrivers/ drivers/ include/ | +9 | 0 replies | |
| 11/48 | target: core: fix preempt and abort for allreg resdrivers/ | +1 | 0 replies | |
| 12/48 | target: core: fix memory leak in preempt_and_abortdrivers/ | +5 | 0 replies | |
| 13/48 | target: core: abort all preempted regs if requesteddrivers/ | +15 | 0 replies | |
| 14/48 | target: core: new key must be used for moved PRdrivers/ | +6 | 0 replies | |
| 15/48 | target: core: remove unused variable in se_dev_entrydrivers/ include/ | +0 | 0 replies | |
| 16/48 | target: core: undepend PR registrant of nacldrivers/ drivers/ drivers/ +4 more | +88 | 0 replies | |
| 17/48 | target: core: make some functions publicdrivers/ drivers/ | +54 | 0 replies | |
| 18/48 | target: core: proper clear reservation on LUN RESETdrivers/ | +1 | 0 replies | |
| 19/48 | target: core: remove superfluous checksdrivers/ | +1 | 0 replies | |
| 20/48 | target: core: proper check of SCSI-2 reservationdrivers/ drivers/ | +14 | 0 replies | |
| 21/48 | target: core: checks against peer node SCSI2 reservationdrivers/ | +10 | 0 replies | |
| 22/48 | target: core: UA on all luns after resetdrivers/ drivers/ drivers/ | +21 | 0 replies | |
| 23/48 | target: core: refactor LUN_RESET codedrivers/ drivers/ drivers/ +1 more | +26 | 0 replies | |
| 24/48 | target: core: pr: use RTPI in APTPLdrivers/ drivers/ drivers/ +1 more | +15 | 0 replies | |
| 25/48 | target: core: pr: have Transport ID storeddrivers/ drivers/ drivers/ +1 more | +68 | 0 replies | |
| 26/48 | target: core: pr: remove se_tpg from pr_regdrivers/ drivers/ drivers/ +4 more | +104 | 0 replies | |
| 27/48 | target: core: fix parsing PR OUT TIDdrivers/ drivers/ drivers/ | +89 | 0 replies | |
| 28/48 | target: core: add function to compare TransportIDdrivers/ drivers/ | +25 | 0 replies | |
| 29/48 | target: core: store proto_id in APTPLdrivers/ drivers/ drivers/ | +22 | 0 replies | |
| 30/48 | target: core: rethink APTPL registrationsdrivers/ | +43 | 0 replies | |
| 31/48 | dlm_ckv: introduce DLM cluster key-value storagedrivers/ drivers/ drivers/ +1 more | +350 | 0 replies | |
| 32/48 | dlm_ckv: add notification servicedrivers/ drivers/ | +194 | 0 replies | |
| 33/48 | dlm_ckv: add key-value storage servicedrivers/ drivers/ | +137 | 0 replies | |
| 34/48 | dlm_ckv: add KV get/set async APIdrivers/ drivers/ | +115 | 0 replies | |
| 35/48 | target: add virtual remote targetdrivers/ drivers/ drivers/ +3 more | +446 | 0 replies | |
| 36/48 | target: cluster: introduce cluster opsdrivers/ drivers/ drivers/ +2 more | +232 | 0 replies | |
| 37/48 | target: cluster: introduce dlm clusterdrivers/ drivers/ | +176 | 0 replies | |
| 38/48 | target: cluster: store PR data in DLM clusterdrivers/ drivers/ drivers/ +3 more | +268 | 0 replies | |
| 39/48 | target: cluster: read PR data from clusterdrivers/ drivers/ drivers/ +4 more | +320 | 0 replies | |
| 40/48 | target: cluster: sync PR for dynamic aclsdrivers/ drivers/ | +21 | 0 replies | |
| 41/48 | target: cluster: sync-up PR data on cluster joindrivers/ | +5 | 0 replies | |
| 42/48 | target: cluster: sync SPC-2 reservationsdrivers/ drivers/ drivers/ +6 more | +104 | 0 replies | |
| 43/48 | target: cluster: allocate UAs on PR syncdrivers/ drivers/ drivers/ +2 more | +121 | 0 replies | |
| 44/48 | target: cluster: support PR OUT preempt and abortdrivers/ drivers/ drivers/ +3 more | +24 | 0 replies | |
| 45/48 | target: cluster: add reset cluster functiondrivers/ drivers/ include/ | +11 | 0 replies | |
| 46/48 | target: cluster: implement LUN reset in DLM clusterdrivers/ drivers/ drivers/ +1 more | +32 | 0 replies | |
| 47/48 | target: cluster: split cluster sync functiondrivers/ | +74 | 0 replies | |
| 48/48 | target: cluster: request data on initial syncdrivers/ | +35 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 00/48] Target cluster implementation over DLM
Hi linux target comminity.
Let's me present RFC of an implementation of cluster features for Target Core that needs for backstore devices shared through cluster nodes.
The patchset is big and of several subsets, but it contains some arguable things and it would take too much time to discsuss them separatelly.
Patches 1-9:
Make RTPI be part of se_tpg instead of se_lun. That is a must because there is no possibility to assign RTPI on a LUN.
That data model is different from SCST and current in LIO but still does not contradict with SAM and even is more according to SAM - a whole TCM is a SCSI Device, and all its ports are SCSI Ports with unique RTPIs.
+ unique identification of TPG through the cluster.
+ possibility of assignment of RPTI.
- number of all TPGs will be limited to 65535.
This patchset was published first time 2 years ago [1]. In previous version the peers RTPIs were put in <device>/alua/... folder. In this version the peers RTPIs are part of TPGs on the remote fabric (patch 35).
Patches 10-29:
Fixes some bugs and deviations from the standard in PR code. Undepend pr_reg from se_nacl and se_tpg to be just a registration holder. Make APTPL registrations (not linked to se_dev_entry) be full-fledged registrations.
Patches 30-34:DLM_CKV module that uses DLM and provides:
* Cluster Lock service (pure wrapper over DLM).
* Cluster Key-Value service in memory storage.
* Cluster Notification service with a blocking acknowledge.
* Cluster membership callbacks.
This module is supposed to be used by TCM and nvmet to implement cluster operations.
Patch 35:
New 'remote' (in fact dummy) fabric module. Configuration on this fabric will provide to TCM a view of TPG/LUN/ACL configuration on a peer nodes.
Patche 36:
Introduce cluster ops and functions to register a cluster ops implementation modules. There could be a several different modules. The device attrib cluster_impl regulates which implementation to use for that device. 'single' is for default (no cluster) implementation.
Patches 37-48:
TCM Cluster over DLM module implementation inspired by SCST.
* Use DLM_CKV Lock service to serialize order of PR OUT commands
* Use DLM_CKV Key-Value storage service to store PR cluster data.
Sync it after successful execution of PR OUT command.
* Use DLM_CKV Notification service to notify (in blocking manner) other
nodes to fetch PR cluster data. The handling of PR OUT command is blocked until other nodes read the cluster PR data.
It provides:
* Cluster lock per LBA for Compare And Write.
* Full support of SCSI-3 Persistent Reservations including PREEMPT AND ABORT and REGISTER AND MOVE.
* Normal PR APTPL imlementation (persistanse over power loss)
* Shared LUN RESET
* Shared SCSI-2 Reservations.
* Unit Attentions for all TPGs in cluster
How to test
1. Setup DLM over worked corosync & pacemaker cluster
$ zypper install libdlm libdlm3 $ crm configure primitive dlm ocf:pacemaker:controld args="-q0 -f0" allow_stonith_disabled=true op monitor interval="60" timeout="60" $ crm configure clone clone-dlm dlm meta interleave=true target-role=Started 2. setup TCM cluster configuration You can use my script [2] that I used to use while developing the patchset. 3. Test what you want. For example, my test report [3].
[1] https://lore.kernel.org/all/[email protected]/
[2] https://pastebin.com/HgCfjywh
[3] https://pastebin.com/AgLSgnWn
Dmitry Bogdanov (39):
target: core: check RTPI uniquity for enabled TPG
target: core: fix preempt and abort for allreg res
target: core: fix memory leak in preempt_and_abort
target: core: abort all preempted regs if requested
target: core: new key must be used for moved PR
target: core: remove unused variable in se_dev_entry
target: core: undepend PR registrant of nacl
target: core: make some functions public
target: core: proper clear reservation on LUN RESET
target: core: remove superfluous checks
target: core: proper check of SCSI-2 reservation
target: core: checks against peer node SCSI2 reservation
target: core: UA on all luns after reset
target: core: refactor LUN_RESET code
target: core: pr: use RTPI in APTPL
target: core: pr: have Transport ID stored
target: core: pr: remove se_tpg from pr_reg
target: core: fix parsing PR OUT TID
target: core: add function to compare TransportID
target: core: store proto_id in APTPL
target: core: rethink APTPL registrations
dlm_ckv: introduce DLM cluster key-value storage
dlm_ckv: add notification service
dlm_ckv: add key-value storage service
dlm_ckv: add KV get/set async API
target: add virtual remote target
target: cluster: introduce cluster ops
target: cluster: introduce dlm cluster
target: cluster: store PR data in DLM cluster
target: cluster: read PR data from cluster
target: cluster: sync PR for dynamic acls
target: cluster: sync-up PR data on cluster join
target: cluster: sync SPC-2 reservations
target: cluster: allocate UAs on PR sync
target: cluster: support PR OUT preempt and abort
target: cluster: add reset cluster function
target: cluster: implement LUN reset in DLM cluster
target: cluster: split cluster sync function
target: cluster: request data on initial sync
Konstantin Shelekhin (1):
scsi: target/core: Unlock PR generation bump
Roman Bolshakov (8):
scsi: target/core: Add a way to hide a port group
scsi: target/core: Set MULTIP bit for se_device with multiple ports
scsi: target/core: Add cleanup sequence in core_tpg_register()
scsi: target/core: Add RTPI field to target port
scsi: target/core: Use RTPI from target port
scsi: target/core: Drop device-based RTPI
scsi: target/core: Add common port attributes
scsi: target/core: Add RTPI attribute for target port
drivers/target/Kconfig | 7 + drivers/target/Makefile | 4 + drivers/target/dlm_ckv.c | 757 +++++++++++++ drivers/target/dlm_ckv.h | 44 + drivers/target/target_cluster_dlm.c | 1012 ++++++++++++++++++ drivers/target/target_core_alua.c | 12 +- drivers/target/target_core_configfs.c | 191 +++- drivers/target/target_core_device.c | 244 ++++- drivers/target/target_core_fabric_configfs.c | 68 +- drivers/target/target_core_fabric_lib.c | 262 +++-- drivers/target/target_core_internal.h | 29 +- drivers/target/target_core_pr.c | 613 +++++------ drivers/target/target_core_pr.h | 33 +- drivers/target/target_core_sbc.c | 12 +- drivers/target/target_core_spc.c | 19 +- drivers/target/target_core_stat.c | 6 +- drivers/target/target_core_tmr.c | 38 +- drivers/target/target_core_tpg.c | 193 +++- drivers/target/target_core_transport.c | 9 +- drivers/target/target_core_ua.c | 1 + drivers/target/tcm_remote/Kconfig | 8 + drivers/target/tcm_remote/Makefile | 2 + drivers/target/tcm_remote/tcm_remote.c | 405 +++++++ drivers/target/tcm_remote/tcm_remote.h | 29 + include/target/target_core_base.h | 56 +- 25 files changed, 3436 insertions(+), 618 deletions(-) create mode 100644 drivers/target/dlm_ckv.c create mode 100644 drivers/target/dlm_ckv.h create mode 100644 drivers/target/target_cluster_dlm.c create mode 100644 drivers/target/tcm_remote/Kconfig create mode 100644 drivers/target/tcm_remote/Makefile create mode 100644 drivers/target/tcm_remote/tcm_remote.c create mode 100644 drivers/target/tcm_remote/tcm_remote.h
Signature · 1 line
2.25.1