scsi: Replace FC-specific jammer with transport-agnostic fault injector
Subsystems (2)
v2 cover scsi: Replace FC-specific jammer with transport-agnostic fault injector
This two-patch series replaces the FC-specific SCSI command jammer introduced in commit 54a5e73f4d6e ("tcm_qla2xxx Add SCSI command jammer/discard capability") with a transport-agnostic initiator-side fault injection module.
The original implementation required LIO configured in target mode with a QLogic qla2xxx HBA, limiting it to FC environments only. tcm_qla2xxx target mode has effectively been retired, making the original approach no longer viable as a general-purpose test tool.
The replacement module (scsi_jammer) operates on the initiator side at the queuecommand level of the SCSI mid-layer. It intercepts commands before they reach any HBA driver by saving and replacing the queuecommand function pointer of the selected Scsi_Host at runtime. This makes it equally effective for FC, FCoE, iSCSI, SAS, and any other transport that presents a Scsi_Host, with no target-side configuration required.
Three injection modes simulate different fabric failure scenarios:
- drop: immediate DID_NO_CONNECT (dead path / cable pull)
- timeout: delayed completion beyond SCSI timeout (slow drain)
- flap: periodic arm/disarm (repeated RSCN events)
The flap mode is particularly useful for testing dm-multipath path reinstatement logic in addition to initial failover.
An optional jam_tur_passthrough knob lets TEST UNIT READY commands pass through to the real driver while all other commands are jammed. This simulates the real-world slow-drain failure mode where the path appears alive to dm-multipath path checkers but data I/O is stalled.
Safety: commands are never silently dropped; every intercepted command is completed via scsi_done() either immediately or from a workqueue timer. The initiator will not panic or be left with orphaned commands regardless of when the module is unloaded.
This patch series was developed with the assistance of Claude AI (Anthropic). The design, testing, and sign-off responsibility remain with the author.
Tested on x86_64 with Emulex lpfc FC HBA, dm-multipath, Linux 7.0.0+. All three injection modes and TUR passthrough verified against active multipath configurations.
Note to reviewers: checkpatch reports 4 CHECKs on patch 2/2, all of which are false positives:
- "Alignment should match open parenthesis": checkpatch miscounts tabs for enum return types; alignment is correct.
- "Lines should not end with a '('": common kernel pattern.
- "Macro argument reuse '_var'" (x2): READ_ONCE/WRITE_ONCE are specifically designed to be safe with macro argument reuse; this is a known false positive for these accessors.
Changes since v1:
- Patch 2: Add jam_tur_passthrough control to pass TEST UNIT READY commands through while jamming all other commands. Simulates slow-drain where path appears alive to multipath but data I/O is stalled.
- Patch 1: unchanged
- Fix duplicate flap_work_fn forward declaration in patch 2
Laurence Oberman (2):
scsi: tcm_qla2xxx: Remove FC-specific SCSI command jammer
scsi: Add transport-agnostic initiator-side fault injector
MAINTAINERS | 6 + drivers/scsi/Kconfig | 22 + drivers/scsi/Makefile | 1 + drivers/scsi/qla2xxx/Kconfig | 9 - drivers/scsi/qla2xxx/tcm_qla2xxx.c | 23 - drivers/scsi/qla2xxx/tcm_qla2xxx.h | 1 - drivers/scsi/scsi_jammer.c | 657 +++++++++++++++++++++++++++++ 7 files changed, 686 insertions(+), 33 deletions(-) create mode 100644 drivers/scsi/scsi_jammer.c
Signature · 1 line
2.54.0
PATCH v2 1/2 scsi: tcm_qla2xxx: Remove FC-specific SCSI command jammer
The jam_host tpg_attrib and CONFIG_TCM_QLA2XXX_DEBUG Kconfig option introduced in commit 54a5e73f4d6e ("tcm_qla2xxx Add SCSI command jammer/discard capability") are superseded by the transport-agnostic initiator-side scsi_jammer module introduced in patch 2/2.
The original implementation had several limitations that motivated this replacement:
- Required LIO configured in target mode with a QLogic qla2xxx HBA, making it unavailable for iSCSI, FCoE, SAS, and any other transport.
- Operated on the target side, meaning a separate target host was needed to test initiator error recovery.
- tcm_qla2xxx target mode has effectively been retired and is no longer a viable dependency for a general-purpose test tool.
- Only supported command discard (drop); no stall or flap modes.
The replacement in patch 2/2 operates on the initiator side at the queuecommand level of the SCSI mid-layer, requires no target-side configuration, and works identically across all transports that present a Scsi_Host.
Note: Documentation/scsi/tcm_qla2xxx.txt was already removed from the tree prior to this patch and does not require deletion here.
Remove CONFIG_TCM_QLA2XXX_DEBUG from Kconfig, the jam_host field from struct tcm_qla2xxx_tpg_attrib, and all associated #ifdef blocks from tcm_qla2xxx.c.
--- drivers/scsi/qla2xxx/Kconfig | 9 --------- drivers/scsi/qla2xxx/tcm_qla2xxx.c | 23 ----------------------- drivers/scsi/qla2xxx/tcm_qla2xxx.h | 1 - 3 files changed, 33 deletions(-)
drivers/scsi/qla2xxx/Kconfig +0 -9
@@ -37,12 +37,3 @@ config TCM_QLA2XXX Say Y here to enable the TCM_QLA2XXX fabric module for QLogic 24xx+ series target mode HBAs. -if TCM_QLA2XXX -config TCM_QLA2XXX_DEBUG - bool "TCM_QLA2XXX fabric module DEBUG mode for QLogic 24xx+ series target mode HBAs" - default n - help - Say Y here to enable the TCM_QLA2XXX fabric module DEBUG for - QLogic 24xx+ series target mode HBAs. - This will include code to enable the SCSI command jammer. -endif
drivers/scsi/qla2xxx/tcm_qla2xxx.c +0 -23
@@ -450,13 +450,6 @@ static int tcm_qla2xxx_handle_cmd(scsi_qla_host_t *vha, struct qla_tgt_cmd *cmd,struct se_cmd *se_cmd = &cmd->se_cmd;struct se_session *se_sess;struct fc_port *sess; -#ifdef CONFIG_TCM_QLA2XXX_DEBUG -struct se_portal_group *se_tpg; -struct tcm_qla2xxx_tpg *tpg; -#endif -int rc, target_flags =TARGET_SCF_ACK_KREF ; -unsigned long flags; -if (bidi) target_flags |=TARGET_SCF_BIDI_OP ;
@@ -475,15 +468,6 @@ static int tcm_qla2xxx_handle_cmd(scsi_qla_host_t *vha, struct qla_tgt_cmd *cmd,return -EINVAL ; } -#ifdef CONFIG_TCM_QLA2XXX_DEBUG - se_tpg = se_sess->se_tpg; - tpg = container_of(se_tpg, struct tcm_qla2xxx_tpg, se_tpg); -if (unlikely (tpg->tpg_attrib.jam_host)) { -/* return, and dont run target_submit_cmd,discarding command */ -return 0 ; - } -#endif - cmd->qpair->tgt_counters.qla_core_sbt_cmd++;spin_lock_irqsave (&sess->sess_cmd_lock, flags);list_add_tail (&cmd->sess_cmd_list, &sess->sess_cmd_list);
@@ -903,9 +887,6 @@ DEF_QLA_TPG_ATTRIB(cache_dynamic_acls);DEF_QLA_TPG_ATTRIB (demo_mode_write_protect);DEF_QLA_TPG_ATTRIB (prod_mode_write_protect);DEF_QLA_TPG_ATTRIB (demo_mode_login_only); -#ifdef CONFIG_TCM_QLA2XXX_DEBUG -DEF_QLA_TPG_ATTRIB (jam_host); -#endif static struct configfs_attribute *tcm_qla2xxx_tpg_attrib_attrs[] = { &tcm_qla2xxx_tpg_attrib_attr_generate_node_acls,
@@ -913,9 +894,6 @@ static struct configfs_attribute *tcm_qla2xxx_tpg_attrib_attrs[] = { &tcm_qla2xxx_tpg_attrib_attr_demo_mode_write_protect, &tcm_qla2xxx_tpg_attrib_attr_prod_mode_write_protect, &tcm_qla2xxx_tpg_attrib_attr_demo_mode_login_only, -#ifdef CONFIG_TCM_QLA2XXX_DEBUG - &tcm_qla2xxx_tpg_attrib_attr_jam_host, -#endif NULL, };
@@ -1030,7 +1008,6 @@ static struct se_portal_group *tcm_qla2xxx_make_tpg(struct se_wwn *wwn, tpg->tpg_attrib.demo_mode_write_protect =1 ; tpg->tpg_attrib.cache_dynamic_acls =1 ; tpg->tpg_attrib.demo_mode_login_only =1 ; - tpg->tpg_attrib.jam_host =0 ; ret =core_tpg_register (wwn, &tpg->se_tpg,SCSI_PROTOCOL_FCP );if (ret <0 ) {
drivers/scsi/qla2xxx/tcm_qla2xxx.h +0 -1
@@ -34,7 +34,6 @@ struct tcm_qla2xxx_tpg_attrib {int prod_mode_write_protect;int demo_mode_login_only;int fabric_prot_type; -int jam_host; };struct tcm_qla2xxx_tpg {
Signature · 1 line
2.54.0
PATCH v2 2/2 scsi: Add transport-agnostic initiator-side fault injector
In reply to Laurence ObermanTesting SCSI error recovery paths — multipath failover, SCSI EH, and path reinstatement — traditionally requires physical fabric disruption:
pulling cables, disabling switch ports, or using vendor-specific tools tied to specific HBA drivers.
This patch introduces scsi_jammer, a transport-agnostic fault injection module that operates on the initiator side at the queuecommand level of the SCSI mid-layer. By saving and replacing the queuecommand function pointer of a selected Scsi_Host at runtime, it intercepts commands before they reach any HBA driver, making it equally effective for FC, FCoE, iSCSI, SAS, and any other transport that presents a Scsi_Host. The original pointer is restored cleanly on disarm or module unload.
This supersedes the FC-specific target-side jammer removed in the previous patch, which required LIO configured in target mode with a qla2xxx HBA and could not be used for iSCSI, FCoE, or other transports.
Three injection modes are provided, controlled via debugfs:
Mode 0 (drop): Commands complete immediately with DID_NO_CONNECT.
Simulates a dead fabric path, triggering immediate multipath failover.
Mode 1 (timeout): Commands are held for jam_msecs milliseconds before
completing with DID_NO_CONNECT. Setting jam_msecs beyond the SCSI command timeout (typically 30s) causes the mid-layer EH to fire naturally, simulating a slow-drain or unresponsive fabric port.
Mode 2 (flap): The jammer is armed for jam_msecs ms then disarmed
for jam_flap_interval ms, repeating until disabled. Simulates repeated RSCN events and a flapping fabric path, exercising both multipath failover and path reinstatement logic.
An optional TUR passthrough mode (jam_tur_passthrough=1) lets TEST UNIT READY commands pass through to the real driver while all other commands are jammed. This simulates the real-world slow-drain failure mode where the fabric is stalling data I/O but the path appears alive to multipath because TURs still succeed, allowing precise testing of dm-multipath path checker behaviour under slow-drain without triggering premature failover.
Debugfs interface at /sys/kernel/debug/scsi_jammer/:
jam_enable w/r 0/1 master arm switch; write resets jam_count
jam_host_no w/r int Scsi_Host host_no to jam
jam_style w/r 0/1/2 injection mode (drop/timeout/flap)
jam_msecs w/r u32 hold duration in ms
(min 100, default 5000)
jam_flap_interval w/r u32 disarmed interval for flap mode
(ms, min 100) jam_tur_passthrough w/r 0/1 1 = pass TURs through, jam all other
commands (slow-drain simulation)
jam_count r/o u64 commands jammed since last jam_enable write
Safety guarantees:
- Commands are never silently dropped. Every intercepted command is completed via scsi_done() with DID_NO_CONNECT, either immediately or from a workqueue after a timer fires.
- All completions occur from workqueue (process) context. The flap timer fires in softirq and only calls queue_work() — it never calls jam_disarm() or any blocking function directly. The actual arm/disarm runs in flap_work_fn() where sleeping is safe.
- The drain path uses a two-phase splice-then-cancel approach ensuring that any entry in the drain list is exclusively owned by the draining thread and cannot be concurrently completed by jam_complete_work.
- On module unload, all pending commands are force-completed before the module exits. The initiator will never be left with orphaned commands regardless of when rmmod is called.
- A 100ms minimum is enforced on all timer intervals to prevent workqueue saturation under misconfiguration.
- TUR passthrough is checked after the jam_enable guard so disarming always takes effect, but before jam_count so passed-through TURs are not counted as jammed commands.
This patch was developed with the assistance of Claude AI (Anthropic). The design, testing, and sign-off responsibility remain with the author.
Tested on x86_64 with Emulex lpfc FC HBA, dm-multipath, Linux 7.0.0+:
- Mode 0: immediate DID_NO_CONNECT, dm-multipath failover confirmed
- Mode 1: SCSI EH triggered at 35s stall, failover and path reinstatement confirmed
- Mode 2: repeated RSCN simulation across multiple flap cycles, dm-multipath failover and reinstatement confirmed, no kernel panic or orphaned commands under sustained I/O
- jam_tur_passthrough=1 + mode 1: dm-multipath path checker keeps path active (TURs pass), data IOs stall until EH fires, slow-drain simulation confirmed
---
--- MAINTAINERS | 6 + drivers/scsi/Kconfig | 22 ++ drivers/scsi/Makefile | 1 + drivers/scsi/scsi_jammer.c | 674 +++++++++++++++++++++++++++++++++++++ 4 files changed, 703 insertions(+) create mode 100644 drivers/scsi/scsi_jammer.c
MAINTAINERS +6 -0
@@ -23913,6 +23913,12 @@ F: Documentation/scsi/scsi-generic.rst F: drivers/scsi/sg.c F: include/scsi/sg.h +SCSI JAMMER +M: Laurence Oberman <[email protected]> +L: [email protected] +S: Maintained +F: drivers/scsi/scsi_jammer.c + SCSI SUBSYSTEM M: "James E.J. Bottomley" <[email protected]> M: "Martin K. Petersen" <[email protected]>
drivers/scsi/Kconfig +22 -0
@@ -1238,6 +1238,28 @@ config SCSI_DEBUG See <http://sg.danny.cz/sg/sdebug26.html> for more information. Mainly used for testing and best as a module. If unsure, say N. + +config SCSI_JAMMER + tristate "SCSI initiator-side fault injector for error recovery testing" + depends on SCSI && DEBUG_FS + default n + help + Loadable module providing transport-agnostic SCSI command fault + injection on the initiator side. Intercepts commands at the + queuecommand level to simulate fabric events such as path loss, + slow drain, and repeated RSCNs (flapping paths). + + Three injection modes are available via debugfs controls: + 0 (drop) - immediate DID_NO_CONNECT, triggers multipath failover + 1 (timeout) - delayed completion to trigger SCSI EH + 2 (flap) - periodic arm/disarm simulating repeated RSCNs + + Works identically for FC, FCoE, iSCSI, SAS and any other transport + using a Scsi_Host. Requires no target-side configuration. + + Controls appear under /sys/kernel/debug/scsi_jammer/ when loaded. + + If unsure, say N. Do NOT enable in production kernels. config SCSI_MESH tristate "MESH (Power Mac internal SCSI) support" depends on PPC32 && PPC_PMAC && SCSI
drivers/scsi/Makefile +1 -0
@@ -155,6 +155,7 @@ obj-$(CONFIG_SCSI_HISI_SAS) += hisi_sas/ # This goes last, so that "real" scsi devices probe earlier obj-$(CONFIG_SCSI_DEBUG) += scsi_debug.o +obj-$(CONFIG_SCSI_JAMMER) += scsi_jammer.o scsi_mod-y += scsi.o hosts.o scsi_ioctl.o \ scsicam.o scsi_error.o scsi_lib.o scsi_mod-$(CONFIG_SCSI_CONSTANTS) += constants.o
drivers/scsi/scsi_jammer.c new file +674 -0
674 lines not on this page · raw message
Signature · 1 line
2.54.0
Quoted 73 lines · Laurence Oberman
This two-patch series replaces the FC-specific SCSI command jammer introduced in commit
54a5e73f4d6e("tcm_qla2xxxAdd SCSI command jammer/discard capability") with a transport-agnostic initiator-side fault injection module.The original implementation required LIO configured in target mode with a QLogic qla2xxx HBA, limiting it to FC environments only.
tcm_qla2xxxtarget mode has effectively been retired, making the original approach no longer viable as a general-purpose test tool.The replacement module (
scsi_jammer) operates on the initiator side at the queuecommand level of the SCSI mid-layer. It intercepts commands before they reach any HBA driver by saving and replacing the queuecommand function pointer of the selectedScsi_Hostat runtime. This makes it equally effective for FC, FCoE, iSCSI, SAS, and any other transport that presents aScsi_Host, with no target-side configuration required.Three injection modes simulate different fabric failure scenarios:
- drop: immediateDID_NO_CONNECT(dead path / cable pull)
- timeout: delayed completion beyond SCSI timeout (slow drain)
- flap: periodic arm/disarm (repeated RSCN events)The flap mode is particularly useful for testing dm-multipath path reinstatement logic in addition to initial failover.
An optional
jam_tur_passthroughknob lets TEST UNIT READY commands pass through to the real driver while all other commands are jammed. This simulates the real-world slow-drain failure mode where the path appears alive to dm-multipath path checkers but data I/O is stalled.Safety: commands are never silently dropped; every intercepted command is completed via
scsi_done()either immediately or from a workqueue timer. The initiator will not panic or be left with orphaned commands regardless of when the module is unloaded.This patch series was developed with the assistance of Claude AI (Anthropic). The design, testing, and sign-off responsibility remain with the author.
Tested on
x86_64with Emulex lpfc FC HBA, dm-multipath, Linux 7.0.0+. All three injection modes and TUR passthrough verified against active multipath configurations.Note to reviewers: checkpatch reports 4 CHECKs on patch 2/2, all of which are false positives:
- "Alignment should match open parenthesis": checkpatch miscounts tabs for enum return types; alignment is correct.
- "Lines should not end with a '('": common kernel pattern.
- "Macro argument reuse '_var'" (x2): READ_ONCE/WRITE_ONCE are specifically designed to be safe with macro argument reuse; this is a known false positive for these accessors.Changes since v1:
- Patch 2: Addjam_tur_passthroughcontrol to pass TEST UNIT READY commands through while jamming all other commands. Simulates slow-drain where path appears alive to multipath but data I/O is stalled.
- Patch 1: unchanged
- Fix duplicateflap_work_fnforward declaration in patch 2Laurence Oberman (2):
scsi:tcm_qla2xxx: Remove FC-specific SCSI command jammer
scsi: Add transport-agnostic initiator-side fault injectorMAINTAINERS | 6 +
drivers/scsi/Kconfig | 22 +
drivers/scsi/Makefile | 1 +
drivers/scsi/qla2xxx/Kconfig | 9 -
drivers/scsi/qla2xxx/tcm_qla2xxx.c | 23 -
drivers/scsi/qla2xxx/tcm_qla2xxx.h | 1 -
drivers/scsi/scsi_jammer.c | 657
+++++++++++++++++++++++++++++
Quoted 2 lines
7 files changed, 686 insertions(+), 33 deletions(-)create mode 100644
drivers/scsi/scsi_jammer.c
Hello
A gentle ping, any interest in looking at this series. If there is no interest, then we should at least retire my old 2016 version that uses tcm_qla2xxx as that is no longer functional
Thanks a lot Laurence