Skip to contentSkip to conversation

megaraid_sas: multiple FALLOC_FL_ZERO_RANGE causes timeouts and resets on MegaRAID 9560-8i 4GB since 5.19

DISCUSSION Vitaly Chikunov 4 participants 8 messages Feb 10, 2024 – Mar 18, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

We started to get timeouts and controller resets since 5.19.5 (vanilla v5.19 is not tested, tests below are on 6.6.15) when several ioctl FALLOC_FL_ZERO_RANGE are issued into device consequentially without delay between them (3-5 is enough to trigger condition). Because of this, for example, mkfs.ext4 extremely slows down when initializing filesystem. This happens on aarch64 (Kunpeng-920) server.

Reproducer:

  # for ((i=0;i<5;i++)); do echo $i; fallocate -z -l 2097152 /dev/sdc; done

Example of dmesg messages after problematic ioctl calls:

  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 Abort request is for SMID: 4753
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d51beacc) tm_dev_handle 0x4
  Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
  Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 task abort FAILED!! scmd(0x00000000d51beacc)
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
  Feb 06 19:45:04 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 Abort request is for SMID: 8293
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d9406c9c) tm_dev_handle 0x4
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 BRCM Debug mfi stat 0x2d, data len requested/completed 0x1000/0x0
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 task abort SUCCESS!! scmd(0x00000000d9406c9c)
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 CDB: Write Same(10) 41 00 03 4c 00 10 00 10 00 00
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting target reset! scmd(0x00000000d51beacc) tm_dev_handle: 0x4
  Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
  Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 target reset SUCCESS!!
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: Power-on or device reset occurred

Excerpt from the controller events log (from storli):

Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: PD 05(e0xfb/s4) Path 5e8b4700e35e2004, CDB: 41 00 00 00 00 00 00 10 00 00, Sense: 6/29/00

Tests was on the latest firmware (at the moment):

Product Name = MegaRAID 9560-8i 4GB
Serial Number = SKC4006982
Firmware Package Build = 52.28.0-5305
Firmware Version = 5.280.02-3972
PSOC FW Version = 0x001A
PSOC Hardware Version = 0x000A
PSOC Part Number = 29211-260-4GB
NVDATA Version = 5.2800.00-0752
CBB Version = 28.250.04.00
Bios Version = 7.28.00.0_0x071C0000
HII Version = 07.28.04.00
HIIA Version = 07.28.04.00
Driver Name = megaraid_sas
Driver Version = 07.725.01.00-rc1

I tried also latest available megaraid_sas driver (07.728.04.00) which is not yet merged into mainline but the problems are not resolved with it.

Thanks,

#1 lore raw

Hi,

Quoted 7 lines · Vitaly Chikunov

We started to get timeouts and controller resets since 5.19.5 (vanilla v5.19 is not tested, tests below are on 6.6.15) when several ioctl FALLOC_FL_ZERO_RANGE are issued into device consequentially without delay between them (3-5 is enough to trigger condition). Because of this, for example, mkfs.ext4 extremely slows down when initializing filesystem. This happens on aarch64 (Kunpeng-920) server.

I am reported that bisect found this commit to cause above mentioned problem:

commit c92a6b5d63359dd6d2ce6ea88ecd8e31dd769f6b
Author: Martin K. Petersen <[email protected]>
AuthorDate: Wed Mar 2 00:35:47 2022 -0500

scsi: core: Query VPD size before getting full page

When from v5.19 this commit is reverted the problem disappears.

Thanks,

Quoted 56 lines

Reproducer:

  # for ((i=0;i<5;i++)); do echo $i; fallocate -z -l 2097152 /dev/sdc; done

Example of dmesg messages after problematic ioctl calls:

  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 Abort request is for SMID: 4753
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d51beacc) tm_dev_handle 0x4
  Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
  Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 task abort FAILED!! scmd(0x00000000d51beacc)
  Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
  Feb 06 19:45:04 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 Abort request is for SMID: 8293
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d9406c9c) tm_dev_handle 0x4
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 BRCM Debug mfi stat 0x2d, data len requested/completed 0x1000/0x0
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 task abort SUCCESS!! scmd(0x00000000d9406c9c)
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 CDB: Write Same(10) 41 00 03 4c 00 10 00 10 00 00
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting target reset! scmd(0x00000000d51beacc) tm_dev_handle: 0x4
  Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
  Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 target reset SUCCESS!!
  Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: Power-on or device reset occurred

Excerpt from the controller events log (from storli):

Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: PD 05(e0xfb/s4) Path 5e8b4700e35e2004, CDB: 41 00 00 00 00 00 00 10 00 00, Sense: 6/29/00

Tests was on the latest firmware (at the moment):

Product Name = MegaRAID 9560-8i 4GB
Serial Number = SKC4006982
Firmware Package Build = 52.28.0-5305
Firmware Version = 5.280.02-3972
PSOC FW Version = 0x001A
PSOC Hardware Version = 0x000A
PSOC Part Number = 29211-260-4GB
NVDATA Version = 5.2800.00-0752
CBB Version = 28.250.04.00
Bios Version = 7.28.00.0_0x071C0000
HII Version = 07.28.04.00
HIIA Version = 07.28.04.00
Driver Name = megaraid_sas
Driver Version = 07.725.01.00-rc1

I tried also latest available megaraid_sas driver (07.728.04.00) which is not yet merged into mainline but the problems are not resolved with it.

Thanks,

#2 lore raw

Vitaly,

I'd appreciate it if you could test the patch Bart referred you to.

Quoted 2 lines

I am reported that bisect found this commit to cause above mentioned problem:

Also, I would like to understand why things fail as a result of the original change.

Could you please send me the output of:

# sg_readcap -l /dev/sdc
# sg_vpd -l /dev/sdc
# sg_vpd -p 0xb0 /dev/sdc
# sg_vpd -p 0xb1 /dev/sdc
# sg_vpd -p 0xb2 /dev/sdc

Thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#3 lore raw

Martin,

Quoted 9 lines · Martin K. Petersen

I'd appreciate it if you could test the patch Bart referred you to.

Quoted 2 lines

I am reported that bisect found this commit to cause above mentioned problem:

Also, I would like to understand why things fail as a result of the original change.

Quoted 7 lines

Could you please send me the output of:

# sg_readcap -l /dev/sdc
# sg_vpd -l /dev/sdc
# sg_vpd -p 0xb0 /dev/sdc
# sg_vpd -p 0xb1 /dev/sdc
# sg_vpd -p 0xb2 /dev/sdc

Here it is:

  # sg_readcap -l /dev/sdc
  Read Capacity results:
     Protection: prot_en=0, p_type=0, p_i_exponent=0
     Logical block provisioning: lbpme=0, lbprz=0
     Last LBA=3907029167 (0xe8e088af), Number of logical blocks=3907029168
     Logical block length=512 bytes
     Logical blocks per physical block exponent=0
     Lowest aligned LBA=0
  Hence:
     Device size: 2000398934016 bytes, 1907729.1 MiB, 2000.40 GB, 2.00 TB
  # sg_vpd -l /dev/sdc
  Supported VPD pages VPD page:
     [PQual=0  Peripheral device type: disk]
    0x00  Supported VPD pages [sv]
    0x80  Unit serial number [sn]
    0x83  Device identification [di]
    0x87  Mode page policy [mpp]
    0x89  ATA information (SAT) [ai]
    0x8a  Power condition [pc]
    0xb0  Block limits (SBC) [bl]
    0xb1  Block device characteristics (SBC) [bdc]
    0xb2  Logical block provisioning (SBC) [lbpv]
    0xb6  Zoned block device characteristics [zbdch]
  # sg_vpd -p 0xb0 /dev/sdc
  Block limits VPD page (SBC):
    Write same non-zero (WSNZ): 1
    Maximum compare and write length: 0 blocks [Command not implemented]
    Optimal transfer length granularity: 0 blocks [not reported]
    Maximum transfer length: 0 blocks [not reported]
    Optimal transfer length: 0 blocks [not reported]
    Maximum prefetch transfer length: 0 blocks [ignored]
    Maximum unmap LBA count: 0 [Unmap command not implemented]
    Maximum unmap block descriptor count: 0 [Unmap command not implemented]
    Optimal unmap granularity: 0 blocks [not reported]
    Unmap granularity alignment valid: false
    Unmap granularity alignment: 0 [invalid]
    Maximum write same length: 0xffff blocks
    Maximum atomic transfer length: 0 blocks [not reported]
    Atomic alignment: 0 [unaligned atomic writes permitted]
    Atomic transfer length granularity: 0 [no granularity requirement
    Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
    Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]
  # sg_vpd -p 0xb1 /dev/sdc
  Block device characteristics VPD page (SBC):
    Nominal rotation rate: 7200 rpm
    Product type: Not specified
    WABEREQ=0
    WACEREQ=0
    Nominal form factor: 3.5 inch
    MACT=0
    ZONED=0
    RBWZ=0
    BOCS=0
    FUAB=0
    VBULS=0
    DEPOPULATION_TIME=0 (seconds)
  # sg_vpd -p 0xb2 /dev/sdc
  Logical block provisioning VPD page (SBC):
    Unmap command supported (LBPU): 0
    Write same (16) with unmap bit supported (LBPWS): 0
    Write same (10) with unmap bit supported (LBPWS10): 0
    Logical block provisioning read zeros (LBPRZ): 0
    Anchored LBAs supported (ANC_SUP): 0
    Threshold exponent: 0 [threshold sets not supported]
    Descriptor present (DP): 0
    Minimum percentage: 0 [not reported]
    Provisioning type: 0 (not known or fully provisioned)
    Threshold percentage: 0 [percentages not supported]

About the patch it will be tested later today.

Thanks,

Quoted 5 lines

Thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#4 lore raw

Hello all,

I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :

commit c92a6b5d63359dd6d2ce6ea88ecd8e31dd769f6b

Author:     Martin K. Petersen <[email protected]>
AuthorDate: Wed Mar 2 00:35:47 2022 -0500

scsi: core: Query VPD size before getting full page

Reverting this commit and building 6.6.80 kernel with the revert patch solved the issue.

Is it possible to receive the patch that was referenced by Bart (I wasn't able to find it) so I can give it a try please ?

Also sharing the output that was asked, not sure if this may help.

sg_readcap -l /dev/sdb :
Read Capacity results:
Protection: prot_en=0, p_type=0, p_i_exponent=0
Logical block provisioning: lbpme=1, lbprz=1
Last LBA=390721967 (0x1749f1af), Number of logical blocks=390721968 Logical block length=512 bytes
Logical blocks per physical block exponent=3 [so physical block length=4096 bytes] Lowest aligned LBA=0
Hence:
Device size: 200049647616 bytes, 190782.2 MiB, 200.05 GB

sg_vpd -l /dev/sdb :
Supported VPD pages VPD page:
[PQual=0 Peripheral device type: disk]

  0x00  Supported VPD pages [sv]
  0x80  Unit serial number [sn]
  0x83  Device identification [di]
  0x87  Mode page policy [mpp]
  0x89  ATA information (SAT) [ai]
  0xb0  Block limits (SBC) [bl]
  0xb1  Block device characteristics (SBC) [bdc]
  0xb2  Logical block provisioning (SBC) [lbpv]

sg_vpd -p 0xb0 /dev/sdb :
Block limits VPD page (SBC):
Write same non-zero (WSNZ): 0
Maximum compare and write length: 0 blocks [Command not implemented] Optimal transfer length granularity: 0 blocks [not reported] Maximum transfer length: 0 blocks [not reported]
Optimal transfer length: 0 blocks [not reported]
Maximum prefetch transfer length: 0 blocks [ignored]
Maximum unmap LBA count: 262143
Maximum unmap block descriptor count: 32
Optimal unmap granularity: 1 blocks
Unmap granularity alignment valid: false
Unmap granularity alignment: 0 [invalid]
Maximum write same length: 0 blocks [not reported]
Maximum atomic transfer length: 0 blocks [not reported]
Atomic alignment: 0 [unaligned atomic writes permitted]
Atomic transfer length granularity: 0 [no granularity requirement Maximum atomic transfer length with atomic boundary: 0 blocks [not reported] Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]

sg_vpd -p 0xb1 /dev/sdb :
Block device characteristics VPD page (SBC):
Non-rotating medium (e.g. solid state)
Product type: Not specified
WABEREQ=0
WACEREQ=0
Nominal form factor: 2.5 inch
MACT=0
ZONED=0
RBWZ=0
BOCS=0
FUAB=0
VBULS=0
DEPOPULATION_TIME=0 (seconds)

sg_vpd -p 0xb2 /dev/sdb :
Logical block provisioning VPD page (SBC):
Unmap command supported (LBPU): 1
Write same (16) with unmap bit supported (LBPWS): 1 Write same (10) with unmap bit supported (LBPWS10): 0 Logical block provisioning read zeros (LBPRZ): 0 Anchored LBAs supported (ANC_SUP): 1 Threshold exponent: 0 [threshold sets not supported] Descriptor present (DP): 0
Minimum percentage: 0 [not reported] Provisioning type: 0 (not known or fully provisioned) Threshold percentage: 0 [percentages not supported]

Thanks for any help!

Kind regards,
Lahfa Samy

#5 lore raw

Samy,

Quoted 3 lines

I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :

Can you try the patch below?

Thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
drivers/scsi/megaraid/megaraid_sas_base.c +7 -2
index 88acefbf9aea..8ced3f1fd427 100644
@@ -2106,6 +2106,9 @@ static int megasas_device_configure(struct scsi_device *sdev,
 	/* This sdev property may change post OCR */
 	megasas_set_dynamic_target_properties(sdev, lim, is_target_prop);
 
+	if (!MEGASAS_IS_LOGICAL(sdev))
+		sdev->no_vpd_size = 1;
+
 	mutex_unlock(&instance->reset_mutex);
 
 	return 0;
@@ -3665,8 +3668,10 @@ megasas_complete_cmd(struct megasas_instance *instance, struct megasas_cmd *cmd,
 
 		case MFI_STAT_SCSI_IO_FAILED:
 		case MFI_STAT_LD_INIT_IN_PROGRESS:
-			cmd->scmd->result =
-			    (DID_ERROR << 16) | hdr->scsi_status;
+			if (hdr->scsi_status == 0xf0)
+				cmd->scmd->result = (DID_ERROR << 16) | SAM_STAT_CHECK_CONDITION;
+			else
+				cmd->scmd->result = (DID_ERROR << 16) | hdr->scsi_status;
 			break;
 
 		case MFI_STAT_SCSI_DONE_WITH_ERROR:
drivers/scsi/megaraid/megaraid_sas_fusion.c +4 -1
index 6c1fb8149553..7d28b5b23751 100644
@@ -2043,7 +2043,10 @@ map_cmd_status(struct fusion_context *fusion,
 
 	case MFI_STAT_SCSI_IO_FAILED:
 	case MFI_STAT_LD_INIT_IN_PROGRESS:
-		scmd->result = (DID_ERROR << 16) | ext_status;
+		if (ext_status == 0xf0)
+			scmd->result = (DID_ERROR << 16) | SAM_STAT_CHECK_CONDITION;
+		else
+			scmd->result = (DID_ERROR << 16) | ext_status;
 		break;
 
 	case MFI_STAT_SCSI_DONE_WITH_ERROR:
Tested-by
#6 lore raw

Martin,

Quoted 2 lines · Martin K. Petersen

Samy,

(I work with Samy.)

Quoted 6 lines
Quoted 4 lines

I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :

Can you try the patch below?

Tested, this works, I do not see the error we reported earlier.

Quoted 2 lines

Thanks!

Thank you for the fast answer.

Kind regards,

Signature · 1 line
Ryan Lahfa
#7 lore raw

Ryan,

Quoted 3 lines

Tested, this works, I do not see the error we reported earlier.

Thanks for testing!

Broadcom: This fix never made it into a driver update. Please submit a formal patch. Thank you!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering