megaraid_sas: multiple FALLOC_FL_ZERO_RANGE causes timeouts and resets on MegaRAID 9560-8i 4GB since 5.19
Subsystems (2)
Hi,
We started to get timeouts and controller resets since 5.19.5 (vanilla v5.19 is not tested, tests below are on 6.6.15) when several ioctl FALLOC_FL_ZERO_RANGE are issued into device consequentially without delay between them (3-5 is enough to trigger condition). Because of this, for example, mkfs.ext4 extremely slows down when initializing filesystem. This happens on aarch64 (Kunpeng-920) server.
Reproducer:
# for ((i=0;i<5;i++)); do echo $i; fallocate -z -l 2097152 /dev/sdc; done
Example of dmesg messages after problematic ioctl calls:
Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 Abort request is for SMID: 4753 Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d51beacc) tm_dev_handle 0x4 Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000 Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 task abort FAILED!! scmd(0x00000000d51beacc) Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00 Feb 06 19:45:04 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 Abort request is for SMID: 8293 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d9406c9c) tm_dev_handle 0x4 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 BRCM Debug mfi stat 0x2d, data len requested/completed 0x1000/0x0 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 task abort SUCCESS!! scmd(0x00000000d9406c9c) Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 CDB: Write Same(10) 41 00 03 4c 00 10 00 10 00 00 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting target reset! scmd(0x00000000d51beacc) tm_dev_handle: 0x4 Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 target reset SUCCESS!! Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: Power-on or device reset occurred
Excerpt from the controller events log (from storli):
Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: Unexpected sense: Encl PD fb Path 5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: PD 05(e0xfb/s4) Path 5e8b4700e35e2004 reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: PD 05(e0xfb/s4) Path 5e8b4700e35e2004, CDB: 41 00 00 00 00 00 00 10 00 00, Sense: 6/29/00
Tests was on the latest firmware (at the moment):
Product Name = MegaRAID 9560-8i 4GB
Serial Number = SKC4006982
Firmware Package Build = 52.28.0-5305
Firmware Version = 5.280.02-3972
PSOC FW Version = 0x001A
PSOC Hardware Version = 0x000A
PSOC Part Number = 29211-260-4GB
NVDATA Version = 5.2800.00-0752
CBB Version = 28.250.04.00
Bios Version = 7.28.00.0_0x071C0000
HII Version = 07.28.04.00
HIIA Version = 07.28.04.00
Driver Name = megaraid_sas
Driver Version = 07.725.01.00-rc1
I tried also latest available megaraid_sas driver (07.728.04.00) which is not yet merged into mainline but the problems are not resolved with it.
Thanks,
Hi,
Quoted 7 lines · Vitaly Chikunov
We started to get timeouts and controller resets since 5.19.5 (vanilla v5.19 is not tested, tests below are on 6.6.15) when several ioctl
FALLOC_FL_ZERO_RANGEare issued into device consequentially without delay between them (3-5 is enough to trigger condition). Because of this, for example, mkfs.ext4 extremely slows down when initializing filesystem. This happens on aarch64 (Kunpeng-920) server.
I am reported that bisect found this commit to cause above mentioned problem:
commit c92a6b5d63359dd6d2ce6ea88ecd8e31dd769f6b
Author: Martin K. Petersen <[email protected]>
AuthorDate: Wed Mar 2 00:35:47 2022 -0500
scsi: core: Query VPD size before getting full page
When from v5.19 this commit is reverted the problem disappears.
Thanks,
Quoted 56 lines
Reproducer:
# for ((i=0;i<5;i++)); do echo $i; fallocate -z -l 2097152 /dev/sdc; doneExample of dmesg messages after problematic ioctl calls:
Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 Abort request is for SMID: 4753 Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d51beacc) tm_dev_handle 0x4 Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 Feb 06 19:44:07 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000 Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 task abort FAILED!! scmd(0x00000000d51beacc) Feb 06 19:44:07 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00 Feb 06 19:45:04 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 Abort request is for SMID: 8293 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting task abort! scmd(0x00000000d9406c9c) tm_dev_handle 0x4 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 BRCM Debug mfi stat 0x2d, data len requested/completed 0x1000/0x0 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 task abort SUCCESS!! scmd(0x00000000d9406c9c) Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#8292 CDB: Write Same(10) 41 00 03 4c 00 10 00 10 00 00 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: attempting target reset! scmd(0x00000000d51beacc) tm_dev_handle: 0x4 Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009 Feb 06 19:45:06 host-226 kernel: megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000 Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: [sdc] tag#4752 target reset SUCCESS!! Feb 06 19:45:06 host-226 kernel: sd 0:2:4:0: Power-on or device reset occurredExcerpt from the controller events log (from storli):
Event Description: PD 05(e0xfb/s4) Path
5e8b4700e35e2004reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: Encl PD fb Path5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: Unexpected sense: Encl PD fb Path5e8b4700e35e201e, CDB: 3c 01 05 00 00 00 00 00 10 00, Sense: b/4b/05 Event Description: PD 05(e0xfb/s4) Path5e8b4700e35e2004reset (Type 03)
Event Description: Drive PD 05(e0xfb/s4) link speed changed
Event Description: Unexpected sense: PD 05(e0xfb/s4) Path5e8b4700e35e2004, CDB: 41 00 00 00 00 00 00 10 00 00, Sense: 6/29/00Tests was on the latest firmware (at the moment):
Product Name = MegaRAID 9560-8i 4GB
Serial Number = SKC4006982
Firmware Package Build = 52.28.0-5305
Firmware Version = 5.280.02-3972
PSOC FW Version =0x001A
PSOC Hardware Version =0x000A
PSOC Part Number = 29211-260-4GB
NVDATA Version = 5.2800.00-0752
CBB Version = 28.250.04.00
Bios Version = 7.28.00.0_0x071C0000
HII Version = 07.28.04.00
HIIA Version = 07.28.04.00
Driver Name =megaraid_sas
Driver Version = 07.725.01.00-rc1I tried also latest available
megaraid_sasdriver (07.728.04.00) which is not yet merged into mainline but the problems are not resolved with it.Thanks,
Vitaly,
I'd appreciate it if you could test the patch Bart referred you to.
Quoted 2 lines
I am reported that bisect found this commit to cause above mentioned problem:
Also, I would like to understand why things fail as a result of the original change.
Could you please send me the output of:
# sg_readcap -l /dev/sdc # sg_vpd -l /dev/sdc # sg_vpd -p 0xb0 /dev/sdc # sg_vpd -p 0xb1 /dev/sdc # sg_vpd -p 0xb2 /dev/sdc
Thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin,
Quoted 9 lines · Martin K. Petersen
I'd appreciate it if you could test the patch Bart referred you to.
Quoted 2 lines
I am reported that bisect found this commit to cause above mentioned problem:
Also, I would like to understand why things fail as a result of the original change.
Quoted 7 lines
Could you please send me the output of:
# sg_readcap -l /dev/sdc # sg_vpd -l /dev/sdc # sg_vpd -p 0xb0 /dev/sdc # sg_vpd -p 0xb1 /dev/sdc # sg_vpd -p 0xb2 /dev/sdc
Here it is:
# sg_readcap -l /dev/sdc
Read Capacity results:
Protection: prot_en=0, p_type=0, p_i_exponent=0
Logical block provisioning: lbpme=0, lbprz=0
Last LBA=3907029167 (0xe8e088af), Number of logical blocks=3907029168
Logical block length=512 bytes
Logical blocks per physical block exponent=0
Lowest aligned LBA=0
Hence:
Device size: 2000398934016 bytes, 1907729.1 MiB, 2000.40 GB, 2.00 TB # sg_vpd -l /dev/sdc
Supported VPD pages VPD page:
[PQual=0 Peripheral device type: disk]
0x00 Supported VPD pages [sv]
0x80 Unit serial number [sn]
0x83 Device identification [di]
0x87 Mode page policy [mpp]
0x89 ATA information (SAT) [ai]
0x8a Power condition [pc]
0xb0 Block limits (SBC) [bl]
0xb1 Block device characteristics (SBC) [bdc]
0xb2 Logical block provisioning (SBC) [lbpv]
0xb6 Zoned block device characteristics [zbdch] # sg_vpd -p 0xb0 /dev/sdc
Block limits VPD page (SBC):
Write same non-zero (WSNZ): 1
Maximum compare and write length: 0 blocks [Command not implemented]
Optimal transfer length granularity: 0 blocks [not reported]
Maximum transfer length: 0 blocks [not reported]
Optimal transfer length: 0 blocks [not reported]
Maximum prefetch transfer length: 0 blocks [ignored]
Maximum unmap LBA count: 0 [Unmap command not implemented]
Maximum unmap block descriptor count: 0 [Unmap command not implemented]
Optimal unmap granularity: 0 blocks [not reported]
Unmap granularity alignment valid: false
Unmap granularity alignment: 0 [invalid]
Maximum write same length: 0xffff blocks
Maximum atomic transfer length: 0 blocks [not reported]
Atomic alignment: 0 [unaligned atomic writes permitted]
Atomic transfer length granularity: 0 [no granularity requirement
Maximum atomic transfer length with atomic boundary: 0 blocks [not reported]
Maximum atomic boundary size: 0 blocks [can only write atomic 1 block] # sg_vpd -p 0xb1 /dev/sdc
Block device characteristics VPD page (SBC):
Nominal rotation rate: 7200 rpm
Product type: Not specified
WABEREQ=0
WACEREQ=0
Nominal form factor: 3.5 inch
MACT=0
ZONED=0
RBWZ=0
BOCS=0
FUAB=0
VBULS=0
DEPOPULATION_TIME=0 (seconds) # sg_vpd -p 0xb2 /dev/sdc
Logical block provisioning VPD page (SBC):
Unmap command supported (LBPU): 0
Write same (16) with unmap bit supported (LBPWS): 0
Write same (10) with unmap bit supported (LBPWS10): 0
Logical block provisioning read zeros (LBPRZ): 0
Anchored LBAs supported (ANC_SUP): 0
Threshold exponent: 0 [threshold sets not supported]
Descriptor present (DP): 0
Minimum percentage: 0 [not reported]
Provisioning type: 0 (not known or fully provisioned)
Threshold percentage: 0 [percentages not supported]About the patch it will be tested later today.
Thanks,
Quoted 5 lines
Thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Hello all,
I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :
commit c92a6b5d63359dd6d2ce6ea88ecd8e31dd769f6b
Author: Martin K. Petersen <[email protected]> AuthorDate: Wed Mar 2 00:35:47 2022 -0500
scsi: core: Query VPD size before getting full page
Reverting this commit and building 6.6.80 kernel with the revert patch solved the issue.
Is it possible to receive the patch that was referenced by Bart (I wasn't able to find it) so I can give it a try please ?
Also sharing the output that was asked, not sure if this may help.
sg_readcap -l /dev/sdb :
Read Capacity results:
Protection: prot_en=0, p_type=0, p_i_exponent=0
Logical block provisioning: lbpme=1, lbprz=1
Last LBA=390721967 (0x1749f1af), Number of logical blocks=390721968 Logical block length=512 bytes
Logical blocks per physical block exponent=3 [so physical block length=4096 bytes] Lowest aligned LBA=0
Hence:
Device size: 200049647616 bytes, 190782.2 MiB, 200.05 GB
sg_vpd -l /dev/sdb :
Supported VPD pages VPD page:
[PQual=0 Peripheral device type: disk]
0x00 Supported VPD pages [sv] 0x80 Unit serial number [sn] 0x83 Device identification [di] 0x87 Mode page policy [mpp] 0x89 ATA information (SAT) [ai] 0xb0 Block limits (SBC) [bl] 0xb1 Block device characteristics (SBC) [bdc] 0xb2 Logical block provisioning (SBC) [lbpv]
sg_vpd -p 0xb0 /dev/sdb :
Block limits VPD page (SBC):
Write same non-zero (WSNZ): 0
Maximum compare and write length: 0 blocks [Command not implemented] Optimal transfer length granularity: 0 blocks [not reported] Maximum transfer length: 0 blocks [not reported]
Optimal transfer length: 0 blocks [not reported]
Maximum prefetch transfer length: 0 blocks [ignored]
Maximum unmap LBA count: 262143
Maximum unmap block descriptor count: 32
Optimal unmap granularity: 1 blocks
Unmap granularity alignment valid: false
Unmap granularity alignment: 0 [invalid]
Maximum write same length: 0 blocks [not reported]
Maximum atomic transfer length: 0 blocks [not reported]
Atomic alignment: 0 [unaligned atomic writes permitted]
Atomic transfer length granularity: 0 [no granularity requirement Maximum atomic transfer length with atomic boundary: 0 blocks [not reported] Maximum atomic boundary size: 0 blocks [can only write atomic 1 block]
sg_vpd -p 0xb1 /dev/sdb :
Block device characteristics VPD page (SBC):
Non-rotating medium (e.g. solid state)
Product type: Not specified
WABEREQ=0
WACEREQ=0
Nominal form factor: 2.5 inch
MACT=0
ZONED=0
RBWZ=0
BOCS=0
FUAB=0
VBULS=0
DEPOPULATION_TIME=0 (seconds)
sg_vpd -p 0xb2 /dev/sdb :
Logical block provisioning VPD page (SBC):
Unmap command supported (LBPU): 1
Write same (16) with unmap bit supported (LBPWS): 1 Write same (10) with unmap bit supported (LBPWS10): 0 Logical block provisioning read zeros (LBPRZ): 0 Anchored LBAs supported (ANC_SUP): 1 Threshold exponent: 0 [threshold sets not supported] Descriptor present (DP): 0
Minimum percentage: 0 [not reported] Provisioning type: 0 (not known or fully provisioned) Threshold percentage: 0 [percentages not supported]
Thanks for any help!
Kind regards,
Lahfa Samy
Samy,
Quoted 3 lines
I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :
Can you try the patch below?
Thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
drivers/scsi/megaraid/megaraid_sas_base.c +7 -2
@@ -2106,6 +2106,9 @@ static int megasas_device_configure(struct scsi_device *sdev,/* This sdev property may change post OCR */ megasas_set_dynamic_target_properties (sdev, lim, is_target_prop); +if (!MEGASAS_IS_LOGICAL (sdev)) + sdev->no_vpd_size =1 ; +mutex_unlock (&instance->reset_mutex);return 0 ;
@@ -3665,8 +3668,10 @@ megasas_complete_cmd(struct megasas_instance *instance, struct megasas_cmd *cmd,case MFI_STAT_SCSI_IO_FAILED :case MFI_STAT_LD_INIT_IN_PROGRESS : - cmd->scmd->result = - (DID_ERROR <<16 ) | hdr->scsi_status; +if (hdr->scsi_status ==0xf0 ) + cmd->scmd->result = (DID_ERROR <<16 ) |SAM_STAT_CHECK_CONDITION ; +else + cmd->scmd->result = (DID_ERROR <<16 ) | hdr->scsi_status;break ;case MFI_STAT_SCSI_DONE_WITH_ERROR :
drivers/scsi/megaraid/megaraid_sas_fusion.c +4 -1
@@ -2043,7 +2043,10 @@ map_cmd_status(struct fusion_context *fusion,case MFI_STAT_SCSI_IO_FAILED :case MFI_STAT_LD_INIT_IN_PROGRESS : - scmd->result = (DID_ERROR <<16 ) | ext_status; +if (ext_status ==0xf0 ) + scmd->result = (DID_ERROR <<16 ) |SAM_STAT_CHECK_CONDITION ; +else + scmd->result = (DID_ERROR <<16 ) | ext_status;break ;case MFI_STAT_SCSI_DONE_WITH_ERROR :
Ryan Lahfa
Tested-byMartin,
Quoted 2 lines · Martin K. Petersen
Samy,
(I work with Samy.)
Quoted 6 lines
Quoted 4 lines
I have just ran into this issue, controller resets and timeouts (running mkfs.ext4 or mkfs.xfs to reproduce) and bisected it to the same commit :
Can you try the patch below?
Tested, this works, I do not see the error we reported earlier.
Quoted 2 lines
Thanks!
Thank you for the fast answer.
Kind regards,
Signature · 1 line
Ryan Lahfa
Ryan,
Quoted 3 lines
Tested, this works, I do not see the error we reported earlier.
Tested-by Ryan Lahfa <[email protected]>
Thanks for testing!
Broadcom: This fix never made it into a driver update. Please submit a formal patch. Thank you!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering