Skip to contentSkip to conversation

SSD SATA 3.3 and Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS

DISCUSSION deloptes 3 participants 11 messages Aug 13, 2023 – Sep 29, 2023

Subsystems (2)
lore
Share & tools
lore

deloptes

author
#0 lore raw

Hi all,

I hope I am in the right news group. If not please point me to a place where someone can give me an advice in which direction I could look for an answer.

I've been using the below mentioned board (see dmidecode) with the below mentioned SAS controllers (see lspci) for many years with normal (rotating/spinning) disk drives.
I now bought 2 SSD disks to replace 2 of the spinning once and I was surprised to find out that the older disks are using 3Gb/s transfer rate while the SSDs are using 1.5Gb/s. The SSDs are reporting SATA 3.3 (see below)

In the mptsas driver I see 3 and 6 but no 3.3 or similar.

https://github.com/torvalds/linux/blob/ae545c3283dc673f7e748065efa46ba95f678ef2/drivers/message/fusion/mptsas.c#L3087C1-L3111C3

If I understand correctly the rate is negotiated by the controller and the mobo. I am wondering where could be the problem. Is it really matter of negotiation i.e. the driver does not understand 3.3 or a technical constrains/incompatibility at 3Gb/s?

Last question: If I would have to replace the controllers, what controller would be recommended?

Thank you in advance

BR

# lspci
01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
08:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
# dmidecode

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: M5A97 EVO R2.0
Version: Rev 1.xx

Handle 0x0028, DMI type 9, 17 bytes
System Slot Information
Designation: PCIEX16_2
Type: x16 PCI Express
Current Usage: Available
Length: Short
ID: 3
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:00:1c.5

# smartctl

=== START OF INFORMATION SECTION ===

Model Family:     WD Blue / Red / Green SSDs
Device Model:     WDC  WDS200T1R0A-68A4W0
...
Firmware Version: 411010WR
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
...
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
#1 lore raw

[+cc mptsas folks]

Quoted 3 lines · deloptes

I hope I am in the right news group. If not please point me to a place where someone can give me an advice in which direction I could look for an answer.

Thanks for your question. I added the mptsas maintainers and related mailing lists.

Quoted 8 lines

I've been using the below mentioned board (see dmidecode) with the below mentioned SAS controllers (see lspci) for many years with normal (rotating/spinning) disk drives.
I now bought 2 SSD disks to replace 2 of the spinning once and I was surprised to find out that the older disks are using 3Gb/s transfer rate while the SSDs are using 1.5Gb/s. The SSDs are reporting SATA 3.3 (see below)

Quoted 8 lines

In the mptsas driver I see 3 and 6 but no 3.3 or similar.

https://github.com/torvalds/linux/blob/ae545c3283dc673f7e748065efa46ba95f678ef2/drivers/message/fusion/mptsas.c#L3087C1-L3111C3

If I understand correctly the rate is negotiated by the controller and the mobo. I am wondering where could be the problem. Is it really matter of negotiation i.e. the driver does not understand 3.3 or a technical constrains/incompatibility at 3Gb/s?

It looks like your SSDs support SATA r3.3 and are capable of 6.0 Gb/s, and the SAS1068E SAS controller is limited to 3.0 Gb/s [1]. I would expect them to negotiate 3.0 Gb/s (as your older drives did) and you're only seeing 1.5 Gb/s. Sorry, I guess I just restated your whole question ;)

I don't know why that would be. Are there any hints in the dmesg log? Can you collect the complete dmesg log with the old drives and again with the new SSDs so we can compare them? I assume you have good cables? I assume the same cables worked at 3.0 Gb/s with the old drives.

I would *expect* that SATA r3.3 would be completely backwards compatible, so since mptsas worked just fine at 3.0 Gb/s with the old SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the new SATA r3.3 drives. But I have no actual knowledge about that.

[1] https://docs.broadcom.com/doc/12352180

Quoted 74 lines

Last question: If I would have to replace the controllers, what controller would be recommended?

Thank you in advance

BR

# lspci
01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
08:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express
Fusion-MPT SAS (rev 08)
# dmidecode

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: M5A97 EVO R2.0
Version: Rev 1.xx

Handle 0x0028, DMI type 9, 17 bytes
System Slot Information
Designation: PCIEX16_2
Type: x16 PCI Express
Current Usage: Available
Length: Short
ID: 3
Characteristics:
3.3 V is provided
Opening is shared
PME signal is supported
Bus Address: 0000:00:1c.5

# smartctl

=== START OF INFORMATION SECTION ===

Model Family:     WD Blue / Red / Green SSDs
Device Model:     WDC  WDS200T1R0A-68A4W0
...
Firmware Version: 411010WR
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
...
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) <<<<<<<<<<<< HERE
Local Time is:    Sun Aug 13 10:38:11 2023 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
author
#2 lore raw
Quoted 10 lines · Bjorn Helgaas

I don't know why that would be.  Are there any hints in the dmesg log? Can you collect the complete dmesg log with the old drives and again with the new SSDs so we can compare them?  I assume you have good cables?  I assume the same cables worked at 3.0 Gb/s with the old drives.

I would *expect* that SATA r3.3 would be completely backwards compatible, so since mptsas worked just fine at 3.0 Gb/s with the old SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the new SATA r3.3 drives.  But I have no actual knowledge about that.

Thank you for your answer. I am also confused and couldn't think of any meaningful reason. This is why I allowed myself to bother you.

I did not change anything - wiring or such. The server has 12 disk bays on the front. Old disks were pulled out and new disks were inserted into the bays.

You (probably much knowable in this matters than me) also assume negotiation should result in 3.0Gb/s. And if I understand correctly it should be not a driver issue.

The only difference I could find out for now is that Rev3.3 introduced PWDIS on Pin 3. To check if the cables provide wiring on P3 I should disassemble the server, but I can do this in September :/ and it is a lot of effort.

I am attaching a portion of the log and dmesg with the relevant information. I see that ASPM is disabled by default (could it be related to P3?).

Thank you all in advance
BR

Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
#3 lore raw

This has nothing to do with the driver and would require analysis from the controller hardware/firmware perspective. The 1068 chip is pretty old and out of support and it will be difficult to get any analysis done on that. If you want to upgrade, the latest 12G SAS-NVMe Tri Mode controllers are better from a long term support perspective.
https://www.broadcom.com/products/storage/host-bus-adapters.

Thanks.

Quoted 37 lines · deloptes
Quoted 10 lines · Bjorn Helgaas

I don't know why that would be. Are there any hints in the dmesg log? Can you collect the complete dmesg log with the old drives and again with the new SSDs so we can compare them? I assume you have good cables? I assume the same cables worked at 3.0 Gb/s with the old drives.

I would *expect* that SATA r3.3 would be completely backwards compatible, so since mptsas worked just fine at 3.0 Gb/s with the old SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the new SATA r3.3 drives. But I have no actual knowledge about that.

Thank you for your answer. I am also confused and couldn't think of any meaningful reason. This is why I allowed myself to bother you.

I did not change anything - wiring or such. The server has 12 disk bays on the front. Old disks were pulled out and new disks were inserted into the bays.

You (probably much knowable in this matters than me) also assume negotiation should result in 3.0Gb/s. And if I understand correctly it should be not a driver issue.

The only difference I could find out for now is that Rev3.3 introduced PWDIS on Pin 3. To check if the cables provide wiring on P3 I should disassemble the server, but I can do this in September :/ and it is a lot of effort.

I am attaching a portion of the log and dmesg with the relevant information. I see that ASPM is disabled by default (could it be related to P3?).

Thank you all in advance
BR

--
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0

author
#4 lore raw

Sathya Prakash Veerichetty wrote:

Thank you for responding

Quoted 3 lines

This has nothing to do with the driver and would require analysis from the controller hardware/firmware perspective.  The 1068 chip is pretty old and out of support and it will be difficult to get any analysis

Well, this is unfortunate. Even if I enable debugging in the kernel (BTW I forgot to mention I use 4.19.288) nothing will be useful?

Quoted 3 lines

done on that.  If you want to upgrade, the latest 12G SAS-NVMe Tri Mode controllers are better from a long term support perspective.
https://www.broadcom.com/products/storage/host-bus-adapters.

Well, I was considering to upgrade, but the investment is not worth it - I calculated > €1000,- for controllers and NVMe disks. If I were to do something like this, I would replace the whole machine as this one is pretty old. The advantage with it is that it consumes around 80-90Watts in total and newer once usually burn up more because of the CPUs.

What I would rather do is to plug in those SSDs directly to the mobo and use the WD Red with 5800RPM in the bays.

Still, if you have any idea on how I can try debuggin it, please let me know. I compiled the 4.19.288 with loggin enabled, but I am not sure this is what I need.

--- Fusion MPT device support
<M> Fusion MPT ScsiHost drivers for SPI
<M> Fusion MPT ScsiHost drivers for FC
<*> Fusion MPT ScsiHost drivers for SAS
(128) Maximum number of scatter gather entries (16 - 128)
<M> Fusion MPT misc device (ioctl) driver
[*] Fusion MPT logging facility

I am attaching the full config below. If it is not asked too much, please have a look and let me know if I missed something.
I will look forward to test it in Sept. when I have access to the server.

Thank you in advance

Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
#5 lore raw
In reply to deloptes
Quoted 24 lines · deloptes
Quoted 10 lines · Bjorn Helgaas

I don't know why that would be.  Are there any hints in the dmesg log? Can you collect the complete dmesg log with the old drives and again with the new SSDs so we can compare them?  I assume you have good cables?  I assume the same cables worked at 3.0 Gb/s with the old drives.

I would *expect* that SATA r3.3 would be completely backwards compatible, so since mptsas worked just fine at 3.0 Gb/s with the old SATA r3.0 drives, it should also work just fine at 3.0 Gb/s with the new SATA r3.3 drives.  But I have no actual knowledge about that.

Thank you for your answer. I am also confused and couldn't think of any meaningful reason. This is why I allowed myself to bother you.

I did not change anything - wiring or such. The server has 12 disk bays on the front. Old disks were pulled out and new disks were inserted into the bays.

You (probably much knowable in this matters than me) also assume negotiation should result in 3.0Gb/s. And if I understand correctly it should be not a driver issue.

Quoted 6 lines

The only difference I could find out for now is that Rev3.3 introduced PWDIS on Pin 3. To check if the cables provide wiring on P3 I should disassemble the server, but I can do this in September :/ and it is a lot of effort.

I am attaching a portion of the log and dmesg with the relevant information. I see that ASPM is disabled by default (could it be related to P3?).

ASPM is a PCIe feature that applies to the PCIe Links between 00:02.0 and 01:00.0 (the first 1068E) and 00:15.0 and 08:00.0 (the second 1068E).

PWDIS is a feature on the SATA cables between the SAS1068E adapters and the SSDs.

PWDIS/P3 should not be related to ASPM. I assume you're referring to the "disabling ASPM on pre-1.1 PCIe device" message. That should happen with both the old r3.0 HDDs and the new r3.3 SSDs.

I wish I had some good ideas for you, but I don't know anything about the SATA side. I googled for "1068 ssd sata 1.5 gb/s" and found a few hints about system firmware, LSI firmware, etc, but nothing concrete.

I think some controllers have a BIOS setup user interface; have you poked around in there?

Bjorn

author
#6 lore raw

Bjorn Helgaas wrote:

Thank you Bjorn for answering and explaining

Quoted 11 lines

ASPM is a PCIe feature that applies to the PCIe Links between 00:02.0 and 01:00.0 (the first 1068E) and 00:15.0 and 08:00.0 (the second 1068E).

PWDIS is a feature on the SATA cables between the SAS1068E adapters and the SSDs.

PWDIS/P3 should not be related to ASPM.  I assume you're referring to the "disabling ASPM on pre-1.1 PCIe device" message.  That should happen with both the old r3.0 HDDs and the new r3.3 SSDs.

Yes, this is what I meant. So it can be ignored as well as the ASPM. I will check the cables, when I can get hands on the machine - just out of curiosity.

Quoted 4 lines

I wish I had some good ideas for you, but I don't know anything about the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few hints about system firmware, LSI firmware, etc, but nothing concrete.

Yeah, if it were so easy, I wouldn't be bothering you.

Quoted 2 lines

I think some controllers have a BIOS setup user interface; have you poked around in there?

I have not seen the bios of the machine for many years. I was looking forward to plug it to a console, so that I can reboot remotely, but for some reason it was not possible. It was may be 5y ago. I will definitely double check this, allthough there will be nothing regarding SATA3.3 there as these were build many years before SATA3.3 saw daylight.

As mentioned in the other posting I will attach those SSDs directly to the mobo. There are 6 SATA ports there. I think this is the best approach. But you know curiosity is a force you can not resist, so I still want to know why?! :)

Thank you anyway
BR

Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
#7 lore raw
Quoted 11 lines · deloptes

Bjorn Helgaas wrote:
...

Quoted 2 lines

I think some controllers have a BIOS setup user interface; have you poked around in there?

I have not seen the bios of the machine for many years. I was looking forward to plug it to a console, so that I can reboot remotely, but for some reason it was not possible. It was may be 5y ago. I will definitely double check this, allthough there will be nothing regarding SATA3.3 there as these were build many years before SATA3.3 saw daylight.

I saw some mention about BIOS knobs that controlled the minimum acceptable SATA link speed or something; that's the kind of thing I wondered about.

Quoted 4 lines

As mentioned in the other posting I will attach those SSDs directly to the mobo. There are 6 SATA ports there. I think this is the best approach. But you know curiosity is a force you can not resist, so I still want to know why?! :)

Haha, yeah, I know! I noticed in your response to Sathya that you're running a 4.19.288 kernel, which is really, really old. If it's practical, the first thing I would try is booting a current kernel, e.g., v6.4, on the chance that something has been fixed since v4.19. I didn't try to compare the mptsas driver to see if it has changed since then, so I don't know whether it's likely.

Bjorn

author
#8 lore raw
Quoted 4 lines · Bjorn Helgaas

I saw some mention about BIOS knobs that controlled the minimum acceptable SATA link speed or something; that's the kind of thing I wondered about.

You surely mean the SAS1068E controller. I'll have to look into the documentation. Thanks for the advice.

Quoted 11 lines
Quoted 4 lines

As mentioned in the other posting I will attach those SSDs directly to the mobo. There are 6 SATA ports there. I think this is the best approach. But you know curiosity is a force you can not resist, so I still want to know why?! :)

Haha, yeah, I know!  I noticed in your response to Sathya that you're running a 4.19.288 kernel, which is really, really old.  If it's practical, the first thing I would try is booting a current kernel, e.g., v6.4, on the chance that something has been fixed since v4.19. I didn't try to compare the mptsas driver to see if it has changed since then, so I don't know whether it's likely.

I'll check and try in any case. Thanks for the advice.

I'll report back in Sept.

BR

Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
author
#9 lore raw
In reply to Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas

Haha, yeah, I know!  I noticed in your response to Sathya that you're running a 4.19.288 kernel, which is really, really old.  If it's practical, the first thing I would try is booting a current kernel, e.g., v6.4, on the chance that something has been fixed since v4.19. I didn't try to compare the mptsas driver to see if it has changed since then, so I don't know whether it's likely.

Hi,
writing politely in reply to your politeness :). I upgraded to 5.15.131. I need a stable system and do not want to jump that far to 6.x.

But it seems this issue is related to a defect in the firmware of the controller. I will try to reflash a newer version during the week. I hope it helps.

Current version is 01.26.00.00 (see [1]). In the readme[2], I read

Phase21 Pre-Alpha Release Version 00.33.01.00 - SAS1_FW_Ph_21 (SCGCQ00177063) Change Summary ( Defects=5) SCGCQ00173656 (DFCT) - 6G SATA drive negotiates to 1.5G speed

I will also check the BIOS settings, when in front of the machine it could be that link speed is enforcable

BR

[1] Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3)
PCI Address:                    00:08:00:00
SAS Address:                    500605b-0-02d0-f3b0
NVDATA Version (Default):       0x2d00
NVDATA Version (Persistent):    0x2d00
Product ID:                     0x2704
Firmware Version:               01.26.00.00
NVDATA Vendor:                  Intel
NVDATA Product ID:              SASUC8I
BIOS Version:                   06.24.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A

[2]
SCG Engineering Release Notice
Product: Firmware
Version: 01.33.00.00
Release Date: 22-AUG-11
Release Type: GCA

Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0
author
#10 lore raw
In reply to Bjorn Helgaas
Quoted 3 lines · Bjorn Helgaas

I wish I had some good ideas for you, but I don't know anything about the SATA side.  I googled for "1068 ssd sata 1.5 gb/s" and found a few hints about system firmware, LSI firmware, etc, but nothing concrete.

Hi,
I did the firmware upgrade on the two controllers and the issue is solved. Interestingly NVDATA Vendor and Product ID changed as well

BEFORE FW v. 1.26 BIOS v. 6.24

        Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3) PCI Address:                    00:08:00:00 SAS Address:                    500605b-0-02d0-f3b0 NVDATA Version (Default):       0x2d00 NVDATA Version (Persistent):    0x2d00 Product ID:                     0x2704 Firmware Version:               01.26.00.00 NVDATA Vendor:                  Intel NVDATA Product ID:              SASUC8I BIOS Version:                   06.24.00.00 BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A

AFTER FW v. 1.33 BIOS v. 6.36

Adapter Selected is a LSI SAS 1068E(B3):

Controller Number:              2
Controller:                     1068E(B3)
PCI Address:                    00:08:00:00
SAS Address:                    500605b-0-02d0-f3b0
NVDATA Version (Default):       0x2d03
NVDATA Version (Persistent):    0x2d03
Product ID:                     0x2704
Firmware Version:               01.33.00.00
NVDATA Vendor:                  LSILogic
NVDATA Product ID:              SAS3081E
BIOS Version:                   06.36.00.00
BIOS Alternate Version:         N/A
EFI BSD Version:                N/A
FCODE Version:                  N/A

smartctl

Model Family:     WD Blue / Red / Green SSDs
Device Model:     WDC  WDS200T1R0A-68A4W0
...
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)
Signature · 1 line
FCD6 3719 0FFB F1BF 38EA 4727 5348 5F1F DCFE BCB0