mptsas problem
Quoted 1 line
I recall seeing some of this problem on the list a few days ago, but I > can't remember what the subject was and was unable to find the thread.
You are perhaps thinking of this:
http://marc.info/?l=linux-scsi&m=120696978819085&w=2
but it seems to be a different issue.
From the message you posted, it looks as though there may be a problem with sda.
Does smartctl -a -d ata /dev/sda show any obvious problems?
Regards,
Richard
Quoted 6 lines · Richard Scobie
Quoted 1 line
I recall seeing some of this problem on the list a few days ago, but
I > can't remember what the subject was and was unable to find the thread.
You are perhaps thinking of this:
Yes.
Quoted 1 line
but it seems to be a different issue.
Ok.
Quoted 2 lines
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Quoted 1 line
Does smartctl -a -d ata
/dev/sdashow any obvious problems?
smartctl doesn't work on sd[a-d] at all:
# smartctl -a -d ata /dev/sdd smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
# smartctl -a -d ata -T permissive /dev/sdd smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
=== START OF INFORMATION SECTION ===
Device Model: [No Information Found] Serial Number: [No Information Found] Firmware Version: [No Information Found] Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 1 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sun Apr 6 21:02:45 2008 EDT
SMART is only available in ATA Version 3 Revision 3 or greater.
We will try to proceed in spite of this.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
Checking for SMART support by trying SMART ENABLE command.
Error SMART Enable failed: Invalid argument
SMART ENABLE failed - this establishes that this device lacks SMART functionality.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. #
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
Quoted 29 lines · Wakko Warner
Quoted 6 lines · Richard Scobie
Quoted 1 line
I recall seeing some of this problem on the list a few days ago, but
I > can't remember what the subject was and was unable to find the thread.
You are perhaps thinking of this:
Yes.
Quoted 1 line
but it seems to be a different issue.
Ok.
Quoted 1 line
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Quoted 1 line
Does smartctl -a -d ata
/dev/sdashow any obvious problems?smartctl doesn't work on sd[a-d] at all:
# smartctl -a -d ata /dev/sdd smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Quoted 5 lines
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
# smartctl -a -d ata -T permissive /dev/sdd
As a smartmontools developer I know that '-d ata' is incorrect in this context. The 'd <command_set> should either not be given or '-d sat' should be given.
This is difficult to explain and hence put in an easily understood option. As far as linux is concerned SCSI commands are being issued to a SCSI device (on a SCSI transport). But those SCSI commands are mostly instances of the SCSI ATA PASS-THROUGH command.
It is further complicated by the fact that a properly implemented SAT layer (and I have never met one) implements the SCSI commands used for SMART support. If that was the case 'smartctl -a -d scsi /dev/sdd' would yield useful output which would be a subset of what 'd sat' would yield.
Clear as mud?
Doug Gilbert
Quoted 22 lines
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
=== START OF INFORMATION SECTION ===
Device Model: [No Information Found] Serial Number: [No Information Found] Firmware Version: [No Information Found] Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 1 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sun Apr 6 21:02:45 2008 EDTSMART is only available in ATA Version 3 Revision 3 or greater.
We will try to proceed in spite of this.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
Checking for SMART support by trying SMART ENABLE command.
Error SMART Enable failed: Invalid argument
SMART ENABLE failed - this establishes that this device lacks SMART functionality.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. #
James Bottomley
maintainerQuoted 41 lines · Douglas Gilbert
Quoted 30 lines · Wakko Warner
Quoted 6 lines · Richard Scobie
Quoted 1 line
I recall seeing some of this problem on the list a few days ago, but
I > can't remember what the subject was and was unable to find the thread.
You are perhaps thinking of this:
Yes.
Quoted 1 line
but it seems to be a different issue.
Ok.
Quoted 2 lines
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Quoted 1 line
Does smartctl -a -d ata
/dev/sdashow any obvious problems?smartctl doesn't work on sd[a-d] at all:
# smartctl -a -d ata /dev/sdd smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/Quoted 5 lines
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
# smartctl -a -d ata -T permissive /dev/sddAs a smartmontools developer I know that '-d ata' is incorrect in this context. The 'd <
command_set> should either not be given or '-d sat' should be given.
Quoted 5 lines
This is difficult to explain and hence put in an easily understood option. As far as linux is concerned SCSI commands are being issued to a SCSI device (on a SCSI transport). But those SCSI commands are mostly instances of the SCSI ATA PASS-THROUGH command.
This is really what you need to use, although implementation of ATA_12 and ATA_16, which are the prescribed taskfile carriers is also done in the SAT layer.
Quoted 6 lines
It is further complicated by the fact that a properly implemented SAT layer (and I have never met one) implements the SCSI commands used for SMART support. If that was the case 'smartctl -a -d scsi
/dev/sdd' would yield useful output which would be a subset of what 'd sat' would yield.
The problem with this, of course, is that the SCSI stats are less comprehensive than the ATA SMART ones, so it's usually better to use the pass through commands if you know you're dealling with an ATA device.
Actually, I'm afraid it's even more complex than this: libata has a SAT layer for all SATA devices it attaches to; libsas also uses this, so you'd think that we only have one SAT layer in Linux. Unfortunately, mptsas and other "smart" SAS cards actually implement their own SAT layer in the firmware which bypasses the linux one.
The Linux one definitely doesn't implement the smart piece of the SAT spec; I'm not at all sure about the various firmware ones.
Quoted 1 line
Clear as mud?
Not quite ... but hopefully I've stirred it enough ...
James
James Bottomley
maintainerQuoted 8 lines · Wakko Warner
Quoted 2 lines
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Actually, I'd say this is a signal for NCQ errors with the drive.
I'm afraid only LSI would be able to say for certain, because the mptsas implements its NCQ handling in firmware. libata-core doesn't show any special workarounds for your device (ST3750640AS) but that doesn't mean there isn't a problem. If it's really an NCQ implementation issue, then clamping the queue depth to 1 is about the only fix, I'm afraid.
James
Quoted 11 lines · James Bottomley
Quoted 8 lines · Wakko Warner
Quoted 2 lines
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Actually, I'd say this is a signal for NCQ errors with the drive.
Unless it's this specific drive firmware, I'd have to disagree. I have 6 of the exact same drives (can't confirm firmware is the same though) in raid5 on an aic9410 sas controller w/o problems. The queue_depth for those are
31. I considered setting that value to the ones I'm having problems with, but I really don't want to go through another 4 hour rebuild.
Quoted 5 lines
I'm afraid only LSI would be able to say for certain, because the mptsas implements its NCQ handling in firmware. libata-core doesn't show any special workarounds for your device (ST3750640AS) but that doesn't mean there isn't a problem. If it's really an NCQ implementation issue, then clamping the queue depth to 1 is about the only fix, I'm afraid.
If it survives another week, I'd say using depth of 1 worked.
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
James Bottomley
maintainerQuoted 11 lines · Wakko Warner
Quoted 10 lines · James Bottomley
Quoted 8 lines · Wakko Warner
Quoted 2 lines
From the message you posted, it looks as though there may be a problem with sda.
It's working fine with /sys/block/sd[abc]/device/queue_depth = 1 (on boot up, as stated before, it's 64)
I performed the same copy again with queue_depth=1 after the array rebuilt. It worked fine then. No errors.
Quoted 7 lines
Quoted 1 line
Actually, I'd say this is a signal for NCQ errors with the drive.
Unless it's this specific drive firmware, I'd have to disagree. I have 6 of the exact same drives (can't confirm firmware is the same though) in raid5 on an aic9410 sas controller w/o problems. The
queue_depthfor those are
31. I considered setting that value to the ones I'm having problems with, but I really don't want to go through another 4 hour rebuild.
Well, yes, different revs of the firmware can behave differently. The libata-core blacklist includes the firmware version as part of the pattern matching.
There's an easy way to verify: smartctl -i will print the firmware version string.
Quoted 7 lines
Quoted 5 lines
I'm afraid only LSI would be able to say for certain, because the mptsas implements its NCQ handling in firmware. libata-core doesn't show any special workarounds for your device (ST3750640AS) but that doesn't mean there isn't a problem. If it's really an NCQ implementation issue, then clamping the queue depth to 1 is about the only fix, I'm afraid.
If it survives another week, I'd say using depth of 1 worked.
James
Quoted 9 lines · James Bottomley
Quoted 7 lines · Wakko Warner
Quoted 1 line
Actually, I'd say this is a signal for NCQ errors with the drive.
Unless it's this specific drive firmware, I'd have to disagree. I have 6 of the exact same drives (can't confirm firmware is the same though) in raid5 on an aic9410 sas controller w/o problems. The
queue_depthfor those are
31. I considered setting that value to the ones I'm having problems with, but I really don't want to go through another 4 hour rebuild.
Quoted 6 lines
Well, yes, different revs of the firmware can behave differently. The libata-core blacklist includes the firmware version as part of the pattern matching.
There's an easy way to verify: smartctl -i will print the firmware version string.
The aic sas one shows 3.AAE
The lsi mptsas can't be queried:
# smartctl -d scsi -i /dev/sdc smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Serial number: 3QD076X8
Device type: disk
Local Time is: Sun Apr 13 13:02:36 2008 EDT
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
#
I tried -d ata, -d sat and not using -d, provided no information. -T permissive didn't work either.
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
James Bottomley
maintainerQuoted 16 lines · Wakko Warner
Quoted 2 lines
There's an easy way to verify: smartctl -i will print the firmware version string.
The aic sas one shows 3.AAE
The lsi mptsas can't be queried:# smartctl -d scsi -i /dev/sdc smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/Serial number: 3QD076X8
Device type: disk
Local Time is: Sun Apr 13 13:02:36 2008 EDT
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
#
Quoted 2 lines
I tried -d ata, -d sat and not using -d, provided no information. -T permissive didn't work either.
That's a bit unfortunate ... it means the LSI firmware SAT layer doesn't support ATA_16. you can try ATA_12 just to make sure (little chance it will work, but just in case):
smartctl -i -d sat,12 /dev/sd<x>
If that doesn't work, I'm afraid you'll need to transfer the drive to a card that does support the command (like the aic).
However, it doesn't have to be a drive firmware fault, it could be some type of corner case NCQ failure triggered by the NCQ handler inside the LSI firmware ... in which case, your only hope for fixing it lies with LSI.
There is one final thought: the reason the aic94xx has a queue depth of 31 is because that's the maximum NCQ can support (well, it's 32 max, but we need one command for error handling). So, if the LSI shows a queue depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
James
Quoted 14 lines · James Bottomley
Quoted 13 lines · Wakko Warner
The aic sas one shows 3.AAE
The lsi mptsas can't be queried:# smartctl -d scsi -i /dev/sdc smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/Serial number: 3QD076X8
Device type: disk
Local Time is: Sun Apr 13 13:02:36 2008 EDT
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
#
Quoted 8 lines
Quoted 2 lines
I tried -d ata, -d sat and not using -d, provided no information. -T permissive didn't work either.
That's a bit unfortunate ... it means the LSI firmware SAT layer doesn't support
ATA_16. you can tryATA_12just to make sure (little chance it will work, but just in case):smartctl -i -d sat,12 /dev/sd<x>
Same thing.
Quoted 2 lines
If that doesn't work, I'm afraid you'll need to transfer the drive to a card that does support the command (like the aic).
The aic is not in the same place as the lsi. I'm not willing to shutdown this machine to do this right now.
Quoted 4 lines
However, it doesn't have to be a drive firmware fault, it could be some type of corner case NCQ failure triggered by the NCQ handler inside the LSI firmware ... in which case, your only hope for fixing it lies with LSI.
Some how, I don't think it is the drive (but could be).
Quoted 3 lines
There is one final thought: the reason the aic94xx has a queue depth of 31 is because that's the maximum NCQ can support (well, it's 32 max, but we need one command for error handling). So, if the LSI shows a queue
That's what I've read.
Quoted 2 lines
depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
IIRC, someone said the LSIs does queue in firmware.
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
Quoted 5 lines · Wakko Warner
Quoted 2 lines
depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
IIRC, someone said the LSIs does queue in firmware.
It was probably me that told you that. Anyways, the lsiutil 100
output you sent me last week indicated youre controller has NCQ
disabled. You need new firmware and controller chip rev in order to
support NCQ. You should contact LSI support to obtain which versions of
firmware and chip rev that is required. Meanwhile your controller will
queuing commands for SATA, only sending one command across the wire at a
time. For SAS, you will not have this issue.
Eric
Quoted 6 lines · Eric
Quoted 4 lines · Wakko Warner
Quoted 2 lines
depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
IIRC, someone said the LSIs does queue in firmware.
Quoted 7 lines
It was probably me that told you that. Anyways, the lsiutil 100
output you sent me last week indicated youre controller has NCQ
disabled. You need new firmware and controller chip rev in order to
support NCQ. You should contact LSI support to obtain which versions of
firmware and chip rev that is required. Meanwhile your controller will
queuing commands for SATA, only sending one command across the wire at a
time. For SAS, you will not have this issue.
Does "chip rev" mean software/firmware or an actual chip? Is there anyway the driver can see this and set the queue_depth to 1?
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
Quoted 5 lines
Does "chip rev" mean software/firmware or an actual chip? Is there anyway
the driver can see this and set thequeue_depthto 1?
When I mean chip, I'm refering to hardware. The revision of your
chip(1064/1068/1078) is located at offset 8 in pci config space. This
also known as PCI_CLASS_REVISION. You can get this by using the tool
called lspci.
Eric
Quoted 9 lines · Eric
Quoted 4 lines
Does "chip rev" mean software/firmware or an actual chip? Is there anyway
the driver can see this and set thequeue_depthto 1?When I mean chip, I'm refering to hardware. The revision of your
chip(1064/1068/1078) is located at offset 8 in pci config space. This
also known asPCI_CLASS_REVISION. You can get this by using the tool
called lspci.
04:02.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)
00: 00 10 54 00 56 01 30 02 01 00 00 01 08 48 00 00
10: 01 50 00 00 04 00 41 d8 00 00 00 00 04 00 40 d8
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 10 30 30
30: 00 00 00 00 50 00 00 00 00 00 00 00 0b 01 40 0a
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 01 98 02 06 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 07 b0 60 10 10 04 43 13
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 05 68 80 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 00 00 00 01 20 00 00 01 30 00 00 00 00 00 00
I'm not that familiar with the pci config space. Are you saying that I need a new chip for the new firmware? If so, considering everything is soldered on the board, I'd have a hard time replacing it! =)
Signature · 2 lines
Lab tests show that use of micro$oft causes cancer in lab animals Got Gas???
Quoted 33 lines · James Bottomley
Quoted 15 lines · Wakko Warner
Quoted 2 lines
There's an easy way to verify: smartctl -i will print the firmware version string.
The aic sas one shows 3.AAE
The lsi mptsas can't be queried:# smartctl -d scsi -i /dev/sdc smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/Serial number: 3QD076X8
Device type: disk
Local Time is: Sun Apr 13 13:02:36 2008 EDT
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
#Quoted 2 lines
I tried -d ata, -d sat and not using -d, provided no information. -T permissive didn't work either.
That's a bit unfortunate ... it means the LSI firmware SAT layer doesn't support
ATA_16. you can tryATA_12just to make sure (little chance it will work, but just in case):smartctl -i -d sat,12 /dev/sd<x>
If that doesn't work, I'm afraid you'll need to transfer the drive to a card that does support the command (like the aic).
However, it doesn't have to be a drive firmware fault, it could be some type of corner case NCQ failure triggered by the NCQ handler inside the LSI firmware ... in which case, your only hope for fixing it lies with LSI.
Quoted 5 lines
There is one final thought: the reason the aic94xx has a queue depth of 31 is because that's the maximum NCQ can support (well, it's 32 max, but we need one command for error handling). So, if the LSI shows a queue depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
If memory serves, I needed to upgrade the firmware on my MPT SAS fusion cards before the SCSI ATA PASS-THROUGH commands worked (properly). I'm a long way away from my hardware at the moment so I can't provide any further information (such as firmware version numbers).
I have just asked Bruce Allen to place a date in the smartctl copyright line because there are so many versions of version 5.38 floating around! smartmontools version 5.38 was recently released but beta (and sometimes partially broken) versions of 5.38 have been available via CVS for over 18 months. That said, the 5.38 versions that mention 2008 (as in "2002-8") are most likely the final release version.
Doug Gilbert
James Bottomley
maintainerQuoted 11 lines · Douglas Gilbert
Quoted 5 lines
There is one final thought: the reason the aic94xx has a queue depth of 31 is because that's the maximum NCQ can support (well, it's 32 max, but we need one command for error handling). So, if the LSI shows a queue depth of 64 it may be queueing internally as well. You could try lowering the lsi queue to 31 and seeing if it makes a difference.
If memory serves, I needed to upgrade the firmware on my MPT SAS fusion cards before the SCSI ATA PASS-THROUGH commands worked (properly). I'm a long way away from my hardware at the moment so I can't provide any further information (such as firmware version numbers).
OK, this prompted me to try mine, and it does work with my mptsas. This is all of my information from the sysfs files in the host directory:
active_mode: Initiator board_assembly: 03-01088-03B board_name: SAS3800X board_tracer: P185695005 can_queue: 127 cmd_per_lun: 7 debug_level: 00000000h device_delay: 00 host_busy: 0 io_delay: 00 proc_name: mptsas sg_tablesize: 128 state: running supported_mode: Initiator unchecked_isa_dma: 0 unique_id: 2 version_bios: 00.00.00.00 version_fw: 01.24.00.00 version_mpi: 105 version_nvdata_default: 2d02h version_nvdata_persistent: 2d02h version_product: LSISAS1068 B0
James