New: SMART responses for SATA disks on SAS get interpreted as errors

http://bugzilla.kernel.org/show_bug.cgi?id=13594

           Summary: SMART responses for SATA disks on SAS get interpreted
                    as errors
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 2.6.30-rc6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: No

Hi,

I just bought a LSI SAS3081E-R which I use against a Supermicro backplane to drive ten Seagate SATA disks (7200.11, 750GB and 1.5GB). I'm using the standard Linux Fusion MPT device driver (CONFIG_FUSION_SAS) under Linux 2.6.30-rc6. Everything seems to work pretty well, with one exception: When I use SMART against the drives (say, smartctl -a /dev/sda) the kernel complains with:

  [  811.091916] sd 0:0:0:0: [sda] Sense Key : Recovered Error [current]
[descriptor]
  [  811.099807] Descriptor sense data with sense descriptors (in hex):
  [  811.106175]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
  [  811.113262]         00 4f 00 c2 00 50
  [  811.117379] sd 0:0:0:0: [sda] Add. Sense: ATA pass through information

available

I've tried upgrading to the newest firmware (1.28.02.00, 05-MAY-2009), but all that changed is that the hex dump was added to the error message.

Whenever this happens, it appears like all the disks “hiccup” and the kernel loses contact with the controller for a small while. If too many of these happen at once, eventually disks start falling off RAIDs, and the entire machine goes down. It looks to me as if these messages should simply not be treated as errors by the kernel -- smartctl explicitly asks for a response even if the command doesn't fail (by setting CK_COND), so the response probably shouldn't be taken as an error.

Quoted 7 lines

  [  811.091916] sd 0:0:0:0: [sda] Sense Key : Recovered Error [current]
[descriptor]
  [  811.099807] Descriptor sense data with sense descriptors (in hex):
  [  811.106175]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
  [  811.113262]         00 4f 00 c2 00 50
  [  811.117379] sd 0:0:0:0: [sda] Add. Sense: ATA pass through information

available

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

Quoted 10 lines

I've tried upgrading to the newest firmware (1.28.02.00, 05-MAY-2009), but all that changed is that the hex dump was added to the error message.

Whenever this happens, it appears like all the disks “hiccup” and the kernel loses contact with the controller for a small while. If too many of these happen at once, eventually disks start falling off RAIDs, and the entire machine goes down. It looks to me as if these messages should simply not be treated as errors by the kernel -- smartctl explicitly asks for a response even if the command doesn't fail (by setting CK_COND), so the response probably shouldn't be taken as an error.

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

http://bugzilla.kernel.org/show_bug.cgi?id=13594

           Summary: SMART responses for SATA disks on SAS get interpreted
                    as errors
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 2.6.30-rc6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: No

Hi,

I just bought a LSI SAS3081E-R which I use against a Supermicro backplane to drive ten Seagate SATA disks (7200.11, 750GB and 1.5GB). I'm using the standard Linux Fusion MPT device driver (CONFIG_FUSION_SAS) under Linux 2.6.30-rc6. Everything seems to work pretty well, with one exception: When I use SMART against the drives (say, smartctl -a /dev/sda) the kernel complains with:

  [  811.091916] sd 0:0:0:0: [sda] Sense Key : Recovered Error [current]
[descriptor]
  [  811.099807] Descriptor sense data with sense descriptors (in hex):
  [  811.106175]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
  [  811.113262]         00 4f 00 c2 00 50
  [  811.117379] sd 0:0:0:0: [sda] Add. Sense: ATA pass through information

available

I've tried upgrading to the newest firmware (1.28.02.00, 05-MAY-2009), but all that changed is that the hex dump was added to the error message.

Whenever this happens, it appears like all the disks “hiccup” and the kernel loses contact with the controller for a small while. If too many of these happen at once, eventually disks start falling off RAIDs, and the entire machine goes down. It looks to me as if these messages should simply not be treated as errors by the kernel -- smartctl explicitly asks for a response even if the command doesn't fail (by setting CK_COND), so the response probably shouldn't be taken as an error.

Quoted 7 lines

  [  811.091916] sd 0:0:0:0: [sda] Sense Key : Recovered Error [current]
[descriptor]
  [  811.099807] Descriptor sense data with sense descriptors (in hex):
  [  811.106175]         72 01 00 1d 00 00 00 0e 09 0c 00 00 00 00 00 00
  [  811.113262]         00 4f 00 c2 00 50
  [  811.117379] sd 0:0:0:0: [sda] Add. Sense: ATA pass through information

available

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

Quoted 10 lines

I've tried upgrading to the newest firmware (1.28.02.00, 05-MAY-2009), but all that changed is that the hex dump was added to the error message.

Whenever this happens, it appears like all the disks “hiccup” and the kernel loses contact with the controller for a small while. If too many of these happen at once, eventually disks start falling off RAIDs, and the entire machine goes down. It looks to me as if these messages should simply not be treated as errors by the kernel -- smartctl explicitly asks for a response even if the command doesn't fail (by setting CK_COND), so the response probably shouldn't be taken as an error.

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #3 from Steinar H. Gunderson <[email protected]> 2009-06-21 18:58:28 --- (In reply to comment #1)

Quoted 4 lines

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

OK.

Quoted 4 lines

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

I tried reporting this to the Linux fusionmpt driver people a while ago, but never received any response (thus this bug)... I guess I'm out of luck,

then,
if there's nothing that can be done for it in the kernel. It's a bit weird, though; one would believe people ran smartd on their systems and discovered this already.

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #3 from Steinar H. Gunderson <[email protected]> 2009-06-21 18:58:28 --- (In reply to comment #1)

Quoted 4 lines

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

OK.

Quoted 4 lines

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

I tried reporting this to the Linux fusionmpt driver people a while ago, but never received any response (thus this bug)... I guess I'm out of luck,

then,
if there's nothing that can be done for it in the kernel. It's a bit weird, though; one would believe people ran smartd on their systems and discovered this already.

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #4 from Anonymous Emailer <[email protected]> 2009-06-21 19:07:13 ---
Reply-To: [email protected]

Quoted 12 lines · On Sun, 2009-06-21 at 18:58 +0000, [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #3 from Steinar H. Gunderson <[email protected]> 2009-06-21 18:58:28 --- (In reply to comment #1)

Quoted 4 lines

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

Quoted 8 lines

OK.

Quoted 4 lines

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

I tried reporting this to the Linux fusionmpt driver people a while ago, but never received any response (thus this bug)... I guess I'm out of luck,

OK, cc'd LSI people, let's see if I get better luck

Quoted 4 lines

then,
if there's nothing that can be done for it in the kernel. It's a bit weird, though; one would believe people ran smartd on their systems and discovered this already.

I can guess that it's some type of firmware mode problem: either it runs for SMART or it runs for normal commands, hence the hiatus. If that's true, you'd likely only see the problem in a large disk setup ... it might also be possible to work around by simply quiescing the card before sending down SMART commands (that would be grossly inefficient, but at least devices wouldn't get errored).

As for the hiccup, I have noticed that with SAS (SCSI) disks from Seagate there is a curious sound and a pause before the response to LOG SENSE SCSI command (the type the smartmontools uses on SCSI disks).

Another annoyance is that the disk must be ready (i.e. spun up) before MODE SENSE and LOG SENSE work, haven't Seagate heard of flash :-)
SCSI standards permit that (i.e. only
a small number of commands have to work when the disk is not ready) but you would think accessing metadata given the disk has spun up once since power up could be accomplished from RAM or flash.

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #4 from Anonymous Emailer <[email protected]> 2009-06-21 19:07:13 ---
Reply-To: [email protected]

Quoted 12 lines · On Sun, 2009-06-21 at 18:58 +0000, [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=13594

--- Comment #3 from Steinar H. Gunderson <[email protected]> 2009-06-21 18:58:28 --- (In reply to comment #1)

Quoted 4 lines

This is a message the kernel prints out on all recovered error returns (except those marked REQ_QUIET). It's purely informational and doesn't affect return processing of the command at all, so the kernel is actually treating this as a successful completion not an error.

Quoted 8 lines

OK.

Quoted 4 lines

So this sounds like the bug ... however, for the LSI card, this bug will be in the SAT layer in the fusion firmware. I can shut the kernel up by making the recovered error processing clause look for 01/00/1D as well as REQ_QUIET, but it won't affect this problem.

I tried reporting this to the Linux fusionmpt driver people a while ago, but never received any response (thus this bug)... I guess I'm out of luck,

OK, cc'd LSI people, let's see if I get better luck

Quoted 4 lines

then,
if there's nothing that can be done for it in the kernel. It's a bit weird, though; one would believe people ran smartd on their systems and discovered this already.

I can guess that it's some type of firmware mode problem: either it runs for SMART or it runs for normal commands, hence the hiatus. If that's true, you'd likely only see the problem in a large disk setup ... it might also be possible to work around by simply quiescing the card before sending down SMART commands (that would be grossly inefficient, but at least devices wouldn't get errored).

I have just replicated the "ATA pass through information available" message report on a similar vintage LSI controller and a SATA disk with a recent smartctl version.

There is no need to report this in the kernel error log, as the smartmontools ATA pass-through (SCSI) command asked for the final state of the ATA registers and the sense buffer is the conduit for that information. That ASC/ASCQ pair basically means "you asked for them and here they are". [reference: sat2r07b.pdf section 12.2.5 table 107 when CK_COND is 1]

As for the hiccup, I have noticed that with SAS (SCSI) disks from Seagate there is a curious sound and a pause before the response to LOG SENSE SCSI command (the type the smartmontools uses on SCSI disks).

As for the hiccup, I have noticed that with SAS (SCSI) disks from Seagate there is a curious sound and a pause before the response to LOG SENSE SCSI command (the type the smartmontools uses on SCSI disks).

Another annoyance is that the disk must be ready (i.e. spun up) before MODE SENSE and LOG SENSE work, haven't Seagate heard of flash :-)
SCSI standards permit that (i.e. only
a small number of commands have to work when the disk is not ready) but you would think accessing metadata given the disk has spun up once since power up could be accomplished from RAM or flash.