Skip to contentSkip to conversation

2.4.19 SCSI core bug?

DISCUSSION Russell King 3 participants 6 messages Sep 11, 2002 – Sep 12, 2002

lore
Share & tools
lore
#0 lore raw

Hi,

I've just been running some of my tests on 2.4.19 on ARM, and came across the following bug, which seems to be somewhere in the bowls of the SCSI layer.

I'm using drivers/acorn/scsi/powertec.c, which uses fas216.c as the core chipset driver (I authored both of these files) which contains some pretty rigorous sanity checking.

Basically, what seems to be happening is that scsi can go wrong after a medium error, and repeatedly pass invalid requests back down to the HBA. My HBA driver dislikes invalid requests thusly:

scsi0.2: bad request buffer length 38400, should be 130048

38400 is SCpnt->request_bufflen.
130048 is the total number of bytes in the SG list.

Dumping out the offending command:

	Read (10) 00 00 00 46 00 00 00 fe 00

Ok, so we were asking for 0xfe 512-byte sectors, which is 130048. So why did SCSI tell me that it wanted 38400 bytes in SCpnt->request_bufflen?

My workaround for this is to forcefully set request_bufflen correctly, which appears to work. However, the SCSI subsystem completely stops making progress at this point. (It just repeats the above command indefinitely.)

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#1 lore raw
Quoted 3 lines · Russell King

Ok, so we were asking for 0xfe 512-byte sectors, which is 130048. So why did SCSI tell me that it wanted 38400 bytes in SCpnt->request_bufflen?

Ok, problem found.

There's a nice loop in scsi_send_eh_cmnd() which just loops endlessly trying to retry a SCpnt command on medium error without restoring it to its pristine state before giving it back to the host, or limiting the number of retries.

The end result is that the SCSI command says N sectors, the request list may say N * sector size bytes, but SCpnt->request_bufflen may be complete rubbish, since the HBA driver is allowed to modify this field during the processing of a command.

We do have a specific function to restore the SCSI command data to a pristine state (scsi_eh_retry_command).

Ok, so there's two bugs here:

1. It's possible for SCSI to completely bring a box to a complete standstill when it encounters a SCSI error.

2. Not retrying the command in its correct state.

I'll look at cooking up a patch for both of these in the next few days or so.

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#2 lore raw
Quoted 11 lines · Russell King [[email protected]] wrote:
Quoted 3 lines · Russell King

Ok, so we were asking for 0xfe 512-byte sectors, which is 130048. So why did SCSI tell me that it wanted 38400 bytes in SCpnt->request_bufflen?

Ok, problem found.

There's a nice loop in scsi_send_eh_cmnd() which just loops endlessly trying to retry a SCpnt command on medium error without restoring it to its pristine state before giving it back to the host, or limiting the number of retries.

The scsi_eh_completed_normally function should be limiting your retries. So you should not be looping endlessly unless the problem of sending down a dirty command is causing another issue.

I have a cleanup patch for 2.5 scsi_error I will add this fix in. scsi_send_eh_cmnd should not be retrying the command it should return to the caller the status and let them decide. We also should create a ?restore_scsi_cb? function that is shared so that it is done consistently.

Eventually the retry policy is going to be changed, but until then we should fix this problem.

-andmike

Signature · 2 lines
Michael Anderson
[email protected]
#3 lore raw
Quoted 5 lines · Mike Anderson

I have a cleanup patch for 2.5 scsi_error I will add this fix in. scsi_send_eh_cmnd should not be retrying the command it should return to the caller the status and let them decide. We also should create a ?restore_scsi_cb? function that is shared so that it is done consistently.

I've got a patch for both of these now. I'm working through some other issues at the moment though (like, why the hell requests for sectors after the sector in error don't have a data phase, and the drive returns status = GOOD, message = COMMAND COMPLETE, leading the kernel to believe that it did transfer data.)

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#4 lore raw
Quoted 7 lines · Russell King
Quoted 5 lines · Mike Anderson

I have a cleanup patch for 2.5 scsi_error I will add this fix in. scsi_send_eh_cmnd should not be retrying the command it should return to the caller the status and let them decide. We also should create a ?restore_scsi_cb? function that is shared so that it is done consistently.

Quoted 5 lines

I've got a patch for both of these now. I'm working through some other issues at the moment though (like, why the hell requests for sectors after the sector in error don't have a data phase, and the drive returns status = GOOD, message = COMMAND COMPLETE, leading the kernel to believe that it did transfer data.)

<mode=fact>
Ok, I've found the cause of this one. Lets assume we're issuing a READ10 command (transfer 1) for blocks N to N + L, and N + B is bad. ( L >= 0, B >= 0, B < L )

We successfully transfer blocks N to N + (B - 1) inclusive to the host without problem. We think B is bad, so we then request N + B + 1 to N + L blocks from the device using a READ10 command (transfer 2.)

The device immediately comes back without any data, but with GOOD status and COMMAND COMPLETED message. In other words, for transfer 2's command, it goes through the states:

	Message Out -> Command Out -> Status In -> Message In

Now, the interesting thing is that the READ10 commands have a special bit - FUA or Force Unit Access. When I set this bit in _all_ READ10 commands, I _do not_ see this behaviour.

<mode=theory>
It turns out that my drive appears to cache the request for transfer 1 (blocks N to N + L), and tries to satisfy the request from its cache. However, since block N + B was in error, it appears fill its cache only for N to N + (B - 1), but reserve N to N + L.

When a request with the FUA bit clear for a block between N + B and N + L is received, it tries to satisfy it from its cache, which doesn't contain the data. It thus responds without a data phase, indicating that the command was successful and without error.

<mode=fact>
This means that, using READ10, it is possible to issue READ commands to the drive, and have the drive respond without any error, yet NOT actually read _or_ transfer any data to the host. Think about what can happen to your ext2fs superblocks / bitmaps / inode tables in this situation, and what happens if something writes back the page to the drive...

I think we have two options:

1. we need to come up with a way of reasonably handing SCSI medium errors if we are going to use the READ10 command with the FUA bit clear.

2. we could just set the FUA bit and bypass the drives on-board cache completely.

Note that the only reason I've found this is because my HBA drives are _really_ pedantic about checking that all expected data does in fact get transferred by the drive.

I wonder how many other drives out there are buggy like this. 8/

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#5 lore raw
Quoted 4 lines

1. we need to come up with a way of reasonably handing SCSI medium errors if we are going to use the READ10 command with the FUA bit clear.

Quoted 8 lines

2. we could just set the FUA bit and bypass the drives on-board cache completely.

Note that the only reason I've found this is because my HBA drives are _really_ pedantic about checking that all expected data does in fact get transferred by the drive.

I wonder how many other drives out there are buggy like this. 8/

What about setting FUA on the next command after an error ?

Would this work or may the disk really keep stale data around ?

I don't have a recent SCSI spec at hand, but is there a command we can send after an error to force a cache flush & invalidate or will it only flush dirty datas to platter and not invalidate ?

Ben.