Skip to contentSkip to conversation

queue barrier support

PATCH James Bottomley 8 participants 15 messages Feb 13, 2002 – Feb 25, 2002

lore
Share & tools
lore
#0 lore raw
Quoted 8 lines · [email protected] said:

[email protected], 2002-02-13 13:42:39+01:00, [email protected]
Add support for SCSI drivers to indicate support for ordered tags
http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/[email protected]?nav= index.html|ChangeSet@-1h

[email protected], 2002-02-13 13:43:04+01:00, [email protected]
Add ordered tag support to the aic7xxx scsi driver

The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than MSG_ORDERED_Q_TAG. You have to scan through the headers to see that these are #defined the same.

A problem (that is probably only an issue for older drives) is that while technically the standard requires all 3 types of TAG to be supported if tag queueing is, some drives really only have simple tag support in their firmware, so you may need to add a blacklist for ordered tags on certain drives.

A further issue is that you haven't added anything to the error recovery code for this. If error recovery is activated for the device at the reset level, all tags will be discarded by the device. The eh will retry the failing command and then the other tagged commands will be re-issued from the scsi_bottom_half_handler (assuming the low level device driver immediately fails them with DID_RESET) in the order in which the low level driver failed them. Thus you have potentially completely messed up the ordering when the commands all get retried.

James Bottomley

#1 lore raw
Quoted 7 lines · James Bottomley
Quoted 5 lines · [email protected] said:

[email protected], 2002-02-13 13:42:39+01:00, [email protected]
Add support for SCSI drivers to indicate support for ordered tags
http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/[email protected]?nav= index.html|ChangeSet@-1h

Quoted 6 lines
Quoted 2 lines

[email protected], 2002-02-13 13:43:04+01:00, [email protected]
Add ordered tag support to the aic7xxx scsi driver

The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than MSG_ORDERED_Q_TAG. You have to scan through the headers to see that these are #defined the same.

Ah, the MSG_ORDERED_TASK did probably not exist when I originally did the patch (a year or so ago, iirc). I'll change the define to follow the rest of the code.

Quoted 5 lines

A problem (that is probably only an issue for older drives) is that while technically the standard requires all 3 types of TAG to be supported if tag queueing is, some drives really only have simple tag support in their firmware, so you may need to add a blacklist for ordered tags on certain drives.

Yes you are right.

Quoted 8 lines

A further issue is that you haven't added anything to the error recovery code for this. If error recovery is activated for the device at the reset level, all tags will be discarded by the device. The eh will retry the failing command and then the other tagged commands will be re-issued from the scsi_bottom_half_handler (assuming the low level device driver immediately fails them with DID_RESET) in the order in which the low level driver failed them. Thus you have potentially completely messed up the ordering when the commands all get retried.

I already have this fixed locally by just maintaining a fifo list of queued commands so we can retry them in the correct order. For 2.5 there's a busy and free list (so no more scanning for a free command either), I would imagine that the easiest for 2.4 is to just maintain a busy list in addition to the current array of pending/free commands.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 8 lines · [email protected] said:

A further issue is that you haven't added anything to the error recovery code for this. If error recovery is activated for the device at the reset level, all tags will be discarded by the device. The eh will retry the failing command and then the other tagged commands will be re-issued from the scsi_bottom_half_handler (assuming the low level device driver immediately fails them with DID_RESET) in the order in which the low level driver failed them. Thus you have potentially completely messed up the ordering when the commands all get retried.

Quoted 5 lines · [email protected] said:

I already have this fixed locally by just maintaining a fifo list of queued commands so we can retry them in the correct order. For 2.5 there's a busy and free list (so no more scanning for a free command either), I would imagine that the easiest for 2.4 is to just maintain a busy list in addition to the current array of pending/free commands.

Quoted 3 lines · [email protected] said:

I was wondering about this, we would need to change the error handler to fail all the requests after the barrier. I was hoping the driver did this for us ;-)

Unfortunately, this is going to involve deep hackery inside the error handler.
The current initial premise is that it can simply retry the failing command
by issuing an ABORT to the tag and resending it (which can cause a tag to move past your barrier). In an error situation, it really wouldn't be wise to try to abort lots of potentially running tags to preserve the barrier ordering (because of the overload placed on a known failing component), so I think the error handler has to abandon the concept of aborting commands and move straight to device reset. We then carefully resend the commands in FIFO order.

Additionally, you must handle the case that a device is reset by something else (in error handler terms, the cc_ua [check condition/unit attention]). Here also, the tags would have to be sent back down in FIFO order as soon as the condition is detected.

Quoted 2 lines · [email protected] said:

Yes, this could get sticky. Does anyone know if other OSes have already done this?

Other OSs (well the ones I've heard about: Solaris and HP-UX) try to avoid ordered tags, mainly because of the performance impact they have---the drive tag service algorithms become inefficient in the presence of ordered tags since they're usually optimised for all simple tags.

James

#3 lore raw
Quoted 5 lines · On Friday, February 15, 2002 10:15:58 AM -0500 James Bottomley…
Quoted 3 lines · [email protected] said:

I was wondering about this, we would need to change the error handler to fail all the requests after the barrier. I was hoping the driver did this for us ;-)

Quoted 9 lines

Unfortunately, this is going to involve deep hackery inside the error handler.
The current initial premise is that it can simply retry the failing command
by issuing an ABORT to the tag and resending it (which can cause a tag to move past your barrier). In an error situation, it really wouldn't be wise to try to abort lots of potentially running tags to preserve the barrier ordering (because of the overload placed on a known failing component), so I think the error handler has to abandon the concept of aborting commands and move straight to device reset. We then carefully resend the commands in FIFO order.

Ok, I'll try to narrow the barrier usage a bit, I'm waiting on the barrier write once it is sent, so I'm not worried about anything done after the ordered tag.

write X log blocks (simple tag)
write 1 commit block (ordered tag)
wait on all of them.

All I care about is knowing that all of the log blocks hit the disk before the commit. So, if one of the log blocks aborts, I want it to abort the commit too. Is this a little easier to implement?

Quoted 5 lines

Additionally, you must handle the case that a device is reset by something else (in error handler terms, the cc_ua [check condition/unit attention]). Here also, the tags would have to be sent back down in FIFO order as soon as the condition is detected.

Quoted 8 lines
Quoted 2 lines · [email protected] said:

Yes, this could get sticky. Does anyone know if other OSes have already done this?

Other OSs (well the ones I've heard about: Solaris and HP-UX) try to avoid ordered tags, mainly because of the performance impact they have---the drive tag service algorithms become inefficient in the presence of ordered tags since they're usually optimised for all simple tags.

I do see that on my drives ;-) The main reason I think its worth trying is because the commit block is directly adjacent to the last log block, so I'm hoping the drive can optimize the commit (even though it is ordered) better when the OS sends it directly after the last log block.

While I've got linux-scsi cc'd, I'll reask a question from yesterday. Do the targets with write back caches usually ignore the order tag, doing the write in the most efficient way possible instead?

-chris

#4 lore raw
Quoted 3 lines · [email protected] said:

While I've got linux-scsi cc'd, I'll reask a question from yesterday. Do the targets with write back caches usually ignore the order tag, doing the write in the most efficient way possible instead?

I'll try to answer, although I'm not quite sure of the basis of the question.

No ordinary SCSI drive that's not on a battery backed circuit should *ever* use writeback caching. They should all come by default as write through. In this case, the tag is acknowleged as completed only when the write has made it to the medium.

If you alter the drive parameter page to turn on write back caching, it's caveat emptor. Since you're now telling the drive that you consider hitting the cache to be equivalent to hitting the medium (because you know better about power failures and the like) it is undefined by the standards how the drives write from the cache to the medium---and you shouldn't care about this if you have arranged your system to do write back caching. They will acknowlege the tag as completed as soon as it hits the cache, and the ordered tag will be order it commits to the cache.

Note, for high end disk arrays, the reverse is usually true since they often have internal battery backups for their caches and drives.

If you lose power to the drive that does write back caching before the cache flushes, all the indications you've given the journalling fs about write commits are voided and the fs is probably inconsistent and not recoverable by a journal replay.

Note also that on system shutdown, most devices that use write back caching are also expecting a cache flush instruction from the node, which Linux doesn't send.

James

#5 lore raw
Quoted 11 lines · On Friday, February 15, 2002 11:51:45 AM -0500 James Bottomley…
Quoted 3 lines · [email protected] said:

While I've got linux-scsi cc'd, I'll reask a question from yesterday. Do the targets with write back caches usually ignore the order tag, doing the write in the most efficient way possible instead?

I'll try to answer, although I'm not quite sure of the basis of the question.

No ordinary SCSI drive that's not on a battery backed circuit should *ever* use writeback caching. They should all come by default as write through. In this case, the tag is acknowleged as completed only when the write has made it to the medium.

Right, friends don't let friends use non-battery back write back caches ;-)

Quoted 9 lines

If you alter the drive parameter page to turn on write back caching, it's caveat emptor. Since you're now telling the drive that you consider hitting the cache to be equivalent to hitting the medium (because you know better about power failures and the like) it is undefined by the standards how the drives write from the cache to the medium---and you shouldn't care about this if you have arranged your system to do write back caching. They will acknowlege the tag as completed as soon as it hits the cache, and the ordered tag will be order it commits to the cache.

Ok, this is what I was expecting. The performance improvement from using the ordered tags on single drives doesn't seem to be very big, and given the problems with failed requests, I'm thinking about dropping the scsi parts of the 2.4 barrier patch, and just worrying about making ide drives flush things correctly. The hard stuff on error recovery can be tackled in 2.5 and (maybe) ported back later.

But, if high end controllers with write back caching see improvements because they can combine the log writes better with the patch, it might be worth keeping, but modified in reiserfs so it flushes IDE caches by default, but only sends ordered tags when the admin mounts with -o barrier_ordered or something.

thanks for the info
-chris

#6 lore raw
Quoted 4 lines · [email protected] said:

I'm thinking about dropping the scsi parts of the 2.4 barrier patch, and just worrying about making ide drives flush things correctly. The hard stuff on error recovery can be tackled in 2.5 and (maybe) ported back later.

That's probably best. I do agree that 2.5 is the place to play around with SCSI error handling first.

I'm willing to help re-do the error handler, since I've always thought that abort isn't a good first line of defence because it actually adds to the command burden of a failing drive.

Jens, if you want to share the code you already have (or point me to the bitkeeper repository where you keep it) I'll look it over.

As far as the back-port to 2.4, as long as you only support SCSI drivers that use the new error handler (so we don't have to worry about the obsolete one) that should be reasonably easy.

James

#7 lore raw
In reply to Chris Mason
Quoted 10 lines · [email protected] said:

Ok, I'll try to narrow the barrier usage a bit, I'm waiting on the barrier write once it is sent, so I'm not worried about anything done after the ordered tag.

write X log blocks (simple tag) write 1 commit block (ordered tag)
wait on all of them.

All I care about is knowing that all of the log blocks hit the disk before the commit. So, if one of the log blocks aborts, I want it to abort the commit too. Is this a little easier to implement?

Actually, I'm afraid not. The FS has the notion of a transaction to help it with this. All the SCSI driver sees is a list of IOs with some requests for ordered tags. If we go into error recovery and have to abort a tag, how do we know which of the outstanding ordered tags is actually the commit block for this transaction? (Remember also that a FS sits on a partition, not a physical SCSI device, so even if you single thread your commits per FS, there could still be multiple FSs on the same physical device).

That's why I think it's safer to abandon the abort entirely. If our first line of error recovery is device reset, we know we've just cleared the entire outstanding tag queue and we can resend all the tags in FIFO order which means that we still don't need a concept of "transaction" at the SCSI level (which is a good thing, I think), all we need to know is the order in which the tags came to us.

Even if some tags were committed but not acknowledged at reset time, it's still safe to resend them in the correct ordered sequence because of transaction idempotence of block writes.

James

#8 lore raw
In reply to Chris Mason
Quoted 3 lines · James Bottomley

Note also that on system shutdown, most devices that use write back caching are also expecting a cache flush instruction from the node, which Linux doesn't send.

Hair splitting: it looks as though Andre Hedrick's IDE patch did this at least. Of course, that does not affect SCSI drives, but since you wrote "Linux", I thought this could be mentioned in this thread again. At least, my ATA drives are powered down some seconds before my ATX PC is, and the kernel says about flushing the cache.

Signature · 4 lines
Matthias Andree

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."         Benjamin Franklin
#9 lore raw
In reply to James Bottomley
Quoted 11 lines · James Bottomley [[email protected]] wrote:

[email protected] said:

Unfortunately, this is going to involve deep hackery inside the error handler.
The current initial premise is that it can simply retry the failing command
by issuing an ABORT to the tag and resending it (which can cause a tag to move past your barrier). In an error situation, it really wouldn't be wise to try to abort lots of potentially running tags to preserve the barrier ordering (because of the overload placed on a known failing component), so I think the error handler has to abandon the concept of aborting commands and move straight to device reset. We then carefully resend the commands in FIFO order.

Quoted 4 lines

Additionally, you must handle the case that a device is reset by something else (in error handler terms, the cc_ua [check condition/unit attention]). Here also, the tags would have to be sent back down in FIFO order as soon as the condition is detected.

I agree on the hacker. You also will need to clean the pipe of the unit attention post the reset or the first command you send down will be coming right back into the error handler and could get you in to a error recovery storm.

Quoted 8 lines

James

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Michael Anderson
[email protected]
#10 lore raw
In reply to James Bottomley
Quoted 7 lines · On Wednesday, February 13, 2002 01:26:12 PM -0500 James Bottomley…
Quoted 5 lines · [email protected] said:

[email protected], 2002-02-13 13:42:39+01:00, [email protected]
Add support for SCSI drivers to indicate support for ordered tags
http://bitmover.com:8888//tmp/v2_logging/athlon.transmeta.com/
torvalds-2002020517305 \ 6-16047-c1d11a41ed024864/[email protected]?nav= index.html|ChangeSet@-1h

Quoted 6 lines
Quoted 2 lines

[email protected], 2002-02-13 13:43:04+01:00, [email protected]
Add ordered tag support to the aic7xxx scsi driver

The rest of the aic7xxx code uses MSG_ORDERED_TASK rather than MSG_ORDERED_Q_TAG. You have to scan through the headers to see that these are

# defined the same.

Jens, my patch from yesterday has this fixed.

Quoted 6 lines

A problem (that is probably only an issue for older drives) is that while technically the standard requires all 3 types of TAG to be supported if tag queueing is, some drives really only have simple tag support in their firmware, so you may need to add a blacklist for ordered tags on certain drives.

Yes, this could get sticky. Does anyone know if other OSes have already done this?

Quoted 9 lines

A further issue is that you haven't added anything to the error recovery code for this. If error recovery is activated for the device at the reset level, all tags will be discarded by the device. The eh will retry the failing command and then the other tagged commands will be re-issued from the scsi_bottom_half_handler (assuming the low level device driver immediately fails them with DID_RESET) in the order in which the low level driver failed them. Thus you have potentially completely messed up the ordering when the commands all get retried.

I was wondering about this, we would need to change the error handler to fail all the requests after the barrier. I was hoping the driver did this for us ;-)

-chris

#11 lore raw
In reply to James Bottomley
Quoted 7 lines · James Bottomley

A problem (that is probably only an issue for older drives) is that while technically the standard requires all 3 types of TAG to be supported if tag queueing is, some drives really only have simple tag support in their firmware, so you may need to add a blacklist for ordered tags on certain drives.

From user space, I hope.

Signature · 1 line
Daniel
#12 lore raw
Quoted 5 lines · James Bottomley

A problem (that is probably only an issue for older drives) is that while technically the standard requires all 3 types of TAG to be supported if tag queueing is, some drives really only have simple tag support in their firmware, so you may need to add a blacklist for ordered tags on certain drives.

Quoted 1 line · [email protected] said:

From user space, I hope.

Well, following the current SCSI black/white list procedure, they would be in the static device_list table in scsi_scan.c, so no.

I agree that it's not a nice or friendly thing (and is certainly prone to delays when you have to get entries into the actual kernel code), but fixing this particular annoyance of the scsi subsystem (although I have actually thought about doing it) would be a project separate from the queue barrier support.

James

#14 lore raw

Re: 929-Emulex, ABTS Command Anamoly!

Quoted 2 lines · In article <[email protected]> you wrote:

Is this the right group to discuss about the ABTS
command ??

no.

emulex is a binary only driver so it should be discussed with emulex if you have problems...