Skip to contentSkip to conversation

Regarding ordered-tag support.

DISCUSSION Tejun Heo 4 participants 16 messages Feb 8, 2006 – Feb 10, 2006 SCSI

lore
Share & tools
lore

Tejun Heo

author
#0 lore raw

Hello, James.

Now that new ordered implementation is in the mainline, we can properly use ordered-tag during barrier sequence. As noted in the barrier doc, the problem with the current SCSI midlayer is that scsi_request_fn() is not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave request queue in the correct order, they can be reordered while they are being issued by SCSI midlayer.

There can be two solutions to this problem.

1. Rewrite scsi_request_fn() and scsi_dispatch_cmd() such that q->queuelock is not released during command issue. This implies grabbing shost->host_lock while holding q->queuelock.

2. Don't allow concurrent command issuing by marking SCSI device during command issuing (e.g. sdev->issue_in_progress)

I prefer #1 as it's more straight-forward. To implement #1, scsi_request_fn() should be reorganized quite a bit, but, if you remember, the scsi_reqfn reimplementation patch I wrote quite a while ago suites this pretty well. What do you think?

Signature · 1 line
tejun
#1 lore raw
Quoted 6 lines · Tejun Heo

Now that new ordered implementation is in the mainline, we can properly use ordered-tag during barrier sequence. As noted in the barrier doc, the problem with the current SCSI midlayer is that scsi_request_fn() is not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave request queue in the correct order, they can be reordered while they are being issued by SCSI midlayer.

Unfortunately, though, this is only half the problem.

The other half is busy/queue_full processing and error recovery

For the first: busy/queue_full processing, the issue is that when we send out a command, we could get a BUSY or QUEUE_FULL return which comes back to us via IRQ context. Unfortunately, we could have multiple commands that do this and a command will be accepted as soon as the busy condition alleviates, so we could see say two commands go down in order, the first one will get BUSY, the second is accepted and only then do we get the IRQ that says BUSY to the first ... now we have out of order execution.

For the second: Depending on the mode page (the QErr bit of the queueing page), most devices fail only a single command. We can set the QErr bit to return every command after the failing one (thus ensuring execution order), but the error handler would have to take them all back and resort them for submission.

I think the first problem has to be solved before we can turn on ordered tag based barriers. I'm ambivalent on the second; we could probably mark ordered tag based barriers as "caveat emptor" while we work on the EH problem in parallel.

James

author
#2 lore raw
Quoted 14 lines · James Bottomley
Quoted 6 lines · Tejun Heo

Now that new ordered implementation is in the mainline, we can properly use ordered-tag during barrier sequence. As noted in the barrier doc, the problem with the current SCSI midlayer is that scsi_request_fn() is not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave request queue in the correct order, they can be reordered while they are being issued by SCSI midlayer.

Unfortunately, though, this is only half the problem.

The other half is busy/queue_full processing and error recovery

Quoted 8 lines

For the first: busy/queue_full processing, the issue is that when we send out a command, we could get a BUSY or QUEUE_FULL return which comes back to us via IRQ context. Unfortunately, we could have multiple commands that do this and a command will be accepted as soon as the busy condition alleviates, so we could see say two commands go down in order, the first one will get BUSY, the second is accepted and only then do we get the IRQ that says BUSY to the first ... now we have out of order execution.

Oh... I see. How many drivers do that? I can't think of good reasons to report BUSY via IRQ for simpler transports (SATA/SPI). Maybe enable ordered-tag selectively?

Quoted 6 lines

For the second: Depending on the mode page (the QErr bit of the queueing page), most devices fail only a single command. We can set the QErr bit to return every command after the failing one (thus ensuring execution order), but the error handler would have to take them all back and resort them for submission.

Actually blk layer will do the sorting part (this is what req->ordcolor is for). Block drivers are only required to not successfully complete later requests while retrying earlier ones, so setting QErr and retrying all on-the-fly requests should do it (no EH code change).

Quoted 4 lines

I think the first problem has to be solved before we can turn on ordered tag based barriers. I'm ambivalent on the second; we could probably mark ordered tag based barriers as "caveat emptor" while we work on the EH problem in parallel.

Signature · 1 line
tejun
#3 lore raw
Quoted 10 lines
Quoted 9 lines

For the first: busy/queue_full processing, the issue is that when we send out a command, we could get a BUSY or QUEUE_FULL return which comes back to us via IRQ context. Unfortunately, we could have multiple commands that do this and a command will be accepted as soon as the busy condition alleviates, so we could see say two commands go down in order, the first one will get BUSY, the second is accepted and only then do we get the IRQ that says BUSY to the first ... now we have out of order execution.

Quoted 3 lines

Oh... I see. How many drivers do that? I can't think of good reasons to report BUSY via IRQ for simpler transports (SATA/SPI). Maybe enable ordered-tag selectively?

This isn't a driver thing - this is a SCSI device thing. The disk array is temporarily out of resources so it BUSY's, or QUEUE_FULL's. But the next i/o may not encounter it.

Quoted 11 lines
Quoted 6 lines

For the second: Depending on the mode page (the QErr bit of the queueing page), most devices fail only a single command. We can set the QErr bit to return every command after the failing one (thus ensuring execution order), but the error handler would have to take them all back and resort them for submission.

Actually blk layer will do the sorting part (this is what req->ordcolor is for). Block drivers are only required to not successfully complete later requests while retrying earlier ones, so setting QErr and retrying all on-the-fly requests should do it (no EH code change).

You're making the assumption that you can set QErr... It's under device control.

-- james s

author
#4 lore raw

Hello,

Quoted 9 lines · James Smart
Quoted 9 lines
Quoted 8 lines

For the first: busy/queue_full processing, the issue is that when we send out a command, we could get a BUSY or QUEUE_FULL return which comes back to us via IRQ context. Unfortunately, we could have multiple commands that do this and a command will be accepted as soon as the busy condition alleviates, so we could see say two commands go down in order, the first one will get BUSY, the second is accepted and only then do we get the IRQ that says BUSY to the first ... now we have out of order execution.

Quoted 7 lines
Quoted 3 lines

Oh... I see. How many drivers do that? I can't think of good reasons to report BUSY via IRQ for simpler transports (SATA/SPI). Maybe enable ordered-tag selectively?

This isn't a driver thing - this is a SCSI device thing. The disk array is temporarily out of resources so it BUSY's, or QUEUE_FULL's. But the next i/o may not encounter it.

I intentionally wrote 'driver' because if a SCSI device determines that it's busy, it would report via CHECK CONDITION. Depending on QErr, the whole task set will be terminated, and such case falls into EH requeueing case (would require changes in scsi_softirq though).

So, I thought this case was driver/controller thing where they report busy condition after issue for a command while accepting later commands.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

For the second: Depending on the mode page (the QErr bit of the queueing page), most devices fail only a single command. We can set the QErr bit to return every command after the failing one (thus ensuring execution order), but the error handler would have to take them all back and resort them for submission.

Quoted 7 lines
Quoted 4 lines

Actually blk layer will do the sorting part (this is what req->ordcolor is for). Block drivers are only required to not successfully complete later requests while retrying earlier ones, so setting QErr and retrying all on-the-fly requests should do it (no EH code change).

You're making the assumption that you can set QErr... It's under device control.

Was it ro field? Didn't know that. I will check it tomorrow. If a device doesn't abort whole taskset, we just can't use ordered-tag for barriers.

Thanks for your comment.

Signature · 1 line
tejun
#5 lore raw
Quoted 4 lines · Tejun

I intentionally wrote 'driver' because if a SCSI device determines that it's busy, it would report via CHECK CONDITION. Depending on QErr, the whole task set will be terminated, and such case falls into EH requeueing case (would require changes in scsi_softirq though).

No, it wouldn't. BUSY is a SPI Message (deprecated on non-SPI transports) and QUEUE_FULL is a status return. The device is entitled to think of the transaction as logically complete (and not impacting error recovery) after either of these returns.

Quoted 5 lines
Quoted 1 line

You're making the assumption that you can set QErr... It's under device control.

Was it ro field? Didn't know that. I will check it tomorrow. If a device doesn't abort whole taskset, we just can't use ordered-tag for barriers.

His point is that it's a field in the mode page of a device. Some devices only have read only mode pages ... still more may not act on this field correctly even if set. Basically it's opening a can of worms.

James

author
#6 lore raw
Quoted 11 lines · James Bottomley
Quoted 4 lines · Tejun

I intentionally wrote 'driver' because if a SCSI device determines that it's busy, it would report via CHECK CONDITION. Depending on QErr, the whole task set will be terminated, and such case falls into EH requeueing case (would require changes in scsi_softirq though).

No, it wouldn't. BUSY is a SPI Message (deprecated on non-SPI transports) and QUEUE_FULL is a status return. The device is entitled to think of the transaction as logically complete (and not impacting error recovery) after either of these returns.

I see. I'll dig docs.

Quoted 11 lines
Quoted 5 lines
Quoted 1 line

You're making the assumption that you can set QErr... It's under device control.

Was it ro field? Didn't know that. I will check it tomorrow. If a device doesn't abort whole taskset, we just can't use ordered-tag for barriers.

His point is that it's a field in the mode page of a device. Some devices only have read only mode pages ... still more may not act on this field correctly even if set. Basically it's opening a can of worms.

Yeah, I kind of doubt whether hardware vendors would implement and test all the options and features whene I read SCSI specs. Considering how many ATA drives have faulty firmware...

Anyways, it seems ordered-tag support should be left alone for the time being. Sad. :-(

Thanks a lot.

Signature · 1 line
tejun
#7 lore raw
Quoted 6 lines · Tejun

Yeah, I kind of doubt whether hardware vendors would implement and test all the options and features whene I read SCSI specs. Considering how many ATA drives have faulty firmware...

Anyways, it seems ordered-tag support should be left alone for the time being. Sad. :-(

Well ... I've always maintained that ordered queueing is not really a good method of implementing barriers in TCQ environments ... primarily because no-one else really does it, so we'd be uncovering all of the bugs.

I suppose I'm a broken record, but I really think linked tasks are the better way to enforce the required ordering guarantees for SCSI TCQ. The problem being that this would present a slightly different API at the block level (you have individual writes that are now ordered, not wholesale barriers).

James

#8 lore raw
Quoted 5 lines · James Bottomley

I suppose I'm a broken record, but I really think linked tasks are the better way to enforce the required ordering guarantees for SCSI TCQ. The problem being that this would present a slightly different API at the block level (you have individual writes that are now ordered, not wholesale barriers).

I strongly agree, since fundamentally, linked tasks is what is really going on.

Jeff

author
#9 lore raw
Quoted 9 lines · Jeff Garzik
Quoted 5 lines · James Bottomley

I suppose I'm a broken record, but I really think linked tasks are the better way to enforce the required ordering guarantees for SCSI TCQ. The problem being that this would present a slightly different API at the block level (you have individual writes that are now ordered, not wholesale barriers).

I strongly agree, since fundamentally, linked tasks is what is really going on.

Hello, James. Hello, Jeff.

Maybe I have misunderstood, but, AFAICS, linked task doesn't assure any ordering relation with respect to other tasks unlike ordered tag. So, it doesn't make much difference compared to simple & stupid ordering by draining. For a barrier to work, the following order should be kept.

pre-barrier writes -> cache flush -> barrier write (FUA) -> other writes

Issuing cache flush and barrier write with ordered tags accomplishes all the ordering requirements (if it works, that is.). I can see linked task can be used for 'cache flush -> barrier write (FUA)' segment but that leaves us with two unhandled ordering requirements.

Another thing I'm curious about linked task is what advantages linked task has over simply issuing and completing the commands sequentially. Commands in a linked task has to be issued and completed sequentially anyway, so I don't really see the difference.

Thanks.

Signature · 1 line
tejun
#10 lore raw
Quoted 4 lines · Tejun Heo

Maybe I have misunderstood, but, AFAICS, linked task doesn't assure any ordering relation with respect to other tasks unlike ordered tag. So, it doesn't make much difference compared to simple & stupid ordering by draining. For a barrier to work, the following order should be kept.

It's a different paradigm from barriers. With a barrier, it's an impassable object in the I/O stream (nothing overtakes it or falls behind it). With a link, there's a set of commands (the linked set, and there may be more than one linked set of commands) that have to be ordered within the link. The point is that most of our problems with barriers are figuring out how to prevent leaks. The linked command paradigm is much more forgiving in this regard.

Quoted 6 lines

pre-barrier writes -> cache flush -> barrier write (FUA) -> other writes

Issuing cache flush and barrier write with ordered tags accomplishes all the ordering requirements (if it works, that is.). I can see linked task can be used for 'cache flush -> barrier write (FUA)' segment but that leaves us with two unhandled ordering requirements.

Right, that's why I said we'd have to change the block layer API to use linked commands.

Quoted 4 lines

Another thing I'm curious about linked task is what advantages linked task has over simply issuing and completing the commands sequentially. Commands in a linked task has to be issued and completed sequentially anyway, so I don't really see the difference.

The fact that tasks can proceed in parallel, only the command components individual linked tasks must be serialised. so it supports the TCQ paradigm.

Really, also, linked tasks is exactly what a JFS wants because most of the updates that would be linked are something like update log entry; modify metadata (or data); remove log entry.

James

author
#11 lore raw

Hello, James.

Quoted 9 lines · James Bottomley

The fact that tasks can proceed in parallel, only the command components individual linked tasks must be serialised. so it supports the TCQ paradigm.

Really, also, linked tasks is exactly what a JFS wants because most of the updates that would be linked are something like update log entry; modify metadata (or data); remove log entry.

I see your point, so it's not really a barrier anymore, but a sequence of ordered writes with an intervening flush independent of other unrelated writes. Some random thoughts...

* All writes in the ordered sequence will have to be done sequentially, but unrelated writes are not affected. This can be overall gain.

* Can be implemented in block layer proper without using SCSI linked task (probably with separate ->ordered_requests queue). What would be the advantage of SCSI linked task?

* As it's finer grained than wholesale barrier, FS layer will need to pay more attention.

Thanks for the explanation. :-)

Signature · 1 line
tejun
#12 lore raw
In reply to Tejun
Quoted 3 lines · Tejun

Oh... I see. How many drivers do that? I can't think of good reasons to report BUSY via IRQ for simpler transports (SATA/SPI). Maybe enable ordered-tag selectively?

All of them. These are scsi *device* returns. The problem is that even if the driver is careful about this, we queue any status done for softirq processing ...

Quoted 11 lines
Quoted 6 lines

For the second: Depending on the mode page (the QErr bit of the queueing page), most devices fail only a single command. We can set the QErr bit to return every command after the failing one (thus ensuring execution order), but the error handler would have to take them all back and resort them for submission.

Actually blk layer will do the sorting part (this is what req->ordcolor is for). Block drivers are only required to not successfully complete later requests while retrying earlier ones, so setting QErr and retrying all on-the-fly requests should do it (no EH code change).

Right ... but at the moment, we don't do that. The eh would have to be changed to gather all the commands for the device (we do it per host) and resubmit them in such a way that the block layer resorts them, wouldn't we ... at the moment we queue each one as they're processed.

James

#13 lore raw
In reply to James Bottomley
Quoted 6 lines · James Bottomley

On Wed, 2006-02-08 at 15:40 +0900, Tejun Heo wrote:
...<snip>...
Unfortunately, though, this is only half the problem.

The other half is busy/queue_full processing and error recovery

Quoted 8 lines

For the first: busy/queue_full processing, the issue is that when we send out a command, we could get a BUSY or QUEUE_FULL return which comes back to us via IRQ context. Unfortunately, we could have multiple commands that do this and a command will be accepted as soon as the busy condition alleviates, so we could see say two commands go down in order, the first one will get BUSY, the second is accepted and only then do we get the IRQ that says BUSY to the first ... now we have out of order execution.

James is also not disclosing other areas that may fail...

- How does multipathing affect this (maybe not dm, but others that exist).
- The implicit expectation of the LLDD and the hardware is that commands are placed on the wire in the same order given to queuecommand. However, no where is this mandated. In general, they do keep order. However, I've seen driver implementations that don't dispatch directly to hardware in queuecommand, allowing the intermediate steps to fail and perhaps disrupt ordering.
- Some adapters may actually be multipath/raid cards, thus affecting command delivery.
- SCSI Transports are not 100% reliable, affecting ordering. For FC, a single Frame corruption could cause a command frame to be dropped thus the next command received and executed out of order. You would not notice an issue until the scsi i/o timeout fires and aborts the i/o. The subsequent i/o may have already completed by that time. Some transports due have command sequence numbering to avoid this, but it's an optional feature.

-- james s

#14 lore raw
Quoted 3 lines · James Smart

James is also not disclosing other areas that may fail...

- How does multipathing affect this (maybe not dm, but others that exist).

Hey, with block level multi-path that's now officially Someone Else's Problem. (And it's also a problem for software RAID as well).

Quoted 6 lines

- The implicit expectation of the LLDD and the hardware is that commands are placed on the wire in the same order given to queuecommand. However, no where is this mandated. In general, they do keep order. However, I've seen driver implementations that don't dispatch directly to hardware in queuecommand, allowing the intermediate steps to fail and perhaps disrupt ordering.

OK ... this acceptance problem is pretty much identical to the BUSY/QUEUE_FULL one.

Quoted 2 lines

- Some adapters may actually be multipath/raid cards, thus affecting command delivery.

This shouldn't be more of a problem than SCSI devices. The issue here is caching. A raid controller with a non-battery backed cache may decide to ack commands before they hit the medium. However, as long as they expose this in the caching mode page and support the FUA bit of 10 byte commands, we should be able to cope (and yes, I know there are drivers that currently don't get this right).

Quoted 6 lines

- SCSI Transports are not 100% reliable, affecting ordering. For FC, a single Frame corruption could cause a command frame to be dropped thus the next command received and executed out of order. You would not notice an issue until the scsi i/o timeout fires and aborts the i/o. The subsequent i/o may have already completed by that time. Some transports due have command sequence numbering to avoid this, but it's an optional feature.

Yes, but again, this is identical to BUSY/QUEUE_FULL. As long as you can get status return from the command in-line you can still preserve ordering.

James

#15 lore raw
Quoted 4 lines · James Bottomley
Quoted 1 line

- How does multipathing affect this (maybe not dm, but others that exist).

Hey, with block level multi-path that's now officially Someone Else's Problem. (And it's also a problem for software RAID as well).

And actually (in the time honoured tradition of telling someone else what they should be doing while blowing your own trumpet) could I just point out that I think that it's impossible to enforce barriers in the face of multi-path (because of the cross path synchronisation requirements). However, it should be quite easy to enforce linked command ordering in these environments by the simple expedient of tagging a link to a single path. Even on path failure for partial link completion, you simply resend the entire segment of link command set that you don't know completed (because writes are idempotent).

James