Skip to contentSkip to conversation

scsi vs ide performance on fsync's

DISCUSSION David Balazic 4 participants 11 messages Mar 6, 2001 – Mar 8, 2001

lore
Share & tools
lore
#0 lore raw
Quoted 14 lines · Linus Torvalds himself
Quoted 10 lines · Alan Cox
Quoted 3 lines
Quoted 2 lines

I don't know if there is any way to turn of a write buffer on an IDE disk.

You want a forced set of commands to kill caching at init?

Wrong model

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the barrier.

Agreed.

Quoted 7 lines

Write buffering is incredibly useful on a disk - for all the same reasons that an OS wants to do it. The disk can use write buffering to speed up writes a lot - not just lower the _perceived_ latency by the OS, but to actually improve performance too.

But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).

ATA , SCSI and ATAPI all have a FLUSH_CACHE command. (*) Whether the drives implement it is another question ...

(*) references :
ATA-6 draft standard from www.t13.org
MtFuji document from ????????

Signature · 4 lines
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -
#1 lore raw
Quoted 15 lines · David Balazic
Quoted 15 lines
Quoted 7 lines

Wrong model

You want a write barrier. Write buffering (at least for short intervals) in the drive is very sensible. The kernel needs to able to send drivers a write barrier which will not be completed with outstanding commands before the
barrier.

Agreed.

Write buffering is incredibly useful on a disk - for all the same reasons that an OS wants to do it. The disk can use write buffering to speed up writes a lot - not just lower the _perceived_ latency by the OS, but to actually improve performance too.

Quoted 5 lines
Quoted 2 lines

But Alan is right - we needs a "sync" command or something. I don't know if IDE has one (it already might, for all I know).

ATA , SCSI and ATAPI all have a FLUSH_CACHE command. (*) Whether the drives implement it is another question ...

(Usually called SYNCHRONIZE_CACHE btw)

SCSI has ordered tag, which fit the model Alan described quite nicely. I've been meaning to implement this for some time, it would be handy for journalled fs to use such a barrier. Since ATA doesn't do queueing (at least not in current Linux), a synchronize cache is probably the only way to go there.

Quoted 3 lines

(*) references :
ATA-6 draft standard from www.t13.org
MtFuji document from ????????

ftp.avc-pioneer.com

Signature · 1 line
Jens Axboe
#2 lore raw
In reply to David Balazic

Hi,

Quoted 6 lines · Jens Axboe

SCSI has ordered tag, which fit the model Alan described quite nicely. I've been meaning to implement this for some time, it would be handy for journalled fs to use such a barrier. Since ATA doesn't do queueing (at least not in current Linux), a synchronize cache is probably the only way to go there.

Note that you also have to preserve the position of the barrier in the elevator queue, and you need to prevent LVM and soft raid from violating the barrier if different commands end up being sent to different disks.

--Stephen

#3 lore raw
In reply to David Balazic
Quoted 10 lines · Stephen C. Tweedie
Quoted 5 lines

SCSI has ordered tag, which fit the model Alan described quite nicely. I've been meaning to implement this for some time, it would be handy for journalled fs to use such a barrier. Since ATA doesn't do queueing (at least not in current Linux), a synchronize cache is probably the only way to go there.

Note that you also have to preserve the position of the barrier in the elevator queue, and you need to prevent LVM and soft raid from violating the barrier if different commands end up being sent to different disks.

Yep, it's much harder than it seems. Especially because for the barrier to be really useful, having inter-request dependencies becomes a requirement. So you can say something like 'flush X and Y, but don't flush Y before X is done'.

Signature · 1 line
Jens Axboe
#4 lore raw
In reply to David Balazic

Hi,

Quoted 5 lines · Jens Axboe

Yep, it's much harder than it seems. Especially because for the barrier to be really useful, having inter-request dependencies becomes a requirement. So you can say something like 'flush X and Y, but don't flush Y before X is done'.

Yes. Fortunately, the simplest possible barrier is just a matter of marking a request as non-reorderable, and then making sure that you both flush the elevator queue before servicing that request, and defer any subsequent requests until the barrier request has been satisfied. One it has gone through, you can let through the deferred requests (in order, up to the point at which you encounter another barrier).

Only if the queue is empty can you give a barrier request directly to the driver. The special optimisation you can do in this case with SCSI is to continue to allow new requests through even before the barrier has completed if the disk supports ordered queue tags.

--Stephen

#5 lore raw
In reply to David Balazic
Quoted 11 lines · Stephen C. Tweedie
Quoted 4 lines

Yep, it's much harder than it seems. Especially because for the barrier to be really useful, having inter-request dependencies becomes a requirement. So you can say something like 'flush X and Y, but don't flush Y before X is done'.

Yes. Fortunately, the simplest possible barrier is just a matter of marking a request as non-reorderable, and then making sure that you both flush the elevator queue before servicing that request, and defer any subsequent requests until the barrier request has been satisfied. One it has gone through, you can let through the deferred requests (in order, up to the point at which you encounter another barrier).

The above should have been inter-queue dependencies. For one queue it's not a big issue, you basically described the whole sequence above. Either sequence it as zero for a non-empty queue and make sure the low level driver orders or flushes, or just hand it directly to the device.

My bigger concern is when the journalled fs has a log on a different queue.

Quoted 4 lines

Only if the queue is empty can you give a barrier request directly to the driver. The special optimisation you can do in this case with SCSI is to continue to allow new requests through even before the barrier has completed if the disk supports ordered queue tags.

Yep, IDE will have to pay the price of a flush.

Signature · 1 line
Jens Axboe
#6 lore raw
In reply to David Balazic

Hi,

Quoted 4 lines · Jens Axboe

On Wed, Mar 07 2001, Stephen C. Tweedie wrote:

My bigger concern is when the journalled fs has a log on a different queue.

For most fs'es, that's not an issue. The fs won't start writeback on the primary disk at all until the journal commit has been acknowledged as firm on disk.

Certainly for ext3, synchronisation between the log and the primary disk is no big thing. What really hurts is writing to the log, where we have to wait for the log writes to complete before submitting the commit write (which is sequentially allocated just after the rest of the log blocks). Specifying a barrier on the commit block would allow us to keep the log device streaming, and the fs can deal with synchronising the primary disk quite happily by itself.

Cheers,
Stephen

#7 lore raw
In reply to David Balazic
Quoted 9 lines · Stephen C. Tweedie
Quoted 4 lines · Jens Axboe

On Wed, Mar 07 2001, Stephen C. Tweedie wrote:

My bigger concern is when the journalled fs has a log on a different queue.

For most fs'es, that's not an issue. The fs won't start writeback on the primary disk at all until the journal commit has been acknowledged as firm on disk.

But do you then force wait on that journal commit?

Quoted 7 lines

Certainly for ext3, synchronisation between the log and the primary disk is no big thing. What really hurts is writing to the log, where we have to wait for the log writes to complete before submitting the commit write (which is sequentially allocated just after the rest of the log blocks). Specifying a barrier on the commit block would allow us to keep the log device streaming, and the fs can deal with synchronising the primary disk quite happily by itself.

A barrier operation is sufficient then. So you're saying don't over design, a simple barrier is all you need?

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to David Balazic
Quoted 7 lines · Stephen C. Tweedie
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Stephen C. Tweedie

For most fs'es, that's not an issue. The fs won't start writeback on the primary disk at all until the journal commit has been acknowledged as firm on disk.

Quoted 6 lines
Quoted 1 line

But do you then force wait on that journal commit?

It doesn't matter too much --- it's only the writeback which is doing this (ext3 uses a separate journal thread for it), so any sleep is only there to wait for the moment when writeback can safely begin:
users of the filesystem won't see any stalls.

Ok, but even if this is true for ext3 it may not be true for other journalled fs. AFAIR, reiser is doing an explicit wait_on_buffer which would then amount to quite a performance hit (speculation, haven't measured).

Quoted 5 lines
Quoted 2 lines

A barrier operation is sufficient then. So you're saying don't over design, a simple barrier is all you need?

Pretty much so. The simple barrier is the only thing which can be effectively optimised at the hardware level with SCSI anyway.

True

Signature · 1 line
Jens Axboe
#9 lore raw
In reply to David Balazic

Hi,

Quoted 7 lines · Jens Axboe
Quoted 4 lines · Stephen C. Tweedie

For most fs'es, that's not an issue. The fs won't start writeback on the primary disk at all until the journal commit has been acknowledged as firm on disk.

But do you then force wait on that journal commit?

It doesn't matter too much --- it's only the writeback which is doing this (ext3 uses a separate journal thread for it), so any sleep is only there to wait for the moment when writeback can safely begin:
users of the filesystem won't see any stalls.

Quoted 2 lines

A barrier operation is sufficient then. So you're saying don't over design, a simple barrier is all you need?

Pretty much so. The simple barrier is the only thing which can be effectively optimised at the hardware level with SCSI anyway.

Cheers,
Stephen

#10 lore raw
Quoted 9 lines · On Wednesday, March 07, 2001 08:56:59 PM +0000 "Stephen C. Tweedie"…

Hi,

Quoted 6 lines · Jens Axboe
Quoted 4 lines · Stephen C. Tweedie

For most fs'es, that's not an issue. The fs won't start writeback on the primary disk at all until the journal commit has been acknowledged as firm on disk.

Quoted 6 lines
Quoted 1 line

But do you then force wait on that journal commit?

It doesn't matter too much --- it's only the writeback which is doing this (ext3 uses a separate journal thread for it), so any sleep is only there to wait for the moment when writeback can safely begin:
users of the filesystem won't see any stalls.

It is similar under reiserfs unless the log is full and new transactions have to wait for flushes to free up the log space. It is probably valid to assume the dedicated log device will be large enough that this won't happen very often, or fast enough (nvram) that it won't matter when it does happen.

Quoted 7 lines
Quoted 3 lines

A barrier operation is sufficient then. So you're saying don't over design, a simple barrier is all you need?

Pretty much so. The simple barrier is the only thing which can be effectively optimised at the hardware level with SCSI anyway.

The simple barrier is a good starting point regardless. If we can find hardware where it makes sense to do cross queue barriers (big raid controllers?), it might be worth trying.

-chris